Better than POSIX?

the probability of having problems varies from filesystem to filesystem, from mount option to mount option within a single filesystem, and is even dependant on kernel tuning parameters that can be set in /etc/sysctl.conf

if you want that guarantee you need to force a write to disk (and make sure that your disk drive doesn't cache or reorder the writes)

it would be nice if there was a write barrier API that let you insert a ordering requirement from userspace without forcing a write to disk, but at this point in time I don't believe that such a API exists.

people were relatively happy with the ~5 second vulnerability window in ext3, but are getting bit by the ~60 second vulnerability window that ext4 defaulted to.

Ok, that's a reasonable problem, and adjusting ext4 to use a smaller window (possibly even by default) is a reasonable thing. the idea that Ted is working on to have ext4 honor the 'how much time am I willing to loose' parameter that was introduced for laptop mode is a good idea in that it's clear what you are adjusting, and it lets you tie all similar types of thing to the one parameter (as opposed to configuring different aspects in different places)

but shortening this time will cost performance. I doubt that the ext4 developers selected the perfect time (I doubt that there is a single perfect time to select), and having it as an easy tunable will let people plot out what the performance/reliability curve looks like. it may be that going from 5 seconds to 10 seconds gains 80% of the performance that going from 5 seconds to 60 seconds gives you (for common desktop workloads ;-) and distros may choose to move the default there.

the problem in this situation is that people are mistakenly believing that this vunerability didn't exist before, it did, it just wasn't as large a window so fewer people were hitting it.
Also they didn't have any other experiances to compare it to. If ext3 had also lost data in similar situations, people would be used to 'if the system crashes and your app didn't do fsync you will loose data modified in the last couple of minutes' and consider that reasonable.

In the actual physical file system image committed to disk; Don't name partially written files.

------

That would pretty much get what everybody wants. I suppose it's much much complicated then that, of course. I'll take "good enough" any day.

Soft updates does not understand hardware write-back caches, NetBSD's journaling does. Given than nearly all commodity systems have write back caches at some level in the I/O path nowadays, and need them for good performance, this is important.

Soft updates was quite complicated code with integration issues, making it very difficult to fix some of the bugs with it. This was mainly down to its structure, which was suited to commercialization (soft updates was initially a commercial offering). It was not necessarily tailored to the BSD systems.

Additionally, journaling gives some nice benefits like no fsck, and fully atomic rename. No fsck is important to NetBSD for the embedded role, and elsewhere is a much bigger story than it was when soft updates came about due to the explosion in storage volume size.

But frankly, since the dawn of ext3 in the 2.2.x kernel series, I've always considered the fact that it didn't leave garbage files around on my laptop (with its dodgy IDE chipset) to be its major benefit. And I never really seriously considered using the other journalling filesystems that only preserve meta-data integrity. So I am unhappy with the choice of names for the options. I'd rather that the default be "data=delayed", and "data=ordered" refer to the allocate-on-commit behavior.

Everywhere else, it's a minority who even know the standard *exists*, let
alone reference it regularly, and it's a very small minority who've
actually read the whole thing.

I, too, wish this was not true (I spend much too much of my time at work
cleaning up after these bozos), but I can see no useful way to fix it. In
a corporate setting mandatory code reviews with a flunk-too-often-and-
you're-fired rule might work, but I wouldn't want to work there: morale
would be appalling.

So now I take a different approach: I write lots of unit tests, and I set up buildbots for any platform I need to support.

This leads to lots of interesting discoveries. For example, under Windows Vista, pretty much any file system operation can fail for no apparent reason, and it may need to be retried until it works.

There are two exceptions to this rule: Security, and data integrity. You can't ensure either with unit tests. You must also understand both the official documentation and the reality of common implementations. (My nastiest surprise so far: There are snprintf implementations on some legacy Unix platforms that ignore the buffer size parameter, exposing every caller of snprintf to an overflow attack. If you rely on snprintf on old Unix platforms, it might be worth writing a unit test that checks to see if it works.)

The kludges for alloc-on-replace that will apparently be enabled by default should deal with the real, problematic cases of loss of pre-existing data quite nicely, and I do think having those as defaults is a good, reasonable compromise. Those who want to be extra-paranoid with either nodelalloc or alloc-on-commit, well, sure, the option is/will be there, but I don't think these might I say somewhat obsolescent modes should be the default.

'course, distros shouldn't use ext4 per default yet anyway for conservativeness sake. Maybe starting next year ;]

but that would be bad. hence there is a journal, and tools like fsck to make sure that a system crash usually does not harm most of your data.

the journal is meant to mean that after a crash the filesystem can be recovered to a valid state, without having to sync after each write.

so ext3 is POSIX plus stuff to make a filesystem safe and useful.

SQLite on the other hand, appears to do a checkpoint (i.e. finish the
updates to all the actual data files) as part of every commit. That is not
necessary in a good design. There is nothing fundamental about filesystem
or database design that requires anything except the journal to be
synchronously updated on transaction commit. If durability is not
required, not even that.

It's amusing to me that Gabriel's example of interruptible read() and write() system calls utterly fails to illustrate his point. The interface complexity of these system calls can be completely hidden by simple C library functions, and the code to implement them is much, much shorter than would be needed in the kernel to re-run the interrupted calls. Furthermore, a user program might well prefer to bypass such help, e.g. in case it could do something else useful before performing more I/O.

That said, Apollo's Aegis, evolved from Unix's predecessor Multics, had a much better interface to read(): it took a pointer to a buffer, but normally would return a pointer into the OS's own file buffer, usually avoiding a copy operation. Unixen didn't typically have a nonblocking read until remarkably recently, and it still doesn't tend to work well with disk files.

Users who get hoist with that particular petard have clearly done it to themselves. "XFS does it too" is really academic.

IOW, AFAICT, the bad reiserfs rep originates in the pre-2.6.6 era, quite
some time ago now, and it's actually a quite stable and mature fs now.
That has certainly been my experience, both bad back then, with corrupt or
zeroed files at boot after a crash pre-data=ordered, and impressively
stable, now and for several years, post-data=ordered.

Again, if you have references otherwise, I'm willing to be corrected, but
I believe reiserfs is actually reasonably similar to ext3 in this regard,
and has been for some years, now.

Duncan

Then, as suggested by a commenter elsewhere, we could add something like a
"rename_unordered" function for those relatively unusual cases where a user
is willing to risk severe data loss to get better performance. In addition,
for portability reasons, we should have an option so that an application
can discover whether or not an fsync is required to get ordered renames.
fsync is very expensive when you don't need (synchronous) durability
semantics.

This is *the* way to achieve atomicity on Unix. It always was.

We didn't use to have journaling filesystems and we never used to expect anything at all to work after a crash. Crashes happened and they often meant hours of work, possibly reinstalling everything.

To discover that ext3 data=ordered just kept on working after a crash was a real eye-opener for me. I never realised before that such robustness was possible (just like I never realised prior to Linux that I could afford my own Unix box) and I am not about to relinquish it! I'm sure I speak for many users here. It *was* an unexpected nice-to-have when we first got it; but it has become a solid requirement.

Delayed allocation without an implicit write barrier before renaming a newly-written file virtually guarantees data loss after a crash with existing applications. It is therefore a regression from the status quo, albeit to something somewhat better than the status of a few years back.

Kudos and thanks to Ted for implementing this must-have write barrier (and also for improving the chances that unsafe truncate-and-write is less likely to hit the inevitable race condition, though IMO he's quite right that applications doing that are broken).

I just wish he wouldn't keep insisting that fsync at application level is the right way to achieve what we want.

POSIX and fsync have nothing to do with it (any journaling filesystem provides much more than POSIX), nor do application authors who forgot to think about recovery after an OS crash. A journaling filesystem *can* guarantee atomic renames, so it *should*, for the sake of users' sanity, not for the sake of a standards document.

Standards can be updated to follow best practice. They often do.

Application writers have long used rename precisely and only to achieve atomicity; it's the only atomic operation the API provides (simulating atomicity with locks and a series of synchronous flushes is a very different matter). As long as the kernel and fs are up, POSIX guarantees that data writes which precede the rename will be visible to all readers after the rename. Post-crash, nothing used to be guaranteed.

This isn't about application developers coding to an API, it's about users wanting reasonable behaviour from their computers even when they abuse them by kicking out power cords or installing proprietary device drivers.

Ext3 has given us a new, much-better-than-POSIX standard of data recoverability. It's a mere implementation detail that it does this in part by effectively preserving the order of operations that POSIX mandates down to the disk level.

Delayed allocation without a write barrier before renames of newly-written files practically guarantees data loss in this extremely common use-case, so it's a regression.

The regression now has been fixed (thanks Ted). No hacking of applications required.

We use a journaling filesystem so that *after* our hardware or our kernel fails -- which is more or less likely depending on the way we use them, but everyone is at risk -- whatever made it to nonvolatile storage is in a sane, recoverable state. NOT MISSING. That way we *only* lose a few seconds' or minutes' work.

[ Or browsing history, some people call that work :-) ]