Luu: Files are hard

That's the way kernel filesystems work. IO ordering is enforced by the layer with the ordering depenencies and it's done by waiting for all dependent IO to complete, then flushing physical caches and/or issuing FUA writes as teh "barrier" to ensure ordering at the physical storage layer is correct.

Dependencies are different for fsync, journal operations, metadata writeback, etc and they are all occurring concurrently in a filesystem. It's far more efficient to have each different subsystem control their IO dependencies directly via the above mechanism than it is to place global ordering barriers in the IO stream as all that such barriers do is create pipeline bubbles that reduce the utilisation of the IO devices. Hence by using a more efficient method of controlling order dependencies the penalty for using "barriers" vs "nobarriers" is now only a couple of percent, simply because we no longer have pipeline bubbles that incorrectly block non-dependent IOs...

You can do exactly the same thing from userspace using AIO+DIO and fsync. AIO+DIO to submit and wait for completion of dependent IO, then fdatasync the file(s) to commit metadata changes and enforce physical IO ordering in the storage. Then you can unplug the barrier and issue all your new IO and the problem is solved. You don't need to block/order any non-dependent IO on the barrier, so other parts of the application can continue oblivious (i.e. at full speed) to the parts of the app that have ordering dependencies.

Yes, i know, it doesn't help apps that use mmap/buffered IO, only deranged monkeys use DIO and crack-addled undead armadillos use AIO, etc. But the reality is that if you are going to step outside POSIX, we already have the tools for apps to implement strict, optimised ordering of their IO....

> I don't even need fsync, which is a killer anyway, because that has all
> sorts of nasty side effects that can bring a system to its knees - call fsync
> when a streaming write is filling buffers faster than the system can flush
> them, and a "compliant" implementation will cause the system to
> effectively hang ...

I'm pretty sure those old fsync/sync livelock problems were pretty much fixed back in 2010 by commit 6e6938b ("writeback: introduce .tagged_writepages for the WB_SYNC_NONE sync stage").

-Dave.

While that might be true(*) I don't think it is an interface that the kernel should provide.
To implement a "write barrier" you need to identify which requests are "before" and which are "after", so you are really asking the kernel identify groups of requests for you and that opens various cans of worms regarding privilege and resource reservation.

I think that the interfaces you want are "make change" "flush changes" and "wait for flush to complete" with the ability to identify "changes" with reasonably precision.
We have "sync_file_range" which allows fairly precise identification of which part of a file you want to flush or wait for. We cannot offer more precision in directories than "all names in the directory". Would there really be value in that?

If you wanted to implement a "barrier" it should be possible in a library, using a separate thread to wait for the "before" writes to complete before initiating the "after" requests. Certainly a genuine attempt at that would be an important precursor to adding any sort of support to the kernel.

(*) However I don't think "All you need is barriers" will ever be a hit song.

Then simple code will get simple semantics. If you know what you are doing and you want to sacrifice safety for performance, you can set a flag to allow the kernel to delay flushing to disk or even reorder writes; or the application could use asynchronous I/O to start with.

As I understand it: You will need to fsync the file. Then, if you changed the file's size (say, by appending to it, or it was created anew), you will need to fsync the directory (which is basically impossible from some languages). Then, you can do your rename, and then finally, you must fsync the directory again to make that rename durable.

Of course, if you're on OS X or iOS, fsync() won't do since it's basically ignored, and if you're writing to a FUSE file system (say, /sdcard on Android), you might not even have fsync() and would need to rely on a global syncfs() or similar and hope the FUSE driver isn't being evil with you.

All disks in use today are, essentially, NAS. That is, they are independent
computers at the end of some kind of network connection. That's true
even for disks physically sitting inside your machine. The network might
not be an ethernet connection in that case, but SATA, USB, SCSI, or
whatever. It's all the same.

You may think you are writing data in a nice, ordered manner. You may
even have gotten the kernel to send the data towards the disk in a well-
defined order using the right combination of fsync and offerings at the
full moon. Maybe the disk has even been told not to do any reordering
whatsoever.

It doesn't matter.

The disk, for reasons of its own, will then make its decisions about what
to do: write this block now and send an acknowledgment? Send the
acknowledgment right away and write later? Write this block before
that block because the latter cannot be written now? Delay writing
this block because it is likely to be rewritten shortly?

There is financial pressure on disk manufacturers to make the decisions
that will cause the drive to appear fast by cutting each and every corner
imaginable.

Until and unless the power goes out suddenly, you will never know the
difference. And then the blame isn't usually stuck to the drive manufacturer.

M.

"Transaction" might be a bad choice of word since it might be
construed to be an ACID transaction, but I don't think it is necessary
to guarantee atomicity or isolation. As used by jgg in the first
comment, "barrier", implying consistency and durability, is probably a
better choice of word.

I can imagine a possible implementation where each txn_OP call would
immediately make the corresponding syscall to start the write, but
register a callback to do the appropriate steps in the "get all the bits
on stable storage" dance once the transaction is committed. Upon commit,
the number of calls to correctly sync would be reduced to the minimum.

So even if my logs and stuff are on different filesystems, possibly across a network, I can write my logfile, say "wait until that's flushed to storage", carry on and update the database itself, say "wait until that's flushed to disk", and then clear the log.

For the sake of a couple of ordering guarantees given to the app, the complexity of the app suddenly becomes MUCH simpler, because it no longer has to worry about whether the data has really been saved (okay, as pointed out elsewhere, hardware can lie, but every level we can push the problem further down the stack, the less likely it is to bite).

Cheers,
Wol

Downside: fragmentation, but with SSDs this shouldn't be a big concern.

Keep per process or per thread list of all file / directory fds with potentially uncommitted changes. When a write barrier is requested, call fsync or fdatasync on all of them and clear the list. If the list gets too large, retire operations from the head of the list as necessary.

I wouldn't call this transaction anything because that implies ACID semantics, which would be much more difficult to implement in a kernel. This is just write barriers with the ability to get completion notification.

The suggested API is more extensive than one might expect in order to allow a single process to handle multiple "write threads" without itself being multi-threaded, or having as many threads as there are application level transactions in process. Each sfd is its own write thread. If you don't want to use sfds explicitly, every thread would have an implicit one. Not necessarily an independent one, but an implicit one.

That means that if all you want is a write barrier you would make one call (or maybe two) and that would be it. Nothing else required.

The real work would be implementing a write barrier internally (i.e. in the kernel) as more than a series of synchronous fsync or sync_file_range operations. There already is the ability to initiate write out (at least for portions of files) without blocking, so basically what you want to do internally is initiate write out for everything on the write thread that hasn't already completed.

Then when the next write operation on the same "write thread" comes along, if there is an outstanding write barrier, you want to wait on write out for all those operations to complete before proceeding. As long as POSIX semantics are in force, you must wait for write out to complete before proceeding with another write operation on the same write thread, or the write barrier is meaningless.

Write threads are important here because there are writes you do not want to block or wait for commit of in any circumstances. That is why you need an API so that different writes can go on different "write threads" or sfds. The implicit sfd makes things simple for standard write(2) calls, and applications and libraries that just do the normal thing, but an explicit write as part of this "write thread" interface is necessary in the general case.

Implementation wise there is an issue with how many uncommitted operations you can track so you can commit them when necessary. I believe that is the biggest issue, especially with implicit write threads - you could easily have thousands of operations on a write thread that haven't been committed to durable storage yet. At some point you have to start combining them or retiring them early or you may have to block synchronously in unexpected places, just to free up space in your uncommitted operation buffer.

One thing that would help there is a filesystem level transaction group number for metadata and certain other operations. Then an internal operation can return the TXG number that the operation is expected to be committed in, and everything in the buffer for the same filesystem with a lower TXG number than the current committed one for that FS can be efficiently discarded - no need to make an inquiry or be notified about the completion of each one. You could just group them together - at most one entry per write thread per filesystem TXG. That is easily a thousand fold reduction in some cases. Then a mere handful of entries - something much more practical to keep in kernel - would suffice most of the time. You would only need more for writes of the sort the FS doesn't plan to commit in the next TXG.