Quote of the week [LWN.net]

At this point we actually provide app developers with what they've been repeatedly asking kernel filesystem engineers to provide them for the past 20 years: a way of overwriting arbitrary file data safely without needing an expensive fdatasync operation on every file that gets modified.
Put simply: atomic writes have a huge potential to fundamentally change the way applications interact with Linux filesystems and to make it *much* simpler for applications to safely overwrite user data.

— Dave Chinner sees an O_PONIES solution

to post comments

How is it O_PONIES?

Posted Dec 12, 2024 14:47 UTC (Thu) by khim (subscriber, #9252) [Link] (27 responses)

Who told you it's O_PONIES solution?

O_PONIES is about how app writers wish that every filesystem behaved like ext3 (in the good ways , of course, not in the bad ways, oh no!) with critical part of the issue in braces, IOW about entirely unrealistic expectation that filesystem may, somehow, do the “good things” while avoiding “bad things” without app explicitly telling what these “good things” and “bad things” are.

Similarly to how “we code to the hardware” people expect that code that is violating language specs would be compiled correctly if it does some “good things” and would be eliminated and removed when it does some “bad things” without developers ever describing what these “good things” or “bad things” are.

Atomic writes, on the other hand, propose and promise the exact opposite: they provide API which does precisely what's written on the tin, without any magic in it.

How is it O_PONIES?

Posted Dec 12, 2024 15:06 UTC (Thu) by corbet (editor, #1) [Link] (26 responses)

The O_PONIES discussion, at its core, was about applications trying to overwrite small files in a safe way — and failing. Atomic writes, if used carefully. could solve that problem.

How is it O_PONIES?

Posted Dec 12, 2024 15:39 UTC (Thu) by khim (subscriber, #9252) [Link] (25 responses)

You omitted one critical distinction. Here is how discussion went:

> The O_PONIES discussion, at its core, was about applications trying to overwrite small files in a safe way — and failing because they were trying to use interface that wasn't designed for that.

It's written very clearly here: It may not be POSIX, but the programmer's intent is clear - no one ever, ever wrote "creat(); write(); close(); rename();" and hoped they would get an empty file if the system crashed during the next 5 minutes. That's what truncate() is for.

That's the complaint: it not about filesystem doing something incorrectly, but about filesystem not understanding the intent.

IOW: app developers ignored the written rules, wrote one thing yet expected the other thing – and then become angry when something have changed and the expected thing stopped being delivered.

Classic story of Hyrum's Law and [ab]use of undefined behavior where something works even if it was never guaranteed and when it breaks – someone becomes upset.

> Atomic writes, if used carefully. could solve that problem.

Sure, but they couldn't implement O_PONIES. Just like other attempts couldn't. Just look on this proposal which was supposed to resolve that issue back then O_PONIES were first implemented:

Magically taking care of filesystem semantics for every use may not be possible (no O_PONIES for you!), but I believe we can help the applications that just want to completely rewrite a file and atomically replace it.

The semantics for O_REWRITE would be:

…

It's very clear that O_PONIES is not about overwriting small files in a safe way but about making thing magically work when API is not used correctly!

O_PONIES is not about O_REWRITE, O_PONIES is not about atomic writers, O_PONIES is “do whatever I meant, not whatever I wrote in the code”.

How is it O_PONIES?

Posted Dec 12, 2024 17:56 UTC (Thu) by Wol (subscriber, #4433) [Link] (9 responses)

> IOW: app developers ignored the written rules, wrote one thing yet expected the other thing – and then become angry when something have changed and the expected thing stopped being delivered.

You've forgotten ext4 ...

Am I right? but it did horrible things in a crash. Istr along the lines of, for example, you opened a (pre-existing) file, *overwrote* the contents, and when it crashed you were left with an empty file. With NOTHING. Which, from the application's PoV, is IMPOSSIBLE.

I don't expect *changing* the contents of a file to *delete the contents completely*.

And if you read Valerie Aurora, she puts it perfectly - "file system writers consider their own (filesystem metadata) more important than the user's data". Or as I put it, "what's the point of knocking 5 minutes off the reboot time after a crash, if the system is then down for a day restoring a backup?"

If I can rely on the system surviving a crash in a logically possible state, I can try and recover. If I know there is going to be random "impossible" damage, the ONLY sensible action after a crash is a restore. And no, I don't call a trashed filesystem after a head crash "impossible" - it's just when there is no POSSIBLE user-space explanation for what happened that makes sense.

Like, as I say, replacing a file's contents IN PLACE only to be left with an empty file ...

Cheers,
Wol

How is it O_PONIES?

Posted Dec 12, 2024 18:46 UTC (Thu) by khim (subscriber, #9252) [Link] (6 responses)

> Am I right? but it did horrible things in a crash. Istr along the lines of, for example, you opened a (pre-existing) file, *overwrote* the contents, and when it crashed you were left with an empty file. With NOTHING. Which, from the application's PoV, is IMPOSSIBLE.

But that's most the majority of filesystems work. Not just Ext4.

The only difference was that Ext4 is perceived as “spiritual descendant of Ext3” which made it “wrong” specifically for Ext4.

> Like, as I say, replacing a file's contents IN PLACE only to be left with an empty file ...

You mean: like it could happen with FAT or with with Ext2 or with pretty much any other filesystem with Ext3 being the only exception?

> And if you read Valerie Aurora, she puts it perfectly - "file system writers consider their own (filesystem metadata) more important than the user's data". Or as I put it, "what's the point of knocking 5 minutes off the reboot time after a crash, if the system is then down for a day restoring a backup?"

IOW: how dare they to do what they promised to do and what I wanted them to do, right?

Indeed, that is the essence of O_PONIES discussion: not about the way to change the configuration safely, but about the fact that programs that were using undocumented tricks to achieve that stopped working.

And that's, most definitely, not what atomic writes are promising. They are just promising to add to the filesystem a way to achieve certain things reliably and efficiently, they don't promise to keep broken programs working.

How is it O_PONIES?

Posted Dec 12, 2024 23:56 UTC (Thu) by Wol (subscriber, #4433) [Link] (5 responses)

> > Am I right? but it did horrible things in a crash. Istr along the lines of, for example, you opened a (pre-existing) file, *overwrote* the contents, and when it crashed you were left with an empty file. With NOTHING. Which, from the application's PoV, is IMPOSSIBLE.

> But that's most the majority of filesystems work. Not just Ext4.

Is it? And if so, WHY!?

Older filesystems just wrote to disk (okay, today that's slow, but they didn't work that way). Okay, ext3 got it right by accident, but it should have got it right by design! Corrupting user data is not acceptable behaviour for a filesystem! And to sabotage user-space attempts to avoid corruption just makes it worse!

As others have pointed out, it's dead easy to write a filesystem that does NOT trash user data. If it's a journalled filesystem, make sure the journal saves *everything* to disk before you update the main copy. If it's Copy-on-Write, flush the user data BEFORE the metadata. Etc etc. Designing a filesystem to "fail safe" isn't rocket science... Saying "posix doesn't require us to protect your data, so we won't bother trying" is a pretty sure way of making any sensible user avoid your filesystem like the plague! The fact ext4 was so cavalier with user data is probably why most serious users moved to ZFS, XFS, or even btrfs. Even if btrfs was rubbish, at least it took reports of data loss seriously, and didn't respond "that's the defaults working AS DESIGNED".

You seem a fan of Rust making sure that most programmers never need to step outside safe mode. Why are you so anti programmers wanting a safe mode for filesystem access? Because if I can't trust the filesystem not to randomly trash my data, that's most certainly not "safe filesystem access".

Cheers,
Wol

How is it O_PONIES?

Posted Dec 13, 2024 1:12 UTC (Fri) by pizza (subscriber, #46) [Link] (2 responses)

> Corrupting user data is not acceptable behaviour for a filesystem!

You keep forgetting we're talking about *full system crashes* here.

"corrupting data" that hasn't been explicitly committed before the crash is to be expected.

> Older filesystems just wrote to disk (okay, today that's slow, but they didn't work that way).

You're right, they didn't -- there were write-back caches in place on both the OS and the disks themselves, and in the absense of an explicit flush/sync they would be re-ordered as needed to maximize throughput and minimize overall latency.

> Designing a filesystem to "fail safe" isn't rocket science...

Doing so in a remotely performant manner very much is.

And it's made even more difficult when application writers don't use the APIs that are in place to ensure data is safely committed.

How is it O_PONIES?

Posted Dec 13, 2024 23:04 UTC (Fri) by Wol (subscriber, #4433) [Link] (1 responses)

> You keep forgetting we're talking about *full system crashes* here.

> "corrupting data" that hasn't been explicitly committed before the crash is to be expected.

I agree. Which is why I'm *not* forgetting about full system crashes. Losing data in flight is normal and to be expected. Tough ...

The problem is when you try and minimise the possibility of data in flight getting lost, and then the filesystem acts to sabotage you. Don't forget, I'm interested in database internals, and making sure data doesn't get lost or corrupted is VERY important. I really don't want to lose a file in the gigabytes because the data and metadata got out of sync!

> > Designing a filesystem to "fail safe" isn't rocket science...

> Doing so in a remotely performant manner very much is.

If you can design a filesystem to protect the metadata, surely it can't be that hard to protect the data as well?

> And it's made even more difficult when application writers don't use the APIs that are in place to ensure data is safely committed.

Is that fsync? Because the biggest complaint is that using fsync knackers performance! When this has come up before, you don't need or want fsync, you just want a barrier so you can reason - at user level - what has been committed and what has been lost. The big problem here, of course, is lower levels that lie to you, but there unfortunately also, if the system lies to you all bets are off.

Barriers lower performance - of course they do. But the problem with programs using barriers like fsync is they knacker performance for EVERYONE. This is another situation where the providers are giving you the choice between an over-the-top expensive solution, and a budget solution that doesn't work, without actually acknowledging the problem.

I don't know what extent filesystems do what I would consider sensible - or even to what extent they can given the deference to Posix, but this is what I'd do ...

I suspect I'm nicking this idea from networking, but ... all processes live above the virtual filesystem, for Posix purposes? Every process (thread, Pid, file, whatever?) is allocated a buffer which the vfs processes on write to provide the posix guarantees. The application will have some way of combining several chosen fds into their own shared buffer, whether by splitting them out of the process buffer or combining individual fd buffers. The vfs then scans all the buffers in a round-robin fashion, merging them into a single stream (or maybe several) down to the next layer. The scan will pass the lesser of the entire buffer, as much until it hits a barrier, or a certain size. If the very first thing in the buffer is a barrier, it will copy the barrier and then carry on with the scan.

Which means processes that don't do much i/o will have everything processed in one hit and suffer minimal performance hit. Processes that use barriers a lot will impede themselves, but not much else. And processes that try to bring the system to a halt with masses of io will have to let other processes queue-jump.

If there's any lower layers between vfs and hard drive, they need to follow the same logic - merging streams and barriers. And the lower layers are free to re-order anything they like BETWEEN barriers, just not across barriers. If the system is lightly loadly, who cares if every second write is a barrier, chances the previous write will already have flushed. If it's heavily loaded and every second write is a barrier, then the only process that's going to suffer is the one writing the barriers. And if the system is heavily loaded but the process writing barriers is only generating maybe 10% of the load, then the system will have a lot of freedom to optimise, and performance won't be impacted that much, because there'll be a lot of io between barriers.

So on a "mixed" server, a database process that's using integrity won't mess up all the other processes on the same system.

Okay this won't work well once it hits a raid layer where streams need to spread out, rather than be merged, but they have their own way of dealing with the problems that brings. (And again, while I can't reason about the raid level messing up, if the filesystem passes stuff to raid in the correct order, then it ain't the filesystem's responsibility if it goes wrong.)

Cheers,
Wol

How is it O_PONIES?

Posted Dec 13, 2024 23:52 UTC (Fri) by pizza (subscriber, #46) [Link]

> The problem is when you try and minimise the possibility of data in flight getting lost, and then the filesystem acts to sabotage you.

Um, no. When you deliberately choose to not follow to the documented API that will give you safety, you are not "trying to minimize the possibility" of data loss! Instead, you are prioritizing performance and rolling the reliability dice, which you _will_ inevitably eventually lose.

> If you can design a filesystem to protect the metadata, surely it can't be that hard to protect the data as well?

Read what I wrote. "Doing so in a remotely performant manner" is _very hard_.

Whining about how filesystem authors need to "nerd harder" doesn't change that basic fact. And there's pretty mucn nothing a filesystem author can do when application writers don't follow API guidelines and system integrators/end-users deliberately use fast-but-unsafe filesystem settings on top of hardware that lies about when writes are completed.

How is it O_PONIES?

Posted Dec 13, 2024 2:35 UTC (Fri) by khim (subscriber, #9252) [Link]

> Okay, ext3 got it right by accident, but it should have got it right by design!

Except ext3 got it wrong by accident. Ext3 was catastrophically slow when used with correct interface. And thus applications writers started experimenting and stumbled upon certain sequence which worked even if wasn't designed to work. And was faster. Woho, let's use that newfound greatness in place of correct interface!

That is how the whole mess started. Everyone was working with their best intentions – but at some point communications have fallen apart.

> And to sabotage user-space attempts to avoid corruption just makes it worse!

Except Ext4 never did that. Ext3 did. That is what you are missing. There was a proper interface that would have guaranteed correctness – apps just have chosen not to use it. Because they wanted to have their cake and eat it, too.

It was bound to explode, at some point and filesystem developers even forked Ext3 into Ext4 to make it clear that it's different beast – but that haven't stopped the whole things from exploding, of course.

It's perfectly natural desire of people to have their cake and to eat it, too. It just never works.

> As others have pointed out, it's dead easy to write a filesystem that does NOT trash user data.

Only if programs using it right it's “dead easy”. Or if you don't disable the guardrails and accept the performance loss, then it's “dead easy”. If applications are using random ideas about how filesystems should behave and you then disabled the guardrails to make it faster… then it's very far from being easy. I would even say that it's impossible

> If it's a journalled filesystem, make sure the journal saves *everything* to disk before you update the main copy.

Which is extremely slow and is the whole reason Ext4 was done differently – but it can do that, if you want.

> If it's Copy-on-Write, flush the user data BEFORE the metadata.

Which is extremely slow and that's why other filesystems don't do that.

> Designing a filesystem to "fail safe" isn't rocket science...

Not if you care about performance.

> Saying "posix doesn't require us to protect your data, so we won't bother trying" is a pretty sure way of making any sensible user avoid your filesystem like the plague!

Obviously not, we know users haven't done that. You may say that all users are insane, and that could very well be true, but that's the world we live in.

> The fact ext4 was so cavalier with user data is probably why most serious users moved to ZFS, XFS, or even btrfs

Except they haven't. And XFS exhibits the exact same behavior as Ext4 thus it's hard to imagine why people would move there if they care about data corruption.

> Even if btrfs was rubbish, at least it took reports of data loss seriously, and didn't respond "that's the defaults working AS DESIGNED".

Except they did. Btrfs in RAID5/6 mode works exactly like Ext4 (with big difference being that it could detect corruptions, but wouldn't prevent them).

> Why are you so anti programmers wanting a safe mode for filesystem access?

I'm not against “wanting a safe mode”. I'm against wanting O_PONIES. Against demands to support something magically without any explanations and specifications.

Please recall that if you enable data=journal on Ext4 then, suddenly, Ext4 would start behaving precisely like you demand and corruptions that we are discussing here would no longer happen.

Surprised? I don't think so. You never stop to think and just demand that others would solve your problems.

> Because if I can't trust the filesystem not to randomly trash my data, that's most certainly not "safe filesystem access".

Obviously. But that's an explicit choice. It's your choice as Ext4 user to pick speed or correctness, but if you have chosen speed then why do you complain then it makes it possible to corrupt things?

That is what prompted the whole O_PONIES thing: while today these applications merely insist on the kernel knowing better and implicitly syncing the data on the applications' behalf, it will not take long before the applications will demand more (e.g., pink ponies).

> You seem a fan of Rust making sure that most programmers never need to step outside safe mode

No, I'm fan of making my choices explicit: I can write safe Rust code (and then compiler has my back) or I may decide to disable the guardrails and write unsafe code – but then I don't expect that compiler would magically save me!

But O_PONIES is precisely about saving you bacon after you have explicitly disabled safety guards!

The root of ext3's O_PONIES problem

Posted Dec 13, 2024 11:45 UTC (Fri) by farnz (subscriber, #17727) [Link]

At core, ext3 had two interacting issues:

Using the correct API to indicate that you wanted data to be safe was extremely slow; ext3 had to write out all dirty filesystem blocks, not just the ones relating to the data you want to make safe.
Not using the correct API made your data safe without a performance cost.

Given this set of tradeoffs, it's not surprising that application developers chose to not tell the filesystem to make their data safe. The trouble is that one of the reasons ext3 was being replaced is that the design decisions that led ext3 here also made it a slow filesystem on faster devices - and those faster devices were coming down from the realm of servers run by people who really know what they're doing (and thus can make xfs, ext4 or other FSes of the era safe), to ordinary consumers.

How is it O_PONIES?

Posted Dec 12, 2024 18:50 UTC (Thu) by pizza (subscriber, #46) [Link]

From an application's PoV the impossibilty is the crash itself; what you're left with are consistency promises made by the rest of the system.

Except in this case, the rest of the system made no such promise in general, nor did the application take the necessary (and well documented) steps to request the behavior they were expecting to obtain such a promise.

Why didn't they? Simple:
a) they didn't know it was necessary
b) they knew but didn't because doing so was "too slow"

How is it O_PONIES?

Posted Dec 12, 2024 18:57 UTC (Thu) by adobriyan (guest, #30858) [Link]

> *overwrote* the contents, and when it crashed you were left with an empty file.

Obvious question is "did it truncate?".

I remember one such crash when KDevelop segfaulted (not oopsed) and left one file empty (as in st.st_size == 0).

How is it O_PONIES?

Posted Dec 12, 2024 20:25 UTC (Thu) by andresfreund (subscriber, #69562) [Link] (14 responses)

> IOW: app developers ignored the written rules, wrote one thing yet expected the other thing – and then become angry when something have changed and the expected thing stopped being delivered.

I'd say that the written rules are *extremely* fuzzy. I don't actually want "creat(); write(); close(); rename();" to magically become durable - that very well might impose unnecessary cost, e.g. if the application can batch some of the syncs for multiple files. But I would like documentation about what fsyncs are actually required. As far as I know the required dance across different filesystems is something like:

fsync_file(oldfd);
newfd = creat();
write(newfd);
fsync_file(newfd);
fsync_directory(dirfd);
rename(old, new);
fsync_file(newfd);
fsync_directory(dirfd);

At the very least it'd be good to have that documented. And perhaps wrapped in a syscall that promises to replace atomically. For many filesystems that could be much more efficient than the above.

How is it O_PONIES?

Posted Dec 12, 2024 21:07 UTC (Thu) by khim (subscriber, #9252) [Link] (13 responses)

> At the very least it'd be good to have that documented. And perhaps wrapped in a syscall that promises to replace atomically. For many filesystems that could be much more efficient than the above.

Nobody objects about that. Having documentation is great and having more efficient way of implementing something is even better. But just read the original 1st April commit:

From the recent discussions all over several mailing lists, and other internet fora, it is obvious that userspace is looking for semantics the kernel does not provide. It is obvious that while today these applications merely insist on the kernel knowing better and implicitly syncing the data on the applications' behalf, it will not take long before the applications will demand more (e.g., pink ponies).

Any applications wishing to inform the kernel that it has these unreasonable assumptions (POSIX does not guarantee this, and thefore any application which makes the assumption that the OS does sync data implicitly is by definition non portable and/or broken) should open all files with the O_PONIES flag.

I have highlighted the part than gave flag it's name.

Does it still looks as if that's something that “atomic writes”, documentation or anything else may provide?

How is it O_PONIES?

Posted Dec 16, 2024 9:54 UTC (Mon) by NYKevin (subscriber, #129325) [Link] (12 responses)

If you're going to accuse application developers of hubris, it might help to quote statements by application developers which display hubris, rather than facetious statements by filesystem developers which display nothing other than frustration.

How is it O_PONIES?

Posted Dec 16, 2024 11:27 UTC (Mon) by khim (subscriber, #9252) [Link] (11 responses)

I'm not “accusing application developers of hubris”, I'm just showing that O_PONIES are very explicitly NOT about “the need for atomic writes” but about about “unreasonable demand of application writers”.

Whether these demands are actually unreasonable or not is big question (I would place blame on the ext3 which botched an API that application writers had to use), but main issue under discussion was whether characterization of atomic writes by Dave Chinner deserves the name of being an O_PONIES solution.

How is it O_PONIES?

Posted Dec 16, 2024 13:49 UTC (Mon) by Wol (subscriber, #4433) [Link] (10 responses)

> Whether these demands are actually unreasonable or not is big question (I would place blame on the ext3 which botched an API that application writers had to use), but main issue under discussion was whether characterization of atomic writes by Dave Chinner deserves the name of being an O_PONIES solution.

Well, given that Posix guarantees ?atomic?, but more importantly SEQUENTIAL writes at the OS interface, my question is why can't we have the SAME guarantee between layers in the disk stack? And no, I don't expect the file system to work at application-level write granularity - that will be a performance killer as everyone knows, but for the application to be able to say "I need a granularity at *this* point". Otherwise data corruption is pretty much guaranteed, and the filesystem has failed at its purpose.

Having watched it from the sidelines, I know half the trouble is finding someone prepared to do the work, but md-raid-5 (and 6) now provides granularity at the stripe level. If the layer above could say "I want a barrier here", mdraid would have no problem providing it.

I got told off for "application developers not wanting to use the proper API". But given that PostGRES still appears to be using fsync, dare I hazard a guess the reason application developers don't "use the proper API" is that the proper API doesn't exist? The problem with fsync, is it provides guarantees application developers don't want, at a price sys-ops don't want to pay!

Cheers,
Wol

Proper API use

Posted Dec 16, 2024 14:18 UTC (Mon) by farnz (subscriber, #17727) [Link]

Firstly, the hardware does not provide sequential writes; all drives with caches require some form of synchronization to guarantee writes to the medium, and will write out-of-order if left to their own scheduling of writes. You generally have two ways to synchronize; the first is to issue a special type of write (not available on all hardware), called "Forced Unit Access" in the SCSI and SATA worlds, where the drive guarantees that when this write completes, the data written by this command (and this command only) is on the medium and not just in cache; these writes guarantee integrity, but are much slower as a result. The second is to issue a complete flush of the device; this is much heavier, but ensures that there's no data in cache but not on the storage medium.

Secondly, fsync is the proper API for requesting that data is on the storage medium; fsync is a barrier operation that does not return until all previous writes to the file are safely on the storage medium. The issue here is that unless you use io_uring, the Linux FS API is synchronous, so you cannot issue further writes to the OS after the fsync in this thread, since fsync does not return until previous writes are safe. The ext3 performance bug was that it could only implement fsync (make writes to one file safe) by syncfs (make all changes to this FS safe), which meant that using the proper API killed your performance on ext3, but was required to save your data on other FSes.

And if fsync is too heavy-weight, there's also fdatasync to sync just data, not metadata and sync_file_range to sync some ranges of data, but no metadata. The difficulty, however, is that you often want the metadata synced, because it's no good having the data definitely on disk, but the metadata not yet pointing to the data - because then the data is not retrievable after a crash.

How is it O_PONIES?

Posted Dec 16, 2024 14:20 UTC (Mon) by khim (subscriber, #9252) [Link] (3 responses)

> Well, given that Posix guarantees ?atomic?, but more importantly SEQUENTIAL writes at the OS interface, my question is why can't we have the SAME guarantee between layers in the disk stack?

We can. But that would be entirely different, non-POSIX based OS. In POSIX we have fdatasync. It's existence (via the exceptio probat regulam in casibus non exceptis principle) implies that regular use of other functions without explicit barriers don't give you any guarantees.

> but for the application to be able to say "I need a granularity at *this* point"

Application is able to do that and it was always able to do the. The real issue was with ext3 inability to cleanly and correctly implement that request. Instead of making data stored on disk just for this application it interpreted that request in the only way it could: it stored the data for all applications and only then returned to the application.

That was slow and application developers, by conducting some experiments, found a way to achieve the desired result without using the proper interface that was designed to achieve that.

> Otherwise data corruption is pretty much guaranteed, and the filesystem has failed at its purpose.

Sure. But most filesystems provide such guarantees only if you use proper interface designed specifically to provide such guarantees!

But app writers, to deal with ext3 that was slow if you use said interface find a “clever way” to bypass it… that only exclusively worked on ext3 – and nowhere else. It haven't worked on ext2, it haven't worked on xfs, it haven't worked on other POSIX systems… it have only ever worked on ext3 – and then by the use of the exact same quirk of implementation that made proper interface slow!

> If the layer above could say "I want a barrier here", mdraid would have no problem providing it.

Sure, but who is supposed to initiate such request? If not applications (they purposefully avoid that to be fast on ext3, remember), then who and why?

> But given that PostGRES still appears to be using fsync, dare I hazard a guess the reason application developers don't "use the proper API" is that the proper API doesn't exist?

It does exist! fsync (well… fdatasync, more precisely) is the proper interface.

Only apps avoided it because it's slow on ext3.

> The problem with fsync, is it provides guarantees application developers don't want, at a price sys-ops don't want to pay!

I think you are conflating different functions from fsync category. aio_fsync, fsync, and fdatasync are different functions. And that excessive price only actually existed on ext3… and that slowness was born out of the exact same design decision that made workaround that applications invented usable!

The problem was in ext3 implementation of fdatasync that, essentially, turned it into a system-wide fsync – and subsequent decision of application writers to just simply avoiding it and using unreliable hack in its place.

How is it O_PONIES?

Posted Dec 16, 2024 16:31 UTC (Mon) by Wol (subscriber, #4433) [Link] (2 responses)

> I think you are conflating different functions from fsync category. aio_fsync, fsync, and fdatasync are different functions. And that excessive price only actually existed on ext3… and that slowness was born out of the exact same design decision that made workaround that applications invented usable!

I wasn't aware of the other versions. It's just that - from what i can tell reading your pages - do these calls guarantee that no writes AFTER the call will be written UNTIL the call completes?

It's clear that you can be confident, when the call returns "success", everything that was written by the application before it called sync has been successfully flushed to disk.

But does it guarantee that anything written AFTER the call to sync, was NOT written until the sync completed? (And yes, that has a cost. But it's in all likelihood a lot less then having to wait until whichever version of sync completes, before you can carry on with whatever you were doing.)

Cheers,
Wol

Not syncing data written after the sync

Posted Dec 16, 2024 17:39 UTC (Mon) by farnz (subscriber, #17727) [Link]

The goal of "no data written after the sync call written before the sync completes" requires a complete queue stall on all modern storage hardware. You must ensure that no writes after the call to sync are issued to hardware before the sync completes; the easiest way to guarantee this is for the application to not issue a write to the OS until fsync returns, since if the writes haven't been issued to the OS, the OS can't issue them to hardware, either.

There's also no performance gain from the application issuing the writes before the sync happens if you want to wait until the sync completes, since the only way the OS can provide the semantic you're asking for is to refuse to do anything with the write until the sync completes. And from the kernel's perspective, it's a lot simpler to make you block in fsync than it is to have special code in write and friends that looks to see if fsync is running on this file, and if fsync is running, it blocks recording the write until the fsync completes.

How is it O_PONIES?

Posted Dec 16, 2024 17:56 UTC (Mon) by khim (subscriber, #9252) [Link]

> And yes, that has a cost. But it's in all likelihood a lot less then having to wait until whichever version of sync completes, before you can carry on with whatever you were doing.

Indeed that's true. For readlly, really, REALLY expensive “enterprise” hardware. SAS devices, U.2/U.3, etc.

Very often these are not even used with regular filesystems, but with entirely proprietary protocols.

> But does it guarantee that anything written AFTER the call to sync, was NOT written until the sync completed?

Given the above… why would that make any sense? If you are an enterprise database then you can use O_DIRECT or even special kernel driver to achieve what you want. If you are normal userspace app designed for normal userspace and typical customer-oriented hardware then you are getting promises that hardware does and can build any abstractions on top of that that you may want.

P.S. You have to remember that storage that actually exists these days it an abstraction, too! Your SSDs don't have these 1KB blocks that your block device is exposed to you. It's all emulated – and things that “enterprise” databases may want but “normal” users can live with are sold for very mouthwatering prices.

How is it O_PONIES?

Posted Dec 16, 2024 14:22 UTC (Mon) by pizza (subscriber, #46) [Link] (4 responses)

> Well, given that Posix guarantees ?atomic?, but more importantly SEQUENTIAL writes at the OS interface, my question is why can't we have the SAME guarantee between layers in the disk stack?

Because there's more than one application/writer in the system whose operations may be interleaved with each other's?

Because those lower levels (OS, disk controllers, disks themselves) aren't aware of individual applications or anything beyond individual reads and writes?

Because those lower levels have caches and re-order writes to be non-sequential for (nearly universally demanded) performance reasons?

Because at those lower levels, the only guarantee they provide is "synchronize *all* writes prior to this point"

Because synchronization is *expensive*?

How is it O_PONIES?

Posted Dec 16, 2024 16:24 UTC (Mon) by Wol (subscriber, #4433) [Link] (3 responses)

> > Well, given that Posix guarantees ?atomic?, but more importantly SEQUENTIAL writes at the OS interface, my question is why can't we have the SAME guarantee between layers in the disk stack?

> Because there's more than one application/writer in the system whose operations may be interleaved with each other's?

And? The OS layer provides a guarantee to applications that their operations will *appear* to be interleaved to other applications. It doesn't have to provide a guarantee that they ARE interleaved. (Unless the application asks for it.)

> Because those lower levels (OS, disk controllers, disks themselves) aren't aware of individual applications or anything beyond individual reads and writes?

So you're saying they ARE aware of individual reads and writes. And? I presume they are also aware of "control writes", such as a call to fsync and friends?

> Because those lower levels have caches and re-order writes to be non-sequential for (nearly universally demanded) performance reasons?

And why can't they cache and re-order writes? It looks like khim's "aio_fsync" is sort-of what I was looking for. I'm not sure, it looks like it might not be enough, but it's close ...

> Because at those lower levels, the only guarantee they provide is "synchronize *all* writes prior to this point"

Do you mean "take each individual write prior to this point and synchronise it"? Or do you mean "take all writes prior to this point, desequentialise reorder and optimise them, and just guarantee that no write AFTER this point will reach the next layer BEFORE any writes that were before this point"?

> Because synchronization is *expensive*?

It CAN be expensive. But if the alternative is a "tar --restore", then that is ALSO expensive.

Aiui, fsync "doesn't return until the io is complete". On a heavily loaded system that can kill my application response, for a guarantee I don't even want! If aio_fsync just lets my application keep running, but passes that "control barrier" in the stream from OS to filesystem to disk driver to disk ... and allows the system to do as much optimisation as it likes BETWEEN BARRIERS. then synchronisation is as cheap or as expensive as I want it to be.

On a lightly loaded system, putting barriers into the stream will probably be unnoticeable. On a heavily loaded system, any application not using barriers is unlikely to notice much if anything. Make the application using barriers suffer the worst consequences of using them - which is what would happen. If my database application calls aio_fsync every couple of seconds, that's still a lot of optimisation per barrier-driven glitch.

The only problem I can see is that any journal really needs to be on the same filesystem/disk as what it is journalling, so that order guarantees definitely apply, but the important thing is I can reason at a granular level as to where my data is, not rely on gold-plated guarantees that make me pay heavily for what I don't want.

Cheers,
Wol

How is it O_PONIES?

Posted Dec 16, 2024 17:42 UTC (Mon) by joib (subscriber, #8541) [Link] (2 responses)

> So you're saying they ARE aware of individual reads and writes. And? I presume they are also aware of "control writes", such as a call to fsync and friends?

The disk devices (ATA, SCSI, NVME, FC, etc etc.) are obviously aware of individual read/writes, however unless you're using O_DIRECT there's no mapping between POSIX read/write/etc and these low level IO's, as the kernel can do all kinds of coalescing etc etc.

The disk devices aren't aware of fsync etc. either. An fsync(), is roughly translated into something like 1. wait for all outstanding writes to the file to complete (that is, they are at least in the disk cache). 2. Send a CACHE_FLUSH operation, which will be reported as completed when all dirty data in the cache has been written to persistent storage. (Well, in reality it's a bit more complicated as the FS needs to ensure correct ordering between file data and the metadata journal, but this is sort of the basic way that it's done.)

Note that in general at the ATA/SCSI/NVME level there's no concept of write ordering either. There used to be some kind of optional ordering support in SCSI but few if any devices implemented it, and Linux had support for that but it was ripped out a long time ago (decade and a half maybe?) as part of the reworking of how the Linux concept of "disk barriers" was implemented. Nowadays if a filesystem wants ordering it needs to implement it itself with wait-for-completion and FUA or CACHE_FLUSH.

> Aiui, fsync "doesn't return until the io is complete". On a heavily loaded system that can kill my application response, for a guarantee I don't even want! If aio_fsync just lets my application keep running, but passes that "control barrier" in the stream from OS to filesystem to disk driver to disk ... and allows the system to do as much optimisation as it likes BETWEEN BARRIERS. then synchronisation is as cheap or as expensive as I want it to be.

I'm not familiar with the implementation of aio_fsync() (or sending fsync via io_uring), but I suspect it doesn't change anything how the fsync is implemented at low level, it just changes of the fsync completion is reported to userspace (that is, use the AIO or io_uring mechanisms rather than the traditional blocking fsync()).

How is it O_PONIES?

Posted Dec 17, 2024 2:11 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

AIO is just a thread pool (or something functionally equivalent to a thread pool), according to aio(7):

> The current Linux POSIX AIO implementation is provided in user space by glibc. This has a number of limitations, most notably that maintaining multiple threads to perform I/O operations is expensive and scales poorly.

It then goes on to say that io_uring is insufficiently mature to replace this thread pool implementation.

In other words: aio_fsync is literally just "fsync, but on a different thread."

Change to fsync with io_uring

Posted Dec 17, 2024 10:00 UTC (Tue) by farnz (subscriber, #17727) [Link]

The critical thing io_uring brings to the table is chained operations; with POSIX I/O, you'd do "write, fsync, write" to get the barrier semantics, where the fsync does not complete until all previous writes to the file have completed (but not, except on ext3, all previous writes to the filesystem). This means that for a database operation, where you want to do "write to journal, confirm journal is written, write to backing store", you end up with 3 system calls, and the potential to be scheduled away between the "confirm journal is written" and "write to backing store".

With io_uring, you'd submit a chain of 3 linked I/Os; first the write, then the fsync, then the next write. The kernel now has the opportunity to know that you've got a "write to backing store" after the fsync, and can thus ensure that that write happens immediately, even if you're due to be scheduled away shortly afterwards, "stealing" timeslice from a database read, or from another write operation.

Now, I doubt that the kernel does anything like that deliberately at the moment. It's something that could be done if it was useful - for example if an io_uring using database was finding that it had N chains in flight all of which had done their "write to journal" step, and all of which were contending to fsync the journal before doing "write to backing store".