Wishful thinking [LWN.net]

Wishful thinking

Posted Mar 16, 2009 22:07 UTC (Mon) by bojan (subscriber, #14302)
In reply to: Wishful thinking by quotemstr
Parent article: Garrett: ext4, application expectations and power management

> First of all, you're most likely never going to see gigabytes of dirty blocks for a single file.

You are right. And nobody will ever need more than 640 kB of RAM ;-)

> They'll have been flushed well before your rename!

Not if someone calls fsync on that directory. Or kernel decides (for whatever reason) that this directory must go out to disk.

But look, you obviously don't want to accept that:

1. This happens.
2. POSIX says what it says.
3. Kernel is allowed to do what POSIX says.

That's OK. The documentation is crystal clear on this. It is all in the manual pages for close() and rename(). Unfortunately, people choose to ignore it.

Sure, it would be nice to have a call that guarantees all this, but thinking that rename() is that call is simply false.

to post comments

Wishful thinking

Posted Mar 16, 2009 22:12 UTC (Mon) by quotemstr (subscriber, #45331) [Link] (5 responses)

And you clearly don't want to accept that we can improve over POSIX with little danger and a large upside. POSIX doesn't make any guarantees about what happens on an unclean shutdown. The kernel could overwrite all your files with pictures of carrots and it'd be POSIX-complaint. Would you be okay with that outcome? After all,

This happens.
POSIX says what it says.
Kernel is allowed to do what POSIX says.

Please stop justifying poor behavior by resorting to POSIX.

Wishful thinking

Posted Mar 16, 2009 22:17 UTC (Mon) by bojan (subscriber, #14302) [Link] (4 responses)

> little danger

Yeah, we've seen that with applications that were losing files on a perfectly good FS.

> Please stop justifying poor behavior by resorting to POSIX.

The behaviour is not poor. It is there for a reason, which you don't want to admit.

Lucky Ted is a nice man, so he put workarounds in place for folks that want to continue using sloppy code on ext4.

Wishful thinking

Posted Mar 16, 2009 22:24 UTC (Mon) by quotemstr (subscriber, #45331) [Link] (3 responses)

Yeah, we've seen that with applications that were losing files on a perfectly good FS.

What are you talking about? A reordering rename is strictly safer than your broken one.

The behaviour is not poor. It is there for a reason, which you don't want to admit.

No, the behavior is dangerous and unintuitive, and there is no sound reason for it to work that way other than to make metadata write-out a little simpler. There is no performance or correctness upside to rename working the way you insist. You don't want to admit that POSIX may allow something that is nevertheless nonsensical.

Wishful thinking

Posted Mar 16, 2009 22:42 UTC (Mon) by bojan (subscriber, #14302) [Link] (2 responses)

> There is no performance or correctness upside to rename working the way you insist.

I'll just answer about correctness. If you take a broken application to a perfectly good system that doesn't order renames, because it doesn't have to, you will lose data. So, there is an upside to programming correctly and according to spec.

I think I answered the performance bit elsewhere, but you don't want to accept it. Which is fine by me.

> You don't want to admit that POSIX may allow something that is nevertheless nonsensical.

POSIX is not nonsensical, it is completely asynchronous and unordered, which is what you don't seem to like. Sure, we could have another mechanism for ordered renames - I don't deny that. It's just that current rename() isn't it, which you don't seem to be able to understand.

Wishful thinking

Posted Mar 16, 2009 22:53 UTC (Mon) by quotemstr (subscriber, #45331) [Link] (1 responses)

If you take a broken application to a perfectly good system that doesn't order renames, because it doesn't have to, you will lose data.

Adhering to POSIX is not the same as being "perfectly good".

POSIX is not nonsensical, it is completely asynchronous and unordered

No, it's perfectly synchronous, ordered, and atomic with respect to a running system. It's only in the case of an unclean shutdown that we disagree, and POSIX really doesn't say much at all about that scenario. My behavior, your behavior, and overwriting with carrots are all perfectly POSIX-compliant with respect to a system that's been shut down uncleanly. To the greatest extent possible, the state of a system after an unclean shutdown should reflect the state the system was in shortly before that shutdown, and an ordered rename goes a long way toward achieving that.

Wishful thinking

Posted Mar 16, 2009 23:29 UTC (Mon) by bojan (subscriber, #14302) [Link]

> Adhering to POSIX is not the same as being "perfectly good".

I love it how you think that your or my opinion actually matters here. What matters is what's been written in the documentation for years now. That is the _only_ objective thing programmers on both ends of this can rely on.

> No, it's perfectly synchronous, ordered, and atomic with respect to a running system.

Yeah, confuse the issue some more, when you don't have anything new to add.

We are discussing here the "ordering of writes to disk on rename". In respect to this, POSIX is asynchronous and unordered. Heck, you cannot even tell if two consecutive writes will be written in that order to disk.

> My behavior, your behavior, and overwriting with carrots are all perfectly POSIX-compliant with respect to a system that's been shut down uncleanly.

Your behaviour is truly what you would like it to be. What you are describing as my behaviour is what the documentation actually says, so it's not mine at all.

As or carrots, that is wrong, because you never fsynced carrots to disk, so they should not be there. But sure - implement it ;-)

> To the greatest extent possible, the state of a system after an unclean shutdown should reflect the state the system was in shortly before that shutdown, and an ordered rename goes a long way toward achieving that.

And also encourages application writers to keep writing broken code, file system writers to put hacks into the system to work around that broken code etc.