This is a regression

Posted Mar 14, 2009 22:50 UTC (Sat) by ikm (guest, #493)
In reply to: This is a regression by bojan
Parent article: Ts'o: Delayed allocation and the zero-length file problem

No one says that the users aren't listening. However, the point was and still is about this problem being a regression from ext3. And it's just unwise to say to users that they should actually "fix all their programs that work with files". That's tail wagging the dog. It's totally unrealistic and not doable in any short- or even mid-term. Why suggest this then? And who is irrational after all?

Users don't care which solution is the right one as long as it *works*. And the solution went to 2.6.30 indeed. Distributors would hopefully backport. Problem solved. Horray. But all the blabbering about how POSIX allows this and stuff is unhelpful to end-user, if surely interesting and inspiring to developers.

This is a regression

Posted Mar 15, 2009 2:02 UTC (Sun) by bojan (subscriber, #14302) [Link] (19 responses)

> However, the point was and still is about this problem being a regression from ext3.

And that is exactly why Ted, being a practical person, reverted to the old behaviour in some situations. Doesn't mean application writers should continue using incorrect idioms.

> It's totally unrealistic and not doable in any short- or even mid-term. Why suggest this then? And who is irrational after all?

Sorry, fixing bugs is irrational?

> But all the blabbering about how POSIX allows this and stuff is unhelpful to end-user, if surely interesting and inspiring to developers.

POSIX isn't blabbering (see http://www.kernel.org/):

> Linux is a clone of the operating system Unix, written from scratch by Linus Torvalds with assistance from a loosely-knit team of hackers across the Net. It aims towards POSIX and Single UNIX Specification compliance.

This is a regression

Posted Mar 15, 2009 3:12 UTC (Sun) by bojan (subscriber, #14302) [Link] (18 responses)

Speaking of regressions, the only _real_ regression here is the fact that fsync on ext3 in ordered mode may lock up your system for a few seconds. I think the real fix sequence for this should be:

1. By default make ext3 ordered mode have fsync as a no-op. People that want current broken behaviour could specify a mount option to get it.

2. Tell folks that they _must_ use fsync in order to commit their data.

3. Once critical mass of applications achieved the above, remove all hacks from ext4, XFS etc.

4. Retire ext3.

This is a regression

Posted Mar 15, 2009 4:53 UTC (Sun) by foom (subscriber, #14868) [Link] (12 responses)

I'm waiting for evilfs: the filesystem that *always* writes the deletion of data to disk synchronously,
but *never* writes any new data (file data, directory data, anything) to a permanent location until
you've called fsync on the file's fd, the containing directory's fd, and the fd of every directory up the
tree to the root (or call sync, of course).

Hopefully it can be the default fs for Ubuntu Jaded Jackal. If anyone complains, I'm sure "But POSIX
says it's okay to do that, the apps are broken for not obsessively calling sync after every write!" will
satisfy everyone. :)

This is a regression

Posted Mar 15, 2009 5:26 UTC (Sun) by bojan (subscriber, #14302) [Link] (9 responses)

Whatever.

All the people here suggesting that well established standards Linux _aims_ to implement should be ignored, should remember the screaming Microsoft had to face from the FOSS community when they started twisting various standards to their own ends.

http://en.wikipedia.org/wiki/Hypocrisy

This is a regression

Posted Mar 15, 2009 12:34 UTC (Sun) by nix (subscriber, #2304) [Link] (6 responses)

Going beyond a standard is not ignoring that standard.

This is a regression

Posted Mar 15, 2009 21:09 UTC (Sun) by bojan (subscriber, #14302) [Link] (5 responses)

So, when the spec says that applications need to call fsync to get data down on disk and they don't, that's going _beyond_ the standard? Sorry, that's falling short of it, ignoring it.

Of course, Ted put hacks into ext4 because application writers missed the above and it will take time to fix it. That's called a workaround.

This is a regression

Posted Mar 15, 2009 23:50 UTC (Sun) by nix (subscriber, #2304) [Link] (4 responses)

Er, no, 'going beyond the standard' is what *ext4* should do; i.e. it
should arrange that even if an app does something under which POSIX
*permits* data loss, that data loss is still considered bad and should be
avoided.

Agreed the apps are buggy, but I think this is a deficiency in POSIX,
rather than anything else.

This is a regression

Posted Mar 16, 2009 0:17 UTC (Mon) by bojan (subscriber, #14302) [Link] (3 responses)

> Er, no, 'going beyond the standard' is what *ext4* should do

And that's going to help the broken application running on another filesystem exactly how? The problem with hypocrisy here is not related to ext4 - it related to application code.

BTW, it is obvious that Ted already decided to make sure ext4 does that. The man is not stupid - he doesn't want the file system rejected over this - no matter how wrong the people blaming ext4 for this are.

> Agreed the apps are buggy, but I think this is a deficiency in POSIX, rather than anything else.

Well, yeah - the spec is, shall we say - demanding. But, it is what it is. We tell Microsoft not to ignore the specs. What makes us so special that we can? I would suggest nothing. If we take the right to demand that from Microsoft, we should make sure we do it ourselves.

This is a regression

Posted Mar 16, 2009 1:07 UTC (Mon) by nix (subscriber, #2304) [Link] (1 responses)

OK, so you consider 'do not want to lose data' to be 'hypocrisy'.

There's no point talking to you at all, IMNSHO.

This is a regression

Posted Mar 16, 2009 2:19 UTC (Mon) by bojan (subscriber, #14302) [Link]

No, I consider telling others to do something and then not doing it ourselves hypocrisy. It think that would be the definition of it, no?

If you don't want to talk to me, then don't. That's OK.

This is a regression

Posted Mar 16, 2009 13:45 UTC (Mon) by ikm (guest, #493) [Link]

>> Er, no, 'going beyond the standard' is what *ext4* should do

> And that's going to help the broken application running on another filesystem exactly how?

It's not. We are talking about fixing problems users start to experience when they switch from ext3 to ext4. None of the other goals, such as fixing all the apps, making all filesystems happy, feeding the hungry and making world a better place are being pursued here. The 2.6.30 fixes do what they are supposed to do, without breaking anything else. So it is a good thing, and I don't understand why you seem to be against it.

Sure, there's lots of stuff which ain't working right, but it's not a subject here. World's not perfect, and it's not going to be any time soon.

This is a regression

Posted Mar 15, 2009 12:57 UTC (Sun) by ikm (guest, #493) [Link] (1 responses)

> All the people here suggesting that well established standards Linux _aims_ to implement should be ignored

Gosh. What people suggest here is that standards should not be used as an excuse for unwanted filesystem behavior.

This is a regression

Posted Mar 16, 2009 0:21 UTC (Mon) by bojan (subscriber, #14302) [Link]

The problem with ignoring the standards is related to the applications, not the filesystem behaviour. ext4 implements POSIX specs just fine, both with and without the patches destined for 2.6.30. It is the applications that do not call fsync before rename that are ignoring the standard.

This is a regression

Posted Mar 15, 2009 12:34 UTC (Sun) by nix (subscriber, #2304) [Link] (1 responses)

Also known as DoSfs for the way it fills your memory up with unwritten
crap generated by apps that didn't call fsync().

(I wonder if we can allow it to write dirty data to disk when under memory
pressure, as well? ;) )

This is a regression

Posted Mar 15, 2009 15:18 UTC (Sun) by dcoutts (guest, #5387) [Link]

It doesn't have to keep everything in memory. It would be a simple variation on btrfs or any similar persistent filesystem. It could write all the new data to disk but not update the root node until you bludgeon it with fsync.

This is a regression

Posted Mar 15, 2009 12:20 UTC (Sun) by alexl (subscriber, #19068) [Link] (4 responses)

> 1. By default make ext3 ordered mode have fsync as a no-op. People that want current broken behaviour could specify a mount option to get it.

Are you crazy? That would break ACID guarantees for all databases, etc.
fsync() is about much more than data-before-metadata.

This is a regression

Posted Mar 15, 2009 21:28 UTC (Sun) by bojan (subscriber, #14302) [Link]

> Are you crazy?

Close to it ;-)

I admit, that was a bit tongue-in-cheek, to point out that current ext3 "lock up on fsync" behaviour is total nonsense.

This is a regression

Posted Mar 16, 2009 14:09 UTC (Mon) by ikm (guest, #493) [Link] (2 responses)

> That would break ACID guarantees for all databases, etc.

Once I had MySQL running on an XFS filesystem, and the system has hanged for some reason. The database got broken so horribly I had to restore it from backups. I wouldn't really count on any 'ACID guarantees' here :) An UPS and a ventilated dust-free environment is our only ACID guarantee :)

This is a regression

Posted Mar 17, 2009 5:41 UTC (Tue) by efexis (guest, #26355) [Link] (1 responses)

Note that mysql isn't always acid compliant and clearly states that fact, eg, when using myisam tables. Converting to innodb should fix that for you. If you were running innodb tables... then shut me up! Hehe never done any testing of this myself. Which storage engine are you using?

This is a regression

Posted Mar 17, 2009 11:59 UTC (Tue) by ikm (guest, #493) [Link]

Yep, it was myisam.