Ted speaks again [LWN.net]

Ted speaks again

Posted Mar 16, 2009 13:17 UTC (Mon) by kleptog (subscriber, #1183)
In reply to: Ted speaks again by forthy
Parent article: Garrett: ext4, application expectations and power management

As far as I can see all this is just changing expectations. Just a few years ago we were *happy* that our filesystems were readable after a crash (after running fsck). Then we progressed to being happy that after a crash we could use the filesystem without waiting hours for the fsck.

Now we're at the stage of worrying about exactly what the files should look like after a crash. Give it a few years and I'm sure we'll find something else to worry about. Also, POSIX was written a long time ago and deliberately vague on some points because they wanted to support many existing systems which all worked slightly differently.

NB: ISTM the solution to the 'lots of little files on ext3' problem is obvious. Create all the new files, then fsync them (fsync on ext3 may be slow, but it wouldn't be as much of a problem this way because all the data would be written out for all the files in one go). Finally rename them all.

to post comments

Ted speaks again

Posted Mar 16, 2009 15:25 UTC (Mon) by drag (guest, #31333) [Link] (1 responses)

> Now we're at the stage of worrying about exactly what the files should look like after a crash. Give it a few years and I'm sure we'll find something else to worry about. Also, POSIX was written a long time ago and deliberately vague on some points because they wanted to support many existing systems which all worked slightly differently.

Well ya. That's progress I guess. People always want better, demand better.

In the case of Linux your traditionally dealing with half-way decent hardware running with UPS and ran by professionals. That is your designing the OS to perform well and reliably when managed by a person who knows, understands, and cares quite a bit about the hardware they are using.

Now with consumer-oriented Linux devices your dealing with people constantly putting excessive demands and loads on the system (especially graphics, which has been a weak point in stability for all systems including Linux) devices that are cheap and mass produced, ran by people that don't even understand what a OS is, have to operate with as low as power usage as possible, and have users with very low tolerances for anything really technical.

In this specific case your having Ubuntu users using unstable graphics drivers with developer versions of the operating system. They were crashing their system frequently; several times a day sometimes. They are doing weird things like over clocking RAM and all that crap.

They were finding that Ext4 was eating a significant portion of their file system, were as with Ext3 it didn't.

But that is just a tip of the iceberg. Your going to deal with mobile phones with batteries that just 'crap out'. Your going to deal with mobile internet devices that get used in abusive environments. Your going to deal with hand held devices that suspend to ram a dozen times a minute.

Try explaining to your grandma or to the guy down the street running a Moblin netbook that their system is not bootable anymore, or they can't use most of their applications, because POSIX doesn't give a shit that users get half their file system blown away when they shut their devices down incorrectly.

I don't know the best way to fix it, whether it's best to:
* Get the Kernel developers to care about maintaining a consistent file system image on the disk at all times
or
* Get the biggest clue stick in the world and collectively drive the "fsync is your friend" point home to all potential Linux developers.
or
* third option

I don't know.

But certainly demands and expectations change. Just like everything else in the computing landscape changes.

Ted speaks again

Posted Mar 16, 2009 16:54 UTC (Mon) by kleptog (subscriber, #1183) [Link]

Honestly, I don't see why POSIX should care. It's a standard that describes an API that can be used by programs that wish to be portable. In principle it could be implemented on anything from the smallest handheld to the largest mainframe. Reliability after a crash is outside the purview of POSIX since the requirements are vastly different in different situations. People writing software for embedded devices don't rely on POSIX to give them crash safety, they read the manuals for the device to see what the manufacturers say they should do.

POSIX compliance is a property of the OS-userspace boundary, crash-safety is a property of an entire system. They're largely orthogonal.

In my opinion it's wrong for people to say that either behaviour is mandated by POSIX. IMHO it's neither mandated nor forbidden. Crash reliability is a contract between you and the OS+hardware+kernel. A ramdisk can be POSIX compliant yet is clearly not crash safe. Leave POSIX out of it, decide what Linux wants to guarantee. POSIX provides a way of guaranteeing a certain reliability but Linux is free to provide additional guarantees if it sees fit.

Maybe something for LSB? I'd like to see the language lawyers work out a way of defining "crash-safety" in a way that doesn't exclude things like ramdisks and several existing filesystems.