Ts'o: Delayed allocation and the zero-length file problem [LWN.net]

Ts'o: Delayed allocation and the zero-length file problem

Posted Mar 13, 2009 19:16 UTC (Fri) by endecotp (guest, #36428)
In reply to: Ts'o: Delayed allocation and the zero-length file problem by jgg
Parent article: Ts'o: Delayed allocation and the zero-length file problem

> you didn't do a fsync on the directory!

Yes, I was just thinking the same thing! Come on Ted, what exactly do you want us to write to be portably safe? I have just added an fsync() to my write() close() rename() code, but I checked man fsync first and it tells me that I need to fsync the directory. So is it:

open()
write()
fsync()
close()
rename()
opendir()
fsync()
closedir()

? Or some re-ordering of that? Is there more? Do I have to fsync() the directories up to the root? Can I avoid all this if I call sync()?

(Log in to post comments)

Ts'o: Delayed allocation and the zero-length file problem

Posted Mar 13, 2009 20:17 UTC (Fri) by alexl (subscriber, #19068) [Link]

I don't think that is quite necessary for durability.

If the metadata is not written out but the data is and then things crash, you will just have the old file as it was, and either a written file+inode with no name (moved to lost+found) or the written file with the temporary name.

As far as i can see syncing the directory is not needed. (Unless you want to guarantee the file being on disk, rather than just not breaking the atomic file replace behaviour.)

Ts'o: Delayed allocation and the zero-length file problem

Posted Mar 13, 2009 21:03 UTC (Fri) by endecotp (guest, #36428) [Link]

OK. So if I want code that's portable to ext2, I need to fsync the directory. Maybe there aren't many people using ext2 these days, but I would like code that's genuinely portable; I do personally care about the various flash filesystems, and when I break things for BSD users they complain. So I guess the directory fsync is needed.

Thinking a bit more about this from the "application requirements" point of view, I can see three cases:

1- The change needs to be atomic wrt other processes running concurrently.
2- The change needs to be atomic if this process terminates (ctrl-C, OOM).
3- The change needs to be atomic if the system crashes.

I can't think of a scenario where the application author would reasonably say, "I need this data to be safe in cases 1 and 2 but I don't care about 3." Can anyone else?

Ts'o: Delayed allocation and the zero-length file problem

Posted Mar 13, 2009 22:45 UTC (Fri) by jgg (guest, #55211) [Link]

> I can't think of a scenario where the application author would reasonably say, "I need this data to be safe in cases 1 and 2 but I don't care about 3." Can anyone else?

It isn't that uncommon really, anytime you want to protect against a failing program and not mess up its output file rename is the best way. For instance programs that are used with make should be careful to not leave garbage around if they crash. rsync and other downloading software does the rename trick too, for the same basic reasons. None of these uses require fsync or other performance sucking things.

The reason things like emacs and vim are so careful is because they are almost always handling critical data. I don't think anyone would advocate rsync should use fsync.

The considerable variations in what FSs do is also why, as an admin, I have a habit of knocking off a quick 'sync' command after finishing some adminy task just to be certain :)

Ts'o: Delayed allocation and the zero-length file problem

Posted Mar 16, 2009 10:37 UTC (Mon) by endecotp (guest, #36428) [Link]

> Come on Ted, what exactly do you want us to write to be portably safe?

Ted seems to have answered this in his second blog post: YES you DO need to fsync the directory if you want to be certain that the metadata has been saved.