User: Password:
Subscribe / Log in / New account

Ensuring data reaches disk

Ensuring data reaches disk

Posted Sep 9, 2011 8:23 UTC (Fri) by k8to (subscriber, #15413)
Parent article: Ensuring data reaches disk

The problem with fsync() is that while it may be necessary to, for example, ensure that your newly created file actually gets contents on disk before you overwrite the origina (the update pattern), there is no guarantee that this has acceptable performance implications.

Say you are updating the state of a system frequently, eg every 30 seconds, and you wish to create a new file representing this new state, and replace the previous one. You can call open(tempname), write(tmpfile), fsync(tmpfile), close(tmpfile), rename(), fsync(dir) and be sure that this data has landed as it shouldat the end of this string of actions, and that your window of data incoherency is at least limited to the time between the rename and the second fsync, if there is any incoherency at all.

Unfortunately, for many filesystems, the first fsync will cause an I/O storm, pushing out a very larger amount of writeback cached data, because the filesystem is not designed for partial synchronization to disk. Similarly the second fsync may cause an I/O storm.

You can continue working, in an awkward fashion, by leaving this blocking call in a thread, but you cannot avoid the fact that the fsync() calls may cause you to be no longer able to meet reasonable workload targets.

If you have this combination of performance sensitivity and correctness, you have to go down the ugly path of specialcasing on a filesystem and platform basis.

(Log in to post comments)

Ensuring data reaches disk

Posted Sep 9, 2011 9:10 UTC (Fri) by trasz (guest, #45786) [Link]

It might be worth mentioning that ZFS has an interesting solution for that: ZIL. Basically, it's a small log intended just for operations that need to be synced to disk; since it's a log, the writes are fast, and it's possible to move it to a separate log device.

Ensuring data reaches disk

Posted Sep 9, 2011 15:59 UTC (Fri) by k8to (subscriber, #15413) [Link]

Yep, on some systems it's as cheap as you could hope!

Ensuring data reaches disk

Posted Sep 23, 2011 20:56 UTC (Fri) by oak (guest, #2786) [Link]

How much using fdatasync() instead of fsync() would help?

Copyright © 2017, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds