LWN: Comments on "Ensuring data reaches disk"

Ensuring data reaches disk

b10s — Sat, 25 Jun 2022 10:48:15 +0000

Hi, thank you the article!

> Now, since the amount of data transferred is already known, and given the nature of network communications (they can be bursty and/or slow), we've decided to use libc's stream functions (fwrite() and fflush(), represented by "Library Buffers" in the figure above) in order to further buffer the data.

But actually reading is done without libc's stream function:

11 ret = read(sockfd, buf, MY_BUF_SIZE);

May I ask, why have you mentioned nature of network here?

---

What if we do fsync() but will not do fflush() in your code example?
The answer most likely - nothing will happen, since fsync flushes OS cache but our data is in libc cache.

---

> Writes using these functions may not result in system calls, meaning that the data still lives in buffers in the application's address space after making such a function call

Except fflush()?

"Flushing output on a buffered stream means transmitting all accumulated characters to the file".
https://www.gnu.org/software/libc/manual/html_node/Flushi...

It sounds like fflush() ends with some system call, isn't it? If so, which? (just write()?)

---

> I/O operations performed against files opened with O_DIRECT bypass the kernel's page cache, writing directly to the storage. Recall that the storage may itself store the data in a write-back cache, so fsync() is still required for files opened with O_DIRECT in order to save the data to stable storage.

Then why O_DIRECT might be needed at all if we still have to call fsync()? What are use cases?

---

Do storages hide their cache? May kernel directly write to drive's stable storage?
I've heard some storages do hide their cache.

Ensuring data reaches disk

farnz — Mon, 09 Nov 2020 18:27:56 +0000

To be fair to btrfs, that's it's USP compared to ext4 - when hardware fails, it lets you know that your data has been eaten at the time of the issue, and not months down the line.

And knowing consumer hardware, chances are very high that it did commit everything properly, and then had a catastrophic failure when there was a surprise power-down. Unfortunately, unless you have an acceptance lab verifying that kit complies with the intent of the spec, it often complies with the letter of the spec (if you're lucky) and no more :-(

Ensuring data reaches disk

zlynx — Mon, 09 Nov 2020 17:11:31 +0000

More fun is consumer grade SSDs that protect their metadata during power-loss but not necessarily the data.

I had to rebuild a btrfs volume because my laptop battery ran down in the bag and on reboot the drive contained blocks saying writes had completed, but those data blocks had old data in them. In other words, data that had been committed to physical storage (or that was CLAIMED by the drive) was no longer present after power-loss. It probably had to fsck or equivalent on the Flash FTL and lost some bits.

btrfs gets very upset about that.

I guess this behavior is still better than some older SSDs which had to be secure-erased and reformatted after losing their entire FTL? I guess.

Ensuring data reaches disk

Wol — Mon, 09 Nov 2020 10:17:29 +0000

Ouch. As a database guy I'm desperate for "queued flush straight barrier", because if you want data integrity that at least makes reasoning possible - "if the transaction log is incomplete, revert; if the data write is complete, continue; if the log is complete and the data write isn't, re-play the log".

If you can't be sure what has or hasn't hit the disk - the nightmare scenario is "part of the log, and part of the data" - then you get the hoops that I believe SQLite and PostgreSQL go through :-(

Cheers,
Wol

How disappointing

Wol — Mon, 09 Nov 2020 10:11:54 +0000

Except I was comparing Unix against proprietary OS's from the likes of DEC, Honeywell, Pr1me, etc.

While Unix was eating the mini-computers' lunch, yes, Windows came along and started eating its lunch ...

Cheers,
Wol

Ensuring data reaches disk

farnz — Mon, 09 Nov 2020 09:56:43 +0000

Yes, such a command is needed, and the various interface specs (ATA, SCSI, NVMe) all have standardised commands for flushing the cache.

At a minimum, you get a FLUSH CACHE or SYNCHRONIZE CACHE type command, which is specified as not completing until all data in the cache is in persistent storage; this is enough to implement fsync() behaviour; beyond that, you can also have forced unit access (FUA) commands, which do not complete until the data written is on the persistent media, and even partial flush commands that only affect some sections of the drive.

There's an added layer of complexity in that some standards have queued flushes which act as straight barriers (all commands before the flush complete, then the flush happens, then the rest of the queue); others have queued flushes that only affect commands issued before the flush in this queue (and can over-flush by flushing data from later commands in the queue), and yet others only have unqueued flushes which require you to idle the interface, wait for the flush to complete, and then resume issuing commands.

How disappointing

jem — Mon, 09 Nov 2020 07:52:10 +0000

The trouble with POSIX is it is based on Unix and, sorry guys, Unix is crap as a commercial OS. It won because it was cheap and good enough.

No, commercial Unix lost to Windows because Windows was cheap and good enough. Only 99 dollars!. (This was not a real ad, though.)

I also don't buy the argument that Unix cost more to maintain per user. Back then, Unix was a multi-user operating system that was centrally administered. Then came DOS and Windows and every user had their individual problems.

avoiding orphan files

Wol — Sun, 08 Nov 2020 23:17:30 +0000

Except it doesn't work - see my comment about hard-linked files ...

Cheers,
Wol

Ensuring data reaches disk

Wol — Sun, 08 Nov 2020 23:15:40 +0000

Not that this is necessarily the way it's done, but linux does have a disk database. I see that often enough watching the raid stuff. It's quite possible (though probably not) that linux looks up the drive characteristics.

Cheers,
Wol

License of example files

Wol — Sun, 08 Nov 2020 23:09:33 +0000

How about "these examples are too simple to be worthy of copyright" ... ?

The law itself sets out a vague line, so just declare that this stuff falls the wrong side of the line.

Cheers,
Wol

How disappointing

Wol — Sun, 08 Nov 2020 23:05:06 +0000

And then what happens if the file is hardlinked, and you want to modify the original file, not make a modified copy...

The trouble with POSIX is it is based on Unix and, sorry guys, Unix is crap as a commercial OS. It won because it was cheap and good enough.

And I curse it regularly because, unlike a lot of people today, I've actually had experience of real commercial OSs. Trouble is, they've died because they cost too much to maintain :-(

(Mind you, I've used real commercial OSs that had those flags to do fancy file-system stuff, and when they have bugs they really do have bugs ...)

Cheers,
Wol

Ensuring data reaches disk

yzou93 — Sun, 08 Nov 2020 21:55:01 +0000

An awesome article.
My question about fsync() is how the OS could control/know the device-internal caching behavior.
When designing a block device hardware, for example if Samsung wants to design a new SSD, is a cache control support for fsync() command issued from OS required?

Thank you.

Ensuring data reaches disk

ppai — Mon, 01 Dec 2014 13:47:25 +0000

I'm wondering about this case:
Directory "a/b/c" already exists.

create("/tmp/whatever")
write("/tmp/whatever")
fsync("/tmp/whatever")
os.makedirs("a/b/c/d/e")
rename("/tmp/whatever", "a/b/c/d/e/obj.data")
fsync("a/b/c/d/e/")

Is it really required to fsync dirs all the way from e to a ? Fsync is totally not necessary for "a/b/c" as it already existed. But after doing a makedirs(), there's no way to know which subtree of "a/b/c/d/e" needs fsync().
Is it reasonable to fsync only the containing directory and expect the filesystem to take care of the rest (to make sure the entire tree makes it to disk) ?

Direct Reads

etienne — Mon, 22 Apr 2013 11:37:00 +0000

> ... data read from disk and not kernel buffers ...

Maybe echo 1, 2 or 3 to /proc/sys/vm/drop_caches ?

Direct Reads

nikm — Sun, 21 Apr 2013 03:16:31 +0000

I am interesting if there is a way to ensure that when you are reading data, these data is directly from the disk. As I wonder in case of reading even if the DIRECT_IO flag is used the data propably is read from kerner buffer. Is that correct?

thanx

avoiding orphan files

droundy — Wed, 25 Jan 2012 20:38:48 +0000

> The write/sync/rename process is hardly an ideal way to implement atomic replacement semantics. There are simply too many potential points of failure.

True, but it's also the only one we've got, right?

code feedback ...

droundy — Wed, 25 Jan 2012 20:34:47 +0000

I prefer to avoid putting buffers on the stack simply to reduce the difficulties associated with buffer overflows. Not for security reasons (most of my programming is scientific), but simply so overwriting the buffer won't trash the stack, making debugging harder. And also to allow valgrind to immediately recognize a buffer overwrite...

Ensuring data reaches disk

oak — Fri, 23 Sep 2011 20:56:34 +0000

How much using fdatasync() instead of fsync() would help?

How disappointing

spitzak — Fri, 23 Sep 2011 00:59:49 +0000

If it is opened with the atomic-replace semantic, I would just have plain close() do the replacement.

There may be a need to somehow "abort" the file so that it is as though you never started writing it. But it may be sufficient to do this if the process owning the fd exits without calling close().

I very much disagree with others that say POSIX should be followed. The suggested method of writing a file is what is wanted in probably 95% of the time that files are written. It should be the basic operation, while "dynamic other processes can see the blocks change as I write them" is an extremely rare operation that should be the one requiring complex hacks.

avoiding orphan files

nybble41 — Mon, 19 Sep 2011 18:17:35 +0000

> Also in the write/sync/rename workflow, what happens if the temp file is on a separate filesystem?

In the write/sync/rename workflow, this is never supposed to occur. The temp file must always be on the same filesystem as the real file for the atomic-rename guarantee to apply.

Naturally, this can be extremely difficult to achieve in some cases. The file may be a symlink, which must be fully resolved to a symlink-free path to determine the real filesystem. The file may be the target of a bind mount, in which case I doubt there is any portable way to determine which filesystem it came from. And there there's the possibility that you can write to the file, but not the directory _containing_ the file...

The write/sync/rename process is hardly an ideal way to implement atomic replacement semantics. There are simply too many potential points of failure.

avoiding orphan files

mathstuf — Mon, 19 Sep 2011 17:15:29 +0000

As someone who uses symlinks to manage dotfiles in another repository, the write/sync/rename workflow is annoying as all hell. Tin (since moved to slrn), gpg, pidgin, weechat (which is why I'm still using irssi), and more all force me to manually copy the file to the real location and remake the symlink. If there was a way to do the workflow and then replay the writes to an fd returned by open() on the original path, it would be much better. So far, it looks as if all of the above solutions still fail for me.

Also in the write/sync/rename workflow, what happens if the temp file is on a separate filesystem? There's a copy involved there, so there is a time when the file is not atomically replaced (unless I'm missing some guarantee by POSIX in this case).

avoiding orphan files

aeriksson — Mon, 19 Sep 2011 12:06:44 +0000

The upshot of using the write/sync/rename workflow is of course that the original file is left untouched until there is a fully comitted replacment ready on disk. The downside is that you need to create a temporary file with a temporary filename while doing it. This is bad for crash recovery, where you'd leave orphan files on the filesystem.

Is there a way to sole that which I have overlooked?

fd=open("",O_UNNAMED);
....
rename_unnamed(fd,"/some/file");

Opening device nodes

bjencks — Fri, 16 Sep 2011 22:50:48 +0000

What are the semantics when you open a device node (either block, e.g. disk, or char, e.g. tape)? Does the kernel ever use page cache for device files? Does O_DIRECT do anything? What about O_SYNC? Does fsync always generate a barrier?

Also, what about the different disk abstraction layers (LVM, dm-crypt, MD RAID, DRBD, etc) -- what's involved in passing an fsync() all the way down the stack?

code feedback ...

bronson — Fri, 16 Sep 2011 19:17:22 +0000

That was painfully true 20 years ago. With modern memory management it doesn't matter much. Within reason of course -- a 20 MB buffer should probably still go on the heap.

code feedback ...

renox — Fri, 16 Sep 2011 13:43:18 +0000

> you forgot to free(buf) after the while loop and in the early returns inside of the loop. might as well just use the stack: char buf[MY_BUF_SIZE];

Is-it such a good idea?
I though that it was better to keep the stack small.

Ensuring data reaches disk

andresfreund — Fri, 16 Sep 2011 10:35:52 +0000

If the storage device has a write cache but no independent power supply you have the problem that you will loose data on power loss because O_DIRECT will only guarantee that the write reaches the device, not that it reaches persistent storage inside that device.
For that you need to issue some special commands - which e.g. fsync() knows how to do.
Besides an O_DIRECT write doesn't guarantee that metadata updates have reached stable storage.

Ensuring data reaches disk

scheck — Fri, 16 Sep 2011 09:16:57 +0000

"Recall that the storage may itself store the data in a write-back cache, so fsync() is still required for files opened with O_DIRECT in order to save the data to stable storage."

Why should I use fsync() for files opened with O_DIRECT and why has the storage device's cache anything to do with it?

Apart from that a very nice and comprehensible article. Thank you.

code feedback ...

jwakely — Wed, 14 Sep 2011 11:59:16 +0000

the cast isn't needed with a C++ compiler either

code feedback ...

vapier — Wed, 14 Sep 2011 05:33:28 +0000

in the past i've been bitten where fwrite was given a char* and size==num bytes to write and nmemb==1 (like in the example here). but perhaps that was a bug in the lower layers (it was a ppc/glibc setup). i do know that size==sizeof(*buf) has always worked for me ;).

code feedback ...

jzbiciak — Tue, 13 Sep 2011 06:08:33 +0000

Err... I guess the read error path returns -1 or 0, which again I think may be an error, unless you wanted to return 0 when the connection drops before "nrbytes" gets read. Oops.

That raises a different question: If you exit early due to the socket dropping, you won't fflush/fsync. Seems like you want a 'break' if read returned 0 and errno != EINTR, don't you?

code feedback ...

jzbiciak — Tue, 13 Sep 2011 06:00:56 +0000

another common mistake: the size/nmemb args are swapped ... the size is "1" (since sizeof(*buf) is 1 (a char)), and the number of elements is "ret". once you fix the arg order, the method of clobbering the value of ret won't work in the "if" check ...

I don't think it's an error. In the example, if fwrite returns anything other than '1', then it reports an error. This is an "all-or-nothing" fwrite. If it fails, 'ret' will be 0, otherwise it will be 1. The semantic is "write 1 buffer of size 'ret' bytes."

I see nothing wrong with this, and it matches the if (ret != 1) statement that follows. Sure, you don't get to find out how many bytes did get written, but the code wasn't interested in that anyway. And, it's one less variable that's "live across call," so the resulting compiler output may be fractionally smaller/faster. (While I can think of smaller microoptimizations, this type of microoptimization is pretty far down the list, I must admit.)

Personally, I think the code might be clearer breaking 'ret' up into multiple variables. For example, if you did switch size/nmemb, you might rewrite the loop like so:

      while (tot_written < nrbytes) {
              int remaining = nrbytes - tot_written;
              int to_read   = remaining > MY_BUF_SIZE ? MY_BUF_SIZE : remaining;

              read_ret = read(sockfd, buf, to_read);
              if (read_ret <= 0) {
                      if (errno == EINTR)
                              continue;
                      return read_ret;
              }
              write_ret = fwrite((void *)buf, 1, read_ret, outfp);
              tot_written += write_ret;
              if (write_ret != read_ret)
                      return ferror(outfp);
      }

Written that way, you could easily add a way to return how many bytes did get written.

Also, the return value is inconsistent. I think "return ferror(outfp)" is wrong. ferror returns non-zero on an error, but it isn't guaranteed to be negative. The other paths through this function return positive values on success, so shouldn't it be simply "return -1;" to match the read error path (which also simply returns -1, and maybe should be written as such)? ie:

      while (tot_written < nrbytes) {
              int remaining = nrbytes - tot_written;
              int to_read   = remaining > MY_BUF_SIZE ? MY_BUF_SIZE : remaining;

              read_ret = read(sockfd, buf, to_read);
              if (read_ret <= 0) {
                      if (errno == EINTR)
                              continue;
                      return -1;
              }
              write_ret = fwrite((void *)buf, 1, read_ret, outfp);
              tot_written += write_ret;
              if (write_ret != read_ret)
                      return -1;
      }

How disappointing

nix — Sun, 11 Sep 2011 23:51:19 +0000

Again, ugh ('with'?). I'd simply say close_replace(), no need for a flag or indeed any parameters at all. This means it has the same prototype as close(), so if anyone wants to choose between calling close() or close_replace() at runtime, they can just use a function pointer.

How disappointing

sionescu — Sun, 11 Sep 2011 22:14:21 +0000

How about "close_with_flags" ?

How disappointing

nix — Sun, 11 Sep 2011 21:52:56 +0000

True, though close2() is a horrible name (as is wait$num() and accept$num()): give it a name that reflects its purpose.

How disappointing

sionescu — Sun, 11 Sep 2011 21:10:24 +0000

Why POSIX ? There are other Linux-specific open flags, did Ulrich object to every one of them ?

The new syscall could be called close2, adding a "flags" parameter - in the spirit of accept4() et al.

How disappointing

nix — Sun, 11 Sep 2011 20:45:55 +0000

Sure. Practicalities: you could do it to open() (though you'd have to get the change into POSIX before Ulrich would let it past), but you could never do that to close() without breaking every C program ever written. You could call it close_replace(), perhaps?

How disappointing

sionescu — Sun, 11 Sep 2011 16:00:42 +0000

No, it's the common way of implementing atomic commit when *modifying* the data, but it's not what I have in mind, which is this:

open(path, O_REPLACE) only allocates a new inode

close(fd, CLOSE_COMMIT) atomically replaces the reference to the old inode with the new inode(just like rename) copying all metadata except for the (a|c|m)time, then calls fsync()

easy, isn't it ?

How disappointing

butlerm — Sun, 11 Sep 2011 00:12:37 +0000

>Who said anything about locking ?

I mention locking because it is the most common way to implement atomic commit semantics, from the perspective of all other processes. Your idea makes great sense as long as you have multiversion read concurrency, so that existing openers can see an old, read only version of the file indefinitely.

POSIX simply has a different solution for that, as I am sure you know - the name / inode distinction, which allows you to delete a file, or rename replace it with a new version without locking other processes out, waiting, or disturbing existing openers.

It is unfortunate of course that there is no standard call to clone an existing file's extended attributes and security context for use in a rename replace transaction - perhaps one should be added, it would be a worthwhile enhancement. Hating UNIX when it is vastly superior to the most widely distributed alternative in this respect seems a bit pointless to me.

code feedback ...

vapier — Sat, 10 Sep 2011 21:02:28 +0000

 5      char *buf = malloc(MY_BUF_SIZE);

you forgot to free(buf) after the while loop and in the early returns inside of the loop. might as well just use the stack: char buf[MY_BUF_SIZE];

11              ret = read(sockfd, buf, MY_BUF_SIZE);

common mistake. the len should be min(MY_BUF_SIZE, nrbytes - written). otherwise, if (nrbytes % MY_BUF_SIZE) is non-zero, you read too many bytes from the sockfd and they get lost.

12              if (ret =< 0) {

typo ... should be "<=" as "=<" doesn't compile.

18              ret = fwrite((void *)buf, ret, 1, outfp);
19              if (ret != 1)

unless you build this with a C++ compiler, that cast is not needed. and another common mistake: the size/nmemb args are swapped ... the size is "1" (since sizeof(*buf) is 1 (a char)), and the number of elements is "ret". once you fix the arg order, the method of clobbering the value of ret won't work in the "if" check ...

27      ret = fsync(fileno(outfp));
28      if (ret < 0)
29              return -1;
30      return 0;

at this point, you could just as easily write:

    return fsync(fileno(outfp));

How disappointing

sionescu — Fri, 09 Sep 2011 22:45:06 +0000

Who said anything about locking ?