COW filesystems and the stall problem

Posted Nov 7, 2013 23:43 UTC (Thu) by jhhaller (guest, #56103)
Parent article: The pernicious USB-stick stall problem

This change may have interesting affects on COW filesystems like btrfs. I have been playing with an interesting combination of mythtv and btrfs. When recording HDTV, mythtv writes about 6GB of data per hour per channel being recorded, or about 16MB/s. Because of the cache/stall effects, mythtv calls fsync once per second, to avoid filling the cache and causing the problems mentioned in the article. While watching live TV, the program is written to disk by one process, and read by another, in a multimedia version of less+tail -f. On some occasions, when fsync runs, there is more than just a little bit of disk I/O, my guess is that the COW semantics are causing more than just the recently written data to be written. I wouldn't otherwise notice this, but the reading process hangs (causing the displayed TV program to hang) until the fsync completes, which can take seconds.

My question is whether writing dirty pages back more quickly will have a big effect on COW filesystem performance. Copying a large file to a COW filesystem may trigger more COW actions than before.

COW filesystems and the stall problem

Posted Nov 11, 2013 0:45 UTC (Mon) by giraffedata (guest, #1954) [Link] (2 responses)

fsync is a rather primitive way to cause the system not to keep memory needlessly dirty.

Lots of fadvise and madvise flags have been developed over the years; doesn't one of them get the writeback happening immediately? I created a TV recorder based on a ca 2004 Linux kernel that used msync(MS_ASYNC) for that purpose (yes, I made it write the file via mmap just so I could use msync).

Of course, if the issue is that you think the recorder might not actually be able to keep up with the data arriving, and want to make sure when that happens it just drops TV data and doesn't cripple the rest of the system too, then you do need something synchronous like fsync.

COW filesystems and the stall problem

Posted Nov 11, 2013 1:59 UTC (Mon) by jhhaller (guest, #56103) [Link] (1 responses)

The point was not how mythtv handles this, but that using more aggressive memory pressure may cause COW filesystems to have lower performance, by forcing large files writes to be flushed sooner than they currently do, with consequent COW filesystem behaviors to create copies of at least the tree data and last group of blocks originally written when the next memory pressure forces another write. For my case, I could adjust the configuration of the directory holding the video files to disable the COW behavior, but that's harder to do for the general case. I suspect msync would cause the same problem for a COW filesystem as fsync.

COW filesystems and the stall problem

Posted Nov 11, 2013 4:16 UTC (Mon) by giraffedata (guest, #1954) [Link]

Yes, I was not responding to the point.

The actual point lost me, because I don't see the connection between the high-volume writing and reading you described and copy-on-write and the stalling of the playback. But I also don't know anything about mythtv or btrfs specifically.

I don't think there's anything inherent in COW that means if you flush a large file write sooner that you make more copies of tree data or some group of blocks, but I may have just totally missed the scenario you have in mind.