The problem is with the dirty buffer caches which can grow quite large compared with the storage I/O capacity, and the limited mechanisms to flush dirty pages to disk. While the amount of dirty pages which can exist without being flushed has been reduced recently, it's still quite large. For a simple example, copy a huge file across a network while watching the disk activity lights. Even with a copy running at 1 Gbps, the disk shows no activity for several seconds, then is solid on for several seconds, and the pattern repeats. While I understand that for some use cases, such as tmp files, that not writing the files which will shortly be deleted is desirable, there are cases which will need to be persisted. For the persistent cases, it would be desirable to start writing dirty pages as they fill, as the lowest priority I/O. Then, when fsync is called, there should be few dirty blocks which need to be written. Perhaps a F_MINIMIZEDIRTYBLOCKS fcntl option is in order.