The intersection of unstable pages and direct I/O

Longtime LWN readers will have encountered the concept of "stable pages" before; it was first covered here nearly 15 years ago. For the most part, the problem that stable pages were meant to solve — preventing errors when user space modifies a buffer that is under I/O — has been dealt with. But recent discussions show that there is one area where problems remain: direct I/O. There is some disagreement, though, over whether those problems are the result of user-space bugs and how much of a performance price should be paid to address them.

Writing a page of data to a block storage device takes time. If a process tells the kernel to perform a write, some time will thus pass before the operation is complete. Should that process modify the under-I/O data while it is being written, the result is a sort of data race with the usual unpredictable results. Either the old or the new data could end up being written; in the worst case, a combination of the old and new data could be written, corrupting the file.

In simple cases, this behavior is less of a problem than it seems; if the application is modifying the pages under I/O, it is likely to write them to persistent storage another time, and the end result will be as it should be. But if, for example, the filesystem is writing checksums alongside the data, changing the data partway through can cause the resulting checksum to not match. Other types of processing during I/O, such as transparent compression or a RAID implementation, can run into the same sort of trouble. In these cases, a badly timed change to a page under I/O can lead to I/O errors or data corruption.

The solution to this problem usually takes the form of stable pages — a guarantee made by the kernel that data will not be changed while it is under I/O. Stability can be achieved by, for example, simply delaying any attempts to modify the data until the I/O operation is complete. For buffered I/O, where data is copied through the page cache, implementing this approach is relatively straightforward, and was done years ago. Delaying memory access can reduce performance, though, so it is usually only done when there is a clear need.

Direct I/O, however, is a special case. When a process requests direct I/O, it wants to transfer data directly between its buffers and the underlying storage device without going through the page cache. Since stable pages are tied to the page cache, direct I/O operations cannot use them. Direct I/O is a relatively advanced, low-level operation, where applications take charge of their own buffer management, so problems from concurrent data changes have generally not been seen. Increasingly, though, there is interest in using direct I/O in conjunction with filesystem-integrity features, or with hardware features like the data integrity field (also known as "protection information" or "PI"), which provides end-to-end checksum protection.

Nobody covers the Linux kernel like LWN; be in the know with a one-month trial subscription, no credit card needed.

In May 2025, Qu Wenruo posted a patch describing problems that come about when user space performs direct I/O to a Btrfs filesystem with checksums enabled, then modifies its buffers while the I/O is underway. As described above, that can create checksum errors. Evidently, QEMU can be configured to use direct I/O and is prone to just this kind of failure. The solution that was adopted (and merged for 6.15), is to simply fall back to buffered I/O on filesystems where checksums are in use. That solves the problem, but at a heavy cost; direct I/O is now simply incompatible with checksums on Btrfs filesystems, and applications that depend on direct I/O for performance will be slowed down accordingly. There is a similar patch under consideration to fall back to buffered I/O for reads as well, since concurrent modifications to the buffer being read into can cause checksum failures.

The Btrfs change was applied without much fanfare; Christoph Hellwig's patch adding a similar fallback to XFS attracted rather more attention. XFS does not support data checksums like Btrfs does, but block devices can perform their own checksumming; they need stable pages to do that work successfully. As already noted, stable pages don't work with direct I/O so, in cases where the underlying block device requires stable pages, Hellwig's patch causes XFS to fall back to buffered I/O even when direct I/O has been requested. He expressed unhappiness about the resulting performance penalty, and suggested a few ways for "applications that know what they are doing" to disable this fallback.

Bart Van Assche had an alternative suggestion: "only fall back to buffered I/O for buggy software that modifies direct I/O buffers before I/O has completed". Dave Chinner also questioned the patch, also suggesting that buffered I/O should only be forced for applications that are "known to corrupt stable page based block devices", of which he suspected there are relatively few. Geoff Beck suggested setting the page permissions to prevent changes to memory while it is under direct I/O.

Hellwig's responses to those suggestions highlight the core of the disagreement around this topic. Is changing a buffer that is under I/O an acceptable thing for an application to do? Hellwig repeatedly insisted that it is indeed acceptable: "Given that we never claimed that you can't modify the buffer I would not call it buggy, even if the behavior is unfortunate". Concurrent modification works in many cases, he said; his objective is to make it work in the remaining cases, even at the cost of severely reduced I/O performance.

Chinner, instead, was adamant that the kernel should not try to support applications that perform this sort of modification:

Remember: O_DIRECT means the application takes full responsibility for ensuring IO concurrency semantics are correctly implemented. Modifying IO buffers whilst the IO buffer is being read from or written to by the hardware has always been an IO concurrency bug in the application.

The behaviour being talked about here is, and always has been, an application IO concurrency bug, regardless of PI, stable writes, etc. Such an application bug existing is *not a valid reason for the kernel or filesystem to disable Direct IO*.

Hellwig, though, reiterated that the kernel has never documented any requirement that buffers be untouched while under direct I/O: "We can't just make that up 20+ years later". He also said that, on RAID devices, modifying the data being written could corrupt an entire stripe, with possible effects on data unrelated to the offending application. At that point, he said, it becomes a kernel bug that could let malicious users corrupt others' data. Jan Kara said that he agreed that concurrent modification of I/O buffers is an application bug, but he added that the kernel needs to do something about the problem anyway.

Darrick Wong, addressing specifically the PI case where the hardware verifies checksums provided by the software, observed that, in most cases, applications will be unable to corrupt their own data by modifying buffers under direct I/O without the corruption being detected by the hardware. In those cases, there is no need to fall back to buffered I/O. The RAID case is worse, though, and must be prevented. Wong suggested adding a new block-device attribute (with the name "mutation_blast_radius") describing how serious a concurrent modification would be and only falling back in the most serious cases. As Hellwig pointed out, though, that solution only addresses the PI case, and would cause QEMU to fail.

While no firm conclusions were reached in this conversation, it seems clear that, regardless of how one views the behavior of applications that modify data that is under I/O, the kernel probably needs to do something to prevent, at a minimum, the worst problems. The real question will be whether it will be possible to provide a way for well-behaved applications to avoid the extra overhead of buffered I/O without creating opportunities for malicious actors. It may turn out that, in the end, some types of storage device will simply not be usable in the direct-I/O mode.

Index entries for this article
Kernel	Block layer/Direct I/O
Kernel	Data integrity
Kernel	Stable pages

---

to post comments