Block-layer error injection

Storage code has to cope with hardware that fails in inconvenient ways, but coaxing a healthy disk into producing those failures on demand, for testing, is usually not possible. The kernel provides several ways to inject block-layer I/O errors, but none of those can select the operation to fail, pick the status code to return, or target a disk directly without employing a stacked device on top. Use of a stacked device means the test runs against the mapper device, not the disk it was meant to exercise. A patch series from Christoph Hellwig adds a configurable error-injection interface that does all three things that the current error-injection code lacks, controlled by a per-disk debugfs file.

Existing methods

The kernel has had a form of block-layer fault injection since 2006, when Akinobu Mita added fault-injection infrastructure. The block-layer portion (called fail_make_request) exposes debugfs knobs for controlling the probability, interval, and number of times that a fault will be injected. Its shortcoming is that it treats every request the same. It cannot tell a read from a write or a discard, cannot restrict failures to a range of sectors, and can only fail a request with BLK_STS_IOERR. The failure happens in submit_bio_noacct(), before the request reaches the driver. The filesystem or other kernel code that submitted the bio structure representing the request sees -EIO. If nothing retries the request on the way up, user space sees EIO from the system call. A single status is limiting for a subsystem in which a media error, a transport error, and a timeout each take a different path through the recovery code.

A second fault-injection mechanism, should_fail_bio(), was added by Howard McLauchlan in 2018 as a hook for BPF programs. That function is annotated with ALLOW_ERROR_INJECTION(), so the kernel's error-injection framework allows BPF programs to override the return value. Before that change, the bio submission path called should_fail_request() to determine whether an error return should be injected and passed only the disk device and a byte count. It now calls should_fail_bio() instead, which takes the bio and calls should_fail_request(), so the older feature behaves as before until a BPF program overrides the call. Such a program can read the bio and decide, for each request, whether to fail it based on the operation type or the sectors involved. That gives should_fail_bio() the selectivity that fail_make_request lacks.

The approach has a limitation, though. BPF programs can only replace a return value. A program can select which bio to fail, but not how it fails: submit_bio_noacct() ignores the value returned and completes the bio with BLK_STS_IOERR either way.

The device-mapper subsystem offers the other common approach, in the form of targets built to fail I/O requests. The simplest, dm-error, maps a region that returns an error for every request it receives. Like fail_make_request, it can produce only BLK_STS_IOERR, and it fails every command routed through the target regardless of operation or sector. To inject failures into I/O requests made to an existing device, a dm-error target must be stacked over that device. The test is then directed at the resulting mapper device, not the device that needs testing.

The dm-flakey and dm-dust targets are more configurable, and can, for example, fail I/O requests intermittently or emulate individual bad sectors. They still share the limitations that matter here: no real choice of error status and no good handling of commands other than reads and writes (such as zone operations or discards). Being device-mapper targets, they are also bound by the alignment rules that device-mapper imposes on the sector ranges a target may cover. dm-dust does not support zoned devices at all, while dm-flakey, like any zoned-capable target, can only confine failures to a whole-zone-aligned region, since a target's range must begin and end on a zone boundary. And, like dm-error, they require a stacked block device on top of the device under test.

The proposed mechanism

Hellwig's proposal, enabled through the new CONFIG_BLK_ERROR_INJECTION Kconfig option, takes a more direct route. When it is turned on, the block layer creates an error_injection file in debugfs, under /sys/kernel/debug/block/, for every registered gendisk. Reading the file lists the injection entries in effect for that disk; writing to it either adds a rule or, with a removeall command, clears every rule for the disk. The series documents the format in Documentation/block/error-injection.rst.

An entry is a short, comma-separated string. Two of its fields are mandatory: op, the operation to fail, taken from the block layer's operation names (READ, WRITE, DISCARD, the zone operations, and so on), and status, the block-layer status to return (IOERR, TIMEOUT, TRANSPORT, and so on). Three more are optional. A start sector and an nr_sectors count confine the failure to a range, leaving unrelated I/O requests to the same disk untouched; they default to sector zero and the remainder of the device, so that, without them, every request of the named operation type is a candidate. A chance value makes the failure probabilistic: with a value of N, a matching request fails with probability 1/N. It defaults to one, meaning the request always fails.

To fail one in every ten reads aimed at sectors 1000 through 1499 of nvme0n1 with a transport error, for example, a test would write:

    $ cd /sys/kernel/debug/block/nvme0n1
    $ echo 'add,op=READ,start=1000,nr_sectors=500,status=TRANSPORT,chance=10' > error_injection

Reading the same file back lists each active rule by its sector range, operation, status, and chance; the entry above appears as:

    1000:1499 op=READ,status=TRANSPORT,chance=10

Implementation details

Almost all of the new code lives in a single file: block/error-injection.c. The hook into the I/O path sits in submit_bio_noacct_nocheck(), which calls blk_error_inject() for every bio before it is submitted. That inline function does nothing unless a static key is enabled; the first rule added on any disk turns it on. A per-disk GD_ERROR_INJECT bit narrows the work further to the disks that carry rules. Only then does the code drop into the out-of-line __blk_error_inject(). The static key was Jens Axboe's suggestion. Without it, every bio would dereference its way to a per-disk state bit that is almost always clear; with it, the branch is patched out entirely when injection is not in use.

__blk_error_inject() walks the target disk's list of rules under RCU, comparing each rule's operation against the bio, checking whether the sector ranges overlap, and, if a chance value was given, rolling a virtual N-sided die. The first rule that matches wins: bi_status is set to that rule's status, bio_endio() completes the bio, and submit_bio_noacct_nocheck() returns without submitting anything to the device. Because a new rule goes at the head of the list and the walk stops at the first match, a newer rule takes precedence over an older one covering the same I/O request; overlapping and even duplicate rules are allowed on purpose.

One case the matcher never fires on is the zero-length bio: the range test cannot match a bio that carries no sectors, so pure cache flushes and ZONE_RESET_ALL operations cannot be failed at all. A special case for them would be easy to add, but a pure flush reaches the block layer as an empty write carrying REQ_PREFLUSH rather than as an operation of its own, so every WRITE rule would start firing on flushes as well. As Hellwig noted in a code comment, making this work properly first requires the block layer to use REQ_OP_FLUSH for pure flushes at the bio level, as it already does in the blk-mq I/O scheduler; a rule could then target flushes directly.

Hellwig's implementation stays small and self-contained, reusing the existing per-disk debugfs infrastructure and not relying on other kernel subsystems. That simplicity was the core of his case against a more elaborate design.

BPF came up in the review as a possible foundation for the mechanism itself; Daniel Gomez proposed a BPF_PROG_TYPE_STRUCT_OPS hook that would keep only the bio_endio_status() call in the kernel and move the matching policy into a loadable program. Hellwig rejected the idea as too much machinery for the job. A program deciding which requests to fail has to read the operation and the sector range out of the bio, and exposing those through BPF type format (BTF) (which is how BPF programs can access the internals of kernel types) would turn the block layer's command and status codes into a stable interface, Hellwig said, which he did not want for internals that need to keep changing. Gomez answered that compile-once-run-everywhere relocations resolve field offsets against the BTF of the running kernel, so a change in layout would not break an existing program. That, Hellwig replied, assumes the fields will still exist in that form at all; the bio iterator is due for rework, and "we have to build up a huge abstraction here first".

A BPF hook would also require libbpf and BTF support inside a test virtual machine. The sector matching would move into the program as well, since Hellwig had found no BPF map type suited to range lookups, so each range to be failed would need code of its own, which he said "makes the thing very hard to use". He had prototyped BPF-based injection himself; as he put it, "it was a mess", and he preferred "about 300 lines of simple code that can be directly used from a shell script".

The patch adding the mechanism was merged for the Linux 7.2 release. Work has continued since. Hellwig fixed a static-key imbalance reported by Le Moal, then added a blktests test: block/044. It creates read and write rules over separate sector ranges, checks that the injected errors reach the caller, and confirms that malformed rules are rejected. The test also loads and unloads scsi_debug with no rules active, exercising the path behind the static-key fix. A later patch from Jackie Liu adding the affected operation to the output in debugfs landed in 7.2-rc4, producing the read-back format shown above. The bio-level flush change that would allow flushes to be failed has yet to appear. Still, users should look forward to expanded options for testing in Linux 7.2.