Adding fs-verity support for Fedora 36?

Adding fs-verity file-integrity information to RPM packages for Fedora 36 is the topic of a recent discussion on the Fedora devel mailing list. The feature would provide a means to install files from RPM packages as read-only files that cannot be read or otherwise operated on if the data in the files changes at any point. The proposal is mostly about making the plumbing available for use cases that are not particularly clear—which has led to some questions and skepticism among those participating in the thread.

fs-verity and RPM

Fs-verity is a kernel feature that is supported by some filesystems; it provides a way to ensure that the contents of a file cannot change on disk. It revolves around a Merkle tree that is created for each file being protected; the tree contains hashes of each data block in the file. When a file is protected by fs-verity, it is marked as read-only and every read operation checks that the block read matches the value stored in the tree; the operation fails if there is no match. In addition, the tree itself can be cryptographically signed to ensure that nothing has been changed underneath the filesystem by, say, accessing the raw block device or image file.

Fedora program manager Ben Cotton posted the Fedora change proposal to add fs-verity support on behalf of the feature owners: Davide Cavalca, Boris Burkov, Filipe Brandenburger, Michel Alexandre Salim, and Matthew Almond. There are several elements to the plan. To start with, the Koji build system needs to be able to create and sign the Merkle tree for each file that gets shipped in the RPM package. The tree itself is not added to the RPM package, just the signed top-level hash for each file.

On the other end, an optional fs-verity RPM plugin would install the Fedora key and enable fs-verity for each file it installs. The filesystem would then recreate the Merkle tree, check it against the signature in the RPM metadata, and store the tree with the file. After that, each access to the file will be checked against the tree, which means that various kinds of operations (e.g. read(), mmap(), execve(), etc.) will only proceed if the data blocks on disk have not changed.

The proposal mainly focuses on the build side of the equation: "Specifically, installing and enabling the fs-verity rpm plugin by default is explicitly considered out of scope here." The overhead of creating the Merkle tree at installation time did not "appear to meaningfully slow down package installs during empirical testing", but there is some (unspecified) cost of creating the tree for every Koji build, of course. The Merkle tree is only stored if the RPM fs-verity plugin is enabled and adds roughly 1/127th (0.8%) to the size of the installed file. All RPMs would get additional metadata, in the form of signatures, if the proposal is adopted, but even that is fairly negligible: "in the vast majority of cases we expect to see minimal to no size increase thanks to RPM header packing".

Reaction

Kevin Fenzi had a number of questions about the proposal. He wondered which keys would be used to sign the Merkle tree at build time and who would work on the changes to the RoboSignatory component that supplies signatures to Koji in order to make all of this happen. In addition, only ext4, F2FS, and, since Linux 5.15, Btrfs (commit) have support for fs-verity; what happens for other filesystems such as XFS?

Cavalca replied, noting that the feature owners plan to work on the changes to RoboSignatory, but would need some help and advice with code review, testing, and deployment. They will be looking into the possibility of using the Fedora package-signing key:

fs-verity needs a RSA key/cert pair for file signing at package signature time. At package install time, the cert needs to be loaded in the appropriate kernel keyring. We've always used a dedicated keypair during testing -- I'm not actually sure if the package signing key could be reused for this, as it's a GPG key, but this is something we should follow up on.

Beyond that, if the filesystem does not support fs-verity, the plugin will fail and there will be no protection of that sort. If it is desired, though, fs-verity support for XFS could be added, he said.

Josh Boyer asked about adding fs-verity support for Fedora when a seemingly similar feature, adding IMA signatures to RPM, was rejected due in part to its impact on RPM size. The Integrity Measurement Architecture (IMA) is a kernel-based mechanism that can be used to detect file tampering, but Salim said that the impact on RPM size is much smaller for fs-verity than for IMA. The idea is that users who do not install the RPM plugin "will be mostly unimpacted" by the change. The proposal compared the two integrity mechanisms, noting a smaller impact at run time as well:

Because fsverity operates on block reads, its runtime cost is small (as it only needs to verify the block that is being accessed), and it can protect from alterations at any point in time.

IMA works by measuring a file as a whole and comparing its signature whenever it’s read of executed. It has a higher runtime cost than fsverity (as it needs to verify the whole file at once) and it cannot detect subsequent alternations.

The proposal also said that IMA provides a rich policy system that can be tied into other security mechanisms, such as SELinux; it might make sense to integrate the two down the road. But Boyer and others were interested in what benefit fs-verity would bring to Fedora and its use cases. Boyer said that the benefit listed in the proposal could largely be fulfilled by IMA, so he wanted to know more. Cavalca said that this particular proposal was "mostly about putting in place the necessary plumbing"; he also described one potential use case:

For example, consider an appliance-like system placed in an untrusted location where you may not be able to control who has physical access (this could be a server, but it could also be a kiosk in an internet point or a school). In this scenario, fs-verity can be one of the building blocks to ensure and maintain system trust.

As with most security solutions, Cavalca continued, fs-verity is no silver bullet, but it is a useful building block:

Once fs-verity is enabled for a given file (which, in the RPM case, happens at package installation time), it cannot be disabled, and the file becomes immutable. One can still rename() or unlink() it (and this is indeed how rpm is able to replace files when upgrading packages), but the actual contents cannot be altered.

Where is this useful? For example, fs-verity can help in the scenario where an attacker has out-of-band access to the storage device (say, they pull a hard drive from a colo'd server or a sdcard from an embedded device, or they boot into a liveusb, or they access a VM image directly from a host).

Let's say that happens, and the attacker changes a few blocks of /bin/ls on the device to make it run nefarious code. When you boot your system again, it would fail at exec() time because the Merkle tree wouldn't match.

But since an attacker can simply replace the files, or add new ones, there is still a missing piece, as Zbigniew Jędrzejewski-Szmek pointed out:

If fs-verity verification prevents me from successfully modifying or replacing /usr/bin/foo or /usr/lib/systemd/system/foo.service, is there anything which hinders just adding /etc/systemd/system/foo.service that does whatever I want?

He asked if there was a Linux Security Module (LSM) available to enforce whatever policies are needed to ensure the integrity of the system. Beyond that, if an attacker can replace a binary like /bin/ls, they may also be able to add new keys to the kernel keyring that would allow their binaries to pass the fs-verity checks.

If the keys are loaded from the file system, can I just drop in a rogue key, similarly to what happens when new keys are distributed as part of distro upgrades?

Florian Weimer described two additional problems that he called "unsolvable". Attackers can reuse binaries from unrelated Fedora packages that have been signed properly, or they can alter the system configuration to disable the feature at boot time. Given those problems, he said, it is unlikely that anyone who actually wants this kind of a feature would use the signatures being added to the RPM files:

The combination of these two unsolvable issues suggests to me that anyone who wants to deploy this is better off with their own trust root, and that approach will include their particular version of key management as well. But this also means that pre-computed file signatures are not [particularly] useful to them. They would have to discard them anyway before deployment.

Burkov agreed that there were some missing pieces, but noted that a message from Roberto Sassu mentioned some ongoing work integrating fs-verity with IMA that might lead to a solution to some of these problems. But, in a similar vein, Lennart Poettering pointed out a number of areas where the threat model is not at all clear—the protections are not really geared to the problems that Fedora users might have:

This protects file contents, not the [metadata], right? So what about the metadata? if I see a fs-verity enabled inode with libssl.so data in it today, and it's a vulnerably version, and I make a hardlink to it, and then it gets replaced by a fixed version (with a slightly updated name) — how you intend to make sure, that i can't fool you into loading my copy of the old file but under the new name?

[...] Is there any protection against downgrades between RPM package versions? Does this in any way protect combinations of binaries/libraries? I mean, pretty much all programs we ship consist of a large number of ELF objects, and you probably need to sign the combination of them, but this model doesn't look like it offers that at all?

There are some areas that need further research, Burkov said, but the threat model is not really specified because the feature is simply meant as plumbing:

The short answer is that just this change as-is has no practical threat model, we viewed it as an enablement step that we believe has a realistic path to broadly authenticated rpm contents. Alternatively, it does give a sophisticated user an opportunity to build something like that for themselves as well.

Poettering expressed further skepticism about the proposal, noting: "Infrastructure that has no consuming feature typically doesn't work. Experience tells us that." But, even if that were overcome, the protection provided is not particularly useful: "If I can have 'ping' and 'poweroff' binaries both signed by Fedora, but then swap them without the signing being able to detect that, why do this at all?"

There were other concerns raised as well. For example, Stephen John Smoogen objected on logistical grounds. Any changes need to be in place before the mass rebuild for Fedora 36, which is currently scheduled for January 19. Given the upcoming holiday season, it is pretty unlikely that those who need to review and sign off on these kinds of changes will even be available to do so. "Even if the changes are trivial ones, this is a very short window to land things." He suggested that Fedora 37 would be a more realistic target.

The wiki version of the change proposal has been updated with further information, along with responses to many of the questions and concerns posted in the thread. But without a real use case that benefits some actual subset of Fedora users, this has the look of a feature that is going nowhere fast. Perhaps what is needed is to join forces with those who are working on IMA to come up with an integrated plan that does provide a threat model and real use cases. The IMA proposal, for example, pointed to the Fedora IoT edition as a possible consumer of that feature, which is more in line with the existing use of fs-verity in Android. Boyd made it clear that Red Hat is interested in seeing IMA support in Fedora, so there would perhaps be some synergy in a combined effort. That approach may find more traction with the Fedora community.