Another Fedora integrity-management proposal

File-integrity management for the Fedora distribution has been the overarching theme of a number of different feature proposals over the last year or so. In general, they have been met with skepticism, particularly with regard to how well the features mesh with Fedora's goals, but also in how they will change the process of building RPM packages. A new proposal that would allow systems to (optionally) perform remote attestation is likewise encountering headwinds; there are several different concerns being raised in the discussion of it.

Background

As is usual for feature proposals, Fedora program manager Ben Cotton posted it to the Fedora devel mailing list on behalf of the feature owner: Roberto Sassu. The change proposal is also on the Fedora wiki. The new feature would use the Digest Lists Integrity Module (DIGLIM) feature, which has been proposed by Sassu as an addition to the kernel's Integrity Measurement Architecture (IMA). Ensuring that file contents and metadata do not change in unexpected ways is IMA's job; DIGLIM is an optimization of sorts to IMA.

IMA has a number of different functions, but at its core it maintains "digests" of file contents and metadata; these digests are cryptographic hashes that can be used to reliably detect file changes. IMA can also use the digests, in combination with the system's Trusted Platform Module (TPM), to calculate a value that proves that the system is running a known set of software. That value can be used to ensure the system has been securely booted or it can be sent elsewhere to remotely attest to the state of the system.

Each file being protected by IMA needs its digest stored with the file, which is normally done using extended attributes in the filesystem. IMA can be configured to check each file before it is accessed to see if its digest still matches the stored value; if not, access can be denied. As files are assessed, their digest can be submitted to the TPM to extend a Platform Configuration Register (PCR); the resulting value is a reflection of the files measured, but it is also affected by the order of the accesses.

According to the DIGLIM proposals (for Fedora and the patch set for the kernel), parallel execution during the assessment results in differing values from the TPM; even if the same code is used, it may result in a different attestation value. DIGLIM provides a mechanism to take a digest value of all of the files installed, instead, and use that for calculating the attestation value. Only files that have digests that were not included in the overall "installation digest" would be used to further extend the PCR in the TPM.

It does so by providing a mechanism to enroll digest values from the installed files into a kernel "digest list", which can then be consulted as files are accessed. If the digest of a file appears on the list, it can be considered to be unchanged and its digest value does not get submitted to the TPM; otherwise, the file has been modified or was not included in the digest list at all, so access could be denied and the file's digest added into the attestation value. The latter would likely mean that the system fails its attestation.

One might expect that adding digests for all of the files installed on a system would be expensive in terms of memory, but it turns out to have fairly modest impact. "A repository of executables and shared libraries digests, of a Fedora 33 minimal installation, occupies less than 800k of memory."

Instead of calculating the digest values for each file, DIGLIM for Fedora would use the values that are already calculated and stored in the RPM header for the packages that get installed. Those values are signed with the GPG key for the Fedora release, so they can be trusted. The kernel needs to be able to verify GPG signatures, which is part of the DIGLIM kernel feature; the Fedora release GPG public key could then be added to the kernel keyring and used for verification. The RPM headers would be processed early in the boot sequence: "A user space parser, executed by the kernel during early boot, parses RPM headers found in /etc/diglim in the initial ram disk (included with a custom dracut script) and uploads them to the kernel."

In part, the idea behind DIGLIM is to provide a way for systems to handle integrity management without the distributions having to be intimately involved in the process. For systems where remote attestation is wanted, there will be ways to achieve it without Fedora (or any other distribution) having to directly manage the process. DIGLIM would also allow the detection of tampering with the installed files, as with IMA, but it can do so with better performance than the standard IMA mechanism, according to the kernel patch description.

There is a fairly long list of kernel prerequisites that need to be maintained "and possibly have them accepted in the upstream kernel". In fact, as the discussion thread made clear, Fedora is unlikely to adopt the feature unless those patches do make their way upstream. There is some additional work that needs done for Fedora so that the RPM headers get processed properly for those systems that wish to enable the feature, which is the other piece of the proposal.

Reaction

The optional nature of the feature, and the generally fairly narrow use cases for it, mean that it will have limited impact for the vast majority of Fedora users. It also means that it will have limited—or no—utility for those users. Some of those with concerns, or objections, seemed to misunderstand that it would only be enabled by user choice, not distribution fiat. For example, Kevin Kofler said: "[...] I do not see how the 'feature' implemented by this Change provides any value at all that does not contradict the very definition of Free Software." He is concerned about being unable to install software built by a third party or by the user themselves.

But, as Mattia Verga put it, the intent is quite different: "It doesn't deny a user to install any software they want, it is about preventing unwanted/unsolicited/malevolent software from being installed without user (admin) approval." Kofler is worried about a slide toward iOS-style centralization of the control of software sources, however, which would obviously run counter to free-software ideals. But few seem to see this effort as headed down that path. Michel Alexandre Salim envisioned something more in keeping with Fedora's goals:

If/when something like this gets shipped, I hope Fedora limits itself to shipping a policy that is the equivalent of SELinux's 'targeted' policy: protect the RPMs that Fedora ships from being tampered with, let users do whatever on top. With an option to turn it off completely or to enforce more strictly.

In fact, Sassu said his work on DIGLIM provides a good example of how third-party code might be handled:

I'm using myself a COPR [Cool Other Package Repo] repository to build the kernel package with the DIGLIM patches. That kernel also includes my own GPG key generated for me by COPR.

This has not been decided yet, but likely the Fedora kernel will contain the official Fedora keys, and the user will decide to add new keys (including those from COPR).

In a longer message, Sassu sought to alleviate some of the concerns and fears about the feature: "[...] its primary goal is to aid the users to satisfy their security needs, and let them decide how this will be done". He noted that the feature is already in production use in the Fedora-based openEuler distribution, but he also recognized that getting the feature upstream was important. Neal Gompa agreed that Fedora generally "does not include non-upstream functionality in its Linux kernel builds", but noted that DIGLIM could be useful:

I also agree that this feature is unlikely to affect people, as this feature will not be enabled by default. It would be extremely useful for people building Fedora-based appliances which need tamper protection for various reasons. And Fedora derivatives (like RHEL/CentOS, Amazon Linux, openEuler, etc.) can benefit from us having the functionality integrated even if we don't enable it by default.

DRM

It is clearly true that the TPM and remote attestation can be used for various digital rights management (DRM)—digital restrictions management, to some—schemes. Nico Kadel-Garcia is convinced that the feature "has no use but digital rights management". But Sassu pointed out that there are other uses:

If you want to enforce an IMA measurement policy instead, access to the files will be always granted, regardless of whether the digest lists are signed or not. IMA, in this case, will simply record the execution of unknown files, in addition to the digest lists you generated.

The IMA measurement list remains in your system, unless you decide that your system should be remotely attested by a remote verifier.

Kadel-Garcia was adamant that it is all about DRM, but Gompa tried to clarify further:

The difference between IMA/verity and DRM is that the former is under the system owner's control (in this case, *you*), and the latter is *not*.

[...] There is a ton of value in user-controlled versions of this capability. And again, none of this is on by default, it's up to *you* to turn it on.

Implementation questions

Zbigniew Jędrzejewski-Szmek had some questions about the feature and its implementation. In particular, the need for a user-space parser that is run by the kernel was questioned, but Sassu said that the helper needs to be run before the init process, so that init (and all of its dependencies) can be integrity checked using the digest list. It is something of a chicken-and-egg problem, which also led Sassu to statically link the helper program; only the digest of that one file needs to be added before it can run to add all of the rest of the digests.

Lennart Poettering thought that the digest information should simply be extracted into a special initrd that can be passed to the kernel. That would avoid having to "upload" the digests to the kernel, but, as Sassu pointed out, that would tie the kernel to a particular digest-list format. The initial version of the feature more or less worked that way, Sassu said; the parser was in the kernel and it read the information from the RPM headers, but every time a new format (e.g. a Debian deb file) needs to be supported, the kernel would have to change.

Poettering was also unhappy with the statically linked helper ("Static linking is a mess."), but Sassu explained that it makes it simpler to bootstrap the integrity checking:

I consider this approach self-contained: everything is needed to bootstrap DIGLIM is contained in the kernel-tools-diglim package. With dynamic linking, you also have to take care of all shared libraries. Since the parser is not yet functional (the kernel is in enforcing mode), you need to generate a digest list in the native format (in the spec file) for every shared library you want to load.

I liked the fact that, once you have the modified kernel with the appropriate GPG keys, and kernel-tools-diglim, you are able to run IMA appraisal without additional effort for its management.

The relationship between the DIGLIM feature and the recently discussed fs-verity change was raised in the January 3 meeting of the Fedora Engineering Steering Committee (FESCo). The main difference, Sassu said, is that DIGLIM works with the existing RPM packages and format, while the fs-verity support would need to add additional per-file information in the RPM header:

DIGLIM adopts the current scheme of RPMs, and verifies with one signature all the files contained in the RPM. Since this data format is not suitable for use by the Linux kernel, for enforcing the integrity policy, DIGLIM extracts the digests and adds them in a hash table stored in kernel memory. Enforcement (it would be better to say security decision) is achieved by doing a lookup in the hash table.

The main advantage is that DIGLIM can achieve its objective, providing reference values, without any change to existing RPMs.

RPM developer Panu Matilainen was happy to see that approach:

Besides not bloating up RPMs with seriously expensive per-file data, this side-steps the other issues associated with both IMA and fs-verity: both require separate signing steps for the file signatures which is a non-trivial cost and complexity, and unlike those the file hashes are covered (and thus protected) by normal rpm-level signatures too.

It seems clear that the DIGLIM feature will not be adopted by Fedora until and unless DIGLIM itself gets merged into the mainline kernel—and maybe not even then. The concerns about locked-down systems and DRM are reasonable to a certain extent, but that is not at all what DIGLIM is targeting. On the other hand, though, it seems like a niche feature, at best; even though it will have a negligible impact for most Fedora users, that is a bit of a double-edged sword. It will not impact most of them because it will not help them or their use cases either.

But there are use cases that will benefit, and the other contenders for integrity management in Fedora seem even more complicated. DIGLIM and fs-verity are both on FESCo's radar (here and here), so we should know the outcome soon. Given the intrusiveness of the fs-verity scheme, and the unclear status of DIGLIM in the mainline, one might guess that both will be pushed off until Fedora 37—at least.