Fedora and supply-chain attacks

The specter of more events like the SolarWinds supply-chain attacks is something that concerns many in our communities—and beyond. Linux distributions provide a supply chain that obviously needs to be protected against attackers injecting malicious code into the update stream. This problem recently came up on the Fedora devel mailing list, which led to a discussion covering a few different topics. For the most part, Fedora users are protected against such attacks, which is not to say there is nothing more to be done, of course.

The SolarWinds attacks subverted the normal update mechanism of various tools to install malware on systems throughout much of the US government; the malware then exploited other flaws to get access to email and other data. Huzaifa Sidhpurwala posted a message to the mailing list on June 11 wondering whether the rekor tool might be useful in helping to prevent similar attacks against Fedora. Rekor comes from the sigstore project that was announced by the Linux Foundation back in March.

Sigstore is meant as a mechanism to securely record cryptographic signatures of binaries, packages, container images, and similar build artifacts in a tamper-proof ledger. The project was founded by Red Hat, Google, Purdue University, and the Linux Foundation. The press release announcement outlines the problem being solved:

Very few open source projects cryptographically sign software release artifacts. This is largely due to the challenges software maintainers face on key management, key compromise / revocation and the distribution of public keys and artifact digests. In turn, users are left to seek out which keys to trust and learn steps needed to validate signing. Further problems exist in how digests and public keys are distributed, often stored on websites susceptible to hacks or a README file situated on a public git repository. sigstore seeks to solve these issues by utilization of short lived ephemeral keys with a trust root leveraged from an open and auditable public transparency logs.

While it may well be true that the majority of projects on GitHub, GitLab, and other project-hosting sites are not signing their artifacts, that is not the case for most Linux distributions. For Fedora, RPM packages are signed and those signatures are checked by the DNF package manager. Other distributions have similar mechanisms. But there are still situations where some kind of external log of the signatures might be useful.

For example, if the Fedora signing keys were to ever be compromised or subverted, attackers could use the keys to sign malicious packages, which DNF would happily accept. But that means the attackers need to be positioned such that they can either intercept the DNF traffic (difficult with HTTPS) or tamper with the package repository. Even without a key compromise, a malicious (or compromised) package repository could hold back some updates to try to ensure that vulnerabilities in those packages continue to be present on the client systems. For the official Fedora mirrors, the metalink feature (originally added to Yum, the DNF predecessor) protects users by skipping mirrors that are more than three days behind in keeping up-to-date with the distribution's package updates.

In the posting, Sidhpurwala noted that Linux distributions, "which are made of thousands of components built from sources they [don't] even have control over", are potentially more vulnerable to supply-chain attacks because of that lack of control. One way that might start to be addressed is to be able "to ensure that there is no distribution point tampering of binaries shipped in Fedora". He wanted Fedora developers to start thinking about how a tool like rekor might be used to help prevent those kinds of problems. It would seem, though, that there is no huge unfilled hole that an immutable log of package hashes would solve, though there are use cases where it could perhaps be helpful.

Björn Persson wondered what "distribution points" were meant: "Repository mirrors? The master repository? mirrors.fedoraproject.org?" He noted that mirrors cannot modify packages (without the key), but that they could perhaps withhold updates; "Is that a thing that Rekor could prevent?" The sigstore web site might have helped answer the latter question, but it was not usable in his eyes:

On that page I can't see anything but a page header and a pointless animation. Even the text that is right there in the HTML code is hidden. Instead it wants me to execute a bunch of Javascript from at least three different domains. When a website expects me to execute some unknown program before they'll even tell me who they are or what they do, then I'm much more likely to just ignore that website.

That is, of course, a common complaint about JavaScript-only web sites these days, but it is perhaps a bit ironic that information about a security-related project is "hidden" in that way. In any case, Neal Gompa filled him in on sigstore, though Gompa was fairly dismissive of the project, at least so far. Persson had mentioned the ability to sign the repository metadata as another level of protection for Fedora, but Gompa complained that it has not been implemented for the Fedora repositories.

Gompa said that openSUSE signs its RPM repository metadata without any problems; in another message, he noted that CentOS added the feature in 2018. But Fedora infrastructure lead Kevin Fenzi pointed out that there have been numerous problems in trying to add the feature to Fedora along the way. But he is not opposed to doing so eventually:

I'm ok with enabling signed repodata when it is possible to do so and don't cause pain and suffering for our users. This has in fact always been my answer to this. It's not a priority for me personally because metalinks handle almost all the cases. The only things that would be improved by signed repodata are local mirrors that sync without checking for the mirror they sync from being up to date in mirrormanager.

Fenzi also described more of the details of the protection that is provided in the default case (where systems get their updates from the Fedora repository mirrors):

In the default configuration you get a metalink from mirrors.fedoraproject.org. That metalink has checksums for the last 2 repomd.xml files in it. If a mirror you go to then offers you a incorrect repomd.xml or any files (which are in turn checksumed in repomd.xml) dnf will treat the mirror as broken and skip to the next one.

So, assuming the Fedora keys are not compromised, which would allow an attacker to sign anything as if it came from Fedora, of course, then the majority of use cases are covered even without adding signed repository data. Local mirrors, which are typically run by organizations to reduce internet traffic and/or to support systems behind a restrictive firewall, might need some additional features, however. (As Fenzi put it: "We could do more for local mirrors indeed."

But, however unlikely it is, there is always a non-zero probability of a signing key compromise; folks unwinding a catastrophic event of that nature might find a log like that created by rekor to be of some use. Luke Hinds, who is the security engineering lead for Red Hat, described how that could work:

Rekor is a transparency log that stores signing manifests into a merkle tree, it uses the same back end technology as certificate transparency

Rekor has what are termed 'types' that allow it to ingest custom schemas and hash those into the transparency log. For those that don't know, a transparency log is immutable / append only datastore that is publicly auditable. One of the types we have is an rpm type to ingest and store signing materials from an RPM package.

Why is this useful? You get a timestamped / tamper resistance record of all signing events. This is very useful for understanding the exact blast radius of a key compromise and monitoring for suspicious events. Most of the time you will not interact with the tlog, it would sit off to side hashing in entries, so it would make no changes to a maintainers working flow or not being in any build path where it could cause an outage. It's the sort of system most would not be aware of, unless something awful happens, and then it has a lot of value, as you have an immutable record of events.

The "not being in any build path" part was confusing and Hinds walked that part back, but the idea is interesting, though it may be of limited utility. If the Fedora build infrastructure were set up to record all of its legitimate signing activities in the log, any package with a valid signature that did not appear there would be a red flag indicating a key compromise. That, of course, assumes that the attacker cannot also add entries to the log, which may or may not be the case.

As Miloslav Trmac pointed out, though, the rekor log may not actually provide anything new: "[...] any other logging solution from that build/signing system (just send it to a write-only log host) would have very similar security properties, it seems to me." In order for it to be truly useful, Trmac said in a follow-up, clients would need to query the log before installing an RPM to ensure that it is on the list. But, of course, an attacker who can create malicious packages with valid signatures may well be able to use the same machinery to add a log entry. Beyond that: "Given the above discussion about signing metadata, presumably a much smaller change, what makes such a Rekor integration more likely to get done?", Trmac asked.

If rekor (or something like it) is truly going to be useful for Fedora, it would seem that a clearer picture of what kinds of threats it would target—and how it would be superior to the existing protections—is needed. It sometimes seems that blockchain-type mechanisms (broadly defined) can be seen as a solution in search of a problem. While sigstore may well have applications for other types of projects—supply-chain attacks are real and dangerous, after all—Linux distributions may not be a good fit for this kind of tamper-proof log. Distribution security is not perfect, by any means, but the developers on those projects have been grappling with these kinds of problems for a really long time at this point. For Fedora, it probably makes more sense to try to fill in some of the holes that have been identified before climbing up on the Merkle-tree craze.

Note: you can avoid this step in the future by logging into your LWN account.