Code authenticity checking

Cryptographically signed binary packages are mainstay of modern Linux distributions, used for verifying packages against both tampering and accidental corruption. But as ever fewer users need to compile their software from source, it is possible to forget that verifying the authenticity and integrity of binary packages depends first on the packagers' ability to verify the upstream source code. In April, Allan McRae posted an entry on his blog that looked at the verifiability of the source code releases made by a number of well-known projects, with less-than-perfect results.

McRae is a core Arch Linux developer, and he undertook his verifiability survey because Arch's packaging tool makepkg recently gained support for checking the PGP signatures of source files. Naturally, such verification is only possible when the source bundle includes a PGP signature, but McRae also looked at four other interrelated factors: whether or not the signing key can be easily verified as authentic, whether checksums for the source files are available, whether the checksums are signed, and whether the checksums are available on a different server than the one hosting the source files. This last criterion is a guard against a server compromise; if an attacker can replace the source files, he or she can also replace the checksum with one that matches the replacement source.

To provide a meaningful sample population for his survey, McRae looked at 63 "core" packages common to most Linux distributions. "For me, that basically means the packages required to build a fairly minimal booting system. This is essentially the package list from Linux From Scratch with a few additions that I see as needed …" His results are presented in a color-coded table; green cells indicate that a package meets all of the verification criteria, red indicates that no verification criteria pass, and yellow indicate some but not all criteria are met. Of the 63 packages examined, ten offered no means of verification and 14 others offered only partial verification.

The packages offering no verification mechanism are file, flex, iana-etc, isl, kbd, libarchive, pkg-config, procps, psmisc, and which. Those packages that partially meet the measured criteria fall into two basic categories, those that provide PGP signatures but have a difficult-to-verify key (gawk, groff, patch, sed, sudo, sysvinit, texinfo, and xz), and those that provide their checksum data on the same server as their files (bzip2, perl, tzdata, and zlib). In addition, gmp and less fell somewhere in between, providing a PGP signature, but making the public key or key ID available only on the same server as the source release.

To be sure, McRae's specific criteria and red/yellow/green assessments draw some rather arbitrary lines—as he observes, several of the projects have other means of verification available, and he admits that the definition of a "readily verifiable" key includes keys signed by keys that he trusts. But the aggregate picture is the important one: most of the packages are green (which is good news), but roughly 15% of them offer no source verification whatsoever (which is far from good). He also notes that the picture seems to rapidly deteriorate "as you move further away from this core subset of packages needed for a fairly standard Linux system".

Best practices

McRae's post was picked up on the oss-security mailing list, where the talk turned to how to establish a set of common guidelines for releasing source code with verifiable authenticity. Alan Coopersmith commented that X.org has received complaints asking it to do more, but without concrete suggestions. "If there was a common standard, with instructions, we'd be far more likely to spend the time to adopt it, than just a 'make signatures appear somewhere, in an unspecified format'". Eric H. Christensen concurred, saying he was interested in establishing a "best practices" recommendation for Red Hat—but, he asked, what really constitutes the best way to disseminate releases? A recommendation would advise against using MD5 for checksumming, he said, and although he favors PGP for signatures, perhaps it has its drawbacks as well.

Indeed, Alistair Crooks replied with a lengthy list of questions one might ask about a PGP-signed release, addressing everything from the key management practices employed by the signing entity to the options specified when generating the key itself (such as whether it is an RSA or DSA key and whether or not it requires a passphrase). A PGP signature proves only that a person with access to the signing key attests that the signed message had a particular value at a particular time, he argued, which does not provide much authentication:

But the majority seemed to feel that PGP in fact provides quite a few gains. Nicolas Vigier and Florian Weimer both commented that key-continuity over multiple releases safeguards against a man-in-the-middle attacker replacing a release. Weimer noted that "hosting sites have been compromised, or serve their content exclusively over a mirror network which literally anyone can join." Kurt Seifried agreed, but acknowledged that the "real problem" with PGP is the cost of implementing it:

Daniel Kahn Gillmor responded that even a simple "the release manager has the OpenPGP key for the project on the keyring of her personal development machine" workflow raises the bar for would-be attackers and would constitute an improvement for the numerous projects that do not currently sign their releases at all. But he still advocated producing guidelines:

He pointed to the existing OpenPGP best practices page at the Debian Grimoire wiki as an example.

For his part, McRae had comparatively simple goals in mind:

Identity cleft

Although a general consensus developed around the idea of crafting a "best practices" recommendation, Crooks's questions about the limitations of PGP signatures raised some valuable points—such as the importance of distinguishing between identity and trust. Some mistook his original email for a call to ditch PGP signatures on source releases, since they do not offer absolute security, but Crooks said that was a misinterpretation. "It's a bit disappointing that my advice (in pointing out ways that PGP can be worked around in order to diminish integrity and security) was categorised as an attack on PGP itself - I shall take that as a reminder that I should be more clear in what I write." Rather, he said, he hoped to "warn against the magic 'it's signed, so it's gospel' myth by pointing out the problems of key management."

The crux of Crooks's argument is that a PGP signature should not be trusted just because it is associated with a known identity (person or team of developers). As he elaborated in one message in the ensuing thread:

Elsewhere in the same message, he noted:

In essence, this is an argument that the signer only earns trust by demonstrating their reliability over time. For individuals, the issue might be the quality of the software (as Crooks discussed above), but to trust the signature on a project's releases, more stringent requirements may be necessary. As he added later: "You actually know very little about the key before and after the signing took place; so you have no way of ascertaining whether the key has been used to sign other things fraudulently."

Add in Crooks's earlier questions about the key options and security on the machines used to sign releases, and proper key management, it would seem, remains an area where there is still work to be done. Crooks did reiterate his support for encouraging PGP signatures on all source releases; he simply cautioned that blind trust in the presence of a signature can lead to a false sense of security.

Then again, a campaign to persuade more upstream software projects to integrate easily-verified PGP signatures and out-of-band checksums into their release process has to start somewhere. There are practical challenges to consider, such as the role that popular hosting providers play in the build and release processes (Stuart Henderson noted that Github and Bitbucket dynamically generate tar archives, which complicates signing releases).

But convincing individual developers and release managers might be the best way to convince hosting services to adapt. Presumably few people enjoy seeing their project marked with the red "No verification available" label. It would certainly be informative to conduct a large-scale examination of McRae's criteria on other popular open source projects. Just over 38% of the core packages McRae examined could not be "readily validated"—a sizable minority, but a minority nonetheless; that gives one hope that the community in general takes source verifiability as a matter worth addressing.