Arch Linux and (the lack of) package signing

The Arch Linux user and developer community has been engulfed in sharply divisive debate recently over the issue of package signing. It started when an Arch user blogged about the distribution's lack of package signatures, the security risk it created, and his own frustrations with the core developers' response to the issue. The ensuing argument has since spread to include Arch's development model and a variety of leadership questions, but the root problem remains unsolved: there is still no mechanism to verify the authenticity of "official" Arch Linux packages.

Arch Linux is a non-commercial distribution that de-emphasizes ease of use in favor of emphasizing the "responsibility" of the user to maintain the system properly, and providing "simple" and "code-correct" tools rather than striving for convenience. That philosophy evidently attracts a healthy number of users and contributors, but to some users, the project in practice misses the mark in the important area of security. Arch Linux practices rolling releases, with new packages pushed out as tarballs made available on the primary project server, and mirrored on close to 100 sites around the globe. Although the official installation guide and download page both emphasize the importance of examining the checksums of downloads, no packages are signed by individual developers or by the main project server.

To blogger IgnorantGuru, this constitutes an unacceptable security risk. In February, he blogged about his concerns, noting that without a method for Arch users to verify that a package is unaltered, packages can be replaced with Trojan-horse-laden code. This can happen if the primary server is compromised, but also at any of the mirrors, at network proxies or caches, on local storage, or even through man-in-the-middle hijacking of the download request. While package authentication might not have been important in Arch's early days, he said, the number of contributors and mirrors has grown so much that without signatures, the distribution and its users are sitting ducks.

Package signing protects against attackers' ability to tamper with or replace an uploaded file by providing an independent way for other people to verify that the binary has not been altered since the time it was signed by the developer. If a file server's security is compromised, a simple checksum (such as an MD5 or SHA1 hash) can be replaced with a new hash that matches the tampered-with binary with virtually no additional work for the attacker. That is because a hash is a function of the file contents alone; a digital signature is a function of both the file and the signer's private key. Thus, its authenticity can be verified using the file and a copy of the corresponding public key from a public key server.

Almost all prominent Linux distributions use package signing to allow users and system administrators to verify the integrity of packages. Typically, package uploaders use a private GPG key to sign the package before it is pushed to the release server, and the user's package installer includes support for automatically retrieving the corresponding public key and checking the signature. Support in the package manager itself is not mandatory, however; simply including a cryptographic signature as part of each package's metadata would permit security-conscious users to manually verify that packages have not been tampered with.

IgnorantGuru made that point in a message to the mailing list for developers of Arch's package manager Pacman. There he also lamented what he said was the low priority given to package security by the developers. The topic had come up before, but no one acted on it, and several of the core Arch developers dismissed the subject as an unimportant one that they were not willing to work on personally. IgnorantGuru subsequently asked for the addition of signatures to package metadata (without adding simultaneous support for checking them), and the core developers replied that he should take the issue up with the maintainers of the Arch mirrors he used.

Process, and the lack thereof

Furthermore, IgnorantGuru lambasted the dismissive attitude of the core developers towards the security issue in general. He quoted several of them directly, including comments such as "For me it makes not a big difference if a package is signed or not. It's a nice to have feature and I would be glad if someone would implement it. But for me it has a very low priority," and "Minor performance issues interest me a hell of a lot more than package signing."

A few, he said, did take the issue seriously and had submitted patches to Pacman, but core developers refused to act on them. In the comment thread that followed, supporters came out on both sides, and IgnorantGuru expounded on his claims of key developers resisting the idea, if not openly trying to derail it. He linked to two bug reports he had filed (the second in response to a comment on the first), noting that Allan McRae seemed to be actively trying to keep package signing improvements out of Arch. McRae, he said, repeatedly claimed that he had no interest in package signatures (and frequently stated that anyone who cared to could submit their code), but sought out every discussion of the topic and tried to quash others' efforts to work on a solution.

In the second first bug report, IgnorantGuru even suggested a lightweight solution that involved only signing the main server's package database, which would not protect users in the event that the main server is compromised, but would at least allow them to verify that a given mirror had not altered the packages it hosted. McRae countered that this approach still left other attack vectors open (such as the master server itself being compromised and the signatures and database forged), thus it was not worth implementing, and that waiting for full support to come to Pacman was preferable. IgnorantGuru responded, in effect, that this line of thinking is making the perfect the enemy of the good, and that adding lightweight signature support provided at least a substantial improvement over "no signatures whatsoever." McRae also argues that the package signing plan in the bug reports is "sub optimal" and proposes a different approach instead, concluding with a list of unanswered questions about the new approach. IgnorantGuru calls this an attempt to derail real work by starting a bike-shedding argument.

In addition to laying out the technical arguments, IgnorantGuru described how his attempts to discuss package signing on the official Arch discussion forum had been shut down by forum moderators. The threads were closed, and then moved to a "dustbin" forum section that is not visible to unregistered users. Subsequently, IgnorantGuru's account was suspended for eight weeks by the moderators, and his IP address blocked.

IgnorantGuru posted a copy of the message that prompted his forum ban on his personal blog. In it, he quotes McRae's assertion that he would accept Pacman patches from other developers, and his assertion that "as usual" no one submitted any. He then describes patches sent by himself and other Arch contributors, and what McRae and other core developers did to prevent merging them. This second post covers similar ground as its predecessor, but the comment thread provides even more detail, as McRae eventually joins in.

As you would expect, there is a lot of heated language between the two, which can make it difficult to separate out the nucleus of each side's complaints. It is clear, however, that other Arch contributors have proposed virtually the same package signing plan as IgnorantGuru in the past, at least as far back as 2008. The core developers have never acted to add signature-checking to Pacman, even though patches have been submitted, yet at the same time have resisted all "interim" solutions as a waste of effort, on the grounds that Pacman is the proper place for package signature work.

For his part, IgnorantGuru posted a Bash script on his own site that allows a user a small amount of integrity checking by comparing the checksums of a package from multiple Arch mirrors. In the weeks since the original post, IgnorantGuru announced that he had given up on the hope that the core developers would ever adopt a package signing solution, and would be moving to a new distribution with a better security model. Most recently he posted about Gnuffy, a fork of Arch Linux that incorporates package signatures and other ideas shunned by the Arch leadership.

Threats

In the final analysis, Arch users are exposed to a security threat both by the distribution's lack of package signing and by the core developer's resistance to adopting it. However much the Arch "philosophy" says each user is responsible for his or her own system, IgnorantGuru is correct in his first blog post when he observes that without signatures, the distribution's infrastructure is vulnerable to every exploit found in every other system on the path between the main project server and the user's PC. Any exploit on the OS used by the HTTP or FTP mirror server can allow an attacker to replace an Arch package, and the user has no way to even detect that the change has been made. A network-based man-in-the-middle attack does not even require finding a root exploit on one of the servers; it could be performed at a router or network access point.

As for McRae and the other core developers resisting outside work to implement package signatures, there are at least two levels at which their recent statements are troubling. First, they (and commenters taking their side) invariably bring up the "stop whining or start coding" retort, or its kinder cousin "patches welcome."

In theory that reply encapsulates the open source approach — anyone with an itch can scratch it — which is true on the surface. But it overlooks the reality of distribution maintainership: some tasks, such as security, are vital to the health of the project, even though they might not be anyone's idea of fun. In addition, as the Arch developers' responses have shown, the gatekeepers to a project can still prevent new code and patches from getting accepted if they so choose.

That of course is the second level at which the core developers' resistance is troubling: the fact that they would prevent security patches from going into the project. Some on the anti-signature camp argue (essentially) that if you don't like any decision made by the core developers, you should leave, and cite Arch's "not for the faint-of-heart" philosophy as justification. But that is in direct opposition to the "patches welcome" response. A distribution cannot invite contributions, but then treat contributors with hostility when they arrive.

Moreover, in Arch's case it hardens the anti-signing camp into a position where it denies the reality of the package-tampering threat. You can already see this in some of the responses to IgnorantGuru and the bug reports he filed: commenters minimizing the statistical likelihood of an attack as grounds for not adopting any package signing plan at all.

Perhaps there is no reason to read ill will into any of McRae or the other Arch developers' resistance to package signing; perhaps they only seem hostile in context. But even if that is the case, by preventing others' code from making it into Pacman and the package database, they are letting Arch Linux users remain sitting ducks, as well as driving them away. It's hard to see how either result improves the project.

[Update: Arch Linux developer (and Pacman lead) Dan McGee strongly disagrees with this article, and has written about it on his blog. Additional comments can be found on our note about his posting as well.]

(Thanks to Roger Leigh for the heads-up on this topic).