The Arch Linux user and developer community has been engulfed in sharply
divisive debate recently over the issue of package signing. It started
when an Arch user blogged about the distribution's lack of package
signatures, the security risk it created, and his own frustrations with the
core developers' response to the issue. The ensuing argument has since
spread to include Arch's development model and a variety of leadership
questions, but the root problem remains unsolved: there is still no
mechanism to verify the authenticity of "official" Arch Linux packages.
Arch Linux is a non-commercial
distribution that de-emphasizes ease of use in favor of emphasizing
the "responsibility" of the user to maintain the system
properly, and providing "simple" and
"code-correct" tools rather than striving for convenience.
That philosophy evidently attracts a healthy number of users and
contributors, but to some users, the project in practice misses the mark in
the important area of security. Arch Linux practices rolling releases,
with new packages pushed out as tarballs made available on the primary
project server, and mirrored on close to
100 sites around the globe. Although the official installation guide and
download page both emphasize the importance of examining the checksums of
downloads, no packages are signed by individual developers or by the main
To blogger IgnorantGuru, this constitutes an unacceptable security risk.
In February, he blogged about his
concerns, noting that without a method for Arch users to verify that a
package is unaltered, packages can be replaced with Trojan-horse-laden code.
This can happen if the primary server is compromised, but also at any of the
mirrors, at network proxies or caches, on local storage, or even through
man-in-the-middle hijacking of the download request. While package
authentication might not
have been important in Arch's early days, he said, the number of
contributors and mirrors has grown so much that without signatures, the
distribution and its users are sitting ducks.
Package signing protects against attackers' ability to tamper with or
replace an uploaded file by providing an independent way for other people
to verify that the binary has not been altered since the time it was signed
by the developer. If a file server's security is compromised, a simple
checksum (such as an MD5 or SHA1 hash) can be replaced with a new hash that
matches the tampered-with binary with virtually no additional work for the
attacker. That is because a hash is a function of the file contents alone;
a digital signature is a function of both the file and the signer's private
key. Thus, its authenticity can be verified using the file and a copy of
the corresponding public key from a public key server.
Almost all prominent Linux distributions use package signing to allow
users and system administrators to verify the integrity of packages.
Typically, package uploaders use a private GPG key to sign the package
before it is pushed to the release server, and the user's package installer
includes support for automatically retrieving the corresponding public key
and checking the signature. Support in the package manager itself is not
mandatory, however; simply including a cryptographic signature as part of
each package's metadata would permit security-conscious users to manually
verify that packages have not been tampered with.
IgnorantGuru made that point in a message
to the mailing list for developers of Arch's package manager Pacman. There
he also lamented what he said was the low priority given to package
security by the developers. The topic had
come up before, but no one acted on it, and several of the core Arch
developers dismissed the subject as an unimportant one that they were not
willing to work on personally. IgnorantGuru subsequently asked for the
signatures to package metadata (without adding simultaneous support for
checking them), and the core developers replied that he should take the
issue up with the maintainers of the Arch mirrors he used.
Process, and the lack thereof
Furthermore, IgnorantGuru lambasted the dismissive attitude of the core
developers towards the security issue in general. He quoted several of
them directly, including comments such as "For me it makes not a big
difference if a package is signed or not. It's a nice to have feature and I
would be glad if someone would implement it. But for me it has a very low
priority," and "Minor performance issues interest me a hell of a lot more
than package signing."
A few, he said, did take the issue seriously and had submitted patches
to Pacman, but core developers refused to act on them. In the comment
thread that followed, supporters came out on both sides, and IgnorantGuru
expounded on his claims of key developers resisting the idea, if not openly
trying to derail it. He linked to two bug reports he had filed (the second in
response to a comment on the first), noting
that Allan McRae seemed to be actively trying to keep package signing
improvements out of Arch. McRae, he said, repeatedly claimed that he had
no interest in package signatures (and frequently stated that anyone who
cared to could submit their code), but sought out every discussion of the
topic and tried to quash others' efforts to work on a solution.
second first bug report, IgnorantGuru even suggested a lightweight
solution that involved only signing the main server's package database,
which would not protect users in the event that the main server is
compromised, but would at least allow them to verify that a given mirror
had not altered the packages it hosted. McRae countered that this approach
still left other attack vectors open (such as the master server itself
being compromised and the signatures and database forged), thus it was not
worth implementing, and that waiting for full support to come to Pacman was
preferable. IgnorantGuru responded, in effect, that this line of thinking is
making the perfect the enemy of the good, and that adding lightweight
signature support provided at least a substantial improvement over "no
signatures whatsoever." McRae also argues that the package signing plan in the
bug reports is "sub optimal" and proposes a different approach
instead, concluding with a list of unanswered questions about the new
approach. IgnorantGuru calls this an attempt to derail real work by
starting a bike-shedding argument.
In addition to laying out the technical arguments, IgnorantGuru
described how his
attempts to discuss package signing on the official Arch discussion forum
had been shut down by forum moderators. The threads were closed, and then
moved to a "dustbin" forum section that is not visible to unregistered
users. Subsequently, IgnorantGuru's account was suspended for eight weeks
by the moderators, and his IP address blocked.
IgnorantGuru posted a
copy of the message that prompted his forum ban on his personal blog.
In it, he quotes McRae's assertion that he would accept Pacman patches from
other developers, and his assertion that "as usual" no one
submitted any. He then describes patches sent by himself and other Arch
contributors, and what McRae and other core developers did to prevent
merging them. This second post covers similar ground as its predecessor,
but the comment thread provides even more detail, as McRae eventually joins
As you would expect, there is a lot of heated language between the two,
which can make it difficult to separate out the nucleus of each side's
complaints. It is clear, however, that other Arch contributors have
proposed virtually the same package signing plan as IgnorantGuru in the
past, at least as far back as 2008. The core developers have never acted
to add signature-checking to Pacman, even though patches have been
submitted, yet at the same time have resisted all "interim" solutions as a
waste of effort, on the grounds that Pacman is the proper place for package
For his part, IgnorantGuru posted a Bash
script on his own site that allows a user a small amount of integrity
checking by comparing the checksums of a package from multiple Arch
mirrors. In the weeks since the original post, IgnorantGuru announced that
he had given up on the hope that the core developers would ever adopt a
package signing solution, and would be moving to a new distribution with a
better security model. Most recently he posted about Gnuffy, a
fork of Arch Linux that incorporates package signatures and other ideas
shunned by the Arch leadership.
In the final analysis, Arch users are exposed to a security threat both
by the distribution's lack of package signing and by the core developer's
resistance to adopting it. However much the Arch "philosophy" says each
user is responsible for his or her own system, IgnorantGuru is correct in
his first blog post when he observes that without signatures, the
distribution's infrastructure is vulnerable to every exploit found in every
other system on the path between the main project server and the user's PC.
Any exploit on the OS used by the HTTP or FTP mirror server can allow an
attacker to replace an Arch package, and the user has no way to even detect
that the change has been made. A network-based man-in-the-middle attack
does not even require finding a root exploit on one of the servers; it
could be performed at a router or network access point.
As for McRae and the other core developers resisting outside work to
implement package signatures, there are at least two levels at which their
recent statements are troubling. First, they (and commenters taking their
side) invariably bring up the "stop whining or start coding" retort, or its
kinder cousin "patches welcome."
In theory that reply encapsulates the open source approach —
anyone with an itch can scratch it — which is true on the surface.
But it overlooks the reality of distribution maintainership: some tasks,
such as security, are vital to the health of the project, even though they
might not be anyone's idea of fun. In addition, as the Arch developers'
responses have shown, the gatekeepers to a project can still prevent new
code and patches from getting accepted if they so choose.
That of course is the second level at which the core developers'
resistance is troubling: the fact that they would prevent security patches
from going into the project. Some on the anti-signature camp argue
(essentially) that if you don't like any decision made by the core
developers, you should leave, and cite Arch's "not for the faint-of-heart"
philosophy as justification. But that is in direct opposition to the
"patches welcome" response. A distribution cannot invite contributions,
but then treat contributors with hostility when they arrive.
Moreover, in Arch's case it hardens the anti-signing camp into a
position where it denies the reality of the package-tampering threat. You
can already see this in some of the responses to IgnorantGuru and the bug
reports he filed: commenters minimizing the statistical likelihood of an
attack as grounds for not adopting any package signing plan at
Perhaps there is no reason to read ill will into any of McRae or the
other Arch developers' resistance to package signing; perhaps they only
seem hostile in context. But even if that is the case, by preventing
others' code from making it into Pacman and the package database, they are
letting Arch Linux users remain sitting ducks, as well as driving them
away. It's hard to see how either result improves the project.
[Update: Arch Linux developer (and Pacman lead) Dan McGee strongly disagrees with this article, and has written about it on his blog. Additional comments can be found on our note about his posting as well.]
(Thanks to Roger Leigh for the heads-up on this topic).
to post comments)