By now most LWN readers will be well aware of the compromise of the systems
running kernel.org. The services provided by kernel.org have been offline
since that time, with the result that the flow of changes into the 3.1-rc
kernel has slowed considerably. Kernel.org will eventually come back,
perhaps with some significant policy changes. But the real effect may be a
wider discussion of security within the development community, which can
only be a good thing.
This compromise is far from the first that we have seen in our community.
Numerous projects and companies have had their systems broken
into at times; in some of those incidents, the attackers have replaced
distributed code with versions containing trojans or backdoors. Think back
to the OpenSSH and sendmail compromises, for example. Kernel.org
suffered a compromise (smaller in extent)
in 2010; there was also an attempt to insert a
backdoor into the kernel source back in 2003. In general, these
attempts have been caught quickly, and there is little (known) history of
compromised code being distributed to users. Cases where backdoors and
other misfeatures have actually been distributed have typically not been
the result of attacks; some readers will remember the InterBase backdoor,
for example, which predated that project's release as open source.
So, while the history of attacks is unnerving, the actual results in terms
of compromised systems have not been all that bad. So far.
Whether this attack on kernel.org has had a worse outcome is not yet
known. As your editor wrote
in a different venue, it is quite unlikely that the mainline Linux source
repository has been corrupted; git makes it almost certain that any such
attempt would be detected quickly. But that article was deliberately
limited in scope; there are many possible attack vectors that do not
involve a direct attempt to corrupt Linus's repository. Ruling out these
other attacks will be harder than verifying the integrity of the mainline
For example, kernel.org distributes tarballs and flat-file
patches that are not as easy to verify. For obvious reasons, comparing
those files against the checksums stored in the same directory is not
considered to be adequate at this point. Kernel.org also serves as a
mirror site for a wide range of other projects and distributions.
Verifying all of those mirrored files will not be a quick exercise.
There is also concern about kernel repositories maintained by other
developers that feed patches into the mainline. It is not uncommon to
create a throwaway branch for merging; that branch is often deleted (or
simply forgotten about) after the pull is done. It is possible that
changes to a throwaway branch between its creation and its pulling into the
mainline could go undetected. There are ways to avoid this possibility -
simply including the commit ID for the head of that branch in the pull
request, for example - but that is not routinely done now.
All recently used branches on all kernel.org-hosted repositories should be
checked for tampering, but to focus on that threat is to miss the bigger
picture. Every tree feeding into the mainline is a possible way for
malicious code to get into the kernel, but the compromise of kernel.org has
not changed the situation much, for a couple of reasons:
- All of those trees originate outside of kernel.org, so each one lives
on at least one other system which is also a target for attack. Often
that other system is a developer laptop. Anybody who has attended a
few developer conferences has seen a long line of laptop bags against
the wall at meals and receptions; it would not be all that hard to
borrow one for the time it takes to drink a beer or two. Those
systems and their owners are also all subject to all the usual forms of
remote attack: corrupt PDF files, social engineering, etc. In many of
these cases, a successful attack is less likely to be detected than it
is on a site like kernel.org.
- In our normal development process, with proper code review and no
compromised systems, we still insert security vulnerabilities into the
kernel - and most other projects as well. So it is a bit of a stretch
to say that we would detect an attempt to deliberately add vulnerable
code through the normal patch submission process. We just don't have
enough people to review code in general; people who are willing and
able to do a proper security review are even harder to come by. The
community could have 100% secure infrastructure and still be
vulnerable to attack.
Kernel.org will be back soon, possibly in a more secure mode. It might
make sense to ask, for example, whether it is really necessary to have 450
shell accounts on such an important system. But it seems clear that a stronger
kernel.org, as important as that is, will not make our security worries go
away. Given the incentives that exist, there will certainly be more
attacks, and some of those attacks will originate in highly competent, well
funded organizations. Those attacks might overwhelm even a reinforced
kernel.org, but attackers need not focus their attention on just that one
What is needed is to make the entire system more robust. A discussion
started at the Linux Plumbers Conference centers around the creation of a
"compilation sandbox" to defend developers (and users) against malicious
code inserted into makefiles or configuration scripts, for example.
Defending against malicious kernels will be rather harder, but it merits
some thought. Someday, perhaps, we'll have static analysis tools that can
find an increasing variety of security problems before they are distributed
to users. There's a lot that can be done to block future attacks.
But, as Bruce Schneier has often said, security efforts focused exclusively
on prevention are doomed to fail; there is a strong need for detection and
mitigation efforts as well. We are not as good in those areas as we should
be; the fact that the kernel.org compromise went unnoticed for days, even
when the system was experiencing unexplained kernel crashes, makes that clear.
We need to improve our ability to detect successful attacks and reduce the
damage that those attacks can cause. Because there will be more attacks,
and some of them will succeed.
This isn't about script kiddies anymore; it hasn't been for a while now.
The compromise of kernel.org needs to be seen as part of a wider pattern of
attacks on high-profile sites - Google, DigiNotar, RSA Security, etc.
At the minimum, large amounts of money are involved; it is not an
exaggeration to say that, in some cases, lives are at stake. The continued
success of free software depends on our ability to deal with this threat,
and to do so without compromising the openness on which our community
depends. It is a hard problem, but not an impossible one. We have solved
many hard problems to get as far as we have; we can deal with this one
to post comments)