On August 31, the world was
kernel.org, the primary repository for kernel code in various stages of
development, had been compromised - though developers with access to the
site had been informed a few days prior. The site was shut down for
"maintenance" when that notice went out, leaving the community without an
important hosting and distribution point. Kernel development has slowed as
a result; the 3.1 kernel, which would have been expected by now, remains
unreleased. Kernel.org is on its way back, but it will almost certainly
never be quite the same.
On October 3, a basic kernel.org returned to the net. Git hosting is back,
but only for a very small number of trees: mainline, stable, and
linux-next. The return of the other trees is waiting for the relevant
developers to reestablish their access to the site - a process that
involves developers verifying the integrity
of their own systems, then generating a new
PGP/GPG key, integrating it into the web of trust, and forwarding the
public key to the kernel.org maintainers. This procedure could take a
while; it is not clear how many developers will be able to regain their
access to kernel.org before the 3.2 merge window opens.
The front-page web interface is back though, as of this writing, it is
not being updated to reflect the state of the git trees. Most other
kernel.org services remain down; some could stay that way for some time.
It is worth remembering that kernel.org only has one full-time system
administrator, a position that has been funded by the Linux Foundation
since 2008. That administrator, along with a number of volunteers, is
likely to be quite busy; some of the less-important services may not return
A full understanding of what happened is also likely to take some time.
Even in the absence of a report on this intrusion, though, there are some
conclusions that can be made. The first is obvious: the threat is real.
There are attackers out there with time, resources, motivation, and
skills. Given the potential value of either putting a back door into the
kernel or adding a trojan that would run on developers' machines, we have
to assume that there will be more attacks in the future. If the restored
kernel.org is not run in a more secure manner, it will be compromised again
in short order.
The site's administrators have already announced that shell accounts will
not be returning to the systems where git trees are hosted. Prior to the
breakin, there were on the order of 450 of those accounts; that is a lot of
keys to the front door to have handed out. No matter how careful all those
developers may be - and some are more careful than others - the chances of
one of them having a compromised machine approach 100%. Keeping all those
shell accounts off the system is clearly an important step toward a higher
level of security.
Kernel.org has its roots in the community and was run the way kernel
developers often run their machines. So, for example, kernel.org tended to
run mainline -rc kernels - a good exercise in dogfooding, perhaps, but it
also exposed the system to bleeding-edge bugs, and, perhaps more
importantly, obscured the real cause of kernel
panics experienced last August, delaying the realization that the
system had been compromised. The kernel currently running on the new systems
has not been announced; one assumes it is something a little better tested,
better supported, and stable. (No criticism is intended by pointing this
out, incidentally. Kernel.org has been run very well for a long time; the
point here is that the environment has changed, so practices need to change
At this point it seems clear that a single administrator for such a
high-profile site is not an adequate level of staffing. Given the
resources available in our community, it seems like it should be possible
to increase the amount of support available to kernel.org. There are
rumors that this is being worked on, but nothing has been announced.
Developers are going to have to learn to pay more attention to the
security of their systems. There are scattered reports of kernel
developers turning up
compromised systems; in some cases, they may have been infected as the
result of excessive trust in kernel.org. Certain practices will have to
change; for that reason, the Fedora project's announcement of a zero-tolerance policy toward
private keys on Fedora systems is welcome. Developers are on the front
line here: everybody is depending on them to keep their code - and the
infrastructure that distributes that code - secure.
There is an interesting question related to that: will kernel developers
move back to kernel.org? These developers have had to find new homes for
their git repositories during the outage; some of them are likely to decide
that leaving those repositories in their new location is easier than
establishing identities in the web of trust and getting back into
kernel.org. Linus has said in the past
that he sees the presence of a kernel.org-hosted tree in a pull request as
a sign that the request is more likely to be genuine. Requiring
that repositories be hosted at kernel.org seems like an unlikely step for
this community, though. It is not entirely clear whether trees distributed
around the net increase the security risk to the kernel, or whether putting
all the eggs into the kernel.org basket would be worse.
One other conclusion would seem to jump out at this point: kernel.org got hit
this time, but there are a lot of other important projects and hosting
sites out there. Any of those projects is just as likely to be a target as
the kernel. If we are not to have a long series of embarrassing compromises,
some with seriously unfortunate consequences, we're going to have to take
security more seriously everywhere. Doing so without ruining our
community's openness is going to be a challenge, to say the least, but it
is one we need to take on. Security is a pain, but being broken into and
used to attack your users and developers is even more so.
to post comments)