One of the biggest events overshadowing the 2011 kernel summit was the
compromise of kernel.org and its slow recovery, so it was fitting that
kernel.org was the first topic to be discussed. A bit more information on
what happened was on offer and future directions were considered. The
future version of kernel.org will be rather more secure than its
H. Peter Anvin started with a history of kernel.org, which was initially
set up when Linus moved to California and started working for Transmeta.
It was split off as a standalone nonprofit organization in 2002. Running
kernel.org had never been anybody's full-time job; it was an all-volunteer
effort until 2008, when the Linux Foundation hired John Hawley as its first
When it was set up, kernel.org was envisioned entirely as a distribution
point, not as a place where development got done. The transition to git
changed things, but nobody appreciated the implications at the time. There
was also a lot of time put into a growing list of web-driven services. The
kernel.org folks had to maintain their own version of gitweb, which would
not have functioned in that environment without the caching layer they
added. Kernel.org also ended up supporting a bugzilla installation,
patchwork, various wikis, the kerneloops.org site, and more. Running
kernel.org had turned into a big, complex job.
On the morning of August 28, 2011, Peter discovered that his personal
server had been compromised. As he dug into the situation, it became clear
that kernel.org had been hit as well. The attack turns out to have been
part of a widespread
credential-stealing network that has been operating for some years now; it
is clear that the site had been owned by this network for some time before
it was discovered. What also seems to be clear is that this was not a
targeted attack; kernel.org was just another on a long list of broken
The attackers operated quietly; the site was not used for
activities like spamming.
What the attackers did do was to snoop operations on ttys and record
usernames and passwords. They installed trojan ssh and sshd binaries and,
seemingly, were able to exploit ssh agent forwarding to move on to new
machines. What they did not do was to mess with the data on kernel.org;
after an extensive investigation, no data tampering has been found. Even
so, kernel.org is coming back without all the old data; developers are
asked to carefully review anything they care about before re-uploading it.
Kernel.org is being rebuilt from the beginning with a much greater
separation of services; it will also be moving fully into the
Linux Foundation. Peter noted that, over the years, the Linux Foundation
has built a lot of credibility; there is a lot of faith in how they serve
the development community. The Linux Foundation is also a lot better at
raising funds to support operations like kernel.org. There will be
additional staff to do the job properly, and no more volunteer
administrators. Only full-time administrators will have root access to the
systems involved; Peter, who is keeping his current day job, will not have
Peter also talked about the building of the new PGP/GPG web of trust. It
is, he said, something that we have needed for a long time. We are past
the time where kernel developers are all able to identify each other.
Locking down kernel.org to the inner core of the development community
would not be a good thing; the site is there for the whole community. That
means there needs to be a way to deal with mundane issues like lost
credentials without actually knowing the people involved.
The old kernel.org would automatically sign tarballs and other files after
they were uploaded. The signature was good for exactly one thing:
verifying that the file involved did, indeed, come from kernel.org. But
people had been reading more than that into kernel.org signatures, which
were seen as some sort of guarantee that the data could be trusted.
will not be happening anymore; developers will need to sign files - with
their own keys - before uploading them. The new web of trust will allow
those signatures to be verified by others.
The new git service will be built on gitolite, a mature package that has
been deployed in a number of places and which has been through a number of
security audits. Access to gitolite will be secured with ssh keys. For
the uploading of tarballs and such, a new tool called "kup" has been
developed. The use of both SSH and GPG keys will be required; the
uncompressed file will be signed, then the tool will manage the creation of
the various compressed formats. The old automatic compression on
kernel.org used a daemon running as root, something that just doesn't seem
like a good idea on the new kernel.org.
For email, only forwarding will be supported. Most mailing lists that were
hosted there will move to vger.kernel.org; only a few, like
firstname.lastname@example.org, will remain. Somebody asked if vger would
remain separate or be pulled into the new site. The answer to that was
clear: vger is maintained by Red Hat's IT group, they know how to deal with
high-bandwidth mailing lists, it all works, and the kernel.org folks are
more than happy to let them keep it. If nothing else, the thought of what
would have happened had all the mailing lists gone down with the rest of
kernel.org was enough to put an end to that discussion.
Other things like web services, will be restored eventually, but the
process of hiring an additional administrator needs to run its course first.
Greg Kroah-Hartman talked briefly about what kernel.org users should be
concerned about. In summary, any credential that touched kernel.org -
passwords typed on that system, or keys stored there - should be considered
to be compromised. He has a list of what may have been exposed for each
user; he will be contacting those users with that information in the near
John Hawley, the current full-time administrator, talked about the design
of the new site. The old kernel.org had a single system, called "hera,"
that was at the center of everything. The new version will be much more
distributed, with hera's functions split out to a number of separate boxes
(some of which will be virtual machines). The site is currently being
built on machines that had originally been delivered to the Linux
Foundation for other purposes; he noted that the firewall box has 24 CPUs
and 32GB of memory, which ought to be enough for the task. Currently only
John has access to the new systems; until somebody else is hired, the
community needs to cross its fingers against the possibility of bus-related
John asked what else the community needs from the new site. The ability to
run hooks attached to git repositories was at the top of the list; a lot of
people miss the mailing lists that carry notifications of new commits. The
come back; the more general hook capability may not. There was also talk
of a new server for shell accounts, though it's not clear what that would
be needed for. Shell accounts will never be returning to the systems that
matter; if a box were set up for that purpose it would likely be, as Ted
Ts'o put it, a "honeypot server."
Ted also talked about a security working group being created within the
Linux Foundation. That group will look at the security of the kernel code
and the processes that create it in the hope of coordinating
security-related efforts and tightening things up.
With regard to processes, Dave Airlie asked if a kernel.org account would
be required to push patches into the mainline. Or will GPG-signed emails
be required? Linus answered that he'll not require signatures; the tools
for checking them are still too painful to use. He would like automatic
checking of signatures on git pulls; that feature may show up at some point
in the future, and he may use it. Linus will also accept pull requests
from hosts other than kernel.org, but with some constraints. He does not
like public hosting services like github; it served well as an emergency
fallback, but he ended up checking every single patch he pulled from there,
a process which isn't going to happen during the merge window. So, he
said, do not send github-based pull requests. Requests from other hosts
known to be trusted - freedesktop.org was named - are OK. But they have to
use the git protocol - no HTTP-based pulls will be considered.
In response to a question about the upcoming merge window, Stephen Rothwell
noted that he is currently sitting on the second-largest linux-next
repository ever seen. There are over 11,000 commits waiting to flood into
the mainline once Linus starts pulling. And that is with a lot of old
trees that may not have found their way back to kernel.org before Stephen
went on vacation in mid-October.
In summary, it has been a difficult time for kernel.org, and the people
involved are looking very tired. But we have been lucky in a sense: it was
not a targeted attack that could have done us some real damage. But it
was a wakeup call that has led to a much-needed redesign of the
system and its support. Kernel.org in the future should be a much stronger
and better-supported resource for the community.
Next: Tracing for large data centers
to post comments)