KS2011: Kernel.org report

H. Peter Anvin started with a history of kernel.org, which was initially set up when Linus moved to California and started working for Transmeta. It was split off as a standalone nonprofit organization in 2002. Running kernel.org had never been anybody's full-time job; it was an all-volunteer effort until 2008, when the Linux Foundation hired John Hawley as its first full-time administrator.

When it was set up, kernel.org was envisioned entirely as a distribution point, not as a place where development got done. The transition to git changed things, but nobody appreciated the implications at the time. There was also a lot of time put into a growing list of web-driven services. The kernel.org folks had to maintain their own version of gitweb, which would not have functioned in that environment without the caching layer they added. Kernel.org also ended up supporting a bugzilla installation, patchwork, various wikis, the kerneloops.org site, and more. Running kernel.org had turned into a big, complex job.

On the morning of August 28, 2011, Peter discovered that his personal server had been compromised. As he dug into the situation, it became clear that kernel.org had been hit as well. The attack turns out to have been part of a widespread credential-stealing network that has been operating for some years now; it is clear that the site had been owned by this network for some time before it was discovered. What also seems to be clear is that this was not a targeted attack; kernel.org was just another on a long list of broken machines.

The attackers operated quietly; the site was not used for activities like spamming. What the attackers did do was to snoop operations on ttys and record usernames and passwords. They installed trojan ssh and sshd binaries and, seemingly, were able to exploit ssh agent forwarding to move on to new machines. What they did not do was to mess with the data on kernel.org; after an extensive investigation, no data tampering has been found. Even so, kernel.org is coming back without all the old data; developers are asked to carefully review anything they care about before re-uploading it.

Kernel.org is being rebuilt from the beginning with a much greater separation of services; it will also be moving fully into the Linux Foundation. Peter noted that, over the years, the Linux Foundation has built a lot of credibility; there is a lot of faith in how they serve the development community. The Linux Foundation is also a lot better at raising funds to support operations like kernel.org. There will be additional staff to do the job properly, and no more volunteer administrators. Only full-time administrators will have root access to the systems involved; Peter, who is keeping his current day job, will not have that access.

Peter also talked about the building of the new PGP/GPG web of trust. It is, he said, something that we have needed for a long time. We are past the time where kernel developers are all able to identify each other. Locking down kernel.org to the inner core of the development community would not be a good thing; the site is there for the whole community. That means there needs to be a way to deal with mundane issues like lost credentials without actually knowing the people involved.

The old kernel.org would automatically sign tarballs and other files after they were uploaded. The signature was good for exactly one thing: verifying that the file involved did, indeed, come from kernel.org. But people had been reading more than that into kernel.org signatures, which were seen as some sort of guarantee that the data could be trusted. Central signing will not be happening anymore; developers will need to sign files - with their own keys - before uploading them. The new web of trust will allow those signatures to be verified by others.

The new git service will be built on gitolite, a mature package that has been deployed in a number of places and which has been through a number of security audits. Access to gitolite will be secured with ssh keys. For the uploading of tarballs and such, a new tool called "kup" has been developed. The use of both SSH and GPG keys will be required; the uncompressed file will be signed, then the tool will manage the creation of the various compressed formats. The old automatic compression on kernel.org used a daemon running as root, something that just doesn't seem like a good idea on the new kernel.org.

For email, only forwarding will be supported. Most mailing lists that were hosted there will move to vger.kernel.org; only a few, like security@kernel.org, will remain. Somebody asked if vger would remain separate or be pulled into the new site. The answer to that was clear: vger is maintained by Red Hat's IT group, they know how to deal with high-bandwidth mailing lists, it all works, and the kernel.org folks are more than happy to let them keep it. If nothing else, the thought of what would have happened had all the mailing lists gone down with the rest of kernel.org was enough to put an end to that discussion.

Other things like web services, will be restored eventually, but the process of hiring an additional administrator needs to run its course first.

Greg Kroah-Hartman talked briefly about what kernel.org users should be concerned about. In summary, any credential that touched kernel.org - passwords typed on that system, or keys stored there - should be considered to be compromised. He has a list of what may have been exposed for each user; he will be contacting those users with that information in the near future.

John Hawley, the current full-time administrator, talked about the design of the new site. The old kernel.org had a single system, called "hera," that was at the center of everything. The new version will be much more distributed, with hera's functions split out to a number of separate boxes (some of which will be virtual machines). The site is currently being built on machines that had originally been delivered to the Linux Foundation for other purposes; he noted that the firewall box has 24 CPUs and 32GB of memory, which ought to be enough for the task. Currently only John has access to the new systems; until somebody else is hired, the community needs to cross its fingers against the possibility of bus-related accidents.

John asked what else the community needs from the new site. The ability to run hooks attached to git repositories was at the top of the list; a lot of people miss the mailing lists that carry notifications of new commits. The lists may come back; the more general hook capability may not. There was also talk of a new server for shell accounts, though it's not clear what that would be needed for. Shell accounts will never be returning to the systems that matter; if a box were set up for that purpose it would likely be, as Ted Ts'o put it, a "honeypot server."

Ted also talked about a security working group being created within the Linux Foundation. That group will look at the security of the kernel code and the processes that create it in the hope of coordinating security-related efforts and tightening things up.

With regard to processes, Dave Airlie asked if a kernel.org account would be required to push patches into the mainline. Or will GPG-signed emails be required? Linus answered that he'll not require signatures; the tools for checking them are still too painful to use. He would like automatic checking of signatures on git pulls; that feature may show up at some point in the future, and he may use it. Linus will also accept pull requests from hosts other than kernel.org, but with some constraints. He does not like public hosting services like github; it served well as an emergency fallback, but he ended up checking every single patch he pulled from there, a process which isn't going to happen during the merge window. So, he said, do not send github-based pull requests. Requests from other hosts known to be trusted - freedesktop.org was named - are OK. But they have to use the git protocol - no HTTP-based pulls will be considered.

In response to a question about the upcoming merge window, Stephen Rothwell noted that he is currently sitting on the second-largest linux-next repository ever seen. There are over 11,000 commits waiting to flood into the mainline once Linus starts pulling. And that is with a lot of old trees that may not have found their way back to kernel.org before Stephen went on vacation in mid-October.

In summary, it has been a difficult time for kernel.org, and the people involved are looking very tired. But we have been lucky in a sense: it was not a targeted attack that could have done us some real damage. But it was a wakeup call that has led to a much-needed redesign of the system and its support. Kernel.org in the future should be a much stronger and better-supported resource for the community.

Next: Tracing for large data centers