On the security of our processes and infrastructure

This compromise is far from the first that we have seen in our community. Numerous projects and companies have had their systems broken into at times; in some of those incidents, the attackers have replaced distributed code with versions containing trojans or backdoors. Think back to the OpenSSH and sendmail compromises, for example. Kernel.org suffered a compromise (smaller in extent) in 2010; there was also an attempt to insert a backdoor into the kernel source back in 2003. In general, these attempts have been caught quickly, and there is little (known) history of compromised code being distributed to users. Cases where backdoors and other misfeatures have actually been distributed have typically not been the result of attacks; some readers will remember the InterBase backdoor, for example, which predated that project's release as open source.

So, while the history of attacks is unnerving, the actual results in terms of compromised systems have not been all that bad. So far.

Whether this attack on kernel.org has had a worse outcome is not yet known. As your editor wrote in a different venue, it is quite unlikely that the mainline Linux source repository has been corrupted; git makes it almost certain that any such attempt would be detected quickly. But that article was deliberately limited in scope; there are many possible attack vectors that do not involve a direct attempt to corrupt Linus's repository. Ruling out these other attacks will be harder than verifying the integrity of the mainline git tree.

For example, kernel.org distributes tarballs and flat-file patches that are not as easy to verify. For obvious reasons, comparing those files against the checksums stored in the same directory is not considered to be adequate at this point. Kernel.org also serves as a mirror site for a wide range of other projects and distributions. Verifying all of those mirrored files will not be a quick exercise.

There is also concern about kernel repositories maintained by other developers that feed patches into the mainline. It is not uncommon to create a throwaway branch for merging; that branch is often deleted (or simply forgotten about) after the pull is done. It is possible that changes to a throwaway branch between its creation and its pulling into the mainline could go undetected. There are ways to avoid this possibility - simply including the commit ID for the head of that branch in the pull request, for example - but that is not routinely done now.

All recently used branches on all kernel.org-hosted repositories should be checked for tampering, but to focus on that threat is to miss the bigger picture. Every tree feeding into the mainline is a possible way for malicious code to get into the kernel, but the compromise of kernel.org has not changed the situation much, for a couple of reasons:

• All of those trees originate outside of kernel.org, so each one lives on at least one other system which is also a target for attack. Often that other system is a developer laptop. Anybody who has attended a few developer conferences has seen a long line of laptop bags against the wall at meals and receptions; it would not be all that hard to borrow one for the time it takes to drink a beer or two. Those systems and their owners are also all subject to all the usual forms of remote attack: corrupt PDF files, social engineering, etc. In many of these cases, a successful attack is less likely to be detected than it is on a site like kernel.org.

• In our normal development process, with proper code review and no compromised systems, we still insert security vulnerabilities into the kernel - and most other projects as well. So it is a bit of a stretch to say that we would detect an attempt to deliberately add vulnerable code through the normal patch submission process. We just don't have enough people to review code in general; people who are willing and able to do a proper security review are even harder to come by. The community could have 100% secure infrastructure and still be vulnerable to attack.

Kernel.org will be back soon, possibly in a more secure mode. It might make sense to ask, for example, whether it is really necessary to have 450 shell accounts on such an important system. But it seems clear that a stronger kernel.org, as important as that is, will not make our security worries go away. Given the incentives that exist, there will certainly be more attacks, and some of those attacks will originate in highly competent, well funded organizations. Those attacks might overwhelm even a reinforced kernel.org, but attackers need not focus their attention on just that one target.

What is needed is to make the entire system more robust. A discussion started at the Linux Plumbers Conference centers around the creation of a "compilation sandbox" to defend developers (and users) against malicious code inserted into makefiles or configuration scripts, for example. Defending against malicious kernels will be rather harder, but it merits some thought. Someday, perhaps, we'll have static analysis tools that can find an increasing variety of security problems before they are distributed to users. There's a lot that can be done to block future attacks.

But, as Bruce Schneier has often said, security efforts focused exclusively on prevention are doomed to fail; there is a strong need for detection and mitigation efforts as well. We are not as good in those areas as we should be; the fact that the kernel.org compromise went unnoticed for days, even when the system was experiencing unexplained kernel crashes, makes that clear. We need to improve our ability to detect successful attacks and reduce the damage that those attacks can cause. Because there will be more attacks, and some of them will succeed.

This isn't about script kiddies anymore; it hasn't been for a while now. The compromise of kernel.org needs to be seen as part of a wider pattern of attacks on high-profile sites - Google, DigiNotar, RSA Security, etc. At the minimum, large amounts of money are involved; it is not an exaggeration to say that, in some cases, lives are at stake. The continued success of free software depends on our ability to deal with this threat, and to do so without compromising the openness on which our community depends. It is a hard problem, but not an impossible one. We have solved many hard problems to get as far as we have; we can deal with this one too.