Posted Sep 2, 2011 16:44 UTC (Fri) by zooko
Parent article: kernel.org compromised
This is an extended and edited version of a long comment I posted at Jon Corbet's blog entry on linux.com. I hope you find it useful.
Dear Jon Corbet:
Everything you wrote in this blog entry was true, but it leaves out some pretty important aspects.
I used git to fetch a copy of the linux kernel from git.kernel.org a few days ago, compiled it, and booted into it on my primary work/dev/play machine (a laptop). What do I do now?
As far as I understand, the answer is the same as if I had fetched the kernel source code with ftp: I either just cross my fingers and gamble that the attackers weren't sophisticated enough to have taken over my laptop and installed a rootkit in the BIOS, or the battery or something, or I shut it down, extract the hard drive, buy a new laptop, and very carefully and painstakingly copy some of the data from the old hard drive.
A lot of the analysis of this attack so far seems to assume that the attackers would change the contents of a git repository on the kernel.org filesystem. There's nothing requiring them to do that. They could cause git pulls to deliver backdoored kernels to users while leaving the persistent copy on disk untouched. They could also choose which users get the real kernel or a backdoored kernelfor example based on the user's IP address or ssh key. Nor are attackers required to leave evidence sitting on the filesystems of those target usersas soon as a user who has just done a git pull from kernel.org runs
make, the payload can compile a backdoored kernel and simultaneously revert the source code to the pristine version so that inspection of the source will show no evidence.
If you did a
git pull and then did not execute any of the resulting code (such as by running
make), then you could inspect for tampering. Git could help with that inspection.
If the kernel.org admins were extremely professional and careful, then they might be able to give me some confidence in which of these two strategies I should take, by doing a good post-mortem and sharing what they learn about what the attacker did and how and when. However, the fact that they appear to take the strategy of investigating and "cleaning up" by running commands on the compromised systems and inviting their users to do so really undermines that confidence.
That's a pretty amateur game plan they've got there. It can't give you confidence that your system isn't still under the control of a sufficiently sophisticated attacker, and it might open up more opportunities for such an attacker to compromise your systems and your users' systems further, to cover their tracks, or to provide you with false clues.
So what am I going to do? I can't really afford to buy a new laptop right now, so I'm going to gamble that the attackers really were as amateur as they seem in allowing themselves to get caught like this, and that there weren't also other, more capable attackers present, and that the operating system I'm running on this laptop right now isn't under their control.
I'm also going to see if I can set up an alternate system (I have a few cool Linux/ARM devices around here that I haven't fully explored yet) without letting any attacker which might control this laptop get control of that system.
I'm also going to stop fetching kernel source code from kernel.org.
To address your main point about git. Its wonderful append-only immutable data semantics could help people like me here, but not the way most people currently use it.
If I want to avoid being exposed to this kind of attack again, I don't exactly need to stop fetching kernel source code from kernel.org, I just need to stop fetching the tag that tells me which secure hash I should be fetching. So, for example, if I could learn from Greg Kroah-Hartman what is the secure hash of the next release of linux 2.6.32, then I could use git (on a known-good system) to fetch the version matching that secure hash from kernel.org without being vulnerable to the people who control kernel.org supplying me with backdoored kernel source.
git clone git://git.kernel.org/pub/scm/linux/kernel/git/longterm/linux-2.6.32.y.git does not offer that same security property, because in that case I would be relying on git.kernel.org to tell me which secure hash I should use to verify the kernel source, and the attackers could supply me with a secure hash that correctly matched their backdoored kernel source.
Or I could check digital signatures with GPG. In that case I would be getting the security benefits from using GPG, and it wouldn't matter whether I were checking a sig on a tarball or on a git tag. Very few people will ever go to the extra effort to check GPG sigs as often as they do git pulls. It would be cool if the checking of the GPG signature were integrated and automatic so that whenever you did a git pull, it would check the signature and stop you right there if the signature was missing or incorrect. We do something equivalent to that in the Tahoe-LAFS secure distributed filesystem.
It would be cool if GregKH had included the commit hash for 18.104.22.168
b0b3e41f6fe88a15bd27f73f6fabef48e0fea2f0in the release announcement. In that case attackers who took over kernel.org but did not take over lwn.net or gmane.org would be unable to trick me into running their backdoored kernel and giving them control of my laptop. People should get into the habit of including the commit hash whenever they are talking about a specific version such as a release. That would really leverage git's integrity properties to protect a lot of people, unlike the current way it is used. (We also do a technique similar to that in Tahoe-LAFSwe embed the secure hash into every URL which points to a specific, immutable version.)
In fact, the same threat that applies to my laptop would also apply to any kernel developer, even Linus, who did a git pull from kernel.org and ran
make. You wrote "Kernel.org may seem like the place where kernel development is done, but its not; its really just a distribution point.". I took that to mean that kernel developers do not do git pulls from kernel.org and instead do it through some other more private hub or do it directly from one another's personal servers or something, but is that true? I doubt it. I'll bet plenty of kernel developers have pulled code from kernel.org, compiled that code, and run it. If they did, then the only reason to think that their development machines are not now under the control of an attacker is because we currently think that the attackers who took over kernel.org were of low skill.
to post comments)