kernel.org compromised

Posted Sep 2, 2011 16:44 UTC (Fri) by zooko (guest, #2589)
Parent article: kernel.org compromised

This is an extended and edited version of a long comment I posted at Jon Corbet's blog entry on linux.com. I hope you find it useful.

Dear Jon Corbet:

Everything you wrote in this blog entry was true, but it leaves out some pretty important aspects.

I used git to fetch a copy of the linux kernel from git.kernel.org a few days ago, compiled it, and booted into it on my primary work/dev/play machine (a laptop). What do I do now?

As far as I understand, the answer is the same as if I had fetched the kernel source code with ftp: I either just cross my fingers and gamble that the attackers weren't sophisticated enough to have taken over my laptop and installed a rootkit in the BIOS, or the battery or something, or I shut it down, extract the hard drive, buy a new laptop, and very carefully and painstakingly copy some of the data from the old hard drive.

A lot of the analysis of this attack so far seems to assume that the attackers would change the contents of a git repository on the kernel.org filesystem. There's nothing requiring them to do that. They could cause git pulls to deliver backdoored kernels to users while leaving the persistent copy on disk untouched. They could also choose which users get the real kernel or a backdoored kernelfor example based on the user's IP address or ssh key. Nor are attackers required to leave evidence sitting on the filesystems of those target usersas soon as a user who has just done a git pull from kernel.org runs make, the payload can compile a backdoored kernel and simultaneously revert the source code to the pristine version so that inspection of the source will show no evidence.

If you did a git pull and then did not execute any of the resulting code (such as by running make), then you could inspect for tampering. Git could help with that inspection.

If the kernel.org admins were extremely professional and careful, then they might be able to give me some confidence in which of these two strategies I should take, by doing a good post-mortem and sharing what they learn about what the attacker did and how and when. However, the fact that they appear to take the strategy of investigating and "cleaning up" by running commands on the compromised systems and inviting their users to do so really undermines that confidence.

That's a pretty amateur game plan they've got there. It can't give you confidence that your system isn't still under the control of a sufficiently sophisticated attacker, and it might open up more opportunities for such an attacker to compromise your systems and your users' systems further, to cover their tracks, or to provide you with false clues.

So what am I going to do? I can't really afford to buy a new laptop right now, so I'm going to gamble that the attackers really were as amateur as they seem in allowing themselves to get caught like this, and that there weren't also other, more capable attackers present, and that the operating system I'm running on this laptop right now isn't under their control.

I'm also going to see if I can set up an alternate system (I have a few cool Linux/ARM devices around here that I haven't fully explored yet) without letting any attacker which might control this laptop get control of that system.

I'm also going to stop fetching kernel source code from kernel.org.

To address your main point about git. Its wonderful append-only immutable data semantics could help people like me here, but not the way most people currently use it.

If I want to avoid being exposed to this kind of attack again, I don't exactly need to stop fetching kernel source code from kernel.org, I just need to stop fetching the tag that tells me which secure hash I should be fetching. So, for example, if I could learn from Greg Kroah-Hartman what is the secure hash of the next release of linux 2.6.32, then I could use git (on a known-good system) to fetch the version matching that secure hash from kernel.org without being vulnerable to the people who control kernel.org supplying me with backdoored kernel source.

However, running git clone git://git.kernel.org/pub/scm/linux/kernel/git/longterm/linux-2.6.32.y.git does not offer that same security property, because in that case I would be relying on git.kernel.org to tell me which secure hash I should use to verify the kernel source, and the attackers could supply me with a secure hash that correctly matched their backdoored kernel source.

Or I could check digital signatures with GPG. In that case I would be getting the security benefits from using GPG, and it wouldn't matter whether I were checking a sig on a tarball or on a git tag. Very few people will ever go to the extra effort to check GPG sigs as often as they do git pulls. It would be cool if the checking of the GPG signature were integrated and automatic so that whenever you did a git pull, it would check the signature and stop you right there if the signature was missing or incorrect. We do something equivalent to that in the Tahoe-LAFS secure distributed filesystem.

It would be cool if GregKH had included the commit hash for 2.6.32.46b0b3e41f6fe88a15bd27f73f6fabef48e0fea2f0in the release announcement. In that case attackers who took over kernel.org but did not take over lwn.net or gmane.org would be unable to trick me into running their backdoored kernel and giving them control of my laptop. People should get into the habit of including the commit hash whenever they are talking about a specific version such as a release. That would really leverage git's integrity properties to protect a lot of people, unlike the current way it is used. (We also do a technique similar to that in Tahoe-LAFSwe embed the secure hash into every URL which points to a specific, immutable version.)

P.S.

In fact, the same threat that applies to my laptop would also apply to any kernel developer, even Linus, who did a git pull from kernel.org and ran make. You wrote "Kernel.org may seem like the place where kernel development is done, but its not; its really just a distribution point.". I took that to mean that kernel developers do not do git pulls from kernel.org and instead do it through some other more private hub or do it directly from one another's personal servers or something, but is that true? I doubt it. I'll bet plenty of kernel developers have pulled code from kernel.org, compiled that code, and run it. If they did, then the only reason to think that their development machines are not now under the control of an attacker is because we currently think that the attackers who took over kernel.org were of low skill.

Regards,

Zooko

to post comments

kernel.org compromised

Posted Sep 2, 2011 17:54 UTC (Fri) by slashdot (guest, #22014) [Link]

Just check that the commit that you compiled from the git repository you downloaded is an ancestor of the current kernel commit.

Of course if you already ran the kernel on the machine storing the downloaded tree, a backdoored kernel could have in principle been programmed to clean up the git repository on disk, but I'd consider that extraordinarily unlikely.

Also, someone else would have likely noticed and reported the discrepancy to LKML.

At any rate, there is always some remote security risk, since for all we know Linus Torvalds himself could have inserted covert backdoors in the kernel source code.

kernel.org compromised

Posted Sep 2, 2011 18:44 UTC (Fri) by abacus (guest, #49001) [Link]

What you wrote makes sense to me, but you might be overestimating the power of the rootkit that infected kernel.org. Apparently what it does is gathering the contents of the .ssh directory and the shell command history files of each user. Source: Mary Heintz, University of Chicaco phalanx2 Infection Report, August 2008.

kernel.org compromised

Posted Sep 2, 2011 18:54 UTC (Fri) by dlang (guest, #313) [Link] (1 responses)

I actually see what the kernel.org folks are doing as one of the best examples of dealing with a compromise that I've seen.

as for your objection that they are running commands on the system to track the extent of the problem, that is something that they pretty much must do, having a spare system with those sort of specs laying around just in case it's needed isn't practical (the thing holds 66 drives for crying out loud, that's _expensive_ and there is a good case to be made for keeping all of that expensive hardware that's been donated in use rather than just sitting around.

they have said that as far as they can tell the git repository has not been tampered with (and they say how they know that), so if you downloaded via git and compiled, you should be good.

it's not just assuming that the filesystem hasn't been changed, it's also the fact that all of the people who pull through the git interface (that you are saying may send trojened source) are automatically doing checks on the data.

yes it's theoretically possible to backdoor git so that it sends valid data to some people and invalid data to other people, but since the git transport layer isn't encrypted, it would be far easier to do this via a man-in-the-middle attack on your network than it would be to compromise kernel.org to do this for you.

kernel.org compromised

Posted Sep 5, 2011 11:53 UTC (Mon) by hein.zelle (guest, #33324) [Link]

> yes it's theoretically possible to backdoor git so that it sends valid
> data to some people and invalid data to other people, but since the git
> transport layer isn't encrypted, it would be far easier to do this via a
> man-in-the-middle attack on your network than it would be to compromise
> kernel.org to do this for you.

As long as we're looking at the theoretical possibilities (can you be 100% sure you're not compromised), that argument doesn't really matter. Also, if the aim is to infect a large number of machines with a modified kernel, it certainly would make sense to use a backdoor in git on the server, instead of using a man-in-the-middle attack on a local network.

In this case I personally would be fine with running that kernel and trusting the research of the kernel.org sysadmins. However, for systems with truly critical security levels, I can imagine that's not acceptable.
I would certainly start with assuming that someone hacking kernel.org is very clever and came extremely well prepared, if only because they might have been.

Independent sources for checksums or signatures sound like a viable solution (to someone who knows very little about security or checksums). That may be worth implementing for those administrators who truly depend on 100% secure verification. As for not running make before verifying what you downloaded - that definitely sounds like the responsibility of the person downloading and building the kernel.

kernel.org compromised

Posted Sep 3, 2011 5:47 UTC (Sat) by PO8 (guest, #41661) [Link]

A Linux Live CD/DVD is a really nice way to inspect your laptop hard disk without having to remove it. Assuming you trust your BIOS to boot from CD when told to, you can be pretty confident that running from a known-good Live CD is safe. You can then use a checksumming tool to verify that the OS files on your hard disk (including the Linux kernel and its modules) match those on the CD. Assuming this all works out, your laptop is now a trusted platform again. For anything where the Live CD mismatches the hard disk, you can move the files from the hard disk aside and replace them with the ones from the Live CD, usually without compromising the usability of the laptop very much.

Of course the usual bootstrap problems apply. How do you get a known-good Live CD given a compromised laptop? How do you get known-good checksums for configuration files and the like that are necessarily configured for your laptop? You can imagine answers to questions like these; ideally, someone would have worked on these answers for common Linux distros, but I'm not aware of much offhand.