LWN: Comments on "kernel.org compromised"

Two-factor authentication

Cato — Sat, 10 Sep 2011 06:51:12 +0000

I think this is a reference to the fact that Yubikey's model is event-based one-time passwords, as with HOTP which it does support as an option. These comments include a response from the vendor explaining more and linking to a third party security analysis: http://www.mnxsolutions.com/security/secure-ssh-and-wordp... - the article talks about using Yubikey to secure SSH and WordPress logins.

kernel.org compromised

arekm — Mon, 05 Sep 2011 13:19:03 +0000

... and now github.com is "new" kernel.org (for linux.git)

kernel.org compromised

hein.zelle — Mon, 05 Sep 2011 11:53:50 +0000

> yes it's theoretically possible to backdoor git so that it sends valid
> data to some people and invalid data to other people, but since the git
> transport layer isn't encrypted, it would be far easier to do this via a
> man-in-the-middle attack on your network than it would be to compromise
> kernel.org to do this for you.

As long as we're looking at the theoretical possibilities (can you be 100% sure you're not compromised), that argument doesn't really matter. Also, if the aim is to infect a large number of machines with a modified kernel, it certainly would make sense to use a backdoor in git on the server, instead of using a man-in-the-middle attack on a local network.

In this case I personally would be fine with running that kernel and trusting the research of the kernel.org sysadmins. However, for systems with truly critical security levels, I can imagine that's not acceptable.
I would certainly start with assuming that someone hacking kernel.org is very clever and came extremely well prepared, if only because they might have been.

Independent sources for checksums or signatures sound like a viable solution (to someone who knows very little about security or checksums). That may be worth implementing for those administrators who truly depend on 100% secure verification. As for not running make before verifying what you downloaded - that definitely sounds like the responsibility of the person downloading and building the kernel.

Two-factor authentication

sitaram — Mon, 05 Sep 2011 10:36:33 +0000

Disclaimer: I am the author and maintainer of gitolite.

If we make the assumption that all 448 users really do not need an actual *shell*, and that they will be mostly doing git push or putting files in some designated area using rsync, you can actually use gitolite to limit what they can do quite handily.

They don't get a shell, their access are limited to whatever repos they've been given access to, and even the rsync command can be access controlled using the same software, limiting users write access to specific directories only.

I've kinda lost track if they found the actual *escalation* vector involved but I'll bet it needed shell on the server.

kernel.org compromised

joey — Sun, 04 Sep 2011 17:58:04 +0000

While this is thuroughly offtopic, if you're interested in recreating original tarballs, gz files, and bz files, see pristine-tar. It's not "easy" in the general case, but it's possible. :)

kernel.org compromised

pebolle — Sat, 03 Sep 2011 19:31:30 +0000

0) Thanks.

1) Note my idea that creating "bzipped tarballs with identical checksums is rather hard" turned out to be entirely incorrect.

2) I was able to create identical bzipped tarballs of linux-2.6.39 and linux-3.0. I also was able to create identical bzip2 versions of a few recent -rc and -stable patches. So it seems the tar an bzip2 formats are more likely to generate reproducible results than I expected. Ditto for the git commands I used to generate their input.

(3) Boring details: for linux-2.6.39 I only needed to add "-c tar.umask=0022" to "git archive" to create an identical tarfile. For the -rc patches I needed to edit one git diff index line (ie, an "index <hash>..<hash> <mode>" line) because one hash abbreviation changed due to, in short, recent additions to the repository. Trivial changes, really. Other files I could easily recreate with rather obvious command lines, like "git diff v3.0..v3.0.4 | bzip2 -9".)

Two-factor authentication

Cato — Sat, 03 Sep 2011 11:00:47 +0000

I found this from 2 years ago - replay attack due to a bug in the Yubico authentication server, since fixed by Yubico: http://www.grennan.com/?p=113

It's up to the authentication server to do the right checks, so perhaps some authentication servers have bugs.

I'd like to see more on this claimed replay attack, too.

kernel.org compromised

PO8 — Sat, 03 Sep 2011 05:47:52 +0000

A Linux Live CD/DVD is a really nice way to inspect your laptop hard disk without having to remove it. Assuming you trust your BIOS to boot from CD when told to, you can be pretty confident that running from a known-good Live CD is safe. You can then use a checksumming tool to verify that the OS files on your hard disk (including the Linux kernel and its modules) match those on the CD. Assuming this all works out, your laptop is now a trusted platform again. For anything where the Live CD mismatches the hard disk, you can move the files from the hard disk aside and replace them with the ones from the Live CD, usually without compromising the usability of the laptop very much.

Of course the usual bootstrap problems apply. How do you get a known-good Live CD given a compromised laptop? How do you get known-good checksums for configuration files and the like that are necessarily configured for your laptop? You can imagine answers to questions like these; ideally, someone would have worked on these answers for common Linux distros, but I'm not aware of much offhand.

kernel.org compromised

forlwn — Sat, 03 Sep 2011 00:37:30 +0000

>>> Its interesting [...]

>> Really?

> no

That's "kind of" interesting, just to know to way how "the story" was being propagated on the specialized media in the net.

Two-factor authentication

jonoberheide — Fri, 02 Sep 2011 23:12:08 +0000

Actually, our Duo Push authentication allows you to approve/deny individual transactions as you see fit, preventing the sort of session-riding attack that you're referring to.

For example, in the follow screenshot, you can see the exact command that an attacker is attempting to execute:

http://blog.duosecurity.com/wp-content/uploads/2011/04/pu...

Obviously, you would deny the attacker's attempted "rm -rf" here. ;-)

Regards,
Jon Oberheide

kernel.org compromised

dlang — Fri, 02 Sep 2011 18:54:49 +0000

I actually see what the kernel.org folks are doing as one of the best examples of dealing with a compromise that I've seen.

as for your objection that they are running commands on the system to track the extent of the problem, that is something that they pretty much must do, having a spare system with those sort of specs laying around just in case it's needed isn't practical (the thing holds 66 drives for crying out loud, that's _expensive_ and there is a good case to be made for keeping all of that expensive hardware that's been donated in use rather than just sitting around.

they have said that as far as they can tell the git repository has not been tampered with (and they say how they know that), so if you downloaded via git and compiled, you should be good.

it's not just assuming that the filesystem hasn't been changed, it's also the fact that all of the people who pull through the git interface (that you are saying may send trojened source) are automatically doing checks on the data.

yes it's theoretically possible to backdoor git so that it sends valid data to some people and invalid data to other people, but since the git transport layer isn't encrypted, it would be far easier to do this via a man-in-the-middle attack on your network than it would be to compromise kernel.org to do this for you.

kernel.org compromised

abacus — Fri, 02 Sep 2011 18:44:12 +0000

What you wrote makes sense to me, but you might be overestimating the power of the rootkit that infected kernel.org. Apparently what it does is gathering the contents of the .ssh directory and the shell command history files of each user. Source: Mary Heintz, University of Chicaco phalanx2 Infection Report, August 2008.

Two-factor authentication

slashdot — Fri, 02 Sep 2011 18:12:59 +0000

As far as I can tell ebirdie's objections apply to your solution as well.

In short, the issue is that a compromised client has full control of the connection after the authentication is done, regardless of whatever fancy mechanism you use to authenticate.

If you don't care about detection, it doesn't even require a compromised client: just software that detects authentication being successfully completed and simulates some keyboard/mouse input that gives the attacker full control of the server and shuts out the administrator from it.

kernel.org compromised

slashdot — Fri, 02 Sep 2011 17:54:21 +0000

Just check that the commit that you compiled from the git repository you downloaded is an ancestor of the current kernel commit.

Of course if you already ran the kernel on the machine storing the downloaded tree, a backdoored kernel could have in principle been programmed to clean up the git repository on disk, but I'd consider that extraordinarily unlikely.

Also, someone else would have likely noticed and reported the discrepancy to LKML.

At any rate, there is always some remote security risk, since for all we know Linus Torvalds himself could have inserted covert backdoors in the kernel source code.

kernel.org compromised

zooko — Fri, 02 Sep 2011 16:44:02 +0000

This is an extended and edited version of a long comment I posted at Jon Corbet's blog entry on linux.com. I hope you find it useful.

Dear Jon Corbet:

Everything you wrote in this blog entry was true, but it leaves out some pretty important aspects.

I used git to fetch a copy of the linux kernel from git.kernel.org a few days ago, compiled it, and booted into it on my primary work/dev/play machine (a laptop). What do I do now?

As far as I understand, the answer is the same as if I had fetched the kernel source code with ftp: I either just cross my fingers and gamble that the attackers weren't sophisticated enough to have taken over my laptop and installed a rootkit in the BIOS, or the battery or something, or I shut it down, extract the hard drive, buy a new laptop, and very carefully and painstakingly copy some of the data from the old hard drive.

A lot of the analysis of this attack so far seems to assume that the attackers would change the contents of a git repository on the kernel.org filesystem. There's nothing requiring them to do that. They could cause git pulls to deliver backdoored kernels to users while leaving the persistent copy on disk untouched. They could also choose which users get the real kernel or a backdoored kernelfor example based on the user's IP address or ssh key. Nor are attackers required to leave evidence sitting on the filesystems of those target usersas soon as a user who has just done a git pull from kernel.org runs make, the payload can compile a backdoored kernel and simultaneously revert the source code to the pristine version so that inspection of the source will show no evidence.

If you did a git pull and then did not execute any of the resulting code (such as by running make), then you could inspect for tampering. Git could help with that inspection.

If the kernel.org admins were extremely professional and careful, then they might be able to give me some confidence in which of these two strategies I should take, by doing a good post-mortem and sharing what they learn about what the attacker did and how and when. However, the fact that they appear to take the strategy of investigating and "cleaning up" by running commands on the compromised systems and inviting their users to do so really undermines that confidence.

That's a pretty amateur game plan they've got there. It can't give you confidence that your system isn't still under the control of a sufficiently sophisticated attacker, and it might open up more opportunities for such an attacker to compromise your systems and your users' systems further, to cover their tracks, or to provide you with false clues.

So what am I going to do? I can't really afford to buy a new laptop right now, so I'm going to gamble that the attackers really were as amateur as they seem in allowing themselves to get caught like this, and that there weren't also other, more capable attackers present, and that the operating system I'm running on this laptop right now isn't under their control.

I'm also going to see if I can set up an alternate system (I have a few cool Linux/ARM devices around here that I haven't fully explored yet) without letting any attacker which might control this laptop get control of that system.

I'm also going to stop fetching kernel source code from kernel.org.

To address your main point about git. Its wonderful append-only immutable data semantics could help people like me here, but not the way most people currently use it.

If I want to avoid being exposed to this kind of attack again, I don't exactly need to stop fetching kernel source code from kernel.org, I just need to stop fetching the tag that tells me which secure hash I should be fetching. So, for example, if I could learn from Greg Kroah-Hartman what is the secure hash of the next release of linux 2.6.32, then I could use git (on a known-good system) to fetch the version matching that secure hash from kernel.org without being vulnerable to the people who control kernel.org supplying me with backdoored kernel source.

However, running git clone git://git.kernel.org/pub/scm/linux/kernel/git/longterm/linux-2.6.32.y.git does not offer that same security property, because in that case I would be relying on git.kernel.org to tell me which secure hash I should use to verify the kernel source, and the attackers could supply me with a secure hash that correctly matched their backdoored kernel source.

Or I could check digital signatures with GPG. In that case I would be getting the security benefits from using GPG, and it wouldn't matter whether I were checking a sig on a tarball or on a git tag. Very few people will ever go to the extra effort to check GPG sigs as often as they do git pulls. It would be cool if the checking of the GPG signature were integrated and automatic so that whenever you did a git pull, it would check the signature and stop you right there if the signature was missing or incorrect. We do something equivalent to that in the Tahoe-LAFS secure distributed filesystem.

It would be cool if GregKH had included the commit hash for 2.6.32.46b0b3e41f6fe88a15bd27f73f6fabef48e0fea2f0in the release announcement. In that case attackers who took over kernel.org but did not take over lwn.net or gmane.org would be unable to trick me into running their backdoored kernel and giving them control of my laptop. People should get into the habit of including the commit hash whenever they are talking about a specific version such as a release. That would really leverage git's integrity properties to protect a lot of people, unlike the current way it is used. (We also do a technique similar to that in Tahoe-LAFSwe embed the secure hash into every URL which points to a specific, immutable version.)

P.S.

In fact, the same threat that applies to my laptop would also apply to any kernel developer, even Linus, who did a git pull from kernel.org and ran make. You wrote "Kernel.org may seem like the place where kernel development is done, but its not; its really just a distribution point.". I took that to mean that kernel developers do not do git pulls from kernel.org and instead do it through some other more private hub or do it directly from one another's personal servers or something, but is that true? I doubt it. I'll bet plenty of kernel developers have pulled code from kernel.org, compiled that code, and run it. If they did, then the only reason to think that their development machines are not now under the control of an attacker is because we currently think that the attackers who took over kernel.org were of low skill.

Regards,

Zooko

kernel.org compromised

nix — Fri, 02 Sep 2011 14:28:45 +0000

The tip isn't signed, but the rcs are, so you know that v3.1-rc4, released Aug 28 2011, is legitimate, and so is all the history leading up to it.

Conclusion: the git tree is not compromised up to that point.

kernel.org compromised

nix — Fri, 02 Sep 2011 14:25:39 +0000

I'm not talking about the PGP signatures for the tarballs. I'm talking about the signed *tags* in the git tree: the object you see via e.g. 'git show v3.0.4'. That is part of the git repo and cannot be forged without access to Greg's private key. Now a hostile attacker could add a fake one, but the key would be different, and Greg would be certain to notice.

value of zero day versus public reputation

copsewood — Fri, 02 Sep 2011 12:04:09 +0000

Whoever would pay you to keep a SHA1 crack as a zero day vulnerability would have to pay more than the value of all the book sales and conference keynote speech fees. Also the kind of organisations who would want you to keep this secret are likely to be more difficult to negotiate with and dangerous to your health if negotiations go wrong than book publishers and conference organisers.

kernel.org compromised

rickmoen — Fri, 02 Sep 2011 08:56:32 +0000

Pebole wrote:

Note: "This signature does not guarantee that the Linux Kernel Archives master site itself has not been compromised."

Well, no code signature ever guarantees that the hosting site hasn't been compromised.

A sentence higher up, immediately after the bit about the signing being automated, is actually quite a bit more significant: "This signature can be used to prove that a file, which may have been obtained from a mirror site or other location, really originated at the Linux Kernel Archives."

A truly careful parsing of that sentence might catch the implication that the signature proves only that the file really originated at kernel.org. However, it'd be really nice if this were more apparent upon casual browsing of tarballs.

Rick Moen
rick@linuxmafia.com

Two-factor authentication

Cato — Fri, 02 Sep 2011 08:22:06 +0000

A lower cost way of having a "separate PC as SSH terminal" might be Qubes, which uses multiple VMs with differing levels of trust - could use one VM for general web browsing, one for shopping/banking, and one for SSH access.
https://lwn.net/Articles/385213/

The problem is that this requires significant work to set up for the owner of the client PC, and it's hard to mandate this setup. However, cutting the number of SSH users from 448 to a handful would make this more practical for kernel.org.

kernel.org compromised

rahulsundaram — Fri, 02 Sep 2011 08:09:30 +0000

Yes, Fedora uses kernel.org tarballs. I am a Fedora contributor although not in the kernel team.

Two-factor authentication

dugsong — Fri, 02 Sep 2011 07:59:32 +0000

Nothing's perfect, but a practical defense is to use your smartphone for out-of-band verification.

Disclaimer: Our company, Duo Security, provides this for free to open-source projects. Because it's the right thing to do!

As an aside, PAM and pubkey auth in OpenSSH are mutually exclusive. We work around this with a simple login_duo utility that doesn't require sshd restart, or even root access.

kernel.org compromised

pebolle — Fri, 02 Sep 2011 07:36:46 +0000

> 0) Does anyone know what the major distributions use as a base for their
> kernel packages: kernel.org tarballs or tarballs created from their copy
> of a git repository? (As far as I know the Fedora kernel packages have a
> tarball as their primary source.)

Well, to answer my own question, if I look at kernel-2.6.40.3-0.fc15.src.rpm (which seems to be the latest kernel pushed for F15) I see it's v.2.6.39 based. And doing a simple md5sum on the copy of linux-2.6.39.tar.bz2 enclosed in that source package shows that is identical to the copy of linux-2.6.39.tar.bz2 I just downloaded for a kernel.org mirror.

Creating bzipped tarballs with identical checksums is rather hard, isn't it? I assume Fedora uses kernel.org tarballs for its packages.

Perhaps someone from the Fedora kernel team could confirm (or deny) that.

Two-factor authentication

job — Fri, 02 Sep 2011 07:12:06 +0000

Replay attacks? That's serious. I believe they are supposed to use some session identifier against that. Do you have more details?

Two-factor authentication

job — Fri, 02 Sep 2011 06:51:55 +0000

There are volume rebates, list price in their online shop is $6750 USD for 450. Put it in perspective, what are the costs of hosting and administering the machines? If that is still to steep I might be a good idea to contact them, I believe there are some free software enthusiasts associated with the company.

kernel.org compromised

pebolle — Fri, 02 Sep 2011 06:43:45 +0000

> What I've basically been hearing amounts to: 'Silly sysadmin,
> don't download kernel tarballs from kernel.org [...]

0) Does anyone know what the major distributions use as a base for their kernel packages: kernel.org tarballs or tarballs created from their copy of a git repository? (As far as I know the Fedora kernel packages have a tarball as their primary source.)

1) What means of verification were there in the pre git era?

2) Anyhow, it turns out it this is all spelled out in detail at http://kernel.org/signature.html . Note:

> This signature does not guarantee that the Linux Kernel Archives
> master site itself has not been compromised. However, if we suffer
> an intrusion we will revoke the key and post information here as
> quickly as possible.

(I assume these lines predate this incident.) So I guess we'll have to wait for a revocation of their key. Not that their key matters much to me any more ...

kernel.org compromised

joey — Fri, 02 Sep 2011 05:52:15 +0000

Junio's blog post, strangely, does not mention sha1 collisions at all. sha1 collisions would allow bypassing exactly the mechanisms he describes.

kernel.org compromised

joey — Fri, 02 Sep 2011 05:49:34 +0000

Git's potential exposure to sha1 collisions is broader than
just files. http://kitenet.net/~joey/blog/entry/size_of_the_git_sha1_...

kernel.org compromised

eupator — Fri, 02 Sep 2011 05:07:19 +0000

A very good point about GKH - users of his stable trees are indeed protected, assuming they check the signatures. Less so, sadly, about Linus - the tip of his tree isn't signed, and even if it was, I don't have a path of trust to his key.

kernel.org compromised

rickmoen — Fri, 02 Sep 2011 04:43:15 +0000

My thanks to you and Duncan for that.

What I've basically been hearing amounts to: 'Silly sysadmin, don't download kernel tarballs from kernel.org and expect to authenticate them by checking the accompanying .sign file using the published gpg key. If you must authenticate a tarball, pull down the sources from the git repo thereby ensuring the sources' integrity, and then tar it up.'

I might now do that, but the point is that many of us have been accustomed to ftp'ing (etc.) Linux kernel tarballs since pre-1.0 days, and many people will naturally interpret the presence alongside those tarballs of .sign files plus the published gpg key as suggesting a means for the public to verify code signature that had been made using a carefully guarded private key. Given that that is not the case, I would humbly suggest that the kernel.org operators, at some point, see if that perceptual pitfall can be eliminated.

Rick Moen
rick@linuxmafia.com

kernel.org compromised

Duncan — Fri, 02 Sep 2011 02:59:57 +0000

That's a possibility that I thought of too, but considering that each object is separately hashed, just getting random bytes into some random firmware doesn't help all that much.

Basically, in addition to that, you'd have to:

1) Insert the payload into the SAME firmware file (a different file will have its own SHA1, so it's gotta be the same file).

2) Somehow, convince Linus to accept a patch that loads that firmware file, either for your specific target if you have one (not out of the realm of possibility), or for a rather large segment of the kernel running population (rather more difficult, given kernel modularity, but it depends on just how large a segment you want, if everyone running a specific NIC or graphics chip is enough, it's not too difficult, but if you want nearly everyone running a Linux kernel, it's VERY difficult indeed!).

Even if that occurs, you then have to wait until it's actually deployed on your target, with some targets not updating for years, hoping it's not caught in the mean time.

So successful attack via firmware is indeed theoretically possible, but still not particularly simple in practice. There's almost certainly faster and less resource intensive compromise methods, so it's unlikely to be used in practice.

Duncan

kernel.org compromised

Duncan — Fri, 02 Sep 2011 02:39:07 +0000

Just confirming via a second source, see the last paragraph in the H-Online coverage here:

http://www.h-online.com/open/news/item/Security-breach-at...

kernel.org compromised

Duncan — Fri, 02 Sep 2011 02:34:58 +0000

You may be doubting wrong. See the last paragraph of the H-Online coverage, here:

http://www.h-online.com/open/news/item/Security-breach-at...

Apparently the signatures are generated on a server @ kernel.org, and it's as yet unclear whether the crackers had access to all the necessary components for signing, or not.

Duncan

kernel.org compromised

BenHutchings — Fri, 02 Sep 2011 00:13:19 +0000

The signing is automatic. As you say, it only proves that the files have not been modified between master.kernel.org and the mirror.

kernel.org compromised

BenHutchings — Fri, 02 Sep 2011 00:11:18 +0000

Yes, binary firmware is definitely a weak point. People tend to think it's more important to have source for software that runs on their CPU than software running on a peripheral processor. However, plenty of those peripheral processors can initiate DMA and compromise the host system unless they are carefully restricted by an IOMMU.

Two-factor authentication

slashdot — Thu, 01 Sep 2011 23:16:48 +0000

Indeed, that method only foils a passive keylogger.

The only truly secure and somewhat practical method is to acquire a cheap low-end machine (preferably with hardware trusted execution path verification) which is exclusively used as a ssh terminal, and blocks all incoming traffic not related to an outgoing tcp connection.

Of course compromise of the server through the services it exposes is still possible.

Two-factor authentication

Cato — Thu, 01 Sep 2011 21:51:19 +0000

The cost would be a good motivation to have far fewer SSH logins on kernel.org. Non-SSH services wouldn't really need 2-factor authentication, as the risk of privilege escalation is much lower.

Two-factor authentication

ebirdie — Thu, 01 Sep 2011 20:15:44 +0000

In case of a trojaned ssh-client does any authentication really help?

The trojan may grap unlocked ssh-key from ssh-agent and grap the key password, if ssh-key is unlocked with -i switch.

In the case PAM and a two-factor authentication is used the trojan may quietly wait until all the connection authentication hoopla is done. The hoopla does not reveal any passwords, OTP or whatever to the attacker yet, but all the attacker needs through the trojaned ssh-client is an authenticated ssh-connection to the target system.

At the target system, while it has a spooky terminal connection via ssh to it, all the attacker need is to wait for terminal with root privileges. From there it is only couple seconds that the target system has rootkit. If all goes well, the rootkit does not crash the system, reveal itself and lets the attacker in.

One can't trust the terminal attached to the ssh-connection, no matter how well the connection was authenticated. Sudo at the target machine can't come to client machine to verify that the calling terminal weren't spooky.

Does this make sense?

kernel.org compromised

rickmoen — Thu, 01 Sep 2011 18:58:00 +0000

As it turned out, the kernel.org regular (not a site admin) who implied that (in a mailing list conversation with me) subsequently allowed as how he really wasn't sure about the signing logic but that he in fact would speculate that the signing doesn't happen on the shared host itself.

Anyway, perhaps we'll hear more-precise details directly from the site admins, when they've caught up on what is doubtless an exhausting (and annoying) cleanup and forensics job and caught some sleep.

I'm already hearing some plausible speculation about how the intruders might have stolen privilege escalation without needing to locally attack software / configurations on kernel.org itself, but let's wait to hear the word from the horse's mouth.

Rick Moen
rick@linuxmafia.com

Two-factor authentication

skvidal — Thu, 01 Sep 2011 17:45:45 +0000

To be fair we've had some.... complications implementing yubikeys in Fedora Infrastructure.

The pam integration is... kinda clunky.
It is also vulnerable to reply attacks in some cases which is not fantastic, either.

personally I've found the google 2-factor auth easier to use.