kernel.org compromised

[Posted August 31, 2011 by corbet]

The main kernel.org page is currently carrying a notice that the site has suffered a security breach. "Earlier this month, a number of servers in the kernel.org infrastructure were compromised. We discovered this August 28th. While we currently believe that the source code repositories were unaffected, we are in the process of verifying this and taking steps to enhance security across the kernel.org infrastructure." As the update mentions, there's little to be gained by tampering with the git repositories there anyway.

to post comments

kernel.org compromised

Posted Aug 31, 2011 23:33 UTC (Wed) by yokem_55 (subscriber, #10498) [Link] (22 responses)

Our fine editor (probably less than cheerful at the moment) has a piece on linux.com explaining for lay folk why the kernel sources couldn't be tampered with.

kernel.org compromised

Posted Aug 31, 2011 23:54 UTC (Wed) by daney (guest, #24551) [Link] (12 responses)

This is based on the axiom that SHA-1 is not breakable. Do we really know that this is the case?

kernel.org compromised

Posted Sep 1, 2011 0:02 UTC (Thu) by dlang (guest, #313) [Link] (2 responses)

given the large numbers of researchers working on looking for weaknesses, and the publicity that comes from even a slight weakening of it, there is no reason to believe that is has been broken.

yes, it's always possible that there is some black hat out there that has broken it and not told anyone about it, but this is extremely unlikely (the black hat could get rich by just publishing this data, no need to do anything more with it)

In any case, I'm pretty sure that the kernel.org team is going to be double-checking everything by using multiple checksum/hash algorithms and the odds of all of them being able to be bypassed is vanishingly slim

kernel.org compromised

Posted Sep 1, 2011 0:53 UTC (Thu) by lutchann (subscriber, #8872) [Link] (1 responses)

> the black hat could get rich by just publishing this data, no need to do anything more with it

Famous, yes, but rich? I suppose you could get a new job with a sexy title and a fat salary, but you'd probably make more money more quickly by keeping your technique a secret and selling it to somebody.

value of zero day versus public reputation

Posted Sep 2, 2011 12:04 UTC (Fri) by copsewood (subscriber, #199) [Link]

Whoever would pay you to keep a SHA1 crack as a zero day vulnerability would have to pay more than the value of all the book sales and conference keynote speech fees. Also the kind of organisations who would want you to keep this secret are likely to be more difficult to negotiate with and dangerous to your health if negotiations go wrong than book publishers and conference organisers.

kernel.org compromised

Posted Sep 1, 2011 0:03 UTC (Thu) by njs (subscriber, #40338) [Link]

If you can break SHA-1, you have a few options:

Announce it to the world and instantly become famous for your mathematical genius, then take your pick of jobs in academia, at the NSA, etc.
Use the secret to screw with world governments, banks, and other such juicy targets. (Risky, but high possible payoff.)
Crack kernel.org, sneak a short-lived and soon-discovered back-door into the Linux upstream sources (which nobody important uses directly), and then wait to see which major government tracks you down first.

I wouldn't lose any sleep over this.

kernel.org compromised

Posted Sep 1, 2011 0:05 UTC (Thu) by jonabbey (guest, #2736) [Link] (4 responses)

It depends on the level of certainty you are asking for in using the term 'know'.

No one has demonstrated the ability to find a collision with a pre-existing text in the open literature yet as far as I know. There are techniques to find generate a collision with less than 2^80 operations, but that's still a ways from attacking Git.

See http://www.schneier.com/blog/archives/2009/06/ever_better...

Even if someone had the ability to generate a collision with a preimage text, they'd have the secondary task of making the colliding text look reasonable to kernel developers. When people were having fun creating md5 collisions, they tended to have pretty long sequences of random bytes in the text, which would be hard to hide in a kernel source file.

Long story short, it's very unlikely that anyone out there has successfully attacked SHA-1 to the degree necessary to be able to attack the kernel's Git repo. If they had, it's unlikely that they'd have made the kind of mistakes that attracted kernel.org's attention.

kernel.org compromised

Posted Sep 1, 2011 7:04 UTC (Thu) by chax (guest, #52122) [Link] (3 responses)

How about making these semirandom bytes appear in a distributable firmware file? ;)

kernel.org compromised

Posted Sep 2, 2011 0:11 UTC (Fri) by BenHutchings (subscriber, #37955) [Link]

Yes, binary firmware is definitely a weak point. People tend to think it's more important to have source for software that runs on their CPU than software running on a peripheral processor. However, plenty of those peripheral processors can initiate DMA and compromise the host system unless they are carefully restricted by an IOMMU.

kernel.org compromised

Posted Sep 2, 2011 2:59 UTC (Fri) by Duncan (guest, #6647) [Link]

That's a possibility that I thought of too, but considering that each object is separately hashed, just getting random bytes into some random firmware doesn't help all that much.

Basically, in addition to that, you'd have to:

1) Insert the payload into the SAME firmware file (a different file will have its own SHA1, so it's gotta be the same file).

2) Somehow, convince Linus to accept a patch that loads that firmware file, either for your specific target if you have one (not out of the realm of possibility), or for a rather large segment of the kernel running population (rather more difficult, given kernel modularity, but it depends on just how large a segment you want, if everyone running a specific NIC or graphics chip is enough, it's not too difficult, but if you want nearly everyone running a Linux kernel, it's VERY difficult indeed!).

Even if that occurs, you then have to wait until it's actually deployed on your target, with some targets not updating for years, hoping it's not caught in the mean time.

So successful attack via firmware is indeed theoretically possible, but still not particularly simple in practice. There's almost certainly faster and less resource intensive compromise methods, so it's unlikely to be used in practice.

Duncan

kernel.org compromised

Posted Sep 2, 2011 5:49 UTC (Fri) by joey (guest, #328) [Link]

Git's potential exposure to sha1 collisions is broader than
just files. http://kitenet.net/~joey/blog/entry/size_of_the_git_sha1_...

kernel.org compromised

Posted Sep 1, 2011 5:47 UTC (Thu) by iabervon (subscriber, #722) [Link]

There are two extents to which SHA-1 could be broken: (1) it could be possible to find one or many collisions, where the attack generates both colliding texts; (2) it could be possible to find a different text that collides with a provided text. Unless you're able to take your pair of texts and have one of them pass code review and the other introduce a back door, (1) is useless for attacking a project hosted in git with a reasonable development strategy. On the other hand, (1) will probably get you EV certs for *.google.com from a reputable CA, because x509 tools don't tend to be suspicious of chunks of random data in the certificate request like Linus is of chunks of random data in a patch. Based on the fact that MD5-based certs *have* seen this attack, and SHA-1 certs have seen less effective attacks, SHA-1 is probably not this broken. On the other hand, MD5 *is* that broken, but it's still not known to be so broken as (2), even with the algorithm changed to be weaker. There's a huge gap between (1) and (2), significantly bigger than between the strengths of different hashes, so it's implausible that (2) would have actually occurred before evidence of (1) being exploited had turned up.

kernel.org compromised

Posted Sep 1, 2011 7:21 UTC (Thu) by raalkml (guest, #72852) [Link] (1 responses)

Not just this axiom. It takes much more than a hash collision to introduce a commit into a git repository or change the recorded history.

See http://git-blame.blogspot.com/2011/08/how-to-inject-malic... for details

kernel.org compromised

Posted Sep 2, 2011 5:52 UTC (Fri) by joey (guest, #328) [Link]

Junio's blog post, strangely, does not mention sha1 collisions at all. sha1 collisions would allow bypassing exactly the mechanisms he describes.

kernel.org compromised

Posted Sep 1, 2011 0:44 UTC (Thu) by hmh (subscriber, #3838) [Link] (1 responses)

I would be more worried about all the stuff in mirrors.kernel.org, binaries at boot.kernel.org, as well as the tarballs and diffs...

kernel.org compromised

Posted Sep 1, 2011 1:12 UTC (Thu) by mtaht (guest, #11087) [Link]

Breakage into mirrors.kernel.org of anything not protected by a sha1 hash worries me. Even if it was temporary - a day or two - then changed back, people that rely on those mirrors of various distros, pieces of those distros could have been compromised unknowingly.

d@bob-desktop:~/git$ host mirrors.us.kernel.org
mirrors.us.kernel.org has address 149.20.4.71
mirrors.us.kernel.org has IPv6 address 2001:4f8:1:10:1997:313:1:0
mirrors.us.kernel.org mail is handled by 10 hera.kernel.org.
mirrors.us.kernel.org mail is handled by 20 zeus1.kernel.org.
mirrors.us.kernel.org mail is handled by 30 zeus2.kernel.org.
mirrors.us.kernel.org mail is handled by 999 bl-ckh-le.kernel.org.

What bothers me right now is that I remember seeing several updates go by in the past month that weren't verifiable via gpg key on several systems I maintain... and I just did installs of them...

kernel.org compromised

Posted Sep 1, 2011 13:06 UTC (Thu) by eupator (guest, #44581) [Link] (6 responses)

This analysis assumes that the compromised git.kernel.org serves the same contents to everyone, which is not necessarily true.

kernel.org compromised

Posted Sep 1, 2011 14:39 UTC (Thu) by epa (subscriber, #39769) [Link]

Obviously instead of releasing "Linux 3.0" Linus should have announced Linux version 02f8c6aee8df3cdc935e9bdd4f2d020306035dbe and instructed everyone to fetch it using git instead of downloading source tarballs. (Really, why are we still doing that?)

kernel.org compromised

Posted Sep 1, 2011 15:25 UTC (Thu) by nix (subscriber, #2304) [Link] (4 responses)

Even if it doesn't, Linus's and Greg's tags are signed and validate against a key fetched long before the compromise: and I very much doubt their private keys are on kernel.org.

kernel.org compromised

Posted Sep 2, 2011 2:34 UTC (Fri) by Duncan (guest, #6647) [Link] (1 responses)

You may be doubting wrong. See the last paragraph of the H-Online coverage, here:

http://www.h-online.com/open/news/item/Security-breach-at...

Apparently the signatures are generated on a server @ kernel.org, and it's as yet unclear whether the crackers had access to all the necessary components for signing, or not.

Duncan

kernel.org compromised

Posted Sep 2, 2011 14:25 UTC (Fri) by nix (subscriber, #2304) [Link]

I'm not talking about the PGP signatures for the tarballs. I'm talking about the signed *tags* in the git tree: the object you see via e.g. 'git show v3.0.4'. That is part of the git repo and cannot be forged without access to Greg's private key. Now a hostile attacker could add a fake one, but the key would be different, and Greg would be certain to notice.

kernel.org compromised

Posted Sep 2, 2011 5:07 UTC (Fri) by eupator (guest, #44581) [Link] (1 responses)

A very good point about GKH - users of his stable trees are indeed protected, assuming they check the signatures. Less so, sadly, about Linus - the tip of his tree isn't signed, and even if it was, I don't have a path of trust to his key.

kernel.org compromised

Posted Sep 2, 2011 14:28 UTC (Fri) by nix (subscriber, #2304) [Link]

The tip isn't signed, but the rcs are, so you know that v3.1-rc4, released Aug 28 2011, is legitimate, and so is all the history leading up to it.

Conclusion: the git tree is not compromised up to that point.

kernel.org compromised

Posted Sep 1, 2011 1:43 UTC (Thu) by rickmoen (subscriber, #6943) [Link] (12 responses)

I'm curious about two points not (to my knowledge) yet covered, probably for the simple reason that there hasn't been enough time for proper forensics:

1. What was the escalation path to root?

2. Completely aside from the git repo contents, were the downloadable *.tar.[gz|bz2] source archives trojaned? Are there any non-site-local mechanisms in place to detect such tampering (other than, of course, the fact that the Linux Kernel Archives OpenPGP key is well known, and some of us bother to check the *.tar.[gz|bz2].sign files?

Rick Moen
rick@linuxmafia.com

kernel.org compromised

Posted Sep 1, 2011 4:57 UTC (Thu) by rickmoen (subscriber, #6943) [Link] (11 responses)

Since asking these questions, I've been informed that downloadable tarballs could have been trojaned and then signed by the intruder on hera, implying that the private signing key + passphrase are normally present there.

In that light, the presence of *.sign files published alongside the tarballs isn't useful for ensuring security integrity of source tarballs on kernel.org. It's useful only for making sure that kernel.org mirrors correctly track the upstream site. Kernel tarballs on kernel.org can be vetted by generating them from an sha1-vetted git repo checkout, but that is currently the only way to check their integrity.

I'm a little surprised at that. Those *.sign files and the published Linux Kernel Archives OpenPGP key thus end up, IMO, being a little misleading.

Rick Moen
rick@linuxmafia.com

kernel.org compromised

Posted Sep 1, 2011 14:23 UTC (Thu) by raven667 (subscriber, #5198) [Link] (10 responses)

Are you sure you have that right? I'd be surprised too I the private key was on the same host the files are distributed from. Are you sure that someone didnt misspeaks and mean that the attacker could have accessed the private key using the credentials of a user on Hera but that the key was on another machine? That would be less surprising.

kernel.org compromised

Posted Sep 1, 2011 18:58 UTC (Thu) by rickmoen (subscriber, #6943) [Link] (9 responses)

As it turned out, the kernel.org regular (not a site admin) who implied that (in a mailing list conversation with me) subsequently allowed as how he really wasn't sure about the signing logic but that he in fact would speculate that the signing doesn't happen on the shared host itself.

Anyway, perhaps we'll hear more-precise details directly from the site admins, when they've caught up on what is doubtless an exhausting (and annoying) cleanup and forensics job and caught some sleep.

I'm already hearing some plausible speculation about how the intruders might have stolen privilege escalation without needing to locally attack software / configurations on kernel.org itself, but let's wait to hear the word from the horse's mouth.

Rick Moen
rick@linuxmafia.com

kernel.org compromised

Posted Sep 2, 2011 0:13 UTC (Fri) by BenHutchings (subscriber, #37955) [Link] (8 responses)

The signing is automatic. As you say, it only proves that the files have not been modified between master.kernel.org and the mirror.

kernel.org compromised

Posted Sep 2, 2011 2:39 UTC (Fri) by Duncan (guest, #6647) [Link]

Just confirming via a second source, see the last paragraph in the H-Online coverage here:

http://www.h-online.com/open/news/item/Security-breach-at...

kernel.org compromised

Posted Sep 2, 2011 4:43 UTC (Fri) by rickmoen (subscriber, #6943) [Link] (6 responses)

My thanks to you and Duncan for that.

What I've basically been hearing amounts to: 'Silly sysadmin, don't download kernel tarballs from kernel.org and expect to authenticate them by checking the accompanying .sign file using the published gpg key. If you must authenticate a tarball, pull down the sources from the git repo thereby ensuring the sources' integrity, and then tar it up.'

I might now do that, but the point is that many of us have been accustomed to ftp'ing (etc.) Linux kernel tarballs since pre-1.0 days, and many people will naturally interpret the presence alongside those tarballs of .sign files plus the published gpg key as suggesting a means for the public to verify code signature that had been made using a carefully guarded private key. Given that that is not the case, I would humbly suggest that the kernel.org operators, at some point, see if that perceptual pitfall can be eliminated.

Rick Moen
rick@linuxmafia.com

kernel.org compromised

Posted Sep 2, 2011 6:43 UTC (Fri) by pebolle (guest, #35204) [Link] (5 responses)

> What I've basically been hearing amounts to: 'Silly sysadmin,
> don't download kernel tarballs from kernel.org [...]

0) Does anyone know what the major distributions use as a base for their kernel packages: kernel.org tarballs or tarballs created from their copy of a git repository? (As far as I know the Fedora kernel packages have a tarball as their primary source.)

1) What means of verification were there in the pre git era?

2) Anyhow, it turns out it this is all spelled out in detail at http://kernel.org/signature.html . Note:

> This signature does not guarantee that the Linux Kernel Archives
> master site itself has not been compromised. However, if we suffer
> an intrusion we will revoke the key and post information here as
> quickly as possible.

(I assume these lines predate this incident.) So I guess we'll have to wait for a revocation of their key. Not that their key matters much to me any more ...

kernel.org compromised

Posted Sep 2, 2011 7:36 UTC (Fri) by pebolle (guest, #35204) [Link] (3 responses)

> 0) Does anyone know what the major distributions use as a base for their
> kernel packages: kernel.org tarballs or tarballs created from their copy
> of a git repository? (As far as I know the Fedora kernel packages have a
> tarball as their primary source.)

Well, to answer my own question, if I look at kernel-2.6.40.3-0.fc15.src.rpm (which seems to be the latest kernel pushed for F15) I see it's v.2.6.39 based. And doing a simple md5sum on the copy of linux-2.6.39.tar.bz2 enclosed in that source package shows that is identical to the copy of linux-2.6.39.tar.bz2 I just downloaded for a kernel.org mirror.

Creating bzipped tarballs with identical checksums is rather hard, isn't it? I assume Fedora uses kernel.org tarballs for its packages.

Perhaps someone from the Fedora kernel team could confirm (or deny) that.

kernel.org compromised

Posted Sep 2, 2011 8:09 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

Yes, Fedora uses kernel.org tarballs. I am a Fedora contributor although not in the kernel team.

kernel.org compromised

Posted Sep 3, 2011 19:31 UTC (Sat) by pebolle (guest, #35204) [Link] (1 responses)

0) Thanks.

1) Note my idea that creating "bzipped tarballs with identical checksums is rather hard" turned out to be entirely incorrect.

2) I was able to create identical bzipped tarballs of linux-2.6.39 and linux-3.0. I also was able to create identical bzip2 versions of a few recent -rc and -stable patches. So it seems the tar an bzip2 formats are more likely to generate reproducible results than I expected. Ditto for the git commands I used to generate their input.

(3) Boring details: for linux-2.6.39 I only needed to add "-c tar.umask=0022" to "git archive" to create an identical tarfile. For the -rc patches I needed to edit one git diff index line (ie, an "index <hash>..<hash> <mode>" line) because one hash abbreviation changed due to, in short, recent additions to the repository. Trivial changes, really. Other files I could easily recreate with rather obvious command lines, like "git diff v3.0..v3.0.4 | bzip2 -9".)

kernel.org compromised

Posted Sep 4, 2011 17:58 UTC (Sun) by joey (guest, #328) [Link]

While this is thuroughly offtopic, if you're interested in recreating original tarballs, gz files, and bz files, see pristine-tar. It's not "easy" in the general case, but it's possible. :)

kernel.org compromised

Posted Sep 2, 2011 8:56 UTC (Fri) by rickmoen (subscriber, #6943) [Link]

Pebole wrote:

Note: "This signature does not guarantee that the Linux Kernel Archives master site itself has not been compromised."

Well, no code signature ever guarantees that the hosting site hasn't been compromised.

A sentence higher up, immediately after the bit about the signing being automated, is actually quite a bit more significant: "This signature can be used to prove that a file, which may have been obtained from a mirror site or other location, really originated at the Linux Kernel Archives."

A truly careful parsing of that sentence might catch the implication that the signature proves only that the file really originated at kernel.org. However, it'd be really nice if this were more apparent upon casual browsing of tarballs.

Rick Moen
rick@linuxmafia.com

Bad week

Posted Sep 1, 2011 2:39 UTC (Thu) by cesarb (subscriber, #6266) [Link] (3 responses)

This has been a bad week.

First Apache, then DigiNotar, now kernel.org...

Bad week

Posted Sep 1, 2011 5:07 UTC (Thu) by jcm (subscriber, #18262) [Link] (2 responses)

I think the takeaway is if you're a big entity you're going to get cracked sooner or later, and the best thing to do is just accept that and move on.

Bad week

Posted Sep 1, 2011 5:45 UTC (Thu) by Cato (guest, #7643) [Link]

I agree with the first part, but big organisations need to spend more time/money on continuous monitoring and intrusion detection rather than just "accept it and move on".

Bad week

Posted Sep 1, 2011 17:26 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link]

I think you mean "design a response plan in advance" rather than "move on". The time to plan for a security breach- or any kind of nearly inevitable disaster- is when it's still a possibility rather than an actual event. Any big target like kernel.org that doesn't have a plan to do when they're cracked is being irresponsible.

Don't give out ssh access

Posted Sep 1, 2011 6:40 UTC (Thu) by slashdot (guest, #22014) [Link] (19 responses)

Well, it's probably not a good idea to give ssh access to 448 people.

Unfortunately, all current kernels including Linux seem to be inherently insecure due to a huge attack surface, so it would be best to acknowledge that, and stop pretending that there is a difference between non-root and root local access.

I'd suggest to only give ssh access to any web/git server to the administrator, and use standard upload mechanisms (git, ftp, WebDAV) for anyone else.

Don't give out ssh access

Posted Sep 1, 2011 9:16 UTC (Thu) by ebirdie (guest, #512) [Link] (18 responses)

>Well, it's probably not a good idea to give ssh access to 448 people.

Does it change much, if the intrusion was through one of the admins?

"a trojan was found on the personal machine of kernel developer H Peter Anvin"
The Register: Kernel.org Linux repository rooted in hack attack
http://www.theregister.co.uk/2011/08/31/linux_kernel_secu...

If I remember correctly, H Peter Anvin was, at least, an admin for kernel.org services.

That would give easy answer, how root access was possibly gained. I find two possibilities. First if sudo authentication was passwordless on utilities with low security and/or sudo authentication had shortcircuit to ssh-login, a single-sign-on, then the intruder did not need to use any vulnerabilities in software code but just found weak spots in methods to use root priveledges. Secondly, if the ssh-client binary were trojaned at H Peter Anvin's use, typed password for sudo were logged as well.

According to the above article the trojan logged pretty much information, which suggests the above might be, what have happened. Ie. the intruder has found from the logs, how to gain root access and rest is just keeping yourself hidden.

This scenario closes your claims, but does not exclude the possibility that root access has had been available through some vulnerability in some software somewhere in the system, but finding the vulnerability to use had put the intruder to higher exposure to be found sooner.

Two-factor authentication

Posted Sep 1, 2011 15:01 UTC (Thu) by Cato (guest, #7643) [Link] (17 responses)

Protecting against a targetted attack on an administrator's system is tough, but it might have helped to require two-factor authentication for all access to the kernel.org servers. Something like Yubikey is not expensive ($25 a key), works with a range of open source software, and can make even a keylogger less useful.

The Fedora project seems to have switched to using this for its project infrastructure, so there is some precedent: http://www.yubico.com/fedora-uses-yubikey-for-strong-two-...

There are many other things that should be done (e.g. intrusion detection to discover injected trojans, ideally on client systems as well) but this is a simple action to take that helps protect against compromised client systems to some degree.

I have no connection with Yubikey, but I have just ordered a couple for use with the excellent LastPass, so I don't feel so paranoid when using its passwords on a Windows PC. Though perhaps I should enable it on Linux as well...

Two-factor authentication

Posted Sep 1, 2011 15:30 UTC (Thu) by yokem_55 (subscriber, #10498) [Link]

Google's 2-factor authentication (the android app for it is open source and straight forward) also can be used in an SSH context using the google_authenticator pam module. Sadly though the regular SSH key-authentication overrides the pam auth methods and so the two can't be combined.....

Two-factor authentication

Posted Sep 1, 2011 16:37 UTC (Thu) by endecotp (guest, #36428) [Link] (4 responses)

> Something like Yubikey is not expensive ($25 a key)

$25 * 448 = $11,200

Two-factor authentication

Posted Sep 1, 2011 17:31 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link]

Now compare that $11,200 to the cleanup cost of having kernel.org hacked. Plus the opportunity cost of having the system down while you respond. And the reputational cost of having such a major system compromised. Good security only looks expensive until it's compared to the cost of insecurity.

Two-factor authentication

Posted Sep 1, 2011 17:32 UTC (Thu) by jonabbey (guest, #2736) [Link]

And cheap at the price.

Two-factor authentication

Posted Sep 1, 2011 21:51 UTC (Thu) by Cato (guest, #7643) [Link]

The cost would be a good motivation to have far fewer SSH logins on kernel.org. Non-SSH services wouldn't really need 2-factor authentication, as the risk of privilege escalation is much lower.

Two-factor authentication

Posted Sep 2, 2011 6:51 UTC (Fri) by job (guest, #670) [Link]

There are volume rebates, list price in their online shop is $6750 USD for 450. Put it in perspective, what are the costs of hosting and administering the machines? If that is still to steep I might be a good idea to contact them, I believe there are some free software enthusiasts associated with the company.

Two-factor authentication

Posted Sep 1, 2011 17:45 UTC (Thu) by skvidal (guest, #3094) [Link] (3 responses)

To be fair we've had some.... complications implementing yubikeys in Fedora Infrastructure.

The pam integration is... kinda clunky.
It is also vulnerable to reply attacks in some cases which is not fantastic, either.

personally I've found the google 2-factor auth easier to use.

Two-factor authentication

Posted Sep 2, 2011 7:12 UTC (Fri) by job (guest, #670) [Link] (1 responses)

Replay attacks? That's serious. I believe they are supposed to use some session identifier against that. Do you have more details?

Two-factor authentication

Posted Sep 3, 2011 11:00 UTC (Sat) by Cato (guest, #7643) [Link]

I found this from 2 years ago - replay attack due to a bug in the Yubico authentication server, since fixed by Yubico: http://www.grennan.com/?p=113

It's up to the authentication server to do the right checks, so perhaps some authentication servers have bugs.

I'd like to see more on this claimed replay attack, too.

Two-factor authentication

Posted Sep 10, 2011 6:51 UTC (Sat) by Cato (guest, #7643) [Link]

I think this is a reference to the fact that Yubikey's model is event-based one-time passwords, as with HOTP which it does support as an option. These comments include a response from the vendor explaining more and linking to a third party security analysis: http://www.mnxsolutions.com/security/secure-ssh-and-wordp... - the article talks about using Yubikey to secure SSH and WordPress logins.

Two-factor authentication

Posted Sep 1, 2011 20:15 UTC (Thu) by ebirdie (guest, #512) [Link] (6 responses)

In case of a trojaned ssh-client does any authentication really help?

The trojan may grap unlocked ssh-key from ssh-agent and grap the key password, if ssh-key is unlocked with -i switch.

In the case PAM and a two-factor authentication is used the trojan may quietly wait until all the connection authentication hoopla is done. The hoopla does not reveal any passwords, OTP or whatever to the attacker yet, but all the attacker needs through the trojaned ssh-client is an authenticated ssh-connection to the target system.

At the target system, while it has a spooky terminal connection via ssh to it, all the attacker need is to wait for terminal with root privileges. From there it is only couple seconds that the target system has rootkit. If all goes well, the rootkit does not crash the system, reveal itself and lets the attacker in.

One can't trust the terminal attached to the ssh-connection, no matter how well the connection was authenticated. Sudo at the target machine can't come to client machine to verify that the calling terminal weren't spooky.

Does this make sense?

Two-factor authentication

Posted Sep 1, 2011 23:16 UTC (Thu) by slashdot (guest, #22014) [Link] (5 responses)

Indeed, that method only foils a passive keylogger.

The only truly secure and somewhat practical method is to acquire a cheap low-end machine (preferably with hardware trusted execution path verification) which is exclusively used as a ssh terminal, and blocks all incoming traffic not related to an outgoing tcp connection.

Of course compromise of the server through the services it exposes is still possible.

Two-factor authentication

Posted Sep 2, 2011 7:59 UTC (Fri) by dugsong (guest, #79624) [Link] (3 responses)

Nothing's perfect, but a practical defense is to use your smartphone for out-of-band verification.

Disclaimer: Our company, Duo Security, provides this for free to open-source projects. Because it's the right thing to do!

As an aside, PAM and pubkey auth in OpenSSH are mutually exclusive. We work around this with a simple login_duo utility that doesn't require sshd restart, or even root access.

Two-factor authentication

Posted Sep 2, 2011 18:12 UTC (Fri) by slashdot (guest, #22014) [Link] (2 responses)

As far as I can tell ebirdie's objections apply to your solution as well.

In short, the issue is that a compromised client has full control of the connection after the authentication is done, regardless of whatever fancy mechanism you use to authenticate.

If you don't care about detection, it doesn't even require a compromised client: just software that detects authentication being successfully completed and simulates some keyboard/mouse input that gives the attacker full control of the server and shuts out the administrator from it.

Two-factor authentication

Posted Sep 2, 2011 23:12 UTC (Fri) by jonoberheide (guest, #71029) [Link]

Actually, our Duo Push authentication allows you to approve/deny individual transactions as you see fit, preventing the sort of session-riding attack that you're referring to.

For example, in the follow screenshot, you can see the exact command that an attacker is attempting to execute:

http://blog.duosecurity.com/wp-content/uploads/2011/04/pu...

Obviously, you would deny the attacker's attempted "rm -rf" here. ;-)

Regards,
Jon Oberheide

Two-factor authentication

Posted Sep 5, 2011 10:36 UTC (Mon) by sitaram (guest, #5959) [Link]

Disclaimer: I am the author and maintainer of gitolite.

If we make the assumption that all 448 users really do not need an actual *shell*, and that they will be mostly doing git push or putting files in some designated area using rsync, you can actually use gitolite to limit what they can do quite handily.

They don't get a shell, their access are limited to whatever repos they've been given access to, and even the rsync command can be access controlled using the same software, limiting users write access to specific directories only.

I've kinda lost track if they found the actual *escalation* vector involved but I'll bet it needed shell on the server.

Two-factor authentication

Posted Sep 2, 2011 8:22 UTC (Fri) by Cato (guest, #7643) [Link]

A lower cost way of having a "separate PC as SSH terminal" might be Qubes, which uses multiple VMs with differing levels of trust - could use one VM for general web browsing, one for shopping/banking, and one for SSH access.
https://lwn.net/Articles/385213/

The problem is that this requires significant work to set up for the owner of the client PC, and it's hard to mandate this setup. However, cutting the number of SSH users from 448 to a handful would make this more practical for kernel.org.

kernel.org compromised

Posted Sep 1, 2011 10:11 UTC (Thu) by forlwn (guest, #63934) [Link] (3 responses)

Its interesting to note extensive similarities in the text from articles that some early IT/Tech news sites apparently with no relation between them, are adopting to report this incident, one of them being a mac fan's page.

http://www.macworld.co.uk/procreative/news/index.cfm?news...

http://www.panarmenian.net/eng/news/76907/

kernel.org compromised

Posted Sep 1, 2011 15:03 UTC (Thu) by dgm (subscriber, #49227) [Link] (2 responses)

> Its interesting [...]

Really?

kernel.org compromised

Posted Sep 1, 2011 16:54 UTC (Thu) by jone (guest, #62596) [Link] (1 responses)

>> Its interesting [...]

> Really?

kernel.org compromised

Posted Sep 3, 2011 0:37 UTC (Sat) by forlwn (guest, #63934) [Link]

>>> Its interesting [...]

>> Really?

> no

That's "kind of" interesting, just to know to way how "the story" was being propagated on the specialized media in the net.

kernel.org compromised

Posted Sep 2, 2011 16:44 UTC (Fri) by zooko (guest, #2589) [Link] (5 responses)

This is an extended and edited version of a long comment I posted at Jon Corbet's blog entry on linux.com. I hope you find it useful.

Dear Jon Corbet:

Everything you wrote in this blog entry was true, but it leaves out some pretty important aspects.

I used git to fetch a copy of the linux kernel from git.kernel.org a few days ago, compiled it, and booted into it on my primary work/dev/play machine (a laptop). What do I do now?

As far as I understand, the answer is the same as if I had fetched the kernel source code with ftp: I either just cross my fingers and gamble that the attackers weren't sophisticated enough to have taken over my laptop and installed a rootkit in the BIOS, or the battery or something, or I shut it down, extract the hard drive, buy a new laptop, and very carefully and painstakingly copy some of the data from the old hard drive.

A lot of the analysis of this attack so far seems to assume that the attackers would change the contents of a git repository on the kernel.org filesystem. There's nothing requiring them to do that. They could cause git pulls to deliver backdoored kernels to users while leaving the persistent copy on disk untouched. They could also choose which users get the real kernel or a backdoored kernelfor example based on the user's IP address or ssh key. Nor are attackers required to leave evidence sitting on the filesystems of those target usersas soon as a user who has just done a git pull from kernel.org runs make, the payload can compile a backdoored kernel and simultaneously revert the source code to the pristine version so that inspection of the source will show no evidence.

If you did a git pull and then did not execute any of the resulting code (such as by running make), then you could inspect for tampering. Git could help with that inspection.

If the kernel.org admins were extremely professional and careful, then they might be able to give me some confidence in which of these two strategies I should take, by doing a good post-mortem and sharing what they learn about what the attacker did and how and when. However, the fact that they appear to take the strategy of investigating and "cleaning up" by running commands on the compromised systems and inviting their users to do so really undermines that confidence.

That's a pretty amateur game plan they've got there. It can't give you confidence that your system isn't still under the control of a sufficiently sophisticated attacker, and it might open up more opportunities for such an attacker to compromise your systems and your users' systems further, to cover their tracks, or to provide you with false clues.

So what am I going to do? I can't really afford to buy a new laptop right now, so I'm going to gamble that the attackers really were as amateur as they seem in allowing themselves to get caught like this, and that there weren't also other, more capable attackers present, and that the operating system I'm running on this laptop right now isn't under their control.

I'm also going to see if I can set up an alternate system (I have a few cool Linux/ARM devices around here that I haven't fully explored yet) without letting any attacker which might control this laptop get control of that system.

I'm also going to stop fetching kernel source code from kernel.org.

To address your main point about git. Its wonderful append-only immutable data semantics could help people like me here, but not the way most people currently use it.

If I want to avoid being exposed to this kind of attack again, I don't exactly need to stop fetching kernel source code from kernel.org, I just need to stop fetching the tag that tells me which secure hash I should be fetching. So, for example, if I could learn from Greg Kroah-Hartman what is the secure hash of the next release of linux 2.6.32, then I could use git (on a known-good system) to fetch the version matching that secure hash from kernel.org without being vulnerable to the people who control kernel.org supplying me with backdoored kernel source.

However, running git clone git://git.kernel.org/pub/scm/linux/kernel/git/longterm/linux-2.6.32.y.git does not offer that same security property, because in that case I would be relying on git.kernel.org to tell me which secure hash I should use to verify the kernel source, and the attackers could supply me with a secure hash that correctly matched their backdoored kernel source.

Or I could check digital signatures with GPG. In that case I would be getting the security benefits from using GPG, and it wouldn't matter whether I were checking a sig on a tarball or on a git tag. Very few people will ever go to the extra effort to check GPG sigs as often as they do git pulls. It would be cool if the checking of the GPG signature were integrated and automatic so that whenever you did a git pull, it would check the signature and stop you right there if the signature was missing or incorrect. We do something equivalent to that in the Tahoe-LAFS secure distributed filesystem.

It would be cool if GregKH had included the commit hash for 2.6.32.46b0b3e41f6fe88a15bd27f73f6fabef48e0fea2f0in the release announcement. In that case attackers who took over kernel.org but did not take over lwn.net or gmane.org would be unable to trick me into running their backdoored kernel and giving them control of my laptop. People should get into the habit of including the commit hash whenever they are talking about a specific version such as a release. That would really leverage git's integrity properties to protect a lot of people, unlike the current way it is used. (We also do a technique similar to that in Tahoe-LAFSwe embed the secure hash into every URL which points to a specific, immutable version.)

P.S.

In fact, the same threat that applies to my laptop would also apply to any kernel developer, even Linus, who did a git pull from kernel.org and ran make. You wrote "Kernel.org may seem like the place where kernel development is done, but its not; its really just a distribution point.". I took that to mean that kernel developers do not do git pulls from kernel.org and instead do it through some other more private hub or do it directly from one another's personal servers or something, but is that true? I doubt it. I'll bet plenty of kernel developers have pulled code from kernel.org, compiled that code, and run it. If they did, then the only reason to think that their development machines are not now under the control of an attacker is because we currently think that the attackers who took over kernel.org were of low skill.

Regards,

Zooko

kernel.org compromised

Posted Sep 2, 2011 17:54 UTC (Fri) by slashdot (guest, #22014) [Link]

Just check that the commit that you compiled from the git repository you downloaded is an ancestor of the current kernel commit.

Of course if you already ran the kernel on the machine storing the downloaded tree, a backdoored kernel could have in principle been programmed to clean up the git repository on disk, but I'd consider that extraordinarily unlikely.

Also, someone else would have likely noticed and reported the discrepancy to LKML.

At any rate, there is always some remote security risk, since for all we know Linus Torvalds himself could have inserted covert backdoors in the kernel source code.

kernel.org compromised

Posted Sep 2, 2011 18:44 UTC (Fri) by abacus (guest, #49001) [Link]

What you wrote makes sense to me, but you might be overestimating the power of the rootkit that infected kernel.org. Apparently what it does is gathering the contents of the .ssh directory and the shell command history files of each user. Source: Mary Heintz, University of Chicaco phalanx2 Infection Report, August 2008.

kernel.org compromised

Posted Sep 2, 2011 18:54 UTC (Fri) by dlang (guest, #313) [Link] (1 responses)

I actually see what the kernel.org folks are doing as one of the best examples of dealing with a compromise that I've seen.

as for your objection that they are running commands on the system to track the extent of the problem, that is something that they pretty much must do, having a spare system with those sort of specs laying around just in case it's needed isn't practical (the thing holds 66 drives for crying out loud, that's _expensive_ and there is a good case to be made for keeping all of that expensive hardware that's been donated in use rather than just sitting around.

they have said that as far as they can tell the git repository has not been tampered with (and they say how they know that), so if you downloaded via git and compiled, you should be good.

it's not just assuming that the filesystem hasn't been changed, it's also the fact that all of the people who pull through the git interface (that you are saying may send trojened source) are automatically doing checks on the data.

yes it's theoretically possible to backdoor git so that it sends valid data to some people and invalid data to other people, but since the git transport layer isn't encrypted, it would be far easier to do this via a man-in-the-middle attack on your network than it would be to compromise kernel.org to do this for you.

kernel.org compromised

Posted Sep 5, 2011 11:53 UTC (Mon) by hein.zelle (guest, #33324) [Link]

> yes it's theoretically possible to backdoor git so that it sends valid
> data to some people and invalid data to other people, but since the git
> transport layer isn't encrypted, it would be far easier to do this via a
> man-in-the-middle attack on your network than it would be to compromise
> kernel.org to do this for you.

As long as we're looking at the theoretical possibilities (can you be 100% sure you're not compromised), that argument doesn't really matter. Also, if the aim is to infect a large number of machines with a modified kernel, it certainly would make sense to use a backdoor in git on the server, instead of using a man-in-the-middle attack on a local network.

In this case I personally would be fine with running that kernel and trusting the research of the kernel.org sysadmins. However, for systems with truly critical security levels, I can imagine that's not acceptable.
I would certainly start with assuming that someone hacking kernel.org is very clever and came extremely well prepared, if only because they might have been.

Independent sources for checksums or signatures sound like a viable solution (to someone who knows very little about security or checksums). That may be worth implementing for those administrators who truly depend on 100% secure verification. As for not running make before verifying what you downloaded - that definitely sounds like the responsibility of the person downloading and building the kernel.

kernel.org compromised

Posted Sep 3, 2011 5:47 UTC (Sat) by PO8 (guest, #41661) [Link]

A Linux Live CD/DVD is a really nice way to inspect your laptop hard disk without having to remove it. Assuming you trust your BIOS to boot from CD when told to, you can be pretty confident that running from a known-good Live CD is safe. You can then use a checksumming tool to verify that the OS files on your hard disk (including the Linux kernel and its modules) match those on the CD. Assuming this all works out, your laptop is now a trusted platform again. For anything where the Live CD mismatches the hard disk, you can move the files from the hard disk aside and replace them with the ones from the Live CD, usually without compromising the usability of the laptop very much.

Of course the usual bootstrap problems apply. How do you get a known-good Live CD given a compromised laptop? How do you get known-good checksums for configuration files and the like that are necessarily configured for your laptop? You can imagine answers to questions like these; ideally, someone would have worked on these answers for common Linux distros, but I'm not aware of much offhand.

kernel.org compromised

Posted Sep 5, 2011 13:19 UTC (Mon) by arekm (guest, #4846) [Link]

... and now github.com is "new" kernel.org (for linux.git)