kernel.org compromised

Posted Sep 1, 2011 1:43 UTC (Thu) by rickmoen (subscriber, #6943)
Parent article: kernel.org compromised

I'm curious about two points not (to my knowledge) yet covered, probably for the simple reason that there hasn't been enough time for proper forensics:

1. What was the escalation path to root?

2. Completely aside from the git repo contents, were the downloadable *.tar.[gz|bz2] source archives trojaned? Are there any non-site-local mechanisms in place to detect such tampering (other than, of course, the fact that the Linux Kernel Archives OpenPGP key is well known, and some of us bother to check the *.tar.[gz|bz2].sign files?

Rick Moen
rick@linuxmafia.com

to post comments

kernel.org compromised

Posted Sep 1, 2011 4:57 UTC (Thu) by rickmoen (subscriber, #6943) [Link] (11 responses)

Since asking these questions, I've been informed that downloadable tarballs could have been trojaned and then signed by the intruder on hera, implying that the private signing key + passphrase are normally present there.

In that light, the presence of *.sign files published alongside the tarballs isn't useful for ensuring security integrity of source tarballs on kernel.org. It's useful only for making sure that kernel.org mirrors correctly track the upstream site. Kernel tarballs on kernel.org can be vetted by generating them from an sha1-vetted git repo checkout, but that is currently the only way to check their integrity.

I'm a little surprised at that. Those *.sign files and the published Linux Kernel Archives OpenPGP key thus end up, IMO, being a little misleading.

Rick Moen
rick@linuxmafia.com

kernel.org compromised

Posted Sep 1, 2011 14:23 UTC (Thu) by raven667 (subscriber, #5198) [Link] (10 responses)

Are you sure you have that right? I'd be surprised too I the private key was on the same host the files are distributed from. Are you sure that someone didnt misspeaks and mean that the attacker could have accessed the private key using the credentials of a user on Hera but that the key was on another machine? That would be less surprising.

kernel.org compromised

Posted Sep 1, 2011 18:58 UTC (Thu) by rickmoen (subscriber, #6943) [Link] (9 responses)

As it turned out, the kernel.org regular (not a site admin) who implied that (in a mailing list conversation with me) subsequently allowed as how he really wasn't sure about the signing logic but that he in fact would speculate that the signing doesn't happen on the shared host itself.

Anyway, perhaps we'll hear more-precise details directly from the site admins, when they've caught up on what is doubtless an exhausting (and annoying) cleanup and forensics job and caught some sleep.

I'm already hearing some plausible speculation about how the intruders might have stolen privilege escalation without needing to locally attack software / configurations on kernel.org itself, but let's wait to hear the word from the horse's mouth.

Rick Moen
rick@linuxmafia.com

kernel.org compromised

Posted Sep 2, 2011 0:13 UTC (Fri) by BenHutchings (subscriber, #37955) [Link] (8 responses)

The signing is automatic. As you say, it only proves that the files have not been modified between master.kernel.org and the mirror.

kernel.org compromised

Posted Sep 2, 2011 2:39 UTC (Fri) by Duncan (guest, #6647) [Link]

Just confirming via a second source, see the last paragraph in the H-Online coverage here:

http://www.h-online.com/open/news/item/Security-breach-at...

kernel.org compromised

Posted Sep 2, 2011 4:43 UTC (Fri) by rickmoen (subscriber, #6943) [Link] (6 responses)

My thanks to you and Duncan for that.

What I've basically been hearing amounts to: 'Silly sysadmin, don't download kernel tarballs from kernel.org and expect to authenticate them by checking the accompanying .sign file using the published gpg key. If you must authenticate a tarball, pull down the sources from the git repo thereby ensuring the sources' integrity, and then tar it up.'

I might now do that, but the point is that many of us have been accustomed to ftp'ing (etc.) Linux kernel tarballs since pre-1.0 days, and many people will naturally interpret the presence alongside those tarballs of .sign files plus the published gpg key as suggesting a means for the public to verify code signature that had been made using a carefully guarded private key. Given that that is not the case, I would humbly suggest that the kernel.org operators, at some point, see if that perceptual pitfall can be eliminated.

Rick Moen
rick@linuxmafia.com

kernel.org compromised

Posted Sep 2, 2011 6:43 UTC (Fri) by pebolle (guest, #35204) [Link] (5 responses)

> What I've basically been hearing amounts to: 'Silly sysadmin,
> don't download kernel tarballs from kernel.org [...]

0) Does anyone know what the major distributions use as a base for their kernel packages: kernel.org tarballs or tarballs created from their copy of a git repository? (As far as I know the Fedora kernel packages have a tarball as their primary source.)

1) What means of verification were there in the pre git era?

2) Anyhow, it turns out it this is all spelled out in detail at http://kernel.org/signature.html . Note:

> This signature does not guarantee that the Linux Kernel Archives
> master site itself has not been compromised. However, if we suffer
> an intrusion we will revoke the key and post information here as
> quickly as possible.

(I assume these lines predate this incident.) So I guess we'll have to wait for a revocation of their key. Not that their key matters much to me any more ...

kernel.org compromised

Posted Sep 2, 2011 7:36 UTC (Fri) by pebolle (guest, #35204) [Link] (3 responses)

> 0) Does anyone know what the major distributions use as a base for their
> kernel packages: kernel.org tarballs or tarballs created from their copy
> of a git repository? (As far as I know the Fedora kernel packages have a
> tarball as their primary source.)

Well, to answer my own question, if I look at kernel-2.6.40.3-0.fc15.src.rpm (which seems to be the latest kernel pushed for F15) I see it's v.2.6.39 based. And doing a simple md5sum on the copy of linux-2.6.39.tar.bz2 enclosed in that source package shows that is identical to the copy of linux-2.6.39.tar.bz2 I just downloaded for a kernel.org mirror.

Creating bzipped tarballs with identical checksums is rather hard, isn't it? I assume Fedora uses kernel.org tarballs for its packages.

Perhaps someone from the Fedora kernel team could confirm (or deny) that.

kernel.org compromised

Posted Sep 2, 2011 8:09 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

Yes, Fedora uses kernel.org tarballs. I am a Fedora contributor although not in the kernel team.

kernel.org compromised

Posted Sep 3, 2011 19:31 UTC (Sat) by pebolle (guest, #35204) [Link] (1 responses)

0) Thanks.

1) Note my idea that creating "bzipped tarballs with identical checksums is rather hard" turned out to be entirely incorrect.

2) I was able to create identical bzipped tarballs of linux-2.6.39 and linux-3.0. I also was able to create identical bzip2 versions of a few recent -rc and -stable patches. So it seems the tar an bzip2 formats are more likely to generate reproducible results than I expected. Ditto for the git commands I used to generate their input.

(3) Boring details: for linux-2.6.39 I only needed to add "-c tar.umask=0022" to "git archive" to create an identical tarfile. For the -rc patches I needed to edit one git diff index line (ie, an "index <hash>..<hash> <mode>" line) because one hash abbreviation changed due to, in short, recent additions to the repository. Trivial changes, really. Other files I could easily recreate with rather obvious command lines, like "git diff v3.0..v3.0.4 | bzip2 -9".)

kernel.org compromised

Posted Sep 4, 2011 17:58 UTC (Sun) by joey (guest, #328) [Link]

While this is thuroughly offtopic, if you're interested in recreating original tarballs, gz files, and bz files, see pristine-tar. It's not "easy" in the general case, but it's possible. :)

kernel.org compromised

Posted Sep 2, 2011 8:56 UTC (Fri) by rickmoen (subscriber, #6943) [Link]

Pebole wrote:

Note: "This signature does not guarantee that the Linux Kernel Archives master site itself has not been compromised."

Well, no code signature ever guarantees that the hosting site hasn't been compromised.

A sentence higher up, immediately after the bit about the signing being automated, is actually quite a bit more significant: "This signature can be used to prove that a file, which may have been obtained from a mirror site or other location, really originated at the Linux Kernel Archives."

A truly careful parsing of that sentence might catch the implication that the signature proves only that the file really originated at kernel.org. However, it'd be really nice if this were more apparent upon casual browsing of tarballs.

Rick Moen
rick@linuxmafia.com