How about the long overdue autopsy on the August 2011 kernel.org compromise?

Posted Nov 11, 2015 22:38 UTC (Wed) by rickmoen (subscriber, #6943)
Parent article: A new Mindcraft moment?

All of this reminds me of something tangential but, I think, very relevant indeed.

The assembled doubtless recall that in August 2011, kernel.org was root compromised. I'm sure the system's hard drives were sent off for forensic examination, and we've all been waiting patiently for the answer to the most important question: What was the compromise vector?

From shortly after the compromise was discovered on August 28, 2011, right through April 1st, 2013, kernel.org included this note at the top of the Site News: 'Thanks to all for your patience and understanding during our outage and please bear with us as we bring up the different kernel.org systems over the next few weeks. We will be writing up a report on the incident in the future.' (Emphasis added.) That comment was removed (along with the rest of the Site News) during a May 2013 edit, and there hasn't been -- to my knowledge -- a peep about any report on the incident since then.

This has been disappointing. When the Debian Project discovered sudden compromise of several of its servers in 2007, Wichert Akkerman wrote and posted an excellent public report on exactly what happened. Likewise, the Apache Foundation likewise did the right thing with good public autopsies of the 2010 Web site breaches.

Arstechnica's Dan Goodin was still trying to follow up on the lack of an autopsy on the kernel.org meltdown -- in 2013. Two years ago. He wrote:

Linux developer and maintainer Greg Kroah-Hartman told Ars that the investigation has yet to be completed and gave no timetable for when a report might be released. [...] Kroah-Hartman also told Ars kernel.org systems were rebuilt from scratch following the attack. Officials have developed new tools and procedures since then, but he declined to say what they are. "There will be a report later this year about site [sic] has been engineered, but don't quote me on when it will be released as I am not responsible for it," he wrote.

Who's responsible, then? Is anyone? Anyone? Bueller? Or is it a state secret, or what? Two years since Greg K-H said there would be a report 'later this year', and four years since the meltdown, nothing yet. How about some information?

Rick Moen
rick@linuxmafia.com

(Log in to post comments)

How about the long overdue autopsy on the August 2011 kernel.org compromise?

Posted Nov 12, 2015 14:19 UTC (Thu) by ortalo (guest, #4654) [Link]

Thank you for the reminder. Unfortunately, I am only a potential reader of the document but obviously, I second your request. And even volunteering, maybe they need some help for the post-mortem analysis.

Less seriously, note that if even the Linux mafia does not know, it must be the venusians; they are notoriously stealth in their invasions.

How about the long overdue autopsy on the August 2011 kernel.org compromise?

Posted Nov 14, 2015 12:46 UTC (Sat) by error27 (subscriber, #8346) [Link]

The compromise vector was made public at the time. http://www.theregister.co.uk/2011/08/31/linux_kernel_secu... Hackers stole an admin's ssh key.

I know the kernel.org admins have given talks about some of the new protections that have been put into place. There are no more shell logins, instead everything uses gitolite. The different services are on different hosts. There are more kernel.org staff now. People are using two factor identification. Some other stuff. Do a search for Konstantin Ryabitsev.

How about the long overdue autopsy on the August 2011 kernel.org compromise?

Posted Nov 14, 2015 15:58 UTC (Sat) by rickmoen (subscriber, #6943) [Link]

error27 wrote:

The compromise vector was made public at the time. http://www.theregister.co.uk/2011/08/31/linux_kernel_security_breach/ Hackers stole an admin's ssh key.

I beg your pardon if I was somehow unclear: That was said to have been the path of entry to the machine (and I can readily believe that, as it was also the exact path to entry into shells.sourceforge.net, many years prior, around 2002, and into many other shared Internet hosts for many years). But that is not what is of primary interest, and is not what the forensic study long promised would primarily concern: How did intruders escalate to root. To quote kernel.org administrator in the August 2011 Dan Goodin article you cited: 'How they managed to exploit that to root access is currently unknown and is being investigated'.

OK, folks, you've now had four years of investigation. What was the path of escalation to root? (Also, other details that would logically be covered by a forensic study, such as: Whose key was stolen? Who stole the key?) This is the sort of autopsy was promised prominently on the front page of kernel.org, to reporters, and elsewhere for a long time (and then summarily removed as a promise from the front page of kernel.org, without comment, along with the rest of the Site News section, and apparently dropped). It still would be appropriate to know and share that knowledge. Especially the datum of whether the path to root privilege was or was not a kernel bug (and, if not, what it was).

Rick Moen
rick@linuxmafia.com

How about the long overdue autopsy on the August 2011 kernel.org compromise?

Posted Nov 22, 2015 12:42 UTC (Sun) by rickmoen (subscriber, #6943) [Link]

I've done a closer review of revelations that came out soon after the break-in, and think I've found the answer, via a leaked copy of kernel.org chief sysadmin John H. 'Warthog9' Hawley's Aug. 29, 2011 e-mail to shell users (two days before the public was informed), plus Aug. 31st comments to The Register's Dan Goodin by 'two security researchers who were briefed on the breach':

Root escalation was via exploit of a Linux kernel security hole: Per the two security researchers, it was one both extremely embarrassing (wide-open access to /dev/mem contents including the running kernel's image in RAM, in 2.6 kernels of that day) and known-exploitable for the prior six years by canned 'sploits, one of which (Phalanx) was run by some script kiddie after entry using stolen dev credentials. Other tidbits:

Site admins left the root-compromised Internet servers running with all services still lit up, for multiple days.
Site admins and Linux Foundation sat on the knowledge and failed to inform the public for those same multiple days.
Site admins and Linux Foundation have never revealed whether trojaned Linux source tarballs were posted in the http/ftp tree for the 19+ days before they took the site down. (Yes, git checkout was fine, but what about the thousands of tarball downloads?)
After promising a report for several years and then quietly removing that promise from the front page of kernel.org, Linux Foundation now stonewalls press queries.

I posted my best attempt at reconstructing the story, absent a real report from insiders, to SVLUG's main mailing list yesterday. (Necessarily, there are surmises. If the people with the facts were more forthcoming, we'd know what happened for certain.)

I do have to wonder: If there's another embarrassing screwup, will we even be told about it at all?

Rick Moen
rick@linuxmafia.com

How about the long overdue autopsy on the August 2011 kernel.org compromise?

Posted Nov 22, 2015 14:25 UTC (Sun) by spender (guest, #23067) [Link]

Phalanx is a rootkit, not an exploit. It uses (or could use) /dev/mem to load itself into the kernel. I think you may have misinterpreted the readme -- it's not saying that /dev/mem was readable/writable by everyone, it's just saying that for users with proper permission to access it, they can read/modify any physical memory. Phalanx itself therefore wouldn't have been the privilege escalation vector. Also, it's unclear from the email you posted, but the "/dev/mem error message" they're talking about was probably the logging from STRICT_DEVMEM. If that's the case, then the rootkit wasn't successful (the Phalanx readme might be referring to older RH/Fedora versions of /dev/kmem restrictions, not the "newer" STRICT_DEVMEM).

Also, it's preferable to use live memory acquisition prior to powering off the system, otherwise you lose out on memory-resident artifacts that you can perform forensics on.

-Brad

How about the long overdue autopsy on the August 2011 kernel.org compromise?

Posted Nov 22, 2015 16:28 UTC (Sun) by rickmoen (subscriber, #6943) [Link]

Thanks for your comments, Brad.

I'd been relying on Dan Goodin's claim of Phalanx being what was used to gain root, in the bit where he cited 'two security researchers who were briefed on the breach' to that effect. Goodin also elaborated: 'Fellow security researcher Dan Rosenberg said he was also briefed that the attackers used Phalanx to compromise the kernel.org machines.' This was the first time I've heard of a rootkit being claimed to be bundled with an attack tool, and I noted that oddity in my posting to SVLUG.

That having been said, yeah, the Phalanx README doesn't specifically claim this, so then maybe Goodin and his several 'security researcher' sources blew that detail, and nobody but kernel.org insiders yet knows the escalation path used to gain root.

Also, it's preferable to use live memory acquisition prior to powering off the system, otherwise you lose out on memory-resident artifacts that you can perform forensics on.

Arguable, but a tradeoff; you can poke the compromised live system for state data, but with the drawback of leaving your system running under hostile control. I was always taught that, on balance, it's better to pull power to end the intrusion.

Rick Moen
rick@linuxmafia.com