Security breach on Linux.com, LinuxFoundation.org

[Posted September 11, 2011 by jake]

From:		The Linux Foundation <info-AT-linuxfoundation.org>
To:		lwn-AT-lwn.net
Subject:		Important: Security Breach on Linux.com, LinuxFoundation.org
Date:		Sun, 11 Sep 2011 10:07:02 +0000 (UTC)
Message-ID:		<20110911100702.C6EF0267@smtp1.linux-foundation.org>

Attention Linux.com and LinuxFoundation.org users,

We are writing you because you have an account on Linux.com,
LinuxFoundation.org, or one of the subdomains associated with these domains.
On September 8, 2011, we discovered a security breach that may have
compromised your username, password, email address and other information you
have given to us. We believe this breach was connected to the intrusion on
kernel.org.

As with any intrusion and as a matter of caution, you should consider the
passwords and SSH keys that you have used on these sites compromised. If you
have reused these passwords on other sites, please change them immediately.
We are currently auditing all systems and will update public statements when
we have more information.

We have taken all Linux Foundation servers offline to do complete
re-installs. Linux Foundation services will be put back up as they become
available. We are working around the clock to expedite this process and are
working with authorities in the United States and in Europe to assist with
the investigation.

The Linux Foundation takes the security of its infrastructure and that of
its members extremely seriously and are pursuing all avenues to investigate
this attack and prevent future ones. We apologize for this inconvenience and
will communicate updates as we have them.

Please contact us at info@linuxfoundation.org with questions about this
matter.

The Linux Foundation

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 11, 2011 16:53 UTC (Sun) by hassock (guest, #78564) [Link] (15 responses)

Maybe they (kernel.org & linux.com) should fire their current
administrators and hire more competent ones?
This is getting ridiculous.

Might not be the admins

Posted Sep 11, 2011 17:01 UTC (Sun) by david.a.wheeler (subscriber, #72896) [Link] (10 responses)

Why assume it's the admins? This could be the result of a 0-day that can be exploited through the network, without authentication. I don't think we know enough yet.

Might not be the admins

Posted Sep 11, 2011 18:20 UTC (Sun) by AlexHudson (guest, #41828) [Link] (9 responses)

I thought they said previously that the working theory was some user's credentials were compromised first - even less of an admin issue imho...

Might not be the admins

Posted Sep 11, 2011 22:22 UTC (Sun) by epa (subscriber, #39769) [Link] (8 responses)

If you offer user accounts on the machine, you must expect that sooner or later one of them will be compromised. That's just common sense, given people's common behaviour of reusing passwords, logging in from insecure Windows PCs and so on. If your system tends to have local root exploits, such that the compromise of one account leads to compromise of all, then you must consider very carefully whether to offer user accounts at all. It's certainly an admin issue.

Might not be the admins

Posted Sep 11, 2011 23:41 UTC (Sun) by elanthis (guest, #6227) [Link] (4 responses)

I like how you manage to call Windows the insecure OS in this story when it's Linux servers that got hacked.

Might not be the admins

Posted Sep 12, 2011 4:55 UTC (Mon) by cmccabe (guest, #60281) [Link] (2 responses)

Just based on this email, we don't know whether the Linux servers were hacked at all. All we know is that the attackers managed to get control of a shell account and escalate that to root. It could have been done through piggybacking on sudo or through an exploit. There's no reason to start a Windows vs. Linux vs. whatever flamewar.

Personally I agree with epa. It's nice to have a server that just does one thing and doesn't offer shell accounts. It will be interesting to see what the admins decide to do to tighten security in the future.

Might not be the admins

Posted Sep 13, 2011 8:18 UTC (Tue) by epa (subscriber, #39769) [Link] (1 responses)

Just based on this email, we don't know whether the Linux servers were hacked at all. All we know is that the attackers managed to get control of a shell account and escalate that to root.

I would call a root exploit being 'hacked', wouldn't you?

Come to think of it, even social engineering to get hold of the root password would count as 'hacking' in my book.

Might not be the admins

Posted Sep 23, 2011 19:27 UTC (Fri) by cmccabe (guest, #60281) [Link]

Well, there are ways to escalate without going through the kernel.

Like this one:
https://bugs.launchpad.net/ubuntu/+bug/127116

Might not be the admins

Posted Sep 13, 2011 8:15 UTC (Tue) by epa (subscriber, #39769) [Link]

Touché! I edited my comment at the last minute to add that bit about 'insecure Windows' in an attempt to soften the tone and deflect any indignant responses from people who are a bit sensitive about these things. But this isn't Slashdot and I need not have bothered.

The point is, user accounts can and do get compromised. If you can't trust your system to keep users properly isolated from each other, then don't give out user accounts. You would instead need to run virtual machines or some other heavily sandboxed environment. It's ugly, and I hate to admit it, but that's how things are.

Might not be the admins

Posted Sep 12, 2011 8:23 UTC (Mon) by AlexHudson (guest, #41828) [Link] (2 responses)

Choice of whether or not to offer user accounts is generally not an admin issue at all: admins are there to provide tools, not dictate what tools are available. User accounts becoming compromised is just a fact of life, it's the cost of running public services.

Might not be the admins

Posted Sep 13, 2011 8:20 UTC (Tue) by epa (subscriber, #39769) [Link] (1 responses)

If the administrator's job is to provide user accounts, it is also his or her job to sandbox them adequately so that giving out one account is not equivalent to giving out root access.

Might not be the admins

Posted Sep 13, 2011 13:49 UTC (Tue) by foom (subscriber, #14868) [Link]

> If the administrator's job is to provide user accounts, it is also his or her job to sandbox them adequately so that giving out one account is not equivalent to giving out root access.

Might be best to give them out on a non-linux machine, then...

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 11, 2011 18:51 UTC (Sun) by rahulsundaram (subscriber, #21946) [Link]

That is a very irresponsible and offensive comment. The kernel.org admins are some of the best ones you can ever find.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 11, 2011 21:14 UTC (Sun) by dlang (guest, #313) [Link] (2 responses)

so if you fire the people who are working on this for free who are you going to get to do the work?

especially since the initial machine that was compromised was a developer's personal machine that they don't even administer.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 11, 2011 22:25 UTC (Sun) by dlang (guest, #313) [Link] (1 responses)

to clarify, I believe that kernel.org is volunteers, linux.com and linuxfoundation.org probably hiretheir admins, but in any case they are two differentgroups.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 11:30 UTC (Mon) by Trelane (subscriber, #56877) [Link]

there's at least one kernel.org admin paid by the linux foundation

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 11, 2011 18:51 UTC (Sun) by tialaramex (subscriber, #21167) [Link] (18 responses)

"consider ... SSH keys that you have used on these sites compromised"

What's going on here? Strictly it might mean keys that were actually stored on the compromised system. There should not be any such keys, it makes no sense to create keys on a system you don't have control over, what useful security assertions could you make about such keys? None. They should certainly be considered compromised, but that's independent of any breach.

More generously it might refer to keys used to access these systems. But in that case it represents a fundamental misunderstanding of how SSH works. SSH keys never leave the system on which they're stored, when you connect to a machine a local agent uses the key (after asking you to unlock it if necessary) to prove you're who you claim to be, but it does not need to transmit the key in order to do this, so it does not.

Further, when you use forwarding to authenticate against more distant machines, again the local agent does all the actual key operations. In this case (ie where ForwardAgent is enabled) an adversary with root on a remote machine can use the key _for the period of time you remain connected only_. In the absence of a serious cryptographic break in the public key algorithm chosen for your keys, there is no further risk once you disconnect.

So, any keys that weren't actually stored on compromised systems are fine, and any that were never should have been trusted (at least, not for more than a loopback connection) in the first place, making the warning pointless on the matter of SSH keys.

Or have I radically misunderstood the intention of this warning?

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 11, 2011 19:40 UTC (Sun) by wichert (guest, #7115) [Link] (2 responses)

A possibly vector to get key details is ssh agent forwarding.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 2:24 UTC (Mon) by fuhchee (guest, #40059) [Link] (1 responses)

Agent forwarding does not forward key data, just the (temporary) capability to generate new authenticated sessions.

Agent forwarding

Posted Sep 12, 2011 10:05 UTC (Mon) by tialaramex (subscriber, #21167) [Link]

More specifically, what's going on is that an additional stream is made available over the SSH connection to the intermediary machine. This stream sends nonces used for authentication one way, and signed messages containing the nonce back the other way.

Suppose you have a desktop PC 'abe' and you routinely use it to connect to a remote system 'sentry' which acts as the gateway to other machines, including a customer database 'enterprise' and the test data repository 'voldemort'. There is no direct access to voldemort -- for defence in depth -- to even attack voldemort's network services, bad guys need to get past the 'sentry' machine.

Your private key K is on abe. When you ssh to sentry, it sends a random nonce value N1 to abe, using K, abe is able to sign the nonce value N1 to prove it knows K, and send this back. You are authenticated.

If agent forwarding is enabled, when you now type ssh voldemort (to further connect from sentry to voldemort) voldemort sends another random nonce value N2 to sentry. Sentry doesn't have any private keys (per my rant above) so it fowards N2 to abe. The local agent on abe signs this using K, it goes back to sentry and then voldemort, and you are authenticated.

Now, suppose a bad guy, Mallory wishes to use your key to connect to the machine 'enterprise'. She has obtained root access on sentry (so she has already significantly penetrated your network defences). She must wait until you are connected to sentry using SSH with agent forwarding enabled. Then she connects to enterprise, and receives a nonce N3. Using her root access, she synthesises a request from sentry to abe, asking abe to sign N3, and it is accepted. Mallory is authenticated to enterprise as you.

BUT this is only possible while you remain logged into the compromised machine sentry, with agent forwarding to that machine enabled. The moment Mallory loses access to this machine, or you disconnect, or you reconnect but with agent forwarding disabled, this attack is impossible. There is no reason to revoke K in this case, it has not been compromised.

Also, Mallory plays a potentially risky game. Out of box default SSH clients give no indication when any of this is happening, but Mallory can't be sure you are so trusting. Your agent on abe is a black box to Mallory, it may report that it was asked to sign N3, prompting an investigation.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 11, 2011 19:42 UTC (Sun) by mallux (guest, #61473) [Link]

Maybe this refers to the SSH host keys of the compromised servers? I read it as a warning that the host key fingerprint would be changing, which would ordinarily cause you to raise an eyebrow if you weren't expecting it.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 11, 2011 21:17 UTC (Sun) by dlang (guest, #313) [Link] (10 responses)

since "SSH is secure" it's very common for people to put their SSH private keys on remote machines for convenience.

also, it may not be clear who's machine was compromised, and so it's a good idea to change them anyway.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 2:26 UTC (Mon) by tseaver (guest, #1544) [Link] (1 responses)

Putting private keys on a remote server "for convenience" is like putting your housekeys under your doormat: it practically begs to be owned.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 6:48 UTC (Mon) by dlang (guest, #313) [Link]

a lot of people don't understand this, they just learn that "SSH is secure"

this includes a lot of security professionals.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 17:42 UTC (Mon) by chrisV (guest, #43417) [Link] (7 responses)

"it's very common for people to put their SSH private keys on remote machines for convenience"

Who on earth does that, and in what way is it convenient? The whole purpose of ssh is that you don't do that.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 17:56 UTC (Mon) by dlang (guest, #313) [Link]

sometimes you want to go from one remote machine to another remote machine and this is the 'obvious' way of making it work.

I didn't say it was a good thing, all I said is that it's unfortunately common.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 19:25 UTC (Mon) by bronson (subscriber, #4806) [Link] (5 responses)

Given the number of smart engineers that have emailed me their private keys or both when I only asked for the public, I don't think dlang is exaggerating.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 22:53 UTC (Mon) by nix (subscriber, #2304) [Link] (4 responses)

Definitely not. I was arguing with a security 'auditor' only last year who said that SSH agents were dangerous and that rather than using them you should copy your private key all over the place. Alas, because he was a highly-paid external consultant and I was not, his view won out and SSH agents were banned in that entire organization.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 23:08 UTC (Mon) by dlang (guest, #313) [Link] (3 responses)

the auditor was right in that SSH agents are dangerous, but you are right that copying the private key around everywhere is also dangerous.I think you are right that putting the private key all over the place is worse.

But frankly, given the vulnerability of desktop software, I are leery of depending on certificate based authentication for anything important

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 13, 2011 11:03 UTC (Tue) by tialaramex (subscriber, #21167) [Link] (2 responses)

Hmm. There are a few scenarios I can think of:

1. User rarely connects to the system at all. In this case storing even a passphrase-protected private key for that user increases your exposure, the bad guys can obtain the protected key and crack the passphrase at their leisure and unknown to you. SSH agents are clearly better here.

2. Users frequently connects, but rarely needs forwarding. In this case the SSH agent capability is probably exposed whenever they're connected, even though they aren't using it. Whereas a pass-phrase protected key would at least have the passphrase protecting it until they used it. I think on balance storing the key locally is no worse in this case.

3. User frequently connects, frequently uses forwarding. In this case the private key, if it were stored, would very often be unlocked. The bad guys would certainly get the key and passphrase, and the passphrase may have been re-used elsewhere, increasing the impact. So SSH agents are much better here, although still dangerous.

It is my opinion that OpenSSH or alternative clients could offer e.g. a middle setting between "ForwardAgent No" and "ForwardAgent Yes" of "ForwardAgent Ask" that might be useful here, then a SSH agent would once again be unquestionably the better option.

One thing where local keys win over SSH agents is if you have people with access to many disparate systems -- the SSH agent doesn't easily distinguish requests that make good sense (e.g. a request from CorpLoginBox for authentication to CorpDBServer) from those that seem unlikely and should be flagged (a request by PublicGitRepo for authentication to CorpDBServer). If ForwardAgent is permitted at all for a machine, then all your keys can be proxied through that machine. However most users don't have such a complicated setup, with multiple SSH identities and forwarding rules, for this to make any difference to them.

Cert auth isn't the solution to all the world's problems, but it completely neutralises password-guessing attacks which to my mind makes it worthwhile just for that. If the bad guys had to use a targeted attack on one of my colleagues, just so that they can successfully connect to a server, I think we're doing pretty well to have forced the bar that high early in the game.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 13, 2011 13:51 UTC (Tue) by foom (subscriber, #14868) [Link] (1 responses)

ssh-agent as distributed by OpenSSH has an "ask" mode, invoked by "ssh-add -c", but it's pretty useless because it doesn't give you any information in the permission dialog about who's asking and for what. Also, none of the alternative ssh agents (like Gnome's) support it last I looked.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 13, 2011 22:10 UTC (Tue) by corvus (guest, #82) [Link]

Here's a 3.5 year old bug report for that:

https://bugs.launchpad.net/ubuntu/+source/gnome-keyring/+...
https://bugzilla.gnome.org/show_bug.cgi?id=525574

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 11, 2011 21:21 UTC (Sun) by pabs (subscriber, #43278) [Link] (1 responses)

What about a laptop with keys on it being compromised? A web browser issue would be the way in and most desktop/laptop computers aren't very hardened so ssh could probably be subverted with a simple "export PATH=/tmp/.muahahaha:$PATH" in ~/.bashrc or whatever.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 9:36 UTC (Mon) by tialaramex (subscriber, #21167) [Link]

Yes, if your laptop has been (or you suspect it may be) compromised, you should (tell others to) disable all the keys which were stored on that laptop, wipe it, and restore from a backup you are confident is unaffected. You might optionally want to first image the laptop drive to try to determine how you were compromised -- in order to take further steps either in protection or action against the trespassers.

But, what does this have to do with linux.com / linuxfoundation.org ?

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 22:01 UTC (Mon) by geofft (subscriber, #59789) [Link]

> What's going on here? Strictly it might mean keys that were actually stored on the compromised system. There should not be any such keys, it makes no sense to create keys on a system you don't have control over, what useful security assertions could you make about such keys? None. They should certainly be considered compromised, but that's independent of any breach.

You can make the useful security assertion that the only people who have access to the key are you and the admins of the server.

Why would you want to make this assertion? Maybe there are other machines controlled by the admins of a server (or a subset thereof) that you'd like to log into.

In any case, what useful security assertions people _do_ make has generally little to do with what they _should_ make.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 2:57 UTC (Mon) by xxiao (guest, #9631) [Link]

does this have something to do with the linux 20 anniversary?

Current lack of information

Posted Sep 12, 2011 6:41 UTC (Mon) by kruemelmo (guest, #8279) [Link] (2 responses)

For me it is irritating that not only linux.com and linuxfoundataion.org are down but also kernel.org is (still?) under maintenance 13 days after the breach.

I hadn't read the kernel.org intrusion analysis and cannot find it now. Suppose it has been on kernel.org meanwhile? Can someone post another link?

The only "official" source of information I know which is still online is lwn.

Current lack of information

Posted Sep 12, 2011 13:07 UTC (Mon) by corbet (editor, #1) [Link]

There has been no analysis of the intrusion posted yet. There are a number of serious people (beyond the kernel.org folks) looking at what happened, but, if they have figured it out yet, I've not heard about it.

Current lack of information

Posted Sep 12, 2011 15:37 UTC (Mon) by dps (guest, #5725) [Link]

While it *is* annoying if you are going to involve the authorities things take longer. There have to be careful steps to preserve evidence to make a criminal case possible. I suspect some of this may involve competent people observing and being able to testify about the authenticity of exhibit A.

In many instances people think this is too much effort and focus on getting back into production, forgoing some information about how the security was breached. how far the crackers got and what, if anything, they did.

Banks are alleged to be willing to lose few million in return for no publicity. Revealing that somebody can move a few million to somewhere they can't trace might reduce your confidence in the safety of any funds you deposited with them.

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 21:29 UTC (Mon) by forlwn (guest, #63934) [Link]

Since the Microsoft job ads campaign all over the world to hire anti-Linux Gurus, I thought that something bad would happen.
It was a matter of time.

"In one of their recent marketing job ads, the words "Linux" and "FOSS", appears more than 20 times !!!"

http://www.linuxtoday.com/it_management/2010091900635NWBZMS

Security breach on Linux.com, LinuxFoundation.org

Posted Sep 12, 2011 21:42 UTC (Mon) by kunitz (subscriber, #3965) [Link]

Bringing back websites like kernel.org is more complex than one might think at first. This task list is very likely incomplete:

Rebuild content making sure that nothing is compromised
Redesign the architecture (Isolate heavy-load servers from user-accessible machines)
Implement new architecture, get new equipment delivered, design and review firewall rules
Implement monitoring tools to discover intrusions
Provision your users with accounts without using old keys and passwords
Execute a penetration test on the new setup

Doing all these things right requires time. Hopefully they have support from folks who have experience in that, particularly the operational side of it.