| |||||||||||||
|
| |||||||||||||
The Fedora-Red Hat Crisis (Datamation)The Fedora-Red Hat Crisis (Datamation)Posted Sep 10, 2008 16:54 UTC (Wed) by BrucePerens (subscriber, #2510)Parent article: The Fedora-Red Hat Crisis (Datamation) The problem is that Red Hat corporate management has abandoned the security process preferred by security professionals: full disclosure. Real security professionals have to verify the details, and make sure they aren't at risk. They don't just trust their vendor to get it right, they check it themselves. Trusting someone inserts a point of failure into their security systems, because not everybody is trustworthy and large companies that have conflicted interest are low on the list of trustworthy entities. The problem is worse because some people have the full details. Fedora insiders, RH employees, and whoever the perpetrator told. These people potentially have power over your system that you would not want them to have, and you don't know enough to do anything about it. Debian handled the SSL issue and their server breach more professionally. It's sad that they have to act as an example for Red Hat. But the fact is that Debian is the prototype for most distribution processes, security or otherwise. They got to most of them first. Bruce
(Log in to post comments)
The Fedora-Red Hat Crisis (Datamation) Posted Sep 10, 2008 19:03 UTC (Wed) by rahvin (subscriber, #16953) [Link]
Speculating the other direction I would assume that RedHat would reveal details that compromise the security of their users. In fact I think to say they wouldn't is very rude.
The management has reasons for doing what they did, to assume the worst and say trust is out the window is just silly. There are very likely to be mitigating circumstances on revealing details. The FBI are likely involved as just one. It would be irresponsible of them to reveal details that aren't going to affect user security but could compromise the investigation or possibly they don't even know the details. Frankly RedHat is a great FOSS company, they deserve some time to get to the bottom, and they deserve trust that they would communicate any details that could compromise user security. I trust them, and I think it's premature to rip into them.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 10, 2008 21:28 UTC (Wed) by smoogen (subscriber, #97) [Link]
Bruce,
1) There is a big difference between disclosing about bad code and a breakin. They are covered under different legal codes and issues. If Debian's servers inside the US were broken into and had been used illegally and Debian had properly reported it.. they would be under a lot more "we can't say things".
2) have you read Spafford's latest on 'full-disclosure'
http://www.cerias.purdue.edu/site/blog/post/security_thro...
While I admit not knowing how they got into RH/Fedora is troubling.. I have dealt with enough break-ins to know that Federal laws that affect public shareholders or an on-going investigation make it impossible to say what is going on beyond "We have a problem, and we are working on it." Especially when improperlly disclosing what happened can get you charged with "intefering with an ongoing investigation", "tampering with evidence", etc.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 10, 2008 21:44 UTC (Wed) by BrucePerens (subscriber, #2510) [Link] Here's a direct quote from the Spafford paper you sent me:The problem occurs when a flaw is discovered and the owners/operators attempt to maintain (indefinitely) the sanctity of the system by stopping disclosure of the flaw. That is not generally going to work for long, especially in the face of determined foes. The owners/operators should realize that there is no (indefinite) security in keeping the flaw secret.The 2006 Debian penetration was to gluck.debian.org, a system hosted at Hewlett-Packard in Colorado. The SSL issue this year very definitely effected systems in the United States. Debian did not have to clam up. Debian is part of SPI, which is a U.S. 501(c)3 non-profit corporation. If Red Hat is constrained by an investigation in progress, they should be able to at least say that much. Let's not try to hide them behind creative misinterpretation of the law. Bruce
The Fedora-Red Hat Crisis (Datamation) Posted Sep 10, 2008 22:09 UTC (Wed) by smoogen (subscriber, #97) [Link]
Actually in the investigations I have dealt with, we were not able to even say we were working with law enforcement until they approved it. They instead told us we could say we had an issue and then later when they decided it would not interfere with their investigation we were able to say "we were working with law enforcement." [By the way, I have no knowledge if Red Hat contacted law enforcement or not.. ]
The difference I am trying to make between the SSL code and the Red Hat issue is that the SSL was a coding error with security implications, and the other was a criminal issue with security implications.
It would be better to compare how the 2006 break-in was dealt with (when was it reported to authorities, did the authorities believe the damage affect national security, stockholders, etc. When was the full-disclosure of the event made in comparison to the detection? etc). In that case RH/Fedora probably comes up wanting again.. but it is a better apples to apples.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 10, 2008 22:40 UTC (Wed) by rahvin (subscriber, #16953) [Link]
It would be HIGHLY unusual if the FBI wasn't involved. They would be risking serious class action shareholder lawsuits if they didn't call the FBI immediately after discovering the breach. RedHat likely has software in some top government agencies, including the NSA, DOD and CIA. Because of this the breach is probably being treated just like a breach at Microsoft would be treated. A full criminal investigation has probably already begun and everyone is gagged because as you said the FBI arrests and prosecutes people that reveal information before they have completed their investigation.
Frankly RedHat's behavior indicates to me that the FBI is involved. The press release reads just like some FBI agent scrubbed it of every single detail that wasn't directly relevant to notifying customers of issues that will affect them. In time we will know what happened, but we need to give them time for the FBI to relax the restrictions. I would say that if we don't have at least a mostly complete release of details in 6 months that it's probably time to start saying they aren't doing things the way they should. Otherwise like I said before I think the author and others are jumping the gun on putting RedHat's feet to the fire. And I think we should at least accept that RedHat has always had pretty good judgement and that if there was anything that directly affects customer security it has been released and the remainder is just details of how the break in occurred which don't necessarily have an impact on customer security.
And Bruce, there is a significant difference between a non-profit corporation who misconfigured software and a publicly traded company dealing with felony computer crimes. RedHat has numerous restrictions and obligations as a result of being publicly traded that Debian doesn't have to deal with. And those restrictions aren't just financial, they deal directly with what shareholders are told, how they are told and how criminal behavior is handled. Making a misstep could cost RedHat millions in SEC fines or shareholder lawsuits whereas there were no such implications to Debian.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 10, 2008 23:05 UTC (Wed) by BrucePerens (subscriber, #2510) [Link] To compare apples to apples, you can see Debian's 2006 server compromise, which occurred to a system in the bastion (outside of the corporate firewall) at HP Fort Collins. They handled it much more openly than Red Hat.The restrictions on publicly traded companies really act both ways. Destroying customer confidence by quashing discussion of a security issue will get you sued just as readily as saying too much about it. Perhaps the bottom line here is that Fedora isn't as good a structure as Debian for maintaining the core of a Linux distribution, because Fedora is too close to the publicly-traded corporation to have freedom of action. Good engineering gets done when you have that freedom.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 0:04 UTC (Thu) by smoogen (subscriber, #97) [Link]
Yes, customer confidence is all a company selling/distributing Linux for the enterprise trades on. The questions I have from the Debian break-in that I could not see were:
My biggest problem with your comments is that you are taking advantage of a person who can't speak for your public schadenfreude. I expected better of you versus cloning ESR. I expected you to write up a treatise of how a public group should handle security incidents.
Instead you seem intent on making non-Debian's regret defending Debian against a-holes who posted "oh what a bunch of dorks" when the SSL issue or the Debian 2006 breach occurred.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 0:17 UTC (Thu) by BrucePerens (subscriber, #2510) [Link] I don't know what Debian's interaction with the police was.Their take on it is public knowledge, just use google. I have seen HP handle a security issue horribly, and did my best to correct the problem while I was at HP. I can't take responsibility for how they behave today, except to note that I was really glad I didn't work there any longer the day their general counsel took the 5th in front of Congress and on national TV. Red Hat is eminently able to speak for itself. They choose not to. I have been making it clear that the Debian way was the prototype for a public group handling security incidents. And hold the ESR insults, please. Debian took its knocks standing up, and IMO that particular incident was dorky, and they owned up to it publicly and then went on with their lives. IMO that was honorable and you were quite right to defend them. What RH is doing today is very different.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 3:05 UTC (Thu) by bojan (subscriber, #14302) [Link]
> Debian is part of SPI, which is a U.S. 501(c)3 non-profit corporation.
I reckon this part alone makes it different. Red Hat is a for-profit, publicly traded company. They turn over millions every year. SPI doesn't.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 6:02 UTC (Thu) by BrucePerens (subscriber, #2510) [Link]
<blockquote>I reckon this part alone makes it different. Red Hat is a for-profit, publicly traded company. They turn over millions every year. SPI doesn't.</blockquote>
That's really the problem, isn't it? Once the money (laws concerning stockholders, management concerns about losing customers or money) get in the way, you can't trust your distribution to do the right thing any longer.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 6:30 UTC (Thu) by bojan (subscriber, #14302) [Link]
I don't think that's a problem at all. By the nature of the relationship between Fedora and Red Hat, an undetected compromise of Fedora code can affect RHEL down the line. Hence, Red Hat have the responsibility to their shareholders even when it comes to this FOSS project they sponsor.
Money talks and all that... Nothing wrong with that.
All that being said, I'll bet $5 that if the root cause of the intrusion was something that required patching upstream (and it may be completely non-applicable), Red Hat folks would already be on it. After all, they run and live on open source software - why on earth would they want something like that go unpatched for the rest of us? After all, many of us are their customers.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 15:33 UTC (Thu) by jspaleta (subscriber, #50639) [Link]
As to the $5 bet:
http://skvidal.wordpress.com/2008/09/09/fedora-security-i...
"Just to dispell this concern. Every package we (fedora infrastructure) have installed or updated on a system since the incident occurred is public and available."
-jef
How can you be sure? Posted Sep 11, 2008 16:04 UTC (Thu) by khim (subscriber, #9252) [Link] I don't think that's a problem at all. Puhlease. I'll bet $5 that if the root cause of the intrusion was something that required patching upstream (and it may be completely non-applicable), Red Hat folks would already be on it. How? "Here, please take this patch: we can't say why you should apply it and how it will work, but you SHOULD apply it" - like this? As others have said: they can't talk about things that went wrong, then don't want to talk about boundaries (what they can say or can not), etc. This will SEVERELY impede interaction with upstream. Will it make impossible to send the patch and get it accepted? Who knows. But I don't hold my breath... After all, they run and live on open source software - why on earth would they want something like that go unpatched for the rest of us? After all, many of us are their customers. The same logic applies to disclosure about current situation. They have used RHEL and Feedora systems, to handle that, right? They were setup by the best professionals available, right? And they STILL were compromised - so probably the same story can be repeated around the world again and again... Disclosure can hurt RedHat but will help customers - exactly as in situation with patch for upstream...
How can you be sure? Posted Sep 11, 2008 22:32 UTC (Thu) by bojan (subscriber, #14302) [Link]
> we can't say why you should apply it and how it will work, but you SHOULD apply it
Say, for instance, it was the kernel problem. They can submit a patch that says: "such and such was fixed, which caused privilege escalation". This does not contain the information about the actual intrusion into their systems, but is a genuine patch with a genuine explanation.
> And they STILL were compromised
It doesn't necessarily follow that a security bug was the root cause of this.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 16:04 UTC (Thu) by smoogen (subscriber, #97) [Link]
Does that mean if one gets a paycheck, one can't be trusted either? Money is going to get in the way somewhere in that case..
So how does one pay for their food/shelter/net-access? Someone has to pay for it, and you are in some way 'in-debted' to them for that access, so are potentially non-trustable to do the right thing.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 0:01 UTC (Thu) by njs (subscriber, #40338) [Link]
Talking about full disclosure here strikes me as odd, because it's a term that usually refers to vulnerabilities rather than attacks, and AFAIK there's no reason to even think they *know* anything to disclose?
In the Debian break-in, the actual vulnerability was a previously-unknown kernel bug; it was only discovered well after the break-in, and only because they found the exploit executable *and* were able to break the encryption on it. I don't know the details, but if the same thing happened again, I bet the encryption wouldn't be breakable -- there's no reason it should be, and the same tools have had years to be refined since then. Debian also made no serious effort to discover the criminal behind the break-in -- or at least if they did, they didn't disclose that.
I'm a Debian loyalist myself; but these criticisms that use hot-button phrases like "full disclosure" without explaining how they apply, and brush aside concerns about legal entanglement, just aren't very convincing to me.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 0:05 UTC (Thu) by BrucePerens (subscriber, #2510) [Link] Well, "We got breached and we can't figure out how" would be useful information, as it would be a call for you and I to start looking for the problem, along with 1000 of our best friends.Pretty much anything that keeps us from hearing that information is sub-optimal.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 0:21 UTC (Thu) by njs (subscriber, #40338) [Link]
I guess we may just have different metrics for the usefulness of information. The "we got breached" part is something we already know, and so for that matter is "our systems have unknown security holes" (I mean... duh? the only people who don't know that are living under rocks). All I can see that announcement adding would be the specific information that some particular people do not know which known-or-unknown bug was used in one particular breach, and -- while that information is sort of interesting in a soap-opera way -- I don't see how it would affect my behavior in the slightest. Maybe there's some utility there that I'm missing?
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 0:40 UTC (Thu) by BrucePerens (subscriber, #2510) [Link] Maybe there's some utility there that I'm missing?This is theoretical, we don't know yet that an undiagnosed exploit was used. But if there was an undiagnosed exploit that potentially effected my system, I'd want a start on protecting myself against it and figuring it out. Having the existing information would certainly help. It's sort of like not wanting your neighbor to try to put out a fire without calling the fire department.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 7:11 UTC (Thu) by njs (subscriber, #40338) [Link]
>This is theoretical, we don't know yet that an undiagnosed exploit was used.
True. If it turns out that RH discovered a previously unknown but used-in-the-wild exploit and then sat on it for some weeks without telling anyone, then everyone will want to lynch them, absolutely.
But I'm assuming that isn't the basis for your argument, because that would mean you were arguing for pre-lynching them *just in case* they were hiding such information, and I respect you more than that :-). It's still *entirely* possible that they don't know the exploit, or it was a known exploit, or it was a previously unknown exploit that they have quietly reported (any number of security fixes have been released since the break-in was discovered, after all). Right?
>But if there was an undiagnosed exploit that potentially effected my system, I'd want a start on protecting myself against it and figuring it out. Having the existing information would certainly help.
But that was my point -- what sort of existing information do you imagine would help? Or make it concrete: in the Debian compromise, the initial announcement basically just said "somehow they got root, we don't know how". That was the only public information available until the details on the exploit were announced ~2 weeks later. How did you act differently during those two weeks? And if you acted differently, then *why*, given that we all know that there are undiagnosed root exploits in all of our boxes?
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 7:38 UTC (Thu) by BrucePerens (subscriber, #2510) [Link] How did you act differently during those two weeks?Well, one thing that was different from this Red Hat thing was that I knew, for sure, that the Debian folks would tell me what went wrong as soon as they could :-) After the two weeks, I changed my kernel, as soon as I found out it was necessary to do that.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 12:54 UTC (Thu) by skvidal (subscriber, #3094) [Link]
From here:
http://skvidal.wordpress.com/2008/09/09/fedora-security-i...
"Something that came up at the board meeting today is that some folks are worried that their systems are not completely patched or current. That fedora infrastructure may have patches applied that we cannot tell people about.
Just to dispell this concern. Every package we (fedora infrastructure) have installed or updated on a system since the incident occurred is public and available."
Everything we have, you have.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 3:10 UTC (Thu) by bojan (subscriber, #14302) [Link]
Look where? I doubt we'd all be given access to machines and their logs just like that. I never heard of anyone using an open method of detecting breaches like this. And that doesn't even touch on privacy issues of people visiting RH/Fedora sites.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 3:34 UTC (Thu) by BrucePerens (subscriber, #2510) [Link] I wasn't expecting logs. What I was expecting was information like "the user appears to have executed a privilege escalation under kernel 2.x.x, and this is what else we found that's directly connected with that user...
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 4:06 UTC (Thu) by rahvin (subscriber, #16953) [Link]
Bruce that is exactly the information authority figures wouldn't want released as it's the information they use during interrogations to get the suspect to reveal details they can't know unless they are one that did it. Not only that, but you previously claimed that you want information on security implications that could effect RedHat/Fedora system installs. Outside what items were directly compromised the details of what the user did are nothing more than the details of the crime. Sure they make a good story but they aren't going to help you know anything about security implications to your installs.
Just admit that you want all the details including the meaty story, not just the facts that directly affect users. My sense in reading your comments is that you want RedHat to reveal all the details of the crime, the pirates of silicon valley type story describing the compromise in detail including a breakdown step by step of what the cracker did and maybe even the commands the cracker used. That's a press story, a crime novel if you will, not anything of value to system admins. As a sys-admin, not a kernel hacker I want to know what was compromised, what I need to do to stay secure and what will happen to update the system. The press release covered that. They laid out what they think happened, they told everyone not to update any package till they could update the signing keys and then they updated the signing keys and issued new updates.
The only thing I can surmise is that you believe they are engaged in a coverup and that more damage was done then they are revealing which would realistically be corporate suicide.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 5:51 UTC (Thu) by BrucePerens (subscriber, #2510) [Link] I want the facts that directly effect users, and was answering how they would be determined if there was a penetration that wasn't fully understood.At least one user I've heard from got compromised ssh packages that were distributed from a corrupt Fedora archive. He doesn't know if they were actually used to penetrate his system. He has little choice but to wipe every password, every private key, etc. He can't get any help from Fedora, because Fedora is itself being kept in the dark. This is just not the way an Open Source product should be operated. If Red Hat is going to work like Microsoft, they're no better than Microsoft. Why the heck did we work on this software for years just to have corporations that screw over their own customers and the Open Source teams that feed them?
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 7:25 UTC (Thu) by njs (subscriber, #40338) [Link]
>At least one user I've heard from got compromised ssh packages that were distributed from a corrupt Fedora archive. He doesn't know if they were actually used to penetrate his system.
Wait, what? The official word from Fedora is that there were no compromised packages whatsoever. (And from RH that there are compromised ssh packages, but that they were not distributed.) If there's information otherwise, then shouldn't he (or you) be contacting Fedora, and possibly also media outlets?
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 7:41 UTC (Thu) by BrucePerens (subscriber, #2510) [Link] Wait, what? The official word from Fedora is that there were no compromised packages whatsoever. (And from RH that there are compromised ssh packages, but that they were not distributed.)I've heard from one person who says his system failed the test script that Red Hat distributed. I don'k know him and can't attest to his reliability. I don't know if it's a false positive.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 7:52 UTC (Thu) by njs (subscriber, #40338) [Link]
Oh, so it was RH? I guess that would be another reason he couldn't get help from Fedora :-). But that's an easy mix-up to make...
If he really did fail the test script, though, then I bet he can get help: presumably the investigators would *love* to figure out how that package got onto his system, since whoever put it there must have contact with the RH attackers... OTOH: random emailer of unknown provenance, making an unverifiable claim that would make news if real but hasn't, and whining about how they can't get help when they would have if they tried? Obviously I don't know him either, but kind of tingles my internet-attention-seeker sense.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 8:02 UTC (Thu) by mspevack (subscriber, #36977) [Link] I don'k [sic] know him and can't attest to his reliability.Then why are you spreading FUD? -1 (Troll)
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 12:58 UTC (Thu) by skvidal (subscriber, #3094) [Link]
If you'd like to pass along his name/email, it'd be handy to know where he obtained this package from.
It wasn't from what is on the master mirror, I can assure you of that.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 22:50 UTC (Thu) by wingo (subscriber, #26929) [Link]
Bruce. That's twice. *Affect.*
Respectfully, Andy.
The Fedora-Red Hat Crisis (Datamation) Posted Sep 11, 2008 12:16 UTC (Thu) by nim-nim (subscriber, #34454) [Link]
If Red Hat/Fedora's problem was linked to a bug or a problem in some code they've been distributing (for which fixed packages were not already available), sure they need to do full disclosure on it.
OTOH if the intrusion was linked to social engineering, system misconfiguration, publicly available updates not applied yet, or internal code/scripts/glue, sorry but it is none of the general's public business. Red Hat doesn't owe anyone external an explication on how they were caught with their pants down, as entertaining as it might be, as long as they're not pushing faulty pants to someone else (and fixed their setup).
That's the difference between "Ford sells A engines that go boom under B conditions" and "through B screwups the maintenance of C equipment was not done properly in the D Ford factory producing A engines, and it had to be shut down for emergency repairs".
So far all the messages we've seen point to the second scenario.
Debian's problems were quite different. It had widely distributed broken software, and had compromised the security of numerous third-party systems (the second part in true both for the SSL screwup and the server breach). So far there's no evidence the Red Hat incident has affected anything but Red Hat internal systems, and I don't doubt many people have checked this independantly in the past month.
you are thinking of the wrong debian problem Posted Sep 11, 2008 20:35 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
debian had a server compromise a year or so ago (related to a developers keys being lost on another machine) and handled it in a much more open manner.
you are thinking of the wrong debian problem Posted Sep 12, 2008 1:16 UTC (Fri) by nim-nim (subscriber, #34454) [Link]
This compromise led to keys to other systems being compromised. They *had* to handle it in a more open manner.
|
|
||||||||||||