Another update on Fedora infrastructure
| From: | "Paul W. Frields" <stickster-AT-gmail.com> | |
| To: | fedora-announce-list <fedora-announce-list-AT-redhat.com> | |
| Subject: | Infrastructure status, 2008-08-19 UTC 0200 | |
| Date: | Tue, 19 Aug 2008 02:07:45 +0000 | |
| Message-ID: | <1219111665.25510.175.camel@victoria> |
Our team has been hard at work for several days now, restoring services in the Fedora infrastructure. We started with what we identified as Fedora's "critical path," those systems required to restore minimum daily operation. That work to be completely finished by the end of the day. We then move on to our other value services to complete them as soon as possible. Please give the infrastructure team the time they need to do this demanding work. They have been doing a spectacular job and deserve the absolute highest credit. The systems that are now back online and usable include the following: * Puppet, Xen and FAS hosts * app1, app3, and app4 * database and proxy servers * the majority of the Xen guest machines * serverbeach5, serverbeach4 * Fedora Hosted** The systems that should be available very soon: * asterisk1 and collab1 * cvs1 * builders, x86 and ppc * Fedora People We know the community is awaiting more detail on the past week's activities and their causes. We're preparing a timeline and details and will make them available in the near future. We appreciate the community's patience, and will continue to post updates to the fedora-announce-list as soon as possible. = = = ** New SSH fingerprint for Fedora Hosted: e6:b3:68:51:98:2d:4c:dc:63:27:46:65:51:d5:f0:7a -- Paul W. Frields, Fedora Project Leader gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://paul.frields.org/ - - http://pfrields.fedorapeople.org/ irc.freenode.net: stickster @ #fedora-docs, #fedora-devel, #fredlug -- fedora-announce-list mailing list fedora-announce-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-announce-list
(Log in to post comments)
Another update on Fedora infrastructure
Posted Aug 19, 2008 9:01 UTC (Tue) by rvfh (guest, #31018) [Link]
Maybe they had a Debian-generated SSH key?
Another update on Fedora infrastructure
Posted Aug 19, 2008 15:17 UTC (Tue) by jwb (guest, #15467) [Link]
A new SSH key fingerprint could just mean that they reinstalled the operating system, or got a new machine and installed fresh on that.
Another update on Fedora infrastructure
Posted Aug 19, 2008 18:58 UTC (Tue) by Los__D (guest, #15263) [Link]
But... Why not move the key to the new system?
Another update on Fedora infrastructure
Posted Aug 19, 2008 17:00 UTC (Tue) by corbet (editor, #1) [Link]
Worthy of note for people speculating in this direction: in the whole password/key reset process that is going on now, they are prohibiting the uploading of DSA keys into Fedora servers.
Another update on Fedora infrastructure
Posted Aug 19, 2008 17:07 UTC (Tue) by jwb (guest, #15467) [Link]
Why would anyone use DSA keys anyway? They have serious flaws which the RSA system avoids. Notably, if you inadvertently sign something using your DSA key and a compromised PRNG, your key is revealed. The attraction of DSA keys seems to be simply that RSA was at one time patented. This seems like a silly reason today.
Another update on Fedora infrastructure
Posted Aug 19, 2008 17:27 UTC (Tue) by tialaramex (subscriber, #21167) [Link]
There's a diversity argument too. If everyone's infrastructure relies on RSA exclusively and then next week someone finds a serious problem in RSA then you've got a massive disaster. Which doesn't add up to an argument for DSA, but it does mean it's not enough to say "RSA is better, we'll just use that". We know that RSA is no /harder/ than the factorisation problem, but we don't have a proof that it isn't /easier/ perhaps /much easier/. We must have alternatives, maybe Elliptic Curve or maybe something quite different.
Another update on Fedora infrastructure
Posted Aug 19, 2008 21:32 UTC (Tue) by danpb (subscriber, #4831) [Link]
The prohibition on uploading new DSA keys is not a new change this week, but was actually introduced to FAS2 a couple of months ago now as per this ticket: https://fedorahosted.org/fedora-infrastructure/ticket/540
Another update on Fedora infrastructure
Posted Aug 19, 2008 13:31 UTC (Tue) by sbergman27 (guest, #10767) [Link]
If and when they ever deign to let their users know what the problem is/was, it's going to be huge. No one would take everything down at once like this, and take a credibility hit of this magnitude unless they were fairly sure that the attacker had his tendrils intertwined in their infrastructure very deeply, indeed. Yeah, I know that the true fans will pat them on the back and say it was great that they responded so quickly. (Standard fall-back procedure.) Others will, understandably, wonder how such a thing was ever allowed to happen at all. Whatever is revealed, it's sure to be riveting.
Another update on Fedora infrastructure
Posted Aug 19, 2008 14:30 UTC (Tue) by kragil (guest, #34373) [Link]
Are the servers running fedora ?? I guess they are running RHEL .. that makes the exploit even scarier. ( OK .. might be wild speculation on my part ;) ) While I am speculating .. :) I also guess all the investment into SELinux did not help. Maybe AppAmor and/or Smack are better suited for human beings.
Gracious!
Posted Aug 19, 2008 14:47 UTC (Tue) by dwheeler (guest, #1216) [Link]
Again, let's give them credit; whatever the problem is, they are clearly taking it really seriously, as evidenced by the extraordinary steps they're taking. And we don't know if AppArmor (etc.) would have done any better. Generally Fedora/Red Hat work especially hard on preventing attack (see their SELinux work, stack protection, etc.); I think it's silly to imply that they don't take security seriously.Here's hoping that it's some sort of serious compiler bug, instead of an attack. However, while I don't have any inside information, I wouldn't bet on that. The sheer secrecy of details suggests a serious attack.
Gracious!
Posted Aug 19, 2008 15:10 UTC (Tue) by kragil (guest, #34373) [Link]
I'm sorry. Speculating is fun though. Maybe they introduced a trojan into the build sources for an update and Linus installed that update and then they changed the kernel source on Linus' box ( because he is running fedora, isn`t he. He shouldn`t say something like that by the way, makes him _obviously_ more vulnerable ;P ) and from now on all kernels will be exploitable by default... B) But seriously: I'm just kidding .. like I said: Speculating is fun.
Gracious!
Posted Aug 19, 2008 16:23 UTC (Tue) by jengelh (subscriber, #33263) [Link]
All this secrecy gets tiring. First the huge bubble about DNS, now Fedora. What I kinda miss in all of this mess is an informant leaking details ;-)
Gracious!
Posted Aug 19, 2008 16:47 UTC (Tue) by mmcgrath (subscriber, #44906) [Link]
Fedora said it: "We know the community is awaiting more detail on the past week's activities and their causes. We're preparing a timeline and details and will make them available in the near future. We appreciate the community's patience, and will continue to post updates to the fedora-announce-list as soon as possible." Paul has told us all that he's going to make this known in the near future. I know I'll be holding Fedora to that :)
Another update on Fedora infrastructure
Posted Aug 19, 2008 16:40 UTC (Tue) by dowdle (subscriber, #659) [Link]
As has been pointed out in a comment to a previous mention of an update notice... the most common thing it could be would be a compromised account of someone who had high administrative rights within their infrastructure. In security the weakest link is usually human and SELinux, AppArmor, etc can't defend against that... nor physical access. Of course I have no idea what the problem is/was. I think this is a testament to Fedora's ability to keep a tight lid on an issue... and keep it from leaking before they are ready to make an announcement. Great job guys!
Another update on Fedora infrastructure
Posted Aug 19, 2008 16:51 UTC (Tue) by Sutoka (guest, #43890) [Link]
This is what I was thinking. It wasn't that long ago that something similar happened in the Debian project, though that was 'only' a developer (and not an administrator IIRC). If one of the Fedora admins, say, had privileged login for several of the main fedora project servers saved on their laptop and then their laptop got stolen, the project may be taking all this as a precautionary attempt to change all their login/keys/etc and they're keeping quiet because they're hoping they can get everything before the person realizes what they got. Going further with this hypothetical, it's possible there were several days before the laptop being stolen and the Fedora projecting finding out so they may simply be worried that any would-be attackers were able to take any credentials on the laptop and spread to other parts of the system as well, and they're using this as an opportunity to do a *complete* audit/reinstalls/upgrades/etc. Then again, this is all speculation so I may be completely off.
Another update on Fedora infrastructure
Posted Aug 20, 2008 1:12 UTC (Wed) by qg6te2 (guest, #52587) [Link]
Yeah, I know that the true fans will pat them on the back and say it was great that they responded so quickly. (Standard fall-back procedure.) Others will, understandably, wonder how such a thing was ever allowed to happen at all.Assuming this is indeed a security problem, the real question to ask would be: if it happened to Fedora, is the infrastructure of Ubuntu, Debian or Suse also vulnerable? Even in the case that this "incident" was something highly specific to Fedora, we as a community can still learn from Fedora's experience.
Another update on Fedora infrastructure
Posted Aug 20, 2008 3:10 UTC (Wed) by sbergman27 (guest, #10767) [Link]
""" if it happened to Fedora, is the infrastructure of Ubuntu, Debian or Suse also vulnerable? """ We cannot know at this point because Fedora ain't talkin'. And usually Red Hat Legal must be consulted and hand down an opinion before they do.
Another update on Fedora infrastructure
Posted Aug 20, 2008 6:14 UTC (Wed) by jd (guest, #26381) [Link]
Standard tactics is to assume the worst and hope for the best. By that, I mean assume that all distros have a vulnerability that may permit root access to an outside user via a service likely to be run on the machine with the key change, but at the same time, don't panic and shut everything down. Use common sense. In this case, if you are running a mission-critical server that is exposed directly to the Internet (rather than via a proxy in a DMZ), double-check you have applied all relevant security updates, ensure unnecessary services are disabled (or run in a honeypot), do a quick check of your security logs for abnormal login failures, and run some auditing tools like SARA, TARA and Nessus. Perhaps get round to installing Tripwire as well. The less critical the server (either in and of itself, or what someone could do if they compromised it), the more of these you can skip and not look like a fool. Likewise, the more shielded it is from a direct attack, the more you should focus on the machines that are at real risk. The chances are good that it's not a genuine risk to other systems, that it's a lost/stolen key, some idiot blogged their password, or even that an admin found a keylogger on their machine that may have predated the last time they ssh'ed in. There are all kinds of "trivial" reasons for a deep clean that won't affect others. For that reason, getting anxious or in a panic won't help. However, there is always the possibility of a real flaw, so take measures that are appropriate to the systems you run. Beyond that, there is nothing you can do - other than cut the network cable or launch tac nukes at the power socket.
Putting things in perspective
Posted Aug 20, 2008 6:43 UTC (Wed) by pr1268 (subscriber, #24648) [Link]
As much speculation we're doing here (and presumably elsewhere on Fedora related forums), it likely doesn't compare to the speculation and rumor milling taking place over the iPhone troubles or the Mobile Me troubles.
Of course, when dealing with proprietary, closed-source hardware and software that malfunctions, all the vendor can do is to start handing out service credits and other freebies to keep customers happy. One article mentioned something similar to the idea that Apple can't disclose squat on the nature of the service troubles since that might open up a flood of class-action lawsuits.
Perhaps I'm comparing apples to oranges here (no pun intended), but certainly developers' lives in Fedoraville aren't as bad right now as those in Cupertino.
Package signing keys
Posted Aug 31, 2008 9:22 UTC (Sun) by kasperd (guest, #11842) [Link]
