|
|
Subscribe / Log in / New account

Another update on Fedora infrastructure

From:  "Paul W. Frields" <stickster-AT-gmail.com>
To:  fedora-announce-list <fedora-announce-list-AT-redhat.com>
Subject:  Infrastructure status, 2008-08-19 UTC 0200
Date:  Tue, 19 Aug 2008 02:07:45 +0000
Message-ID:  <1219111665.25510.175.camel@victoria>

Our team has been hard at work for several days now, restoring services
in the Fedora infrastructure. We started with what we identified as
Fedora's "critical path," those systems required to restore minimum
daily operation.  That work to be completely finished by the end of the
day. We then move on to our other value services to complete them as
soon as possible.

Please give the infrastructure team the time they need to do this
demanding work. They have been doing a spectacular job and deserve the
absolute highest credit.

The systems that are now back online and usable include the following:
* Puppet, Xen and FAS hosts
* app1, app3, and app4
* database and proxy servers
* the majority of the Xen guest machines
* serverbeach5, serverbeach4
* Fedora Hosted**

The systems that should be available very soon:
* asterisk1 and collab1
* cvs1
* builders, x86 and ppc
* Fedora People

We know the community is awaiting more detail on the past week's
activities and their causes.  We're preparing a timeline and details and
will make them available in the near future.  We appreciate the
community's patience, and will continue to post updates to the
fedora-announce-list as soon as possible.

= = =
** New SSH fingerprint for Fedora Hosted:
   e6:b3:68:51:98:2d:4c:dc:63:27:46:65:51:d5:f0:7a

-- 
Paul W. Frields, Fedora Project Leader
  gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233  5906 ACDB C937 BD11 3717
  http://paul.frields.org/   -  -   http://pfrields.fedorapeople.org/
  irc.freenode.net: stickster @ #fedora-docs, #fedora-devel, #fredlug

-- 
fedora-announce-list mailing list
fedora-announce-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-announce-list



(Log in to post comments)

Another update on Fedora infrastructure

Posted Aug 19, 2008 9:01 UTC (Tue) by rvfh (guest, #31018) [Link]

Maybe they had a Debian-generated SSH key?

Another update on Fedora infrastructure

Posted Aug 19, 2008 15:17 UTC (Tue) by jwb (guest, #15467) [Link]

A new SSH key fingerprint could just mean that they reinstalled the operating system, or got a
new machine and installed fresh on that.

Another update on Fedora infrastructure

Posted Aug 19, 2008 18:58 UTC (Tue) by Los__D (guest, #15263) [Link]

But... Why not move the key to the new system?

Another update on Fedora infrastructure

Posted Aug 19, 2008 17:00 UTC (Tue) by corbet (editor, #1) [Link]

Worthy of note for people speculating in this direction: in the whole password/key reset process that is going on now, they are prohibiting the uploading of DSA keys into Fedora servers.

Another update on Fedora infrastructure

Posted Aug 19, 2008 17:07 UTC (Tue) by jwb (guest, #15467) [Link]

Why would anyone use DSA keys anyway?  They have serious flaws which the RSA system avoids.
Notably, if you inadvertently sign something using your DSA key and a compromised PRNG, your
key is revealed.

The attraction of DSA keys seems to be simply that RSA was at one time patented.  This seems
like a silly reason today.

Another update on Fedora infrastructure

Posted Aug 19, 2008 17:27 UTC (Tue) by tialaramex (subscriber, #21167) [Link]

There's a diversity argument too. If everyone's infrastructure relies on RSA exclusively and
then next week someone finds a serious problem in RSA then you've got a massive disaster.

Which doesn't add up to an argument for DSA, but it does mean it's not enough to say "RSA is
better, we'll just use that". We know that RSA is no /harder/ than the factorisation problem,
but we don't have a proof that it isn't /easier/ perhaps /much easier/. We must have
alternatives, maybe Elliptic Curve or maybe something quite different.

Another update on Fedora infrastructure

Posted Aug 19, 2008 21:32 UTC (Tue) by danpb (subscriber, #4831) [Link]

The prohibition on uploading new DSA keys is not a new change this week, but was actually
introduced to FAS2 a couple of months ago now as per this ticket: 

https://fedorahosted.org/fedora-infrastructure/ticket/540

Another update on Fedora infrastructure

Posted Aug 19, 2008 13:31 UTC (Tue) by sbergman27 (guest, #10767) [Link]

If and when they ever deign to let their users know what the problem is/was, it's going to be
huge.  No one would take everything down at once like this, and take a credibility hit of this
magnitude unless they were fairly sure that the attacker had his tendrils intertwined in their
infrastructure very deeply, indeed.  Yeah, I know that the true fans will pat them on the back
and say it was great that they responded so quickly. (Standard fall-back procedure.)  Others
will, understandably, wonder how such a thing was ever allowed to happen at all.

Whatever is revealed, it's sure to be riveting.

Another update on Fedora infrastructure

Posted Aug 19, 2008 14:30 UTC (Tue) by kragil (guest, #34373) [Link]

Are the servers running fedora ??

I guess they are running RHEL .. that makes the exploit even scarier. ( OK .. might be wild
speculation on my part ;) )

While I am speculating .. :)

I also guess all the investment into SELinux did not help. Maybe AppAmor and/or Smack are
better suited for human beings.

Gracious!

Posted Aug 19, 2008 14:47 UTC (Tue) by dwheeler (guest, #1216) [Link]

Again, let's give them credit; whatever the problem is, they are clearly taking it really seriously, as evidenced by the extraordinary steps they're taking. And we don't know if AppArmor (etc.) would have done any better. Generally Fedora/Red Hat work especially hard on preventing attack (see their SELinux work, stack protection, etc.); I think it's silly to imply that they don't take security seriously.

Here's hoping that it's some sort of serious compiler bug, instead of an attack. However, while I don't have any inside information, I wouldn't bet on that. The sheer secrecy of details suggests a serious attack.

Gracious!

Posted Aug 19, 2008 15:10 UTC (Tue) by kragil (guest, #34373) [Link]

I'm sorry. Speculating is fun though.

Maybe they introduced a trojan into the build sources for an update and Linus installed that
update and then they changed the kernel source on Linus' box ( because he is running fedora,
isn`t he. He shouldn`t say something like that by the way, makes him _obviously_ more
vulnerable ;P ) and from now on all kernels will be exploitable by default... B)

But seriously: I'm just kidding .. like I said: Speculating is fun.

Gracious!

Posted Aug 19, 2008 16:23 UTC (Tue) by jengelh (subscriber, #33263) [Link]

All this secrecy gets tiring. First the huge bubble about DNS, now Fedora. What I kinda miss
in all of this mess is an informant leaking details ;-)

Gracious!

Posted Aug 19, 2008 16:47 UTC (Tue) by mmcgrath (subscriber, #44906) [Link]

Fedora said it:

"We know the community is awaiting more detail on the past week's
activities and their causes.  We're preparing a timeline and details and
will make them available in the near future.  We appreciate the
community's patience, and will continue to post updates to the
fedora-announce-list as soon as possible."

Paul has told us all that he's going to make this known in the near future.  I know I'll be
holding Fedora to that :)

Another update on Fedora infrastructure

Posted Aug 19, 2008 16:40 UTC (Tue) by dowdle (subscriber, #659) [Link]

As has been pointed out in a comment to a previous mention of an update notice... the most
common thing it could be would be a compromised account of someone who had high administrative
rights within their infrastructure.  In security the weakest link is usually human and
SELinux, AppArmor, etc can't defend against that... nor physical access.

Of course I have no idea what the problem is/was.

I think this is a testament to Fedora's ability to keep a tight lid on an issue... and keep it
from leaking before they are ready to make an announcement.  Great job guys!

Another update on Fedora infrastructure

Posted Aug 19, 2008 16:51 UTC (Tue) by Sutoka (guest, #43890) [Link]

This is what I was thinking. It wasn't that long ago that something similar happened in the
Debian project, though that was 'only' a developer (and not an administrator IIRC). If one of
the Fedora admins, say, had privileged login for several of the main fedora project servers
saved on their laptop and then their laptop got stolen, the project may be taking all this as
a precautionary attempt to change all their login/keys/etc and they're keeping quiet because
they're hoping they can get everything before the person realizes what they got.

Going further with this hypothetical, it's possible there were several days before the laptop
being stolen and the Fedora projecting finding out so they may simply be worried that any
would-be attackers were able to take any credentials on the laptop and spread to other parts
of the system as well, and they're using this as an opportunity to do a *complete*
audit/reinstalls/upgrades/etc.

Then again, this is all speculation so I may be completely off.

Another update on Fedora infrastructure

Posted Aug 20, 2008 1:12 UTC (Wed) by qg6te2 (guest, #52587) [Link]

Yeah, I know that the true fans will pat them on the back and say it was great that they responded so quickly. (Standard fall-back procedure.) Others will, understandably, wonder how such a thing was ever allowed to happen at all.

Assuming this is indeed a security problem, the real question to ask would be: if it happened to Fedora, is the infrastructure of Ubuntu, Debian or Suse also vulnerable? Even in the case that this "incident" was something highly specific to Fedora, we as a community can still learn from Fedora's experience.

Another update on Fedora infrastructure

Posted Aug 20, 2008 3:10 UTC (Wed) by sbergman27 (guest, #10767) [Link]

"""
if it happened to Fedora, is the infrastructure of Ubuntu, Debian or Suse also vulnerable?
"""

We cannot know at this point because Fedora ain't talkin'.  And usually Red Hat Legal must be
consulted and hand down an opinion before they do.

Another update on Fedora infrastructure

Posted Aug 20, 2008 6:14 UTC (Wed) by jd (guest, #26381) [Link]

Standard tactics is to assume the worst and hope for the best. By that, I mean assume that all
distros have a vulnerability that may permit root access to an outside user via a service
likely to be run on the machine with the key change, but at the same time, don't panic and
shut everything down. Use common sense.

In this case, if you are running a mission-critical server that is exposed directly to the
Internet (rather than via a proxy in a DMZ), double-check you have applied all relevant
security updates, ensure unnecessary services are disabled (or run in a honeypot), do a quick
check of your security logs for abnormal login failures, and run some auditing tools like
SARA, TARA and Nessus. Perhaps get round to installing Tripwire as well.

The less critical the server (either in and of itself, or what someone could do if they
compromised it), the more of these you can skip and not look like a fool. Likewise, the more
shielded it is from a direct attack, the more you should focus on the machines that are at
real risk.

The chances are good that it's not a genuine risk to other systems, that it's a lost/stolen
key, some idiot blogged their password, or even that an admin found a keylogger on their
machine that may have predated the last time they ssh'ed in. There are all kinds of "trivial"
reasons for a deep clean that won't affect others. For that reason, getting anxious or in a
panic won't help. However, there is always the possibility of a real flaw, so take measures
that are appropriate to the systems you run.

Beyond that, there is nothing you can do - other than cut the network cable or launch tac
nukes at the power socket.

Putting things in perspective

Posted Aug 20, 2008 6:43 UTC (Wed) by pr1268 (subscriber, #24648) [Link]

As much speculation we're doing here (and presumably elsewhere on Fedora related forums), it likely doesn't compare to the speculation and rumor milling taking place over the iPhone troubles or the Mobile Me troubles.

Of course, when dealing with proprietary, closed-source hardware and software that malfunctions, all the vendor can do is to start handing out service credits and other freebies to keep customers happy. One article mentioned something similar to the idea that Apple can't disclose squat on the nature of the service troubles since that might open up a flood of class-action lawsuits.

Perhaps I'm comparing apples to oranges here (no pun intended), but certainly developers' lives in Fedoraville aren't as bad right now as those in Cupertino.

Package signing keys

Posted Aug 31, 2008 9:22 UTC (Sun) by kasperd (guest, #11842) [Link]

What's the status of the package signing keys? I saw announcements indicating that it would be replaced as well, but I haven't seen new public keys released anywhere? (What's even the procedure for replacing package signing keys? It isn't obvious to me how to get them to client machines and get the old ones removed).


Copyright © 2008, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds