On August 22, the Fedora Project released an "
confirming what most observers had, by then, suspected: the project had
suffered a major security breach. The attacker got as far as a system used
to sign packages distributed by Fedora. That, of course, is something
close to a worst-case scenario: if an intruder has control over such a
system, it's a relatively small step to capture the package signing key and
the passphrase used to employ that key. And those, in turn, could be used
to create hostile packages which would be accepted as genuine by Fedora
Fortunately for the Fedora Project (and its users), an audit has determined
that nobody made use of the key while the intruder was present. So, even
if some means for capturing something as transient as the passphrase were
in place, that passphrase was not exposed, and, thus, cannot have been
Needless to say, the project is changing its package signing key anyway.
Interestingly, the Fedora Project was not the only target in this attack: Red Hat,
too, was compromised. Unlike Fedora, Red Hat did not issue a statement
specifically about this intrusion; instead, the information was included in
security update. In this case, the attacker was more successful, to
the point of being able to sign "...a small number of OpenSSH
packages relating only to Red Hat Enterprise Linux 4 (i386 and x86_64
architectures only) and Red Hat Enterprise Linux 5 (x86_64 architecture
only)." This language deserves to be questioned: it is only
necessary to sign a single openssh package (certainly qualifying as a
"small number") to compromise thousands of RHEL hosts, and the "only" terms
describe what must be a large majority of deployed RHEL systems. Seriously
scary, but Red Hat has been able to convince itself that none of the
compromised packages were fed out to RHEL subscribers. So this attack,
too, failed - but not by much.
Needless to say, disclosures like this raise more questions than they
answer. The one question that Fedora and Red Hat will have to answer at
some point is this: how did the intruder get in? One assumes that Fedora
and Red Hat are running their own distributions on their internal systems;
it thus stands to reason
that, if those distributions contained a vulnerability that allowed the
attacker to get in, many other systems will also be vulnerable. If,
instead, this compromise is the result of administrator or developer error
(or, say, of a lost laptop), administrators responsible for Fedora and RHEL
systems can breathe a little easier. Either way, they deserve to know how
this series of events came to pass.
Some people would like to have that information immediately. Beyond that,
there has, predictably, been a fair amount of grumbling (also
predictably, from a relatively small number of people) on the Fedora lists
on how this incident was handled. Your editor, too, has argued that some information
took too long to emerge. He will now argue that, while Fedora still has
more to disclose, the project has said enough to give itself some breathing
room while it struggles to put its infrastructure back together and figure
out what really happened. There's all kinds of good reasons why more
information may not be immediately forthcoming, including the obvious
possibility that nobody really knows yet how the intruder gained access. There is
little to be gained from hammering on Fedora at this point at this time.
That said, anything the project can say to tell its users whether they
should be worried about an undisclosed vulnerability in their systems would
be most welcome, and sooner would be better than later.
Meanwhile, what can be done, and what Fedora board member Jeff
Spaleta, in particular, has been
pushing for, is to think about how things should be handled the next time.
Did we have a communication problem? Maybe. But communication
problems are not equivalent to trust issues. But considering that
was a first of its kind event for us as a project, I don't think
its necessarily unexpected to see some miscommunication. I don't
think any of us, either inside Red Hat or outside had talked
through how this sort of thing should be handled. I don't remember
a serious public discussion about how to deal with communication of
an event like this before having an event like this. And I'm not
going to let the assumption stand that to do things differently
should have been obvious to those in a position to deal with the
If people want things to be better, if god forbid something like
this happens again, then a serious effort to write a communication
process has to be written up and it must be agreeable to legal as a
workable process that won't set off any legal liability landmines.
The Fedora Project should certainly write up a policy for situations like
this. It would be good for Fedora's users, but it could have an effect far
beyond that: such a policy could serve as an example for other distributors
as well. It is, after all, probably safe to say that Fedora is not the
only distributor which has, thus far, neglected to put plans in place for
this sort of disaster.
We all need such plans. For better or for worse, distributors have come to
occupy an important position with regard to the security of much of the
net. Millions of systems run packages signed by Linux distributors; they
depend, implicitly, on the security of the process used to create those
packages. That process is not a small one; it can involve hundreds of
developers, before even counting all of those involved in upstream
development projects. The consequences of a failure anywhere in that chain of trust
can be severe. It is not surprising that the distribution system was
attacked; perhaps the only real surprise is that it has not happened more
often - that we know of. These attacks will happen again; distributors
need to have a firm idea of how they will respond.
A related subject is worth a quick mention: as of this writing, the Fedora
Project has issued no security updates since August 12, almost a full
two weeks. A number of significant vulnerabilities, including the postfix symbolic link
vulnerability, remain unpatched for Fedora users. Red Hat has done
better, but not by much. Linux users depend heavily on the distributor
security update process, and, for these two distributions, that process has
been severely disrupted. If there had been a truly serious vulnerability
disclosed during this time, people charged with keeping Fedora and RHEL
systems secure might have found themselves in a difficult position. One
need not be overly paranoid to envision this type of disruption being done
intentionally as part of a zero-day attack on the net.
This incident should serve as a sort of wake-up call for both distributors
and their users. Distributors wanting to retain their users' trust should
be thinking about documenting things like:
- How the packaging chain is kept secure. It would be good to know
how many people are able to sign packages, and how they gain access to
the systems where this signing is done.
- What sort of plans the distributor has in place for dealing with
security problems. One can only assume that Red Hat dedicated (and
continues to dedicate) a large amount of staff time to understanding
and recovering from this incident. Would other distributors be willing
and able to do the same?
- What are the plans for dealing with a major security breach? How
might a critical security update be propagated during a time when the
integrity of the packaging system has been compromised?
- Should something go wrong, when and how will information be
communicated to the wider community?
Conversely, anybody who is deploying an important Linux-based system should
be asking such questions when choosing a distribution for that system. If
the system requires high-assurance guarantees in this regard, it probably
makes sense to look at the vendors who are willing to provide such
guarantees for a fee. But, again, the lesson we have learned from recent
events is that the time to ask these questions is now, and not when
something has gone wrong and people are running around in circles.
As a whole, Linux users have been very well served by distributors since
the very beginning. The distributors pull together thousands of software
releases and integrate them into a coherent whole; they then make the
results available, often for free. They provide fixes when things break,
and most of them pay particular attention to fixing security-related
problems. And they have done a top-quality job of not being used as a
conduit for hostile software. It's a great system, and that has not
changed. But we have learned something about how heavily we depend on that
system, and how it can fail. Proper application of the lessons from this
episode should help us all to be more secure in the future.
to post comments)