One week of infrastructure issues [LWN.net]

By Jonathan Corbet
August 20, 2008

On August 14, Fedora leader Paul Frields sent out a terse announcement regarding "an issue in the infrastructure systems" supporting the project. This "issue" could lead to some service outages, for which he apologized. Also included in the note was this ominous warning:

We're still assessing the end-user impact of the situation, but as a precaution, we recommend you not download or update any additional packages on your Fedora systems.

As this article is written (August 20, just barely in time for the LWN weekly publication deadline), there have been a couple of uninformative updates, but the situation persists and nobody seems to know what is really going on. The Fedora team, it would seem, is quite good at keeping secrets when the need arises. As a result, Fedora users worldwide have spent almost a full week wondering what has happened and whether they need to be worried about it.

In such a situation, there is a delightful amount of space for wild speculation. Your editor does not usually start his drinking binge until after publication, but, for the purposes of interpreting the following, one should assume that it was already well underway. This "issue" could be explained by any of the following:

Maybe a Fedora developer - on a drinking binge of his own, perhaps - tripped over a power cord. The resulting mess not only deprived an important server of power, but said developer, on his way toward the floor, managed to take the entire rack down with him. Ever since, the infrastructure team has been trying to reassemble a set of working systems from the rubble.
Last month, Fedora slipped a small patch into gcc designed to ensure that the results from the most recent board election - where one slot went to a candidate who was not a Red Hat employee - would never be repeated. But the patch was botched, and most mathematical operations in gcc-compiled programs have been returning random numbers ever since. Now the Fedora team is trying to quietly replace the broken binaries before anybody notices.
It turns out that the rights to the Fedora name had never actually been secured, and the real owner got an injunction shutting the project down. As soon as all the branding has been changed, Fedora will be reborn as Leopard-Skin Pillbox Hat Linux. Just wait until you see the new desktop themes.
The package signing key has been compromised, as have the build servers. For the last six months, every version of Firefox shipped by Fedora has reported account names, passwords, and credit card numbers to a server located on a ship in international waters near Colombia. The openssh client has been similarly modified. The Fedora team has been slow to get an explanation out because it takes time to relocate your home and family to an undisclosed location on a different continent.
A vulnerability in RPM has enabled the creation of a large ecosystem of hostile mirrors operated by competing criminal groups. Most Fedora users have been installing compromised updates for the last year or so.
No less than three Fedora system administrators turned out to be the type of people who will give out their password for a bar of chocolate. The provider of sweets really only wanted to fix the longstanding claws-mail dependency problems in Rawhide, but the project hit the panic button anyway.
The Fedora team simply wanted to take a vacation in an undisclosed location on a different continent and didn't want to deal with a bunch of email on their return.

The real point of this being, of course, that none of us know what is going on, creating a situation described by Alan Cox as "leaving people in the dark assuming the worst - a very bad way to create long term trust." Distributors occupy a crucial part of our ecosystem; they absolutely need to have the trust of their users. There is just too much that can go wrong at that level.

One can only assume that something fairly serious has happened. By all accounts, the Fedora team has been working flat-out to get things resolved as quickly as possible; they seem to be doing an exceptional job under a great deal of pressure. They have undoubtedly earned a big round of thanks - and lots of beers - from the Fedora community as a whole.

But Fedora's leadership appears to have failed here. If Fedora users need to be concerned about the software running on their systems, they should have been told by now. If they can relax and stop worrying, they should have been told that as well. Instead, the Fedora user community has been left wondering for nearly a week while the infrastructure they count on is torn down and rebuilt from the beginning. Given that, Fedora users have shown a tremendous amount of patience and restraint; the user community clearly has a high degree of confidence in the project in general, and has been willing to wait until the project is ready to come clean.

To retain that confidence, the Fedora project will have to tell the full story in a clear manner - and sooner would certainly be better. A good explanation of why Fedora users were made to wait so long before hearing anything about how this "infrastructure issue" affects them will also be needed. Fedora users are concerned about what has happened so far, but their real response will be determined by what Fedora does next.

to post comments

One week of infrastructure issues

Posted Aug 21, 2008 0:23 UTC (Thu) by ofeeley (guest, #36105) [Link] (5 responses)

[...]none of us know what is going on [...]But Fedora's leadership appears to have failed here.

Although we've been "waiting and not seeing" for a week now it's still impossible to make such a judgement.

If Fedora users need to be concerned about the software running on their systems, they should have been told by now. If they can relax and stop worrying, they should have been told that as well.

Supposing it's a difficult to fix vulnerability affecting many parts of the Fedora infrastructure and although it's not believed that any packages were compromised it's still a small possibility and in order to quantify that possibility the exact details need to be shared which could lead to further attacks? All speculation is premature at this stage and certainly strong conclusions about failure of leadership is really pre-judging things.

One week of infrastructure issues

Posted Aug 21, 2008 1:21 UTC (Thu) by sbergman27 (guest, #10767) [Link] (1 responses)

While I cannot know exactly what the problem is, it seems almost certainly to be serious and
security related.  It think it's a pretty fair guess that we are all actually waiting on Red
Hat Legal to make a decision regarding what should be made known, when it should be made
known, and how it should be phrased.  I'm very pro Red Hat.  But realistically, this could
turn out to be a *very* serious issue for them.  It's not their commercial product. But Red
Hat is ultimately liable for Fedora.

One week of infrastructure issues

Posted Aug 21, 2008 20:08 UTC (Thu) by smoogen (subscriber, #97) [Link]

Or its dealing with law enforcement on something. That is usually the biggest gag issue.

One week of infrastructure issues

Posted Aug 21, 2008 9:06 UTC (Thu) by liljencrantz (guest, #28458) [Link] (2 responses)

Not impossible at all, in my opinion. In what way would Fedora security be significantly
compromised by a short, non-specific statement explaining whether there is a chance that
packages that have been distributed to users have been compromised or not. Thats mainly what
people want to know. 

In fact, if there is a chance that packages that have been distributed to users have been
altered by a third party, it is extremely troubling that Fedora has not gone public with this
information.

One week of infrastructure issues

Posted Aug 21, 2008 11:18 UTC (Thu) by motk (subscriber, #51120) [Link] (1 responses)

... then it's probably not an issue then, or else they would have done so.

One week of infrastructure issues

Posted Aug 22, 2008 9:27 UTC (Fri) by liljencrantz (guest, #28458) [Link]

If that is the case, why have they not simply gone out and said that?

A simple statement along the lines of «We've had a major security breach, that we can not
disclose at this time. We have determined that packages distributed by the Fedora project have
not been compromised. You Fedora installation is not at risk. Please bare with us while we fix
this.» would put a lot of peoples mind at ease. Like the article said, this is a bit of a
failure to communicate from Fedoras leadership.

One week of infrastructure issues

Posted Aug 21, 2008 1:12 UTC (Thu) by sbergman27 (guest, #10767) [Link] (19 responses)

"""
By all accounts, the Fedora team has been working flat-out to get things resolved as quickly
as possible; they seem to be doing an exceptional job under a great deal of pressure. 
"""

Keep in mind that these are the same people who said (for how many years?) that they were
working hard on setting up a public Subversion repository.

One week of infrastructure issues

Posted Aug 21, 2008 1:26 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (18 responses)

What are you talking about? I don't recall any plans to setup a public subversion repository
within Fedora Project and certainly not one for years. Can you provide a reference?

One week of infrastructure issues

Posted Aug 21, 2008 1:40 UTC (Thu) by sbergman27 (guest, #10767) [Link] (17 responses)

You cannot possibly mean that you don't remember.  Although it turns out it was actually a CVS
repository, rather than Subversion, which had been promised since July 2003 (when Fedora was
still called the Red Hat Linux Project) and remained unrealized until, I believe, sometime in
2006, despite a persistent public clamoring and grumbling about the issue the entire time.
Does that jog your memory or do you still need links?

One week of infrastructure issues

Posted Aug 21, 2008 1:55 UTC (Thu) by sbergman27 (guest, #10767) [Link] (2 responses)

Actually, it looks like it was not 2006, but spring of 2007.  So 3 years and 10 months for the
promised CVS server.

One week of infrastructure issues

Posted Aug 21, 2008 6:48 UTC (Thu) by wtogami (subscriber, #32325) [Link] (1 responses)

Um... November 2004 the CVS server went public for Fedora Extras where we collaboratively
maintained thousands of packages around Core before the distro fully merged prior to Fedora 7.

One week of infrastructure issues

Posted Aug 21, 2008 21:02 UTC (Thu) by sbergman27 (guest, #10767) [Link]

Warren, yes. (I'll take your word on the date for Extras.  Sounds believable.)  Since the
early days, Fedora has had many auxiliary repositories like "Extras".  And yes, "Extras" was
kinda sorta special. But what people were clamoring for (for almost 4 years) was for the
"Core" of Fedora Core to open up.  You know, the "community distro" itself.  Not the auxiliary
repos.  And *you know that very well*.  How could you not?  At this late date, isn't it really
best to just admit it and go on?  As I recall, it was only *very shortly* before the end of
May 2007 release of F7 that the merged cvs repo was completed.  

http://lwn.net/Articles/232958/

And I could dig up plenty of "What the hell is taking the Fedora guys so long?" articles over
the 4 year period, including at least one pretty good one from LWN, IIRC.  Want me to?

One week of infrastructure issues

Posted Aug 21, 2008 2:02 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (13 responses)

"You cannot possibly mean that you don't remember"

Of course, I can quite honestly claim that. Why not?

"Although it turns out it was actually a CVS
repository, rather than Subversion"

Ah, there in lies the rub. I knew for a fact that Subversion was never really considered a
serious contender for anything in Fedora Project although it is a supported option in
fedorahosted.org. So your claim of subversion repository threw me off the mark. 

Also opening up existing infrastructure that has existed for over a decade within a
organization and in many cases rewriting it to be community facing is a entirely different
ball game from merely setting up a cvs repository. For a open setup, you need public facing
build systems for exampleand that was written from scratch for Fedora, twice even. Sure, there
was a well known delay during the initial stages of the project but it was also a non trivial
effort to put it mildly. You seem to be drawing parallels between very different situations.

One week of infrastructure issues

Posted Aug 21, 2008 2:41 UTC (Thu) by sbergman27 (guest, #10767) [Link] (12 responses)

"""
Ah, there in lies the rub.
"""

I find this surprising.  The issue was a high profile one for a very long time.  You really
did not make the connection because I said Subversion instead of cvs?  And does it really
matter if the server it took them nearly 4 years to set up was cvs or subversion?

"""
Also opening up existing infrastructure that has existed for over a decade within a
organization and in many cases rewriting it to be community facing is a entirely different
ball game from merely setting up a cvs repository.
"""

That was the party line during most of that time.  I will agree that this is a somewhat
different situation.  But not for the party-line reason you restate here.  Red Hat did not
really want to open up Fedora that much.  They felt they had too much riding on it for that.
(I remember a Red Hat official even coming close to saying it in so many words.)  In this case,
Red Hat also likely has a keen self interest, and are likely to want to get the servers up as
soon as possible.

BTW, that is not intended to come off as anti Red Hat.  I greatly respect Red Hat.  But the
low priority foot dragging was pretty obvious.  You can bet that if "opening up existing
infrastructure that has existed for over a decade within a organization" had been perceived as
critical to the RHEL product, they could have done it in a matter of months if not less.

One week of infrastructure issues

Posted Aug 21, 2008 3:03 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

"You really did not make the connection because I said Subversion instead of cvs?"

Yes and also so because it is not the same people like you claimed and I am not sure your
dates are right either. I am very specific that way and that is one of the reason I ask for
references. 

"And does it really matter if the server it took them nearly 4 years to set up was cvs or
subversion?"

Simply setting up a cvs or subversion system wouldn't result in much without a open
buildsystem, compose tools etc to go along with it and these things take time. It took until
Fedora 7 in fact to convince everybody including internally to really unify all these
infrastructure bits. Like you admit, this situation is pretty different so I am not even sure
why you bought it up in the first place.

One week of infrastructure issues

Posted Aug 21, 2008 13:21 UTC (Thu) by interalia (subscriber, #26615) [Link] (1 responses)

Well I have to say that I also felt it took a long time for Fedora to open up. I'm just
posting as a disinterested Debian user with nothing against Red Hat or Fedora, but it took an
age, with little feeling of progress from the vantage point of someone who just generally
follows Linux and distro news (ie. a typical LWN reader). In hindsight it was perhaps simply a
failure to manage expectations, but I can understand the frustration that other people felt.

One week of infrastructure issues

Posted Aug 21, 2008 17:26 UTC (Thu) by jspaleta (subscriber, #50639) [Link]

"Well I have to say that I also felt it took a long time for Fedora to open up."

No one is going to deny that it took longer than would have hoped to clear the huddles
presented in opening up what use the be completely internal infrastructure used to build rhl
and build an open community infrastructure that would allow the eventual merging of Fedora
Core and Extras. I personally remember a few rounds of discussion concerning issues with
compliance of the Sarbanes-Oxley Act which came into effect in 2002.. a new law that affected
every single publicly traded company in the US. Maybe in hindsight, the discussions concerning
sarbox compliance as it relates to opening up things for Fedora were unneeded..but in the
context of what was happening at the time.. it was probably an unavoidable reality which
slowed the process down.

I'm sure there's a great coffeetable book to be made of the anecdotal stories related to the
building of the Fedora Project. I'd probably buy it too..as long as the proceeds were churned
into the Fedora Scholarship program. I'd love to read seth's and gafton's recollections of
the process of getting a public cvs system up and running for Fedora Extras.. with a lovely
cartoon illustration of them in a boxing ring fighting about it.

Yes, it was a painful process, and an important one, but in a way that most probably don't
think about. The great achievement is not the building of an external community around Fedora.
Debian proved well before Fedora came into its own that you can build a community of
volunteers and get something significant done. There was a community already in place at
Fedora.us before RHL was ended and the Fedora project began. The community organization was
already happening on its own..that's how cool the open ecosystem is.. RHL users were already
self-organizing at Fedora.us before Red Hat decided to take the plunge.

No the most important result of what Fedora as a project has achieved is the deep
internalization of an open development culture inside of Red Hat itself. The fact that Red
Hat continually re-invests in the community in ways that allow technology to grow outside of
its direct control. The most recent battle in the culture war was won with the release of
Spacewalk as an open technology. And now Red Hat is taking its experience earned in that hard
fought cultural war and helping its own customers better understand how to internalize and
benefit from the same open development culture that powers Debian or Fedora or hundreds of
other individual project pieces out in the ecosystem.

As members of a larger open development ecosystem.. larger than Fedora or Debian or whatever
project you have a personal commitment to... we are better off now with Red Hat as a full
partner, than we would be by encouraging Red Hat, the corporate entity, to stand outside of
that process.

We can nitpick previous mistakes in the process of how we got to where we are forever, but its
not particularly helpful. Hell, its not even the same group of people in the discussion that
was happening in 2002, there are a lot more voices now, because the project has grown so much.
People who use to be external 'community' are now 'red hat' and there are new external voices
doing new work and pushing things forward in new ways. What matters is understanding how
Fedora exists right now and whether its doing its job to push open innovation forward in a way
that every single person using a linux distribution benefits from the work.

-jef

One week of infrastructure issues

Posted Aug 21, 2008 11:09 UTC (Thu) by motk (subscriber, #51120) [Link] (8 responses)

You really know not of what you speak here, and it shows. I recall the old build system and
source repositories, and they were definitely not something you'd just switch over to the
other side of the DMZ and ring the bell for dinner. It took a lot of time and effort for the
ISIT team to build out infrastructure to bootstrap everything, for Legal to ensure that
nothing lurked in there that could either bite RH or encumber the nascent Fedora project, for
community structures to be formalised - all the hard work you seem to casually discount.

You really are making a habit of sinking the slipper into anything vaguely Red Hat-ish,
despite your protestations otherwise.

One week of infrastructure issues

Posted Aug 21, 2008 21:17 UTC (Thu) by sbergman27 (guest, #10767) [Link] (7 responses)

"""
It took a lot of time and effort for the ISIT team to build out infrastructure to bootstrap
everything, for Legal to ensure that nothing lurked in there that could either bite RH or
encumber the nascent Fedora project,
"""

Sun Microsystems opened the very proprietary Solaris in far less time.  I guess they just have
smarter engineers.

"""
You really are making a habit of sinking the slipper into anything vaguely Red Hat-ish,
"""

I'm very pro-RedHat.  I haven't a single server at any of my client sites which is not either
CentOS or Fedora. I tend to be critical of Fedora where it is deserved. (And yes, I am
migrating the remaining Fedora servers to CentOS for reasons of practicality and pragmatism.)

If my comments about Red Hat seem negative, please imagine how you would act if your company
was as visible, had lots of cash in the bank that some litigious entities might want a slice
of, and had a 3.2 billion dollar market cap to protect.  Red Hat does the right thing whenever
they possibly can because they truly believe, in their corporate heart, that if they stick to
the straight and narrow they will be rewarded.  And that has worked quite well, indeed, for
them, so far.  And I am gratified to behold it.  But they are not stupid.  And I'm quite
certain that they consult legal before every action they make.  And if some things need to
remain secret to mitigate liability...

One week of infrastructure issues

Posted Aug 21, 2008 23:06 UTC (Thu) by jspaleta (subscriber, #50639) [Link] (6 responses)

"Sun Microsystems opened the very proprietary Solaris in far less time. "

Sun opened up its build infrastructure they use to actually build the versions of OpenSolaris
that they provide in binary form? I think you are confused between offering an open source
codebase.. and building an open community infrastructure. They are not the same things and
one is easier to do than the other.

Now I may be wrong, but it seems to me that opensolaris development is gated through Sun
developers.
These pages would certainly suggest it works like that:
http://www.opensolaris.org/os/communities/participation/
http://opensolaris.org/os/bug_reports/request_sponsor/

If you want to contribute fixes or enhancements to OpenSolaris you file bugs or RFE's in the
opensolaris bug tracker and those patches are reviewed by Sun employees for integration. In
fact you actually have to sign over joint copyright ownership to Sun for any contributions you
make..before you are allowed to contribute anything.

You want to contribute, you send in a patch...it gets reviewed, the Sun employee applies it
for you. Actually its more complicated than that.. you actually have to request a Sun employee
sponsor you and give you the go ahead to actually work on a patch.

Besides the whole, must give Sun joint copyright detail and the whole you must request a
sponsor detail, the basic mechanism by which non-Sun employeed contributors can contribute to
OpenSolaris compares directly to community participated in Fedora before the Core/Extras
merge...as far back as RHL 6.x if not prior to that. But I only speak of history that I have
personally lived so I won't speak to anything before RHL 6.x.

If filing bugs and patches in a bug tracker is bar you want open source distributions to
meet... RHL met that a long long time ago. With Fedora as of Fedora 7, community involvement
goes well beyond that, with community members having equal access to actually commit code for
the entire infrastructure which builds the distribution. But we've even moved beyond that
now. Now, there are community members shoulder to shoulder with redhat employees as part of
the infrastructure and release engineering teams. Community members aren't just contributing
the bits... community members are responsible for grinding the bits and cranking out the
sausage every six months. At every level where responsibility can be shared inside Fedora, Red
Hat shares it with community members... because its a partnership. What other software
company is making that level of commitment that Red Hat is making to work with community as a
development partner?

-jef

Outside of Debian, what other linux distribution gives that much access to

One week of infrastructure issues

Posted Aug 22, 2008 0:06 UTC (Fri) by gman (guest, #40493) [Link] (1 responses)

While that process has certainly been true of the past, things are progressing. OpenSolaris has now had a successful migration from Teamware to Mercurial internally, and the next phase is to push that gate outside the firewall. Once this happens, people will be able to commit directly to the kernel (though, much of the review process will still be in place). The bug tracker is still internal (due to confidentiality issues), though we now have a bugzilla instance (defect.opensolaris.org) that many project groups are using.

There is obviously still a significant amount of work to be where we want to be.

One week of infrastructure issues

Posted Aug 22, 2008 2:54 UTC (Fri) by jspaleta (subscriber, #50639) [Link]

Yes opening up infrastructure and and shaking up an existing established workflow inside a
company fence line takes time and non-trivial effort.  Both from an engineering standpoint and
from a legal standpoint.  If doing that work is OpenSolaris's intentions then I applaud the
effort and I look forward to each of the incremental steps as  they take place.  Its hard work
clearing internal company hurdles to make those incremental steps happen.  

Just what ever you do, don't reach for or build a new set of proprietary tools to form the
basis of your community infrastructure and community contribution process.  Don't build your
own launchpad.  If you can avoid that trap, and take advantage of existing open tech or if you
have to build your own open tech... it will pay off when your community wants to help adjust
the community facing tools, because they'll be able to step in and do it for themselves. 

Human code review isn't the issue, someone will always have to be the final say on some issue
and for something like the kernel that person should be someone who is deeply invovled in its
development. Not going to get any arguments from me there.  Cultivating deeply technical
community contribution is a process in itself. It starts with code access, and then it grows
via development process participation, and then in culminates in development process
leadership in new areas...areas your company didn't really think about at the beginning of the
process of opening up. 

-jef

One week of infrastructure issues

Posted Aug 23, 2008 21:05 UTC (Sat) by sbergman27 (guest, #10767) [Link] (3 responses)

"""
Sun opened up its build infrastructure they use to actually build the versions of OpenSolaris that they provide in binary form?
"""

No. I was responding to motk's statement:

"It took a lot of time and effort for the ISIT team to build out infrastructure to bootstrap everything, for Legal to ensure that nothing lurked in there that could either bite RH or encumber the nascent Fedora project"

You might recognize that as the stock argument for why it takes so long to open large bodies of closed source code. In response, I pointed out that Sun managed to open a gigantic and completely closed code base (with a long history), in far less time than it took Fedora/Redhat to open up the supposedly 100% open-source build infrastructure of a 100% open-source project for public use. Different goals, yes. But opening Solaris was at least an order of magnitude more ambitious than what took Fedora so very, very long to accomplish.

It's hard to believe that people whom I know *must* be familiar with the situation are actually trying to deny the obvious and major foot-dragging that occurred for years.

One week of infrastructure issues

Posted Aug 24, 2008 1:04 UTC (Sun) by motk (subscriber, #51120) [Link] (2 responses)

Thanks for implying that I'm a liar. I really appreciate your contributions to the public FOSS discourse.

One week of infrastructure issues

Posted Aug 24, 2008 4:20 UTC (Sun) by sbergman27 (guest, #10767) [Link] (1 responses)

"""
Thanks for implying that I'm a liar.
"""

motk,

What do you mean by that? I disagreed with your argument, but never intended to imply that you were a liar. I do perceive some general desire by some to sweep certain things under the rug at this late date, when the evidence is not still hanging in everyone's faces. But that's a different thing.

All's well that ends well. But that does not mean that all was always well. Best to admit past mistakes and go on, I think...

One week of infrastructure issues

Posted Aug 24, 2008 18:22 UTC (Sun) by rahulsundaram (subscriber, #21946) [Link]

"I pointed out that Sun managed to open a gigantic and completely closed code base (with a long history), in far less time than it took Fedora/Redhat to open up the supposedly 100% open-source build infrastructure of a 100% open-source project for public use."

It wasn't open build infrastructure at all before and what was used internally was completely unusable outside of Red Hat. It had to be written from scratch before it was used for Fedora as I have already indicated earlier. It wasn't 100% open source either. That's just two of the several things you have got wrong in this discussion. This discussion seems completely unrelated to the article at this point even if it was tangentially related earlier and If you got more questions, feel free to email me and I will happy to tell you all about it.

More speculation

Posted Aug 21, 2008 1:14 UTC (Thu) by mrons (subscriber, #1751) [Link] (1 responses)

I wonder if the problem found with yum in RH enterprise has anything to do with the fedora
problem:

http://lwn.net/Articles/294013/

More speculation

Posted Aug 21, 2008 1:27 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

Considering that yum-rhn-plugin is specific to RHEL and is not even included within Fedora, I
think it is a safe bet, that it is not related.

This is very troubling

Posted Aug 21, 2008 1:17 UTC (Thu) by JoeBuck (subscriber, #2330) [Link] (1 responses)

I've run Red Hat and Fedora systems forever, I use them at work and at home, and I have relationships that go back to the Cygnus days. But I am deeply troubled by the lack of communication. It is corrosive to the trust that Fedora and Red Hat need to be seen as credible, and people will start to assume the worst.

Please, Fedora, say something. Tell us what you know, as well as what you don't know.

This is very troubling

Posted Aug 21, 2008 9:12 UTC (Thu) by nix (subscriber, #2304) [Link]

Well, if this speculation is right:

Last month, Fedora slipped a small patch into gcc designed to ensure that the results from the most recent board election - where one slot went to a candidate who was not a Red Hat employee - would never be repeated.

... then GCC has mystic mind-control powers (which of course work only on people who aren't GCC hackers, because of quantum). So once that nefarious patch is fixed, you will be the only person who is concerned. ;)

Far more likely....

Posted Aug 21, 2008 9:32 UTC (Thu) by stevan (guest, #4342) [Link] (3 responses)

And I thought it was because the Fedora team had had an investment of $100 million from
Microsoft and were too busy buying new equipment and pouring bubbly into their ears.

;-|

Let's trust them to be doing the right thing and keep suspicion for when it's required.

S

Far more likely....

Posted Aug 21, 2008 11:12 UTC (Thu) by motk (subscriber, #51120) [Link] (2 responses)

Exactly. There are no doubt sound reasons for all of this; if RH has not earned some community
trust then no-one can be said to have done so, or ever do.

Far more likely....

Posted Aug 21, 2008 21:53 UTC (Thu) by sbergman27 (guest, #10767) [Link] (1 responses)

In security, ten "Attaboys" is worth one "Oh, shit". (That's an ancient proverb at the company
I used to work for.)

I have a strong suspicion that the Fedora guys are having, or are about to have, an "Ex-Lax
moment".

Far more likely....

Posted Aug 21, 2008 23:55 UTC (Thu) by motk (subscriber, #51120) [Link]

You really should try and get out of the crackrockery sometime. Or just give up and become a
ZDNet pundit.

One week of infrastructure issues

Posted Aug 21, 2008 13:34 UTC (Thu) by cde (guest, #46554) [Link] (1 responses)

It is my belief that key Fedora administrators just got abducted by aliens and are currently
being probed in a sanitory manner. Hopefully they will be returned to earth soon enough.

One week of infrastructure issues

Posted Aug 21, 2008 14:51 UTC (Thu) by jprindiv (guest, #51354) [Link]

On their way back they had a stopover on the Brain Slug Planet and liked it so much that they decided to stay of their own free will.

One week of infrastructure issues

Posted Aug 22, 2008 15:06 UTC (Fri) by Klavs (guest, #10563) [Link] (1 responses)

They finally leaked some information about what happened: 
https://www.redhat.com/archives/fedora-announce-list/2008...

One week of infrastructure issues

Posted Aug 22, 2008 21:18 UTC (Fri) by Brenner (guest, #28232) [Link]

Thanks.

Following the link we have http://www.redhat.com/security/data/openssh-blacklist.html, with
this excerpt:

[ In connection with the incident, the intruder was able to sign a small
number of OpenSSH packages relating only to Red Hat Enterprise Linux 4
(i386 and x86_64 architectures only) and Red Hat Enterprise Linux 5 (x86_64
architecture only). ]

Does anyone knows how RH can be sure that _only_ the openssh packages listed in their
openssh-blacklist-1.0.sh checker have been signed by the intruder ?