User: Password:
|
|
Subscribe / Log in / New account

Fedora Board meeting minutes (2008-AUG-26)

From:  John Poelstra <poelstra-AT-redhat.com>
To:  fedora-advisory-board-AT-redhat.com
Subject:  Fedora Board Recap 2008-AUG-26
Date:  Tue, 02 Sep 2008 17:49:11 -0700
Message-ID:  <48BDDF07.8070104@redhat.com>
Archive-link:  Article

https://fedoraproject.org/wiki/Board/Meetings/2008-08-26

== Roll Call ==

* Attendees: John Poelstra, Paul Frields, Jesse Keating, Matt Domsch, 
Jef Spaleta, Bill Nottingham, Chris Tyler, Karsten Wade, Spot Callaway, 
Seth Vidal
* Regrets: Harald Hoyer

== Discussion About Incident Handling ==
* Could other groups have been brought into knowledge of the incident 
earlier?
* Could the Fedora Board have been notified or kept in the loop better?
** Would probably require signed NDAs which most are not in favor of
* Event was complicated by co-announcement made by Red Hat
* Ongoing tension between Fedora being able to act independently and Red 
Hat being liable for Fedora's actions
* Could Community Architecture Group be involved earlier to help 
facilitate communication?
* Don't want to get into a situation where every Fedora decision or 
announcement has to be vetted through Red Hat executive levels
* Create a predefined flow-chart or decision tree that explains steps 
that we will take in similar situations
** one potential flow through could be Red Hat Legal
** get advanced agreement from all parties involved
** include time limits where appropriate to speed up the response time 
and make the decision  work flow more efficient.
** standardize types of messages that should be published and how often
** one path might be the necessity of shutting down the entire 
infrastructure--would need to enable the ability to efficiently do that 
if not already present
** Cross-link to established industry security standards
** one condition of agreeing to process flow is that actions could be 
initiated without requiring constant sign-off which is the intention 
behind advanced agreement
* FESCo to discuss proposal from release engineering about updating 
package signing keys on Wednesday (2008-08-27) at 18:00 UTC: 
http://lists.fedoraproject.org/pipermail/rel-eng/2008-Aug...
** board members should be aware of and attend as appropriate

== Next Meetings ==
* No board meeting on September 2, 2008--follows holiday weekend and 
some people are away
* Move IRC and Board Q&A meeting to September 9, 2008
* Next regular board meeting September 16, 2008


(Log in to post comments)

Fedora Board meeting minutes (2008-AUG-26)

Posted Sep 3, 2008 18:53 UTC (Wed) by rgmoore (✭ supporter ✭, #75) [Link]

And AFAIK they still haven't released a single software update- security or otherwise- since the "incident" started. With the exception of the low content "status updates" and these minutes, it's as though the whole project dropped off the map. I don't want to sound like a whiner or a troll, but this is not how to run a community oriented project.

Fedora Board meeting minutes (2008-AUG-26)

Posted Sep 3, 2008 19:26 UTC (Wed) by skvidal (guest, #3094) [Link]

The lack of a release of updates is just b/c of having to resign and figure out how to rekey everyone's systems relatively painlessly.

Unfortunately, resigning 15000+ packages per architecture is extremely time consuming.

Fedora Board meeting minutes (2008-AUG-26)

Posted Sep 4, 2008 3:54 UTC (Thu) by bojan (subscriber, #14302) [Link]

> how to rekey everyone's systems

I think we were lucky this time, given that Fedora key didn't actually get compromised. If it had been compromised, users would have a tough time establishing a new trust using a compromised key or by applying RPMs by hand after verifying that the new key is OK also by hand.

If Fedora signing system, rpm and yum allowed multiple alternative signatories (at least for fedora-release package), it would have been trivial to quickly deliver a new fedora-release package signed by sufficient number of alternative signatories. This would in turn change trusted keys in /etc/pki/rpm-gpg, deliver revocation certificate for the old key and enable seamless transition to the new Fedora key. And all without ever trusting a compromised or an already revoked key or forcing users to manually apply anything.

Fedora Board meeting minutes (2008-AUG-26)

Posted Sep 3, 2008 19:48 UTC (Wed) by jspaleta (subscriber, #50639) [Link]

The Board minutes are always terse.

If community members feel that Board members need to elaborate on a particular item, people are encouraged to raise questions on the fedora-advisory-board mailinglist in reply to the minutes posting. I would encourage the editor here to link to the fab list posting of the minutes instead of the wiki in the future, to give a better indication as to where follow-up discussion is expected.

But on to the point....
The process is in motion to roll out new signing keys. That process was openly discussed as part of the public FESCo meeting. For reference see
the FESCo minutes and associated links:
http://fedoraproject.org/wiki/Extras/SteeringCommittee/Me...

and subsequent discussion here with regard to implementation:
https://www.redhat.com/archives/fedora-devel-list/2008-Au...

I will remind you that FESCo is a completely elected body, with public meetings. It's probably not wise to just link to the Board minutes without also linking to FESCo minutes concerning a topic that is within FESCo jurisdiction. Especially when the Board minutes specifically state that FESCo is going to be taking up the issue in their next meeting.

FESCo is where the bulk of the week to week decision making is happening concerning the distribution and packaging....not the Board. The Board discussion revolved around the issues associated with developing a project wide incident reporting process that everyone can rely on. It did not focus on post-incident infrastructure or release engineering activities, which continue apace. As the minutes stated, Board members were encouraged to attend the public FESCo meeting specifically to be on hand for the more technical discussion regarding the plan for rolling out new signing keys.

It might be worthwhile for people to reacquaint themselves with the Board and FESCo mandates to get a better idea of which body deals with which issues:
https://fedoraproject.org/wiki/Board
https://fedoraproject.org/wiki/Development/SteeringCommittee

In the case of the specific issues regarding package updates, FESCo has oversight over greenlighting the plan to move forward because it directly impacts the distribution. They have done so, and things are in motion.. new keys exist, packages are being re-signed, people are working on the mechanism to get the new key out to users, so when the newly signed packages hit a public repository mirror, people will be able to verify them with the new key.

-jef

Fedora Board meeting minutes (2008-AUG-26)

Posted Sep 3, 2008 20:08 UTC (Wed) by tuna (guest, #44480) [Link]

I have had many updates in Rawhide since last weekend. Which messed up packagekit, but that is another story....

Fedora Board meeting minutes (2008-AUG-26)

Posted Sep 3, 2008 21:45 UTC (Wed) by pr1268 (subscriber, #24648) [Link]

* Could the Fedora Board have been notified or kept in the loop better?

Most certainly seems like they should have been.

* Ongoing tension between Fedora being able to act independently and Red Hat being liable for Fedora's actions
* Don't want to get into a situation where every Fedora decision or announcement has to be vetted through Red Hat executive levels
** one potential flow through could be Red Hat Legal

Whose show is this?? Red Hat's or Fedora's?

** Cross-link to established industry security standards

One would hope that a big "brand-name" distro two big "brand-name" distros such as Red Hat and Fedora would do so already.

I'm glad I don't use Red Hat or Fedora. Any Linux distribution that has to worry about petty issues as these instead of focusing on making technical, performance, and usability enhancements to both the kernel and userspace applications seems like either (1) crisis management or (2) misguided priorities. Not to mention that it seems like FESCo (whatever that stands for) is having to tap-dance around Red Hat's legal department (and likely RH's marketing group as well).

Granted, I do admire FESCo's efforts to work through whatever issues caused the security breach from several weeks ago, but I just needed to vent my feelings here. Go ahead and bring on the flames--it's ironic that I'm listening to Flamethrower by the J. Geils Band as I type this!

Fedora Board meeting minutes (2008-AUG-26)

Posted Sep 4, 2008 0:00 UTC (Thu) by smoogen (subscriber, #97) [Link]

Why vent if you aren't using the distro? I mean you have a distro you are happy with... so why all the schadenfreude? Why the need to pee on others versus explaining why yours serves you better?

Why vent?

Posted Sep 4, 2008 1:16 UTC (Thu) by pr1268 (subscriber, #24648) [Link]

I dunno... Maybe it's because I actually did use Red Hat a while back. As in the 1998-vintage version 5.1 or so. That's where I first learned about (and how to use) Linux. And then they decided to pull the plug on free (as in beer) releases and fork the whole distro into an "enterprise" version and Fedora. I didn't (and still don't) totally appreciate how that went about. While I do see that Red Hat and Fedora have worked through some of those kinks, I still see a lot of a single organization managing two distinct and discrete distributions.

Or, maybe I'm just feeling a little insecure for using Slackware (NOT!). But, I admire you pointing out how my venting is slightly misguided. And nice use of the word schadenfreude!

Why vent?

Posted Sep 4, 2008 9:18 UTC (Thu) by nim-nim (subscriber, #34454) [Link]

> I dunno... Maybe it's because I actually did use Red Hat a while back. As
> in the 1998-vintage version 5.1 or so. That's where I first learned about
> (and how to use) Linux. And then they decided to pull the plug on free (as
> in beer) releases and fork the whole distro into an "enterprise" version
> and Fedora.

So you want a gratis show, but complain Red Hat pays Fedora lawyers in your stead?

Why vent?

Posted Sep 4, 2008 13:55 UTC (Thu) by oblio (guest, #33465) [Link]

"I'm glad I don't use Red Hat or Fedora. Any Linux distribution that has to worry about petty issues as these instead of focusing on making technical, performance, and usability enhancements to both the kernel and userspace applications seems like either (1) crisis management or (2) misguided priorities."

"Or, maybe I'm just feeling a little insecure for using Slackware (NOT!)"

Slackware, aka the "1 man show distribution"?! I'd be ashamed if I were you to make such comments. Slackware has NO upstream contribution. No usability enhancements or such. Red Hat has invested more money in Open Source projects that we all use, than almost all other parties except IBM, Sun and Novell. As per priorities, IBM, Sun and Novell are doing what they're doing partly because of the pressure from Red Hat, the true and original "Open Source company".

This is a rant by a Debian/Ubuntu user. I still know who does the hard work in this community and who pays for it (at least partly, of course there are a lot of companies involved) ;)

And I still know where some respect is due.

Why vent?

Posted Sep 4, 2008 14:47 UTC (Thu) by pr1268 (subscriber, #24648) [Link]

Slackware, aka the "1 man show distribution"?!

Nowadays, Slackware is the work of a bunch of online volunteers. Patrick Volkerding still does some coordination work, but it's certainly not a "1 man show" any longer.

No usability enhancements or such. Red Hat has invested more money in Open Source projects that we all use

That's odd... Slackware runs substantially faster than Red Hat and Fedora on equivalent hardware. And it's easier to use (my opinion). Maybe RH and Fedora need to retool their investment strategy.

Why vent?

Posted Sep 4, 2008 18:37 UTC (Thu) by oblio (guest, #33465) [Link]

"Slackware runs substantially faster than Red Hat and Fedora on equivalent hardware. And it's easier to use (my opinion). Maybe RH and Fedora need to retool their investment strategy."

You claim to be an old Linux user. I thought you understood by now that

faster != better

I won't bother to comment on their investment strategy considering that 1-5% of Slackware (call it an educated guess) is probably written by Red Hat people. 1-5% of the numbers of lines of code, doing 50% of the work (parts of GCC, GDB, Linux, glibc, GTK, and many, many more).

Why vent?

Posted Sep 4, 2008 19:06 UTC (Thu) by pr1268 (subscriber, #24648) [Link]

I will concede that some parts of Slackware are actually Debian/Ubuntu/Fedora contributions (mostly in the area of init scripts and various service daemons), but overall, Slackware is comprised of unadulterated upstream packages. I will also concede that the larger distros' developers often contribute to these packages. As do Slackware users.

As for your "faster != better" comment, well, I can likely get some support for the statement that bloat != better, and I'm convinced that the bigger-name distros are all about bloat. Which likely fuels my perception that Slackware is faster than $BIG_NAME_DISTRO.

And besides, the one huge annoyance that prompted me to write that Slackware is faster was/is the system responsiveness "stutter" I vividly remember experiencing with FC2/3 (caused by page faults according to a fellow LUG member). A friend told me that this still happens with Fedora 8 (usually while running Mozilla Firefox). I suppose that'll never get fixed...

I will even argue that Slackware versions 11.x/12.x have become bloated relative to earlier versions, but its bloat still can't hold a candle to the bloat I've seen Red Hat and Fedora go through.

Why vent?

Posted Sep 4, 2008 19:25 UTC (Thu) by smoogen (subscriber, #97) [Link]

Well of course they are more bloated than earlier versions... upstream software has bloated quite a bit. Developers usually write for the most current hardware and usually have larger than normal amounts of CPU/memory.

Even then though, Slackware is a conservative distribution which is usually a full cycle behind what is seen in Fedora/Ubuntu. It is aimed at people who aren't buying new computers every year and do not want every bit of new eye-candy. It is also conservative about what compilation options it will use (UTF-8 support was not a high priority so why turn it on)... which makes it zippier than those that turn it on.

By the time Slackware gets 13/14 it will be as bloated as Fedora is now, and less bloated then Fedora will be then. [And no, I am not alluding to Slackware being behind the curve etc... it knows its market and caters to it well.]

Venting and SchadenFreude

Posted Sep 4, 2008 1:23 UTC (Thu) by AnswerGuy (subscriber, #1256) [Link]

Why not? First venting about issue one sees in the periphery of one's own experiences can serve to remind yourself and others of how important those issues are to you. By saying "seeing this validates my descision to go with an alternative" one is communicating to others what things they may want to consider (in choosing their own alternatives, or in prioritizing their own work).

As for Shadenfreude ... you can interpret his post as such. However, it strikes me as somewhat defensive on your part to call him (or her) out on it.

The response to this incident was and continues to be a bit of a debacle. It's continuing to erode the confidence of a large number of users. Key members of the Linux community ... those who work on any distribution ... should pay close attention to how this plays out and do their utmost to learn the lessons that can be gleaned therefrom.

For example, earlier in this thread someone mentioned that (re)-signing over 15K RPMs is a major undertaking and will take time. Granted. However, signing the 800 or so core packages that are part of a default server installation is at least an order of magnitude, or two, less intensive. Lesson: get the core stuff vetted and re-signed ASAP and re-assure your users thereby. Corollary: provide clear instructions for fetching (vetted?) .src RPMs for other packages so that users can fetch, build and install those (non-core) packages during the interim.

If I was my show I would have started by spinning up a small number of new (or completely re-imaged) servers, putting a small set of core packages and some new "checkme" scripts or packages thereon, and clearly labeling these as such. Those should host new ISO images with new keys and links that check only based on the new key rings. Thus my customers could, as their option, download and run just the "checkme" scripts/packages (and perhaps just the replacement/rebuilds of OpenSSH and the full suite of OpenSSL and other crypto and libc libraries on which it depends). Alternatively they could download these newly vetted ISO images and opt for a fresh re-installation.

Then I would migrate each of the old insfracture components over to the new infrastructure in a clearly documented fashion --- keeping a bright line quarantine between old (suspect) and new. I would do this even if I though it wasn't technically necessary --- even if it's purely to restore public confidence.

(At the same time I'm painfully aware that there's no guarantee that the new infrastructure would be uncompromised. I'm also aware that there's no guarantee that Debian or Ubuntu's infrastructures are secure. We must, all of us, realize that some cracker or group of crackers might have hooks in any code that we have not personally reviewed ... and even that there may be subtle hooks or vulnerabilities even in code we've thoroughly read and could recite by heart. Such is the nature of computer security. I don't trust any security "guru" who claims "this time we're sure").

Venting and SchadenFreude

Posted Sep 4, 2008 2:55 UTC (Thu) by smoogen (subscriber, #97) [Link]

I will disagree on the helpfulness of venting. It usually brings people further into conflict over trivialities versus dealing with the real problems.

Your suggestions though make a good sense on how things should be looked at in the future. At some point a breakin will occur at SuSE, Ubuntu, or Debian and build servers will be compromised... having a set and community vetted methods of how distros can regain trust of users and customers in the future is important.

Fedora Board meeting minutes (2008-AUG-26)

Posted Sep 4, 2008 9:15 UTC (Thu) by nim-nim (subscriber, #34454) [Link]

>> one potential flow through could be Red Hat Legal
> Whose show is this?? Red Hat's or Fedora's?

What an abysmally clueless comment. That there is a legal resource gracefully made available to the project is not a problem but a strength.

Given the way the USA loves to fatten trial-happy parasites, lawyers are a must for a major distribution (and they're not cheap). Fedora's happen to be provided by RH. While independent legal counsel might sound better, who would pay for them? (If it was via a foundation financed by RH, would they be really independent? If they were provided by some other organisation, do you think no one would contest this organisation agenda?)

Lawyers are part of the infrastructure nowadays. And spontaneous infrastructure generation in the absence of generous sponsors has not been observed so far.

Fedora Board meeting minutes (2008-AUG-26)

Posted Sep 4, 2008 0:05 UTC (Thu) by smoogen (subscriber, #97) [Link]

Actually these notes pretty much follow the flow of a lot of meetings after a security problems. Ooops servers cracker, but told not to say whats going on due to X, Y, or Z group. And then realizing that you didn't have a process to cover this and so lots of confusion, miscommunication, and headaches occur.

Good organizations have a meeting to go over what went wrong, and start working on the fixes. Great organizations actually look over things in 6 months time and see if the process is written and works (and continue to do so regularly). Most organizations go through this over and over and over again...


Copyright © 2008, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds