Weirdness at CentOS

[Posted July 30, 2009 by corbet]

The CentOS.org front page currently carries an open letter from several CentOS developers to Lance Davis, the person who controls most CentOS resources. "You seem to have crawled into a hole ... and this is not acceptable. You have long promised a statement of CentOS project funds; to this date this has not appeared. You hold sole control of the centos.org domain with no deputy; this is not proper. You have, it seems, sole 'Founders' rights in the IRC channels with no deputy ; this is not proper." Evidently attempts to contact Lance have been unsuccessful for some time.

to post comments

Open distro meta data

Posted Jul 30, 2009 14:37 UTC (Thu) by kragil (guest, #34373) [Link] (6 responses)

All distros should provide meta data about updates, CVEs fixed, VCS commits, etc in an open standardized format. That way an independent entity (like e.g. Distrowatch) could monitor the status of the project.

This data could really help make informed decisions based on facts about which distro to choose for a specific task and it probably help identify events or processes like this one before they are anouced.

Open distro meta data

Posted Jul 30, 2009 14:58 UTC (Thu) by kragil (guest, #34373) [Link]

And obviously the meta data should include how many people are in charge ..

Open distro meta data

Posted Jul 30, 2009 19:38 UTC (Thu) by nevyn (guest, #33129) [Link] (4 responses)

Pretty sure all the data you'd want for RHEL is already published at Red Hat/security/data/metrics. And on a client machine a significant amount of that data is available to yum/yum-security so you can do "yum update-minimal --security" etc.

Of course that doesn't tell you what the window is for RHEL => CentOS, but that's been very small (the 5.3 update was/is the only real mark against them time wise, and that was just bugfixes).

And it's hard to compare that to Ubuntu etc. as they don't publish anything but PR statements saying they are much better than everyone else.

Open distro meta data

Posted Jul 30, 2009 20:55 UTC (Thu) by kragil (guest, #34373) [Link] (3 responses)

Red hat does a good job with regards to security and only security, but the data is not available from most other distros and the format is not good for aggregation.

I think all commercial vendors should get together and publish metadata about the work they are doing as open data.

A distrowatch-like website that can answer questions like:

"Which distro updated the kernel and bind the most and fixed its security issues the fasted in the last three years."

Currently the data for that is too hard to get from most distros. I think this data would give SLES, RHEL, Ubuntu, Debian a clear advantage over these fire and forget distros that just make a release and never do any maintenance.

Open distro meta data

Posted Jul 31, 2009 12:37 UTC (Fri) by jond (subscriber, #37669) [Link] (2 responses)

I wonder who would seriously make a decision based soley on that data, aggregated in one place. It would seem very daft to make a distro decision for any sized site based soley on criteria that can be aggregated (and presumably by unvetted, vendor-supplied data at that).

Open distro meta data

Posted Jul 31, 2009 22:28 UTC (Fri) by kragil (guest, #34373) [Link]

I wouldn*t pick a distro _soley_ based on this data, but I would probably discard a lot of distros based on it. That can be very useful too.

And I don't see any problem that is data would be vendor supplied. Lying would't work in a FOSS ecosystem.

Open distro meta data

Posted Aug 8, 2009 21:15 UTC (Sat) by dsas (guest, #58356) [Link]

The security release date can be checked by checking the date that the updated package was released.

Also I don't think anyone said anything about solely using that data to make a distro choice.

Weirdness at CentOS

Posted Jul 30, 2009 14:52 UTC (Thu) by z00dax (guest, #47373) [Link]

http://planet.centos.org/ has more details on exactly what the issues are that built upto this stage. One thing that is important to note here is that the entire Development Team, Core members and the Infrastructure team of the CentOS project are onboard, aware and fully support this move. The project is not going away anytime soon. This is an administrative issue that needs resolving, right away, and we are going to do our best to make sure there is little or no impact to users.

Disclaimer: I am one of the signatories to that Open Letter.

Weirdness at CentOS

Posted Jul 30, 2009 15:54 UTC (Thu) by sbergman27 (guest, #10767) [Link] (13 responses)

As much as I appreciate all the work that the CentOS guys do, posting this open letter on the website was a very poor judgment call. CentOS is not Gentoo. CentOS is not Debian. CentOS is (supposed to be) an Enterprise Class OS for conservative admins. It makes conservative admins very nervous to see things like this on the front page of their OS maintainer's site:

"""
Please do not kill CentOS through your fear of shared management of the project.

Clearly the project dies if all the developers walk away.

Please contact me, or any other signer of this letter at once, to arrange for the required information to keep the project alive at the 'centos.org' domain.
"""

Certainly, one would never see that at the site of the upstream, well known, North American Linux vendor. (CentOS EATS YOUR BRANE, anyone?) They wouldn't allow such a thing even on the Fedora site. And for very good reason.

Weirdness at CentOS

Posted Jul 30, 2009 16:28 UTC (Thu) by djao (guest, #4263) [Link]

I agree the letter sounds bad, but after reading more, I'm not that worried.

It says on the front page, right next to the letter, "CentOS is not Dead or going away." Anyone who knows the least bit about how free/open source works knows that, if all the developers walk away as a group, the result is a fork, not a shutdown. There are rare exceptions in cases where the project is so bad that no salvaging is possible, but rebuilding and mirroring Redhat rpms is just fundamentally not that hard a thing to do. (I am not making light of CentOS's contributions, just saying this is a relatively known and manageable task compared to what some other community projects do.)

Worst case is, they'll move to another domain, everyone will have to edit their yum configuration to point to the new domain, and life will move on. Given that the nature of the complaint is Lance's inactivity, it doesn't seem that Lance is likely or inclined to actively sabotage the project, nor does he have the necessary bits required to do so (such as the GPG signing key).

For those sysadmins who are unable or unwilling to handle the dynamics of free/open source community projects, paid Redhat support is the best option, and indeed this is why Redhat exists.

Weirdness at CentOS

Posted Jul 30, 2009 16:46 UTC (Thu) by drag (guest, #31333) [Link] (11 responses)

The way I feel about it is such:

If you don't understand CentOS enough that this sort of thing would scare the shit out of you... then your probably better off not using it. Not because it's scary, but becuase if you have little understanding or desire for understanding then your probably should pay Redhat for support.

Weirdness at CentOS

Posted Jul 30, 2009 18:00 UTC (Thu) by sbergman27 (guest, #10767) [Link] (10 responses)

I still have a few CentOS servers. And I must say that while this is certainly not the end of the world, it's does affect my confidence in CentOS being around at the end of the the CentOS4 or CentOS5 life-cycle. And I would prefer that my customers not see this foolishness.

However, as I've finished my migrations of XDMCP servers away from the Fedora three ring circus, my CentOS servers were next on the list anyway. So I really don't have to worry about headlines like the "Weirdness at CentOS" or "CentOS Project Administrator Goes Missing-in-Action" headlines that I've seen today, even on non-Linux sites.

Perhaps he was kidnapped by the City of Tuttle...

Weirdness at CentOS

Posted Jul 30, 2009 18:08 UTC (Thu) by einstein (subscriber, #2052) [Link]

> Perhaps he was kidnapped by the City of Tuttle...

rofl... that's the funniest thing I've heard all week :)

Weirdness at CentOS

Posted Jul 30, 2009 18:20 UTC (Thu) by drag (guest, #31333) [Link] (3 responses)

> it's does affect my confidence in CentOS being around at the end of the the CentOS4 or CentOS5 life-cycle.

It doesn't for me. Worst case is that you have to point your yum repositories at a different URL.

Weirdness at CentOS

Posted Jul 30, 2009 18:57 UTC (Thu) by sbergman27 (guest, #10767) [Link] (2 responses)

What does being able to point to a different URL (presumably at Red Hat, after paying for a commercial support contract) have to do with having confidence that CentOS will be around in 3 to 6 years' time? Others (e.g. Whitebox Linux and at least one other) have demonstrated that tracking RHEL is not trivial by trying to do it and then backing out. (And of course I'm not that concerned about them possibly having to move to centoslinux.org or whatever. That's not the point I'm making.)

At any rate, if this incident does not have at least *some* effect on your opinion of how the project is being run, then I have to wonder about your judgment. For example, this single point of failure should not have been allowed by policy in the first place. Are there any others waiting in the wings?

That probably comes off as more sensationalist than I would normally be. But as you're pushing the point, I'm responding.

Weirdness at CentOS

Posted Jul 30, 2009 19:18 UTC (Thu) by drag (guest, #31333) [Link]

> What does being able to point to a different URL (presumably at Red Hat, after paying for a commercial support contract) have to do with having confidence that CentOS will be around in 3 to 6 years' time? Others (e.g. Whitebox Linux and at least one other) have demonstrated that tracking RHEL is not trivial by trying to do it and then backing out. (And of course I'm not that concerned about them possibly having to move to centoslinux.org or whatever. That's not the point I'm making.)

Nope. That is not what I meant at all.

I meant that the worst case is if CentOS folks can't get the Centos.org domain under their control then they will just use a different domain name. So I will have to use that in my Yum repo.

Just because they can't get rights to the DNS name does not mean that the project itself is in risk of dying. It just mean it's going to be a huge PITA for them to migrate over to another domain. (mailing lists, security certificates, developer contacts and all that)

Weirdness at CentOS

Posted Jul 30, 2009 19:59 UTC (Thu) by ewan (guest, #5533) [Link]

CentOS is not the only rebuild; there's also Scientific Linux, and that's not going away.

Weirdness at CentOS

Posted Jul 30, 2009 21:31 UTC (Thu) by muwlgr (guest, #35359) [Link]

... TUTTLE
... TUTTLE
... TUTTLE
... BUTTLE !

Weirdness at CentOS

Posted Jul 31, 2009 0:52 UTC (Fri) by pr1268 (guest, #24648) [Link] (2 responses)

Perhaps he was kidnapped by the City of Tuttle...

Aww, you just made my day!! Thanks!

Weirdness at CentOS

Posted Jul 31, 2009 1:20 UTC (Fri) by sbergman27 (guest, #10767) [Link] (1 responses)

Actually, I live just 20 minutes or so from Tuttle. (Seriously.) I could drive down to their City Hall, ask for Jerry Taylor, and *demand* that he release Lance immediately. :-)

I think he's left office now, though.

Weirdness at CentOS

Posted Jul 31, 2009 1:29 UTC (Fri) by jordanb (guest, #45668) [Link]

I hope they quietly dismissed him for being an incompetent moron (who wasn't even smart enough to find a competent staff member when he didn't understand something) as soon as the bruhaha died down.

Weirdness at CentOS

Posted Aug 4, 2009 20:12 UTC (Tue) by rickmoen (subscriber, #6943) [Link]

"This whole Buttle/Tuttle confusion was obviously planned from the inside."

-- Rick Moen
rick@linuxmafia.com

Weirdness at CentOS

Posted Jul 30, 2009 18:14 UTC (Thu) by jordanb (guest, #45668) [Link] (4 responses)

It occurs to me that all this is benefiting Red Hat very much. Both the actions of the original developer and the letter on the front page of the website (and associated coverage).

Weirdness at CentOS

Posted Jul 30, 2009 18:28 UTC (Thu) by drag (guest, #31333) [Link] (2 responses)

Hopefully it will.

It'll possibly help remind people of the benefits of paying professionals to support your environment.

Either that or convince people why Debian kicks-ass.

Weirdness at CentOS

Posted Jul 31, 2009 0:04 UTC (Fri) by spiro (guest, #54657) [Link] (1 responses)

uh, yeah, because debian is free from in-fighting and political differences...

debian slur

Posted Aug 1, 2009 0:00 UTC (Sat) by brian (subscriber, #6517) [Link]

In Debian, raucous discussion is the process.

Weirdness at CentOS

Posted Jul 30, 2009 18:42 UTC (Thu) by ixs (subscriber, #47170) [Link]

Yeah. Someone over at slashdot already mentioned that he was asking a friend over at Red Hat and this friend couldn't confirm or deny that Red Hat has kidnapped Lance.

Clearly, the Scientific Linux guys will be next...

Weirdness at CentOS

Posted Jul 30, 2009 18:41 UTC (Thu) by sbergman27 (guest, #10767) [Link] (1 responses)

Remember this? Could this be related, I wonder?

http://lwn.net/Articles/340130/

Weirdness at CentOS

Posted Jul 30, 2009 18:47 UTC (Thu) by range (guest, #51876) [Link]

No, it really is not related at all. Those two things have nothing to do with each other.

Btw., I am the one who wrote the mail you linked to and I have signed the open letter.

Weirdness at CentOS

Posted Jul 30, 2009 20:03 UTC (Thu) by BackSeat (guest, #1886) [Link] (10 responses)

It seems to me that comments fall into two categories:

1) "I'm not worried because ... " + explanation of how Open Source projects evolve, survive, etc

2) "I don't want my customers seeing this"

Even if one understands that CentOS, under one name or another, won't disappear overnight, some customers - or worse, potential customers - won't see it that way. It's unprofessional at best.

It would be easy for someone researching the validity of choosing Linux as a business platform that CentOS and, by extension, Linux is run by amateurs. It doesn't matter what the truth is: perception is everything.

Weirdness at CentOS

Posted Jul 30, 2009 20:42 UTC (Thu) by drag (guest, #31333) [Link] (2 responses)

If you say so.

I know, however, that the likelihood that customers:
A) Understand what CentOS is.
while simultaneously:
B) Not understand how CentOS is run.
And...
C) Don't trust your judgement yet are willing to give you money.

Are much much much smaller group then it would seem at first blush. Within 3-4 days nobody is going to remember this thing happening, if they knew at all, except for a small group of people.

Weirdness at CentOS

Posted Jul 30, 2009 20:49 UTC (Thu) by NightMonkey (subscriber, #23051) [Link] (1 responses)

CentOS has customers?

Weirdness at CentOS

Posted Jul 30, 2009 21:34 UTC (Thu) by drag (guest, #31333) [Link]

No. People that use CentOS have customers.

The question is how much this flap is going to color the perceptions of your customers (and potential customers) towards _you_ if you are using CentOS to house their services.

I don't think it's a big deal as the chances that customer simultanously have enough of a clue to know what CentOS is while at the same time not understanding what is going on is rather small.

Weirdness at CentOS

Posted Jul 30, 2009 21:37 UTC (Thu) by jspaleta (subscriber, #50639) [Link] (6 responses)

So using this same logic.. you would applaud a company for continuing to tell customers everything is going swimmingly when it is in fact not? Hmm, that's interesting.

No comment or worse deliberately craft misinformation is better for a customer and a potential customer? If organizations don't talk about the problems and are doing their utmost to project a false facade how does anyone make accurate judgements as to confidence.

I think you and I have very different definitions of professionalism.

-jef

Weirdness at CentOS

Posted Jul 30, 2009 21:54 UTC (Thu) by sbergman27 (guest, #10767) [Link] (5 responses)

Disclosure is fine and expected. But they could certainly have done without all the "CentOS will die" stuff, Jef.

Plus the fact that all of this was avoidable with even minimal attention to policy. It's not like they haven't had over five years.

Although I must say that it is ironic that you would champion the cause of disclosure. Had any good "infrastructure issues" lately? :-)

Weirdness at CentOS

Posted Jul 30, 2009 22:44 UTC (Thu) by jspaleta (subscriber, #50639) [Link] (1 responses)

The statement regarding the conditions of the death of the project are factually correct. If the developers all walk away...the project folds.
Do you dispute that? It's not even hyperbole...its the truth. It's like stating that if the fusion process in the core of the sun stops..the sun will die. Its the truth. Knowing that, am I afraid that the sun will die anytime soon? I'm pretty confident that it will continue. Is it unprofessional of me as a former fusion researcher to point the fact that the Sun's fusion process is critical?

So here we have a large group of CentOS developers saying that the project depends on developers to continue. If the developers aren't happy and the developers walk away...that would be a significant problem. CentOS is run by volunteers and everyone should keep that in mind...especially those people..like yourself perhaps.. professional who are making a business of leveraging the volunteer work when selling services to customers...without contributing back to that effort.

I use CentOS at work. Am I concerned that the developers are going to enmasse up and quit. Nope. Will CentOS go through a re-branding and a domain name...maybe. The trademarked name CentOS might die but the developers will continue the work apace. And lets face it, CentOS doesn't have the strongest brand in the world. The people who are leveraging it are leveraging it because its compatible with RHEL...it really doesn't matter what its called.

And since you brought it up...I am very much a champion of disclosure. Are you not satisified with the final summary of information Paul put out over events?
https://www.redhat.com/archives/fedora-announce-list/2009...

Are you not satisfied with the follow-up effort to create and publicly publish an incident procedure that will be followed in the future?
http://infrastructure.fedoraproject.org/csi/security-poli...

If the only thing that leaves you unsatisfied is the explanation as to the delayed public disclosure due to an ongoing investigation I'm not going to be able convince you otherwise. You either accept that explanation or you do not. The new incident response procedure should help prevent unnecessary delays if this ever happens again.

-jef

Weirdness at CentOS

Posted Jul 31, 2009 2:16 UTC (Fri) by sbergman27 (guest, #10767) [Link]

"""
If the only thing that leaves you unsatisfied is the explanation as to the delayed public disclosure...
"""

Seven months delayed public disclosure. Sure, they had excuses in hand, carefully worded by Red Hat's PR department and approved by Red Hat Legal. But other distros, Debian for example, don't seem to have that "can't do" attitude exhibited by Fedora under similar circumstances, and don't seem to need the carefully worded excuses. And no, there is really no way to know if what was (finally) disclosed was the full truth. Any presumed confidence one might have had of that was lost sometime between August and March.

"""
...I'm not going to be able convince you otherwise.
"""

As long as there are such clear examples of other distros executing responsible and timely disclosure, in contrast to Fedora's behavior, I'm quite sure you won't. I guess we'll just have to see what happens next time Fedora's infrastructure gets hacked.

Weirdness at CentOS

Posted Jul 31, 2009 9:07 UTC (Fri) by marcH (subscriber, #57642) [Link] (2 responses)

> Disclosure is fine and expected. But they could certainly have done without all the "CentOS will die" stuff, Jef.

You cannot make the headlines without any keyword like "die", "sex", "war",...

Weirdness at CentOS

Posted Aug 1, 2009 9:58 UTC (Sat) by nix (subscriber, #2304) [Link] (1 responses)

It's a good thing LWN isn't a trashy tabloid. We'd see headlines line

UFO War on Fairer Sex --- Developers Driven Screaming into Sea

Weirdness at CentOS

Posted Aug 1, 2009 14:51 UTC (Sat) by sbergman27 (guest, #10767) [Link]

Yeah, what really irked me about that episode was that all the streaming video was in H.264. If even the OSS world defaults to proprietary codecs, does Theora have any chance at all?

Weirdness at CentOS

Posted Jul 30, 2009 21:00 UTC (Thu) by pheldens (guest, #19366) [Link] (1 responses)

It's may not be very elegant, but to break the stalemate, I think seeking publicity is a good move. To prevent this they should have had the funding public in the first place. And rentals in name of the organisation, not individuals.

Weirdness at CentOS

Posted Jul 31, 2009 11:04 UTC (Fri) by tialaramex (subscriber, #21167) [Link]

A reason (I have no close ties to CentOS) for being reluctant to put everything in the name of the organisation is that you can be distracted for a little while and turn around to find that two people you never really liked that much, but who seemed competent, have taken over everything and have decided to "refocus" your Whatever organisation as an outreach group for the cult you didn't know they were both members of.

They have the legal paperwork, the $41 387.18 donated by people who are really into Whatever is now under their control and they just deleted all the Whatever files that you'd promised users would be on the whatever.example site forever to make room for a Flash abomination advertising the cult.

IF you can afford expensive lawyers and IF you didn't make any foolish mistakes like signing something without reading it, you MIGHT be able to undo this sort of thing. But even if you can, your name (as the founder / all round famous guy connected with project Whatever) will be dragged through the mud.

Overall I agree that an organisation is a better choice, but the most important thing is vigilance, everything needs oversight, and it's tough for volunteers to stay interested in stuff like auditing accounts, reading dull corporate paperwork and so on. Of course the CentOS guys didn't even _get_ accounts to audit, but having them and not checking wouldn't be an improvement.

Weirdness at CentOS

Posted Jul 31, 2009 15:48 UTC (Fri) by zaitcev (guest, #761) [Link] (1 responses)

Maybe Lance is just dead. Can happen to anyone. In that case it's kinda too late to berate him. Do we have any evidence that he is not, in fact, dead?

Not dead

Posted Aug 1, 2009 3:53 UTC (Sat) by eru (subscriber, #2753) [Link]

Have a look at what http://www.centos.org/ is currently displaying: "Note: The CentOS Development team had a normal meeting today and Lance Davis was in attendance." Apparently the problem has been resolved.