CentOS turbulence and enterprise Linux tradeoffs

CentOS must seem like a dream distribution to many. Its users get the benefit of the massive team of developers that Red Hat has working on the Red Hat Enterprise Linux product without having to pay for any of it. CentOS offers a level of stability that cannot be found in any of the more community-oriented distributions; even Debian Stable requires its users to upgrade more often than CentOS does. Hosting providers have a solid, supported platform to sell to many thousands of customers, and it does not cost them even a single devalued US dollar. Many, many sites depend on CentOS, so anything which threatens the stability of that foundation is certain to raise a number of eyebrows. Unfortunately, that is exactly what happened at the end of July.

CentOS has never been the most transparent of projects; its lists do not carry the kind of open discussion that can be found with Debian, Fedora, or (increasingly) openSUSE. Most CentOS users perhaps worry little about where their software comes from, but there are those who have tried to help the project and bring its workings more into the open. One of those, well-known RPM packager Dag Wieers, threw in the towel in June:

Problems within the project became more public on July 30, when a disturbing open letter was posted on centos.org. The immediate issue was the disappearance of project founder Lance Davis, whose last post on the centos-devel mailing list was in April, 2008. Evidently Lance hadn't been heard from for some time in other parts of the project as well. A missing founder can be a problem, but it gets worse: when Lance vanished from sight, he took with him control over the project's domain name and IRC channels.

Lance also had control over the project's finances. There has been a lot less noise concerning this part of the problem, but the fact remains: nobody seems to know where the money which has flowed into the project (via donations and web advertising) has gone. Quoting Dag Wieers again:

Naturally enough, this issue failed to resolve itself; eventually the other key CentOS contributors were forced to go public with their concerns. The move appears to have been entirely effective: Lance was flushed out from wherever he was hiding and met with the team. Ownership of the domain name has been transferred. The CentOS project appears to be back on track, and, perhaps, headed toward a more democratic mode of operation.

Little is being said about the financial side, beyond this:

So the management of future revenue into the project should be handled in a more open sort of way.

One could argue that CentOS users had little to worry about. In the worst possible scenario, the active CentOS developers could have forked the distribution and moved to a new domain, perhaps without even changing the name of the project. Such a move could certainly be successful. But users who have picked a distribution known for stability might just feel a little concerned about being told to change their repository pointers to a different location run by a group claiming to be the "real" CentOS. A certain amount of disruption would have been guaranteed.

There is a lesson here: use of a distribution like CentOS has its risks. A system running CentOS is relying on the efforts of a relatively small group of volunteers; these volunteers are not obligated to continue to provide support to anybody. The project's governance and processes are on the murky side - even if it looks like things are about to get better. CentOS is fully dependent on Red Hat for security updates, and it necessarily imposes a delay between the release of Red Hat's fix (which discloses any vulnerability which wasn't already in the open) and the availability of a fix for CentOS. For the curious: here is the observed delay time a few recent updates:

Package	Delay (days)
seamonkey	1
bind	1
python	2
tomcat	8
firefox	7
libtiff	7
dhcp	1
httpd	5

Sometimes updates pass through the CentOS system quickly, but other times the performance is not quite as good; the "critical" firefox update languished for a full week.

The point of the above text is not to criticize CentOS: that project has done an outstanding job of providing a highly stable and well-supported distribution to the community for free. How can anybody criticize that? The point, instead, is that there are tradeoffs associated with any distribution choice. A Linux user who feels the need for contractually-assured service backed up by a well-funded support operation and faster security updates would be well advised to consider purchasing support from one of the companies operating in that area.

For those who do not need that level of support, instead, distributions like CentOS provide great value. A more open CentOS looks like it should be able to provide greater value yet. Also encouraging are the suggestions that CentOS could work more closely with Scientific Linux, another RHEL rebuild with very similar goals. All told, there appears to be a good chance that the recent turbulence will lead to a more solidly founded CentOS which will continue to be a firm platform for many thousands of deployed systems well into the future.

---

to post comments

Six of one—and no mention of the other?

Posted Aug 4, 2009 1:30 UTC (Tue) by sladen (guest, #27402) [Link] (1 responses)

Here's my last interaction with Lance Davis, all documented publicly on the {kde,gnome}-uk mailing lists:

• 2008-October/msg00012.html (12:07:45 +0100, question sent)
• 2008-October/msg00013.html (12:16:20 +0100, reply received)

Why did I (as a third party) even have to write this email? ...Because nobody had bothered to ask Lance Davis for what they wanted. Three hours later—unprompted—Lance Davis replied again, with an update that (gratis) booth space was sorted.

Sometimes my belief in human beings as willing communicators able to just ask for what they desire, simply astounds me.

If LWN allows the phrase "flushed out" into an article in the future, I do hope to read at least one quote from both sides—or a note covering why it was not been possible. In short, articles should aim to be balanced summary, because we already read the witch-hunt announcements first-hand, and they didn't need copyediting.

Six of one—and no mention of the other?

Posted Aug 5, 2009 2:24 UTC (Wed) by spiro (guest, #54657) [Link]

> Sometimes my belief in human beings as willing communicators able to just ask for what they desire, simply astounds me.

My impression is that many people consider asking a question confrontational and this politically correct society wants to avoid confrontation more than ever before. Or maybe this was always the case, but the wonders of globalisation have brought us into contact with a wider variety of people.

Sorry about dawdling off topic slightly.. :)

CentOS turbulence and enterprise Linux tradeoffs

Posted Aug 4, 2009 3:55 UTC (Tue) by jordanb (guest, #45668) [Link] (16 responses)

> Its users get the benefit of the massive team of developers that Red Hat
> has working on the Red Hat Enterprise Linux product without having to pay
> for any of it.

> A Linux user who feels the need for contractually-assured service backed
> up by a well-funded support operation and faster security updates would be
> well advised to consider purchasing support from one of the companies
> operating in that area.

Honest to god Corbet, have you figured out a new revenue stream or what? The shilling in this article is so bald-faced it's embarrassing.

I don't use REHL. I had to use Fedora for a while and that experience still has me waking up in the middle of the night in cold sweats. So I have a little bit of trouble seeing what the "massive team of developers" at Red Had provides, besides drama like Dreppering and failing to maintain RPM -- not to mention mysterious outages and apparent security breaches that get muffled.

I've known a few people in the hosting space who use CentOS. From what they've said they use it because:

* They have enough in-house expertise that they don't need the corporate security blanket of a shrinkwraped support agreement.

* But proprietary "enterprise" tools (in the web-hosting space, cPanel, mostly) have standardized on REHL, so you need either REHL or a work-alike like CentOS no matter how much your workers would rather be deploying Debian.

Red Hat has muscled itself into the S&P by becoming "The Linux Company" in the heads of corporate types. Certainly this helps them land contracts with companies who need a support agreement because they don't have sufficient IT on their own, and I imagine Red Hat probably adds value in those relationships (I haven't heard nearly as many bad things about them as I've heard of other such companies, like IBM Global Services).

Another side to that, though, is that whenever a proprietary developer decides to "target Linux," there's no question which distro they're going to pick. They're going to go with REHL. Without groups like CentOS, companies like cPanel would be doing a lot of sales work for Red Hat. In fact, they likely are anyway, but at least those groups with sufficient technical competence can use CentOS to avoid paying a "Red Hat Tax" for a decision made by one of their vendors.

Did he touch a nerve?

Posted Aug 4, 2009 5:23 UTC (Tue) by felixfix (subscriber, #242) [Link] (2 responses)

Such a rabid response over perfectly natural commentary! Did you wake up on the wrong side of the bed or what?

I can't remember ANY response as off base as yours. Shilling is a bizarre complaint in the first place, to consistently mistype "RHEL" doesn't help your cause, and then to just ramble on without saying anything.

You need some coffee or time off or *something*.

Did he touch a nerve?

Posted Aug 4, 2009 18:05 UTC (Tue) by ncm (guest, #165) [Link]

It needed to be said, but it could have been said much more gently, to better effect.

My employer uses CentOS, too, for precisely the same reasons. Companies like Oracle for which no equivalent to CentOS exist, or can, have been obliged to offer low-cost licenses to third-party developers for similar reasons. Red Hat has no need for this because CentOS provides that service.

Security Fix Delays

Posted Aug 5, 2009 3:45 UTC (Wed) by ncm (guest, #165) [Link]

I should note, further, that in my employer's typical use of CentOS, delays in delivery of security updates rarely affect them. That is the nature of third-party development: it just has to work well enough to compile against. It doesn't have to be fast or secure, or to run on modern hardware. The only iron law is that it needs to be link-compatible with the (mutual) customers' RHEL installations.

CentOS turbulence and enterprise Linux tradeoffs

Posted Aug 4, 2009 9:46 UTC (Tue) by epa (subscriber, #39769) [Link]

I don't use REHL. I had to use Fedora for a while and that experience still has me waking up in the middle of the night in cold sweats.

This seems a bit unfair; you do realize that the two distributions have different goals? Perhaps you think that Fedora is so terrible that nobody associated with it could ever be capable of making a reasonable enterprise distribution, but to me that is like thinking that a bad tennis player would automatically be bad at rowing. I would suggest you can't judge the quality of RHEL except by trying it, or at least trying CentOS. (I happen to think Fedora is pretty good at what it does.)

CentOS turbulence and enterprise Linux tradeoffs

Posted Aug 4, 2009 9:55 UTC (Tue) by bvdm (guest, #42755) [Link] (8 responses)

RHAT is pretty well regarded and is in fact the posterboy for being a good open-source corporate citizen. So I don't see what your beef with them is. CentOS will logically also disappear if RHAT ever fails, so what's wrong with ppl paying their dues?

Also, Corbet's observations are entirely valid, whatever minute benefit they may have for RHAT here on LWN.

Sorry, but I don't see how your post is helpful to other readers. Which is kinda the point of posting, right?

CentOS turbulence and enterprise Linux tradeoffs

Posted Aug 4, 2009 14:46 UTC (Tue) by drag (guest, #31333) [Link] (7 responses)

There is still some lingering resentment over Redhat deciding to eliminate free-as-beer ISO image downloads years ago. People thinking that Redhat screwed over it's users and other such nonsense.

I've seen it here and there. The above comment is probably a example of it.

CentOS turbulence and enterprise Linux tradeoffs

Posted Aug 5, 2009 19:31 UTC (Wed) by vonbrand (subscriber, #4458) [Link] (6 responses)

The "free as in beer" Red Hat downloads got replaced by Fedora, mostly a name change at first. The problem is that what "enterprise" users want is radically different from what "hobby" users demand. Trying to please both just gets you into a bind, and infuriates everybody. Current state of affairs is near optimal, AFAICS: For personal use, you have Fedora (rapid development, always the latest gizmo ready to try/use); for non-critical servers you can run CentOS or Scientific Linux (Red Hat goes out of its way to make sure that building a clone distribution is easy by segregating branded stuff into a few easily replaced packages, they distribute full source even where they aren't forced to do so by the packages' license, most stuff developed in-house or even bought from third parties has been GPLed by them); if you need solid support you pay for it via Red Hat Enterprise Linux.

CentOS turbulence and enterprise Linux tradeoffs

Posted Aug 6, 2009 9:29 UTC (Thu) by epa (subscriber, #39769) [Link] (5 responses)

FWIW, I run Fedora on a server and I'm pretty happy with it. I don't need the OS to be 'certified' for Oracle or SAP or other expensive proprietary software. Every six months it takes about a day of my time to upgrade it, mostly spent in testing and in rebuilding any locally modified packages. If I had more than one server to maintain then of course I would need to spend some effort to automate and centralize the administration.

CentOS turbulence and enterprise Linux tradeoffs

Posted Aug 7, 2009 1:28 UTC (Fri) by vonbrand (subscriber, #4458) [Link] (4 responses)

We used to have Fedora on servers too, but the update timing became all wrong for us (smack in the most busy time in the term), so...

CentOS turbulence and enterprise Linux tradeoffs

Posted Aug 8, 2009 11:41 UTC (Sat) by dag- (guest, #30207) [Link] (3 responses)

Even the Fedora project in the FC3 era understood that running their own infrastructure on Fedora was taking more resources than they could spend. That's when they moved to CentOS for their infrastructure instead.

Lots of Fedora developers also admit using CentOS for everything where they need stability or simply don't want to update every X months. There is no controversy anymore. It's the same codebase with a different deployment target (for the same audience).

Within companies you can also see companies mix CentOS and RHEL, depending on the support needs they have for different systems. Business or mission-critical solutions running on RHEL, and testing or development systems running on CentOS.

CentOS turbulence and enterprise Linux tradeoffs

Posted Aug 8, 2009 20:30 UTC (Sat) by nirik (subscriber, #71) [Link] (2 responses)

I could be wrong, but I don't think Fedora infrastructure uses CentOS anywhere, they use
mostly RHEL and a few Fedora machines that need to be fedora for composes, etc.

CentOS turbulence and enterprise Linux tradeoffs

Posted Aug 9, 2009 15:13 UTC (Sun) by dag- (guest, #30207) [Link] (1 responses)

It doesn't matter, my point is still valid.

I said they moved from Fedora to CentOS in the FC3 era. It's possible they moved to RHEL. The point is that the Fedora project is not using Fedora for their servers. Whether it is CentOS or RHEL now is not relevant.

CentOS turbulence and enterprise Linux tradeoffs

Posted Aug 9, 2009 20:01 UTC (Sun) by rahulsundaram (subscriber, #21946) [Link]

Actually, Fedora Project was just using external infrastructure which happened to be running something else at that point but yes, Fedora doesn't claim to be for everyone either.

Massive team of developers

Posted Aug 4, 2009 15:08 UTC (Tue) by man_ls (guest, #15091) [Link] (1 responses)

So I have a little bit of trouble seeing what the "massive team of developers" at Red Had provides

For a start, Red Hat is the biggest employer of kernel developers. For example in 2.6.30: 11% of the changesets and 5% of the lines come from it, not to speak of 42% of signed-off lines.

But as others have pointed out Red Hat is mostly known as the source of CentOS packages and security fixes. It also publishes almost all its work as free software and is a model company in the area. I am a Debian user and prefer it to Red Hat, but there is a lot of value in having a corporate provider. There are a lot of repackagers so nobody is forced to pay Red Hat for support; people do it because they want to. What is your problem?

Massive team of developers

Posted Aug 4, 2009 16:21 UTC (Tue) by eparis123 (guest, #59739) [Link]

It's not only the kernel. They have smart engineers in each level of the OS stack.

Kernel, SELinux, gcc, glibc, DBUS, gnome, pulseaudio, *kit stuff, ...

No one matches them in this, not even Novell.

CentOS turbulence and enterprise Linux tradeoffs

Posted Aug 5, 2009 12:12 UTC (Wed) by nim-nim (subscriber, #34454) [Link]

> In fact, they likely are anyway, but at least those groups with sufficient
> technical competence can use CentOS to avoid paying a "Red Hat Tax" for a
> decision made by one of their vendors.

You can use Centos without Red Hat but you can not create a Centos-like system without someone paying a Red Hat. So the "Red Hat Tax" moniker is stupid. Anyone is free to try to build another distro for ISVs to standardise on (and in fact many tried and continue trying). ISVs do not particularly like Red Hat (some like Oracle and IBM have been hostile to it many times in the past). They chose RHEL as supported platform because they don't want or can not convince their customers a better job could be done more cheaply another way.

In fact, the day customers like you convince ISVs they're ready to manage the OS directly, without passing the cost to ISVs another way (such as requiring them to support XX badly defined Gentoo or Debian versions without increasing their own software prices, or complaining their Linux does not support latest hardware now there is no Red Hat-paid developper to write drivers for it), ISVs will happily dump Red Hat.

People want Oracle or Symantec or insert-isv-name-here to use something else than RHEL, without paying ISVs more, and without explaining how the numerous engineers Red Hat pays to enhance Linux and do support will be replaced. ISVs wisely decide not to embark on this lunacy. TANSTAAFL.

CentOS turbulence and enterprise Linux tradeoffs

Posted Aug 4, 2009 12:02 UTC (Tue) by qg6te2 (guest, #52587) [Link]

A Linux user who feels the need for contractually-assured service backed up by a well-funded support operation and faster security updates would be well advised to consider purchasing support from one of the companies operating in that area.

It can be argued that getting a paid Red Hat service subscription in effect also supports a lot of open source development. RH financially supports a considerable number of engineers working on open source projects, as well as providing infrastructure for some projects. (No, I'm not a RH employee)

CentOS turbulence?

Posted Aug 4, 2009 22:03 UTC (Tue) by mmcgrath (guest, #44906) [Link] (26 responses)

CentOS is a distribution with conflicts and people that don't always agree on everything. Some people identified a risk and took steps to correct it and then did so. I don't see how that is unique to any other distribution. While yes, they did write an open letter which is a bit unorthodox. But it seems to have accomplished exactly what the authors were hoping and did so in very little time.

I guess I don't get what corbet's trying to accomplish here, perhaps this FUD would be more warranted had Lance took CentOS.org and done something irrational with it. Except that never happened. They hit a bump, put a plan together, executed the plan and things are right back on track where most of the public thought things were a couple weeks ago.

FUD?

Posted Aug 4, 2009 23:44 UTC (Tue) by corbet (editor, #1) [Link] (22 responses)

Excuse me, but what, exactly, qualifies as FUD here? Certainly that was not my intent. It seems to me like some people are reading things into this article that I didn't want to be there. I guess that's my fault, as I'm the author, but still...

No, it's the readers at fault, not the writer

Posted Aug 5, 2009 0:05 UTC (Wed) by felixfix (subscriber, #242) [Link] (2 responses)

You can only dumb things down so far before throwing up your hands in disgust. You don't dumb things down at all and that is the joy of LWN. These fools come here expecting a dumbed down site where they can add dumb comments, and you scare them. They are used to being kings of scum ponds and here is crystal clear Colorado spring water, and the only response they can come up with is to piss in it so it looks familiar.

Don't you dare dumb things down :-)

No, it's the readers at fault, not the writer

Posted Aug 5, 2009 2:35 UTC (Wed) by corbet (editor, #1) [Link] (1 responses)

Perhaps I have touched a couple of sensitive toes, but I also really hesitate at calling LWN subscribers "dumb." The joy of writing for this readership is that you all are anything but that. There's been enough unhappiness about this article that I don't think I can write it off as reader dumbness, unfortunately. I don't plan to dumb things down by any stretch, but it's worth trying to understand what happens when I go wrong.

That said, I still don't understand what constitutes "FUD" in this article...

No, it's the readers at fault, not the writer

Posted Aug 6, 2009 14:47 UTC (Thu) by liljencrantz (guest, #28458) [Link]

I find your attitude laudable. I don't find the reporting in this particular article biased or inaccurate, but I am not that informed on the subject, so I wouldn't really know. But your willingness to accept errors on your will definitely help keep you accurate and respected in the future.

FUD?

Posted Aug 5, 2009 2:38 UTC (Wed) by spiro (guest, #54657) [Link] (15 responses)

I wouldn't stress about comments like that. Most people don't completely understand what is going on. Many people *think* they understand, but don't. Some don't even bother to try to understand and just panic irrationally (as seen on the CentOS discussion list).

You've reported the events as well as anyone could have without having behind the scenes knowledge of the troubles the developers have been facing.

Assumptions of FUD, I can only guess, come from reaching conclusions not evidenced in your article.

Demand More.

Posted Aug 5, 2009 3:13 UTC (Wed) by mmcgrath (guest, #44906) [Link] (14 responses)

This article had far too many subjective bits in it. Doing minimal research from mailing lists and public sayings without a single attempt as far as I can tell to interview any parties involved. Referring to CentOS as a "dream" os then describing all the things that went wrong or could have gone wrong with it, talking about lead developers leaving, calling it non transparent without any data to back that up, mentioning that centos takes longer to get updates out then RHEL (ORLY?). It was just irresponsible reporting is all.

If corbet wanted to write an opinion piece, fine. But he threw some data together that made it seem like the foundation of centos was Lance and was crumbling even though it clearly wasn't. Turns out LWN readers aren't dumb after all, some of us even keep in touch with the CentOS devs regularly and saw the open letter for what it was. A successful attempt at change. The core development team set out do so something and accomplished it very quickly. Turns out that's what the story was and it's been missed by every major news outlet that covered it because a success story isn't juicy enough here.

Demand More.

Posted Aug 5, 2009 7:28 UTC (Wed) by bvdm (guest, #42755) [Link] (4 responses)

The posting of the open letter on centos.org was an extreme measure. The content of the letter illustrated a profound crisis and most commentators found it quite shocking.

Compared with how the mainstream IT market flies off their handles whenever something sensational happens at Apple or Microsoft, Corbet's reporting on this has been factual and sober.

Seriously guys, the CentOS guys made a very tough call. The worst fallout was not here on LWN though - not by any measure.

Also, CentOS users have to acknowledge the moral reality that they are enjoying what others are essentially sponsoring (RHEL subscribers). This is their GPL-given right, but the defensiveness in some of the comments here are a bit perplexing when one considers this fact.

Demand More.

Posted Aug 5, 2009 9:14 UTC (Wed) by hppnq (guest, #14462) [Link] (3 responses)

I was a bit surprised by the article, and found it somewhat lacking with respect to relevant facts and insights. Project governance is not an easy topic but I had expected somewhat more ambitious than "If you need serious support, you should pay for serious support". There must be a whole world out there where CentOS plays a significant role as the platform of choice for small IT companies and consultants, whose customers do not really care about the implementation or its name.

Surely, it is understood by CentOS users that there is no certainty the project will exist tomorrow, because of the dependence on the ability of specific people in the community to keep contributing to it. I feel the Free Software way out of forking or continuing a project may have been understressed, and this episode may have been a nice opportunity to look into the viability that option a bit more seriously. Instead, the article looked at the delays in applying fixes, which seems irrelevant.

So I wouldn't share the article's main conclusion, that there is a lesson to be learned here. CentOS is not chosen despite its weaknesses, but because of its strong points.

Demand More.

Posted Aug 5, 2009 14:00 UTC (Wed) by bvdm (guest, #42755) [Link] (2 responses)

I think that you are taking a very narrow view of Corbet's intent with the article.

You seem to imply that Corbet is arguing against CentOS's legitimate right or potential to continue to exist. My reading is that he is simply commenting on the validity of Red Hat's business model of offering payed support. And backing it up with real numbers. Timely updates is quite important for many security-sensitive businesses.

"If you need serious support, you should pay for serious support" is an entirely valid statement. You seem to have skipped over the word "serious". It's right there, twice! :) Cost-conscious companies out there that are yet paying for RHEL support obviously have different needs from that of most CentOS users.

Your presupposition seems to be that free software is more than pro-freedom; that it is necessarily anti-commercial. I am confident that even RMS will disagree with you on this.

Demand More.

Posted Aug 5, 2009 20:23 UTC (Wed) by hppnq (guest, #14462) [Link] (1 responses)

Ehhh... Perhaps I was not clear and you misunderstood what I wrote? It was me who put the "serious" there twice!

Demand More.

Posted Aug 6, 2009 5:25 UTC (Thu) by bvdm (guest, #42755) [Link]

No you were not clear. When you put that phrase in quotation marks it had every appearance of being a direct quote.

What's the problem?

Posted Aug 5, 2009 9:49 UTC (Wed) by Cato (guest, #7643) [Link] (5 responses)

I don't see where the subjective parts are - this was simply pulling together some publicly available comments with a helpful and considered analysis of the tradeoffs of the 'RHEL rebuild' approach used by CentOS. I don't agree that LWN should have to interview people on articles like this - maybe a good idea as a followup, but the CentOS team were blogging publicly anyway so I don't see the need.

It was pretty clear from the article and the links that CentOS itself was not crumbling, but did go through a shaky patch that might have required a new website and that users change their repository pointers. If LWN can't report and analyse events like this, what can it cover?

There was some data about CentOS being slower than RHEL in delivering updates - since it depends entirely on RHEL updates this can hardly be a surprise, but it's useful to have this data compiled.

People seem upset by an implicit criticism of CentOS in the article, but it was simply looking at the drawbacks of a community run project, and highlighting some of the weaknesses in project organisation that it seems are being rapidly fixed.

What's the problem?

Posted Aug 5, 2009 21:55 UTC (Wed) by jspaleta (subscriber, #50639) [Link] (4 responses)

"I don't agree that LWN should have to interview people on articles like this - maybe a good idea as a followup"

I think this points to a general problem with the technical laypress that goes well beyond the reporting here. The technical laypress lacks well established journalist standards, nor does its readership expect them to apply any. Technical articles tend to blend fact based reporting and editorial content with no effort to distinguish either.

But on to the point about the need for personal interviews. They are important if your goal is to provide a clear summary of events or is meant to be constructive analysis. Any time you plan to "make news" with an interpretation of a set of public record events its a reasonably good idea to talk to as many of the individuals involved. Communications have context and the assumption that you can cobble together snippets of public record communications and paint an approximate picture of reality may not actually be adequate. It's like charting a course through icebergs just by relying on spotting the tips of the icebergs without knowing where the ice is beneath the waterline.

At the very least LWN could give the key individuals in this article like Dag Wieers a free subscription (if they don't already have one) and an opportunity to rebut any editorial content the Corbet put in to fill the gaps they feel was inappropriate.

-jef

What's the problem?

Posted Aug 5, 2009 22:42 UTC (Wed) by jake (editor, #205) [Link] (2 responses)

> Communications have context and the assumption that you can cobble
> together snippets of public record communications and paint an
> approximate picture of reality may not actually be adequate.

While I think there is some truth to this, quotes are certainly not a panacea. Humans can't really help themselves from "spinning" things the way they think they should be, as opposed to the way they are (or *were*), especially when they know they are being quoted. Quotes *can* add to a story, but don't always. The public record of what went on is often much more enlightening.

> At the very least LWN could give the key individuals in this article
> like Dag Wieers a free subscription (if they don't already have one)
> and an opportunity to rebut any editorial content the Corbet put in
> to fill the gaps they feel was inappropriate.

While that might not be a bad idea, there is a far simpler solution. We provide a means for subscribers to send a link to anyone they might wish to. That way, a non-subscriber who is involved in a particular article can read and comment if they so desire.

jake

What's the problem?

Posted Aug 6, 2009 0:03 UTC (Thu) by jspaleta (subscriber, #50639) [Link] (1 responses)

Everyone spins...journalists...historians...person-on-the-street...everyone has bias.

There was a point in time, before the 24-hour news cycle, when journalists were trained to try to demarcate the boundary of subjective bias from verifiable fact. A lot of journalistic content nowadays does a very poor job of keeping those concepts separate. This article is most likely mediocre in that regard. But that being said, it's a bit hypocritical to imply that its not worth the effort to contact individuals for comment because of the bias such comments might inject.

A journalist's bias in cherry picking from the public record is no better than anyone else's, if anything its less meaningful and more damaging than other sources of bias. Well trained journalists hedge against their own bias by making sure individuals in a story get a chance to comment. It's something old-fashioned newspaper readers expected..it's something new-media readership seems to no longer value. So the readership is as much to blame for the general quality of the technical laypress reporting as the reporters are. It's unfair to reasonably expect journalistic content to rise above established expectations on what qualifies as newsworthy or informative.

-jef

What's the problem?

Posted Aug 6, 2009 0:42 UTC (Thu) by jake (editor, #205) [Link]

> A journalist's bias in cherry picking from the public record is no
> better than anyone else's, if anything its less meaningful and more
> damaging than other sources of bias. Well trained journalists hedge
> against their own bias by making sure individuals in a story get a
> chance to comment.

Hmm, I don't think I can agree with that. A journalist's job is to try to portray an accurate picture of the events and issues at hand. The reader's job is to decide whether they believe that, on the whole, a particular journalist generally does that. If not, the journalist loses credibility and readers, perhaps eventually their job as well. Then, perhaps, they go into talk radio :)

More seriously, it is up to the journalist to determine what the right tools are for the job at hand. And, again, for readers to judge them on those choices.

In a 100-post thread, who should be contacted for comments? The fact is, they had a chance to comment, and did.

Everyone certainly has biases, but the journalistic tradition that quotes must always be sought from those engaged in the debate certainly has its limitations as well. Just the choice of who to ask for comments injects a bias into things. Bias can't be escaped.

I guess I just don't see the quote issue as black and white as you seem to.

jake

What's the problem?

Posted Aug 11, 2009 12:25 UTC (Tue) by jschrod (subscriber, #1646) [Link]

Umm, you recognize that Dag is commenting in this thread, do you? (account name "dag-") And he doesn't see Jon's report as FUD.

FWIW: I very seldomly use CentOS, but as an independent and infrequent user, I couldn't read any FUD into that report either. For me it read as: "Seems there were some turbulences that were taken care of, as in so many projects."

Demand More.

Posted Aug 6, 2009 9:58 UTC (Thu) by seyman (subscriber, #1172) [Link] (1 responses)

> mentioning that centos takes longer to get updates out then RHEL (ORLY?)

From what I understand, Centos takes the updates that RHEL releases and rebuilds them on Centos before releasing them. With this method, I don't see any possible way they could release updates faster than RHEL.

Am I missing something?

Demand More.

Posted Aug 7, 2009 14:48 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]

Nobody expects it to be faster. There is a substantial difference in how much the delay is between rebuilds like Scientific Linux and CentOS however and that is what the conversation is all about.

Demand More.

Posted Aug 10, 2009 18:54 UTC (Mon) by mcopple (subscriber, #2920) [Link]

One cannot report on events objectively, at least not on events that are themselves highly subjective. If all he is to do is dispassionately parrot the comments of others, with no interpretation or comment, then he has no reason to write an article at all -- we could simply read the source material for ourselves.

Yes, he primarily used e-mail traffic and public statements as his source material. However, they were still primary sources, and probably more indicative of how the participants were feeling at the time than a post-event interview would be.

This story clearly has many more angles than the one Corbet chose to focus on, but for the sake of readability, he can only focus on one at a time. I think the CentOS experience would make great source material for a more in-depth article on how volunteer distributions respond to change. One could look at several different distributions that have faced similar challenges over the years -- Gentoo, Mandrake/Mandriva, etc.

FUD?

Posted Aug 8, 2009 18:15 UTC (Sat) by dag- (guest, #30207) [Link] (2 responses)

I don't think there's any FUD in the article. There are concerns about the project and you need to understand the difference between CentOS and RHEL. What you get and what you don't get.

I've done many CentOS presentations promoting CentOS on the basis that there are some benefits with CentOS, but there are also drawbacks compared to RHEL and there is no point to not put them on the table when you want to convince people.

The last thing you want is users that are disappointed afterwards because of what was promised. You can find the presentation slides (including notes) of the presentation on the CentOS wiki at:

http://wiki.centos.org/Events/Presentations
http://wiki.centos.org/Events/Presentations?action=Attach...

FUD?

Posted Aug 8, 2009 21:36 UTC (Sat) by mmcgrath (guest, #44906) [Link] (1 responses)

> I don't think there's any FUD in the article.

Just so we're clear. After reading this article you feel no fear, uncertainty or doubt about using CentOS or the future of CentOS?

-Mike

FUD?

Posted Aug 9, 2009 15:05 UTC (Sun) by dag- (guest, #30207) [Link]

You mean, FUD as in a marketing or political strategy ?

http://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt

You are probably right that the article reveals some uncertainty wrt. the CentOS project as it currently is organized. And sure the delay in updates is far from ideal for certain users. Maybe this information is new to some users, but that doesn't mean it shouldn't be reported on.

But I don't think there is any strategy behind it, I'd prefer a more positive article myself about CentOS. But in the light of things I agree with the article. It's up to the project to improve what can be improved and clearly communicate what is inherent to the project. I prefer users can make an honest decision with all the facts laid out before them.

And that's what I tried to do with my CentOS presentations. It's not all good news. But that doesn't mean it is "FUD".

PS If it is unclear, I am "Dag Wieers" from the article

CentOS turbulence?

Posted Aug 10, 2009 18:14 UTC (Mon) by mcopple (subscriber, #2920) [Link] (2 responses)

FUD? Hardly.

Reporting facts -- which is exactly what Jon did -- is not FUD. It is journalism. And unlike the spin doctors of Corporate America, Corbet stuck to the facts, and drew logical conclusions from them.

I only wish we could clone him and his staff so we could get even more articles!

CentOS turbulence?

Posted Aug 10, 2009 19:50 UTC (Mon) by mmcgrath (guest, #44906) [Link] (1 responses)

> and drew logical conclusions from them.

That's funny, I thought it was my job as the reader to draw conclusions... not journalists. Editorials and opinion pieces draw conclusions. Perhaps my standards were set a bit too high for this piece.

CentOS turbulence?

Posted Aug 13, 2009 6:46 UTC (Thu) by botsie (guest, #1485) [Link]

> That's funny, I thought it was my job as the reader to draw conclusions... not journalists.
> Editorials and opinion pieces draw conclusions. Perhaps my standards were set a bit too high for
> this piece.

Traditionally, the front page of LWN is the editorial page. I respect Jon's opinion and I'm interested
in reading it -- even if I don't agree with it.