LWN: Comments on "Wade: Balancing the needs around the CentOS platform"

Wade: Balancing the needs around the CentOS platform

ceplm — Wed, 30 Dec 2020 23:03:06 +0000

At least they know I am not a dog. Hopefully.

Wade: Balancing the needs around the CentOS platform

geert — Wed, 30 Dec 2020 10:11:48 +0000

On the Internet, nobody knows when you are sarcastic (or not ;-)

Wade: Balancing the needs around the CentOS platform

ceplm — Wed, 30 Dec 2020 01:17:14 +0000

And of course Red Hat releases RHEL always exactly on the schedule.

Wade: Balancing the needs around the CentOS platform

jhhaller — Mon, 28 Dec 2020 17:15:53 +0000

The effects on the CentOS SIGs will be interesting. For example, RDO and Ceph SIG have been based on CentOS. In the case of RDO, it required repackaging newer versions of various Python packages which OpenStack required but weren't in CentOS. In theory, these new packages could be added to CentOS Stream, assuming it didn't break some other package which continued to need the older version. It will certainly provide challenges to support for older releases of RDO for any significant period of time. But, I get the feeling that the move to Stream is going to make anyone dependent on any of the SIGs for production workloads move to RHEL and supported versions of the SIG packages, or to work directly with the upstream projects for the SIGs, which will be more work.

Wade: Balancing the needs around the CentOS platform

amacater — Thu, 24 Dec 2020 15:51:02 +0000

LWN has a good institutional memory: I just realised that my comment history on LWN is preserved. For fun, I went back to some of my earliest comments in 2003 round about Red Hat 9: there is an analogue for almost every single comment in the streams round CentOS made back then. Interestingly, I suggested a five year cycle for Red Hat then and was shot down by rahulsundaram

The comments round changing from Red Hat to Debian then almost exactly parallel advice people are asking about today. See https://lwn.net/Articles/56861/ and the comments there.

Wade: Balancing the needs around the CentOS platform

pizza — Wed, 23 Dec 2020 01:54:04 +0000

> Given that the interval between major RHEL releases seems to also be around 5 years

RH has said that future RHEL releases will have a 3-year cadence going forward, so there will presumably be a 2-year overlap where both CentOS Stream N and N+1 are actively supported.

Wade: Balancing the needs around the CentOS platform

motk — Wed, 23 Dec 2020 01:53:21 +0000

I was there as an insider (in a peripheral sense) and this is a correct summation of events.

Wade: Balancing the needs around the CentOS platform

cesarb — Tue, 22 Dec 2020 23:32:25 +0000

...and unfortunately, I just noticed an annoying detail buried in the FAQ (https://wiki.centos.org/FAQ/CentOSStream#What_happens_whe...): unlike RHEL and former CentOS, which has/had 10 years of support, CentOS Stream will have 5 years of support (the "Full Support Phase" is only 5 years, not 10 years).

Given that the interval between major RHEL releases seems to also be around 5 years, this means that there will be only a tiny window between a new CentOS Stream major release and the EOL of the previous one. That is, if an user of CentOS Stream wants to keep receiving updates, they will have to migrate to a new release (with major changes) in a hurry.

Unless this is changed, it will mean CentOS Stream is not a viable option for most use cases (other than beta-testing RHEL). Other distributions with a 5 year lifetime have a longer overlap (for instance, releasing every 2 or 3 years).

Wade: Balancing the needs around the CentOS platform

pizza — Tue, 22 Dec 2020 15:23:27 +0000

> That's a bit extreme. I do think that a lot of people prefer, or even demand, justly or not, that the free software also has to be free as in free beer.

No, you misunderstand me. The overwhelming majority of the users don't care that CentOS (or whatever) is "Free Software". They only care that they are getting high-quality software (and ongoing updates) for free.

They do not care about "Free Software principles." They only care that _they_ don't have to pay for it.

For them, software is an expense that must be minimized, not an investment.

They do not care about long-term "sustainability" or supporting its authors/contributors, because doing so would cost money.

(IMO/IME as a two-decade veteran of this industry and Free Software author/contributor)

> How many of the CentOS users are also contributors to the very same software, tooling, knowledge, ... community in general?

I'd bet the $25 I have in my wallet right now that they are a small fraction of a percent of the overall userbase.

(FWIW, FOSS upstreams don't actually tend to care terribly much about RHEL and its derivatives, as any software they ship is outdated and only gets more so during the EL lifecycle -- if anything, this actually _increases_ the burden for upstreams)

Wade: Balancing the needs around the CentOS platform

farnz — Tue, 22 Dec 2020 13:15:39 +0000

I note a lot of people describing the users of the free CentOS builds as "prospective customers"; one thing that's coming out loud and clear in these discussions is that a significant fraction of the CentOS users who are not willing to switch to CentOS Stream largely object to having to pay (whether by giving Red Hat money, or by comparing the CentOS Stream packages to the RHEL 8 announcements and only letting through the packages they want).

This has me wondering what fraction of the aggrieved CentOS users are actually going to pay for what they want (noting that I count spending time making Rocky Linux a functional thing as paying - it doesn't have to be money spent), and what fraction of them are simply complaining that the nice thing that they got for free is no longer on offer.

Wade: Balancing the needs around the CentOS platform

smurf — Tue, 22 Dec 2020 12:32:28 +0000

> it's a moving target

Literally, with the switch to CentOS Stream.

I'm all right with RedHat no longer supporting the CentOS effort, in general that is. Their choice, and there are free alternatives. What I'm not all right with is cutting off CentOS 8 in a year (i.e. way earlier than CentOS 7). That kind of about-face treatment of their users (prospective customers …) doesn't exactly instill confidence in any other of their promises WRT supporting free software.

Wade: Balancing the needs around the CentOS platform

LightDot — Tue, 22 Dec 2020 11:41:59 +0000

That's a bit extreme. I do think that a lot of people prefer, or even demand, justly or not, that the free software also has to be free as in free beer. But on the other hand, I also think that many would not be running it if it was free as in free beer but not free software.

Red Hat needs to balance the fact that it is them that take other people's free software without payment, release it as an operating system and charge for it. Having a for profit company tightly integrated in the community that is otherwise mostly non-incorporated and mostly nonprofit in various ways, is never going to be without friction.

Red Hat does it by producing parts of the very same software, tooling, knowledge and by contributing all this back to the community. They do it by hiring people from the community. They do it in many different ways. Nobody is denying that. A balancing act of quid pro quo. I find them good at it... well, most of the time.

To simplify, Red Hat does charge its customers but parts of the profits are funneled back into the community. And that is fine.

To take this one step further, I also think that viewing CentOS users as freeloaders, beggars can't be choosers and whatnot, is not only rude but often inaccurate. How many of the CentOS users are also contributors to the very same software, tooling, knowledge, ... community in general? I think Red Hat is well aware that CentOS users deserve to have access to the final product not only because of the licensing requirements but also as a part of the balancing act.

Balance is the key here, and it's a moving target. Periods of imbalance will occur. As long as the Red Hat's relationship with the free software community tends to balance itself, as long as everybody involved strives for it, everything will be fine in the long run.

Wade: Balancing the needs around the CentOS platform

eduperez — Tue, 22 Dec 2020 11:41:14 +0000

In my personal experience, CentOS was just the "gratis" alternative to RHEL: when a vendor came saying "you need a RHEL box to run our software", the answer was always "is CentOS enough?".

Wade: Balancing the needs around the CentOS platform

Wol — Tue, 22 Dec 2020 10:13:16 +0000

Well, I'm not sure of the date, but I started with SuSE 5 back in the 90s I think (my very first distro was Universe linux with kernel 1.3), and SUSE is still my go-to distro that I support for anyone else.

Slackware lives in my toolbox, and my desktop runs gentoo - for probably over 10 years.

Cheers,
Wol

Wade: Balancing the needs around the CentOS platform

tuna — Tue, 22 Dec 2020 09:04:33 +0000

They can probably get discounts from Red Hat as well if they want to negotiate.

Wade: Balancing the needs around the CentOS platform

tuna — Tue, 22 Dec 2020 08:57:36 +0000

Commas and punctuations (, .) where also designed to make text easier to read :)

Anyway, Fedora Workstation works really good for me and for the less computer literate people around me. But you obviously have different experiences.

Wade: Balancing the needs around the CentOS platform

efitton — Tue, 22 Dec 2020 04:56:35 +0000

I am not asking Fedora to do the work, I am pointing out that it makes sense for me to use a distro that already is doing that work.

I am i3 and Cinnamon. Mint or Majora make sense for me. Given that why would I use Fedora? Their choosing to use their resources elsewhere is fine, but it certainly isn't going to draw me there.

Wade: Balancing the needs around the CentOS platform

efitton — Tue, 22 Dec 2020 04:51:14 +0000

Ah. That makes sense but not a lot of twenty year loyalty to a distribution users I wouldn't think. I started on Redhat 5.0 before Fedora existed. But I found Suse to be a much better KDE 3 distro. I very much felt the pain of KDE 4.x and at the time having transitioned to teaching found myself using Linux less and less. A true believer 100% with no dual boot to pretty rarely to be honest.

Anyhow, I have touched Debian, Manjora, and Mint since and it all pretty much reinforces my thinking that you go to the distro of the DE you like. Especially when it is less "default" and more blood, sweat, tears, and testing.

i3 and Cinnamon will never be as polished or supported in Fedora as Mint or Manjora and working in a Windows shop I have zero distro loyalty.

Wade: Balancing the needs around the CentOS platform

pizza — Tue, 22 Dec 2020 04:41:39 +0000

> Why would I want to use a distribution where a breakage in my environment is considered somewhere between irrelevant and not a show stopper?

Because there aren't enough volunteers that care about "your environment" to do the development, integration, testing, and bugfixing necessary to make it something supportable.

> If Fedora offered me some meaningful improvement compared to a different distro that supports other options equally instead of second class citizens than I am sure I could make it work.

There's no magic silver bullet here; "supporting other options equally" requires a _lot_ more work, and there's a (very) finite supply of time and bodies that can be directed at those other options. "Volunteering others to do work" tends to not actually result in success.

Wade: Balancing the needs around the CentOS platform

mathstuf — Tue, 22 Dec 2020 04:27:29 +0000

I started using Linux back when Fedora Core 5 and 6 (installed 5, started using 6 day-to-day when XP decided the motherboard didn't have an Ethernet connection anymore), so 2006-ish. I started out in KDE, kept with it until 4.3 (I installed Fedora 9 with KDE 4.0.3 and the pulseaudio migration during the beta period, so it wasn't a "KDE 4 is such a disappointment" exodus) when my eeePC 700 crashed mid-presentation when it ran out of memory. I started piecing things together in late 2009, early 2010 to get my setup to be consistent across all machines, beefy and portable. Maybe if I had started doing this after Arch's popularity spike, I'd be there, but that's not how it worked out. Gentoo was the "tinkerer's distro" when I was in college and I wasn't interested in the experiences my friends had with making their laptops into portable stoves during `emerge world`.

At this point, my setup has been pretty constant since 2010 after the Xmonad migration. The only big thing since then has been a systemd --user session migration in 2012/2013. Wayland is the next big thing, but the motivation is lacking due to a dearth of "usable" compositors for the way I've trained myself on this setup.

At this point, I'm a Fedora packager (if a mostly absent one, but I also maintain mostly hands-off packages at this point). Fedora has everything I need at my fingertips and copr makes it easy to build things that aren't packaged (mostly due to a lack of time/motivation to do it properly). It's also nice that when I'm on a CentOS machine/container I'm not in a completely alien environment. I am slowly learning dpkg stuff where needed, but it just feels inadequate compared to dnf and Debian polices on how things work are not my favorite anyways; Canonical doesn't make improvements to those either. So I don't know where I'd go if Fedora ended up disappointing me. Probably Arch, but I'll only leave if forced to at this point.

Wade: Balancing the needs around the CentOS platform

efitton — Tue, 22 Dec 2020 04:26:55 +0000

I am sure users would have been disappointed without the ten year promise, but you make point well that the nasty surprise is what makes this particularly ugly.

Wade: Balancing the needs around the CentOS platform

efitton — Tue, 22 Dec 2020 03:12:07 +0000

If you are piecing things together with various tiling window managers, may I ask why you chose Fedora?

Wade: Balancing the needs around the CentOS platform

efitton — Tue, 22 Dec 2020 03:09:45 +0000

Why would I want to use a distribution where a breakage in my environment is considered somewhere between irrelevant and not a show stopper? However breakage in Gnome is all hands on deck. If Fedora offered me some meaningful improvement compared to a different distro that supports other options equally instead of second class citizens than I am sure I could make it work. But the Workstation release is optimized for a Gnome experience. If I was a Gnome user I would likely chose Fedora. But I am not. I see frustration with no benefit.

Wade: Balancing the needs around the CentOS platform

mathstuf — Tue, 22 Dec 2020 02:22:10 +0000

Why would the pre-fabbed environment set not being the Default™ be a reason not to use a distro? I use Fedora and have cobbled together my own "environment" going from ratpoison to Xmonad with a pile of various X-based applications and systemd units to glue it all together. Fedora having a default of Gnome really doesn't matter at all to me and I don't see why it would trouble anyone else since the packaging teams for the various DEs do a great job on the whole.

Wade: Balancing the needs around the CentOS platform

efitton — Tue, 22 Dec 2020 01:57:44 +0000

I wasn't aware that there was a singular upstream desktop. Four successful forks of Gnome along with continued KDE, XFCE, and other DE development along with and the creation of new window managers would very much indicate that it is not.

Frankly my interest in Fedora as a potential user is zero given that any environment I am interested in would be a "spin" at best.

Wade: Balancing the needs around the CentOS platform

pizza — Mon, 21 Dec 2020 23:00:09 +0000

> So, once gain, could the people smearing Red Hat here please tell us how they think our ideals regarding free software should be funded?

They care about "software, for free", not "free software".

(in fairness, most folks/organizations only ever cared for the former)

Wade: Balancing the needs around the CentOS platform

pebolle — Mon, 21 Dec 2020 22:38:55 +0000

> It is more like terabytes of data these days and we are finding fewer and fewer mirrors across the board.

Fun fact: I thought about dialling down the hyperbole in my comment. Silly me! Thanks a ton for your reply!

So, once gain, could the people smearing Red Hat here please tell us how they think our ideals regarding free software should be funded?

Wade: Balancing the needs around the CentOS platform

jccleaver — Mon, 21 Dec 2020 22:34:52 +0000

To some extent you're combining very different things.

RHEL has a paid service where you can stay on a prior point release and just get updates for that. Many RHEL customers take advantage of this until their 3rd party hardware is validated with whatever changed RH put into the point release. Many others just take the updates as they come.

CentOS does not have a concept of staying on a point release, but *that doesn't mean point releases aren't a thing.* If you download an ISO point it at an install tree and don't pull updates, you're getting a set thing. If you mirror a CentOS tree with updates and choose not to pull in CR/Stream now, then you're essentially getting updates only for that branch. EL-derivative users have plenty of ways that they've been handling release management, since they're unable to use upstream RHN tooling... And whatever it is they're doing, it works for them at risk levels they're okay with.

The removal of CentOS Linux forces downstream users into a continuous Stream paradigm which many didn't necessarily want. Those that did want it already had it available to them in the form of CR/Fastrak/Stream repos they could enable at any time. Furthermore, all of the packages going into CentOS Stream are already well past RH QA and are basically just previews of official RHEL releases to come.

There is zero upside for CentOS users in this flip except that now CentOS Stream is a rolling, unstable update set dropping packages randomly whenever something is pushed and now CentOS Linux+Updates doesn't exist.

RedHat management has decided to enforce a dichotomy between paid (or "free, but licensed") support and "rolling preview" and is pulling the rug out of the enormous userbase that lived in the middle ground and, by definition, was comfortable there. And then muddles the message, insults mailing list users, and says to send in emails before going on holiday vacations.

It really is a phenomenal slap in the face, and the reverberations will last long beyond this decision cycle.

Wade: Balancing the needs around the CentOS platform

smoogen — Mon, 21 Dec 2020 22:28:42 +0000

It is more like terabytes of data these days and we are finding fewer and fewer mirrors across the board. Mainly the amount of traffic that all the CI's out there generate acts like a DDOS even with a thousand mirrors spread around the world. Various ISPs and schools find their outbound budget eaten up. They also find out that space is a lot larger these days. As one ISP leaving put it.. well we just thought it would be a couple of gigabytes like the ISO.. not terabytes of data with a lot of churn. Bandwidth and diskspace in servers is not cheap and is the first thing an administrator will chop off if they aren't seeing a 'bonus' from it.

Wade: Balancing the needs around the CentOS platform

johannbg — Mon, 21 Dec 2020 22:23:45 +0000

I'm perfectly aware of what Fedora workstation is supposed to be and here's the thing computers were originally designed to makes peoples lives easier not harder so it does not matter if things work for you the "IT professional that is subscribed to LWN" and can accept,tolerate and workaround bugs from the ground up, the desktop environment needs to work for those that are technology challenge and dont want to spend hours in, either setting up their OS environment or fighting it and that it has always failed to do on Linux these 2 decades I've been using it...

Wade: Balancing the needs around the CentOS platform

pebolle — Mon, 21 Dec 2020 20:49:02 +0000

> And the work for those updates comes primarily from Red Hat.

So true.

I'm getting increasingly agitated by the people basically wanting for free stuff paid for by, in this case, Red Hat. None of them ever mention money. They only mention vague promises, the spirit of a license, or whatever. Apparently they're on the moral high ground here. No need to mention basic facts.

But I have a vague feeling that The Document Foundation is not sure how to improve funding. And the news from Mozilla is far from good, as far as I can tell. (I'm agitated so my view might be a bit off. Feel free to contradict me.)

What happens next if, say, ISP's and universities decide that mirroring gigabytes of free software is not in their interest?

Wade: Balancing the needs around the CentOS platform

tuna — Mon, 21 Dec 2020 20:44:49 +0000

The point of Fedora Workstation is that it is not a "box of things" but an integrated system where everything works. You obviously think it fails at that, but it works really well for me.

Wade: Balancing the needs around the CentOS platform

mbunkus — Mon, 21 Dec 2020 20:09:14 +0000

> So is the level of support your company needs from the OS vendor. (personally we expect none)

And that's untrue, I'd wager. If all you want out of a distro is 10 years of stability, you could install any distro & keep it unchanged (by not installing updates). However, that's most likely not what you're actually doing or what you actually want, which is having 10 years of updates on top of the software changing as little as possible. And the work for those updates comes primarily from Red Hat. You do actually rely on and expect the OS vendor's support, just not for your individual issues — but for security & bug fixes in general.

Wade: Balancing the needs around the CentOS platform

JoeBuck — Mon, 21 Dec 2020 17:04:48 +0000

You would think that would be true. Nevertheless, the large electronic design companies generally run CentOS, not RHEL, and they demand and get significant discounts from the EDA providers.

Wade: Balancing the needs around the CentOS platform

mathstuf — Mon, 21 Dec 2020 15:59:53 +0000

Eh. We use CentOS as an "old" distribution to build Linux packages against a "sufficiently old" glibc for maximum compatibility. The devtoolset packages make using newer compilers with it way easier too. I don't think our CentOS usage will change with this announcement (as Stream works for us). I expect there are others out there that used it as we have used it.

Wade: Balancing the needs around the CentOS platform

wittenberg — Mon, 21 Dec 2020 14:21:22 +0000

It doesn't matter that newer versions are "better" or even "more secure" to a large group of people who don't know much about computers, and don't want to. They want to be able to keep doing things that work for them.
This is even more important to people using Linux in places like medical devices or infrastructure. In the case of infrastructure, they are often building systems that will run for over 20 years, and taking them down in the middle of that time is incredibly disruptive. In the case of medical devices, updating software may require re-accreditation, which is very expensive.

Wade: Balancing the needs around the CentOS platform

smurf — Mon, 21 Dec 2020 13:05:45 +0000

I mostly agree with you, and "ewaste" is indeed a problem given the number of mobile phones produced each year (among a heap more examples), but in the context of this discussion?? AFAIK there are no devices that ship with a not-upgradeable CentOS.

Wade: Balancing the needs around the CentOS platform

smurf — Mon, 21 Dec 2020 13:01:40 +0000

Well you might argue that, as CentOS is free, there's no "customer" in the economic sense of the word.

Wade: Balancing the needs around the CentOS platform

cesarb — Mon, 21 Dec 2020 12:25:23 +0000

> So why did they want that extremely low rate of change? CentOS Stream is not that much higher a rate of change than CentOS 8, after all.

Perhaps because they never looked. I am one of these; I hadn't looked closely at CentOS Stream, but now that I have, at least for my use cases it seems pretty good: it should be basically CentOS without the annoying delay every minor release. There seems to be a bit of an extra risk due to seeing some of the intermediate states between the minor releases, but that should not be much of a problem. And the rate of change is not a problem; CentOS Stream would have smaller jumps, but the end result after X years would be identical between traditional CentOS and CentOS Stream.

I'll probably wait a couple of months (to not rock the boat during the new year holidays and payments) and then switch the few CentOS 8 systems I manage to CentOS Stream. If it works fine, I might recommend that my coworkers do the same. And that also leaves extra time for the migration instructions (at the bottom of https://www.centos.org/centos-stream/) to be enhanced; they already changed once that I've seen (the original instructions didn't have the "dnf swap" step).

Wade: Balancing the needs around the CentOS platform

amacater — Mon, 21 Dec 2020 12:17:28 +0000

That's not quite how it works: every time you apply updates to CentOS from a mirror / Red Hat from Satellite, you upgrade to the latest point release by default. If you really want to install CentOS 7.0 in July 2014 and never upgrade at all you get no security / functionality fixes [and this might be true for an isolated HPC cluster, for example]. If this is what you want - fine - otherwise with the first "yum update" you run and accept, you get later security fixed packages (and kernels, for example, update every few months at least). If you don't want that, you have to run yum update, take a note of the changed packages and install / block them one by one.

If you've got some absolute dependency on another program/kernel driver/piece of certified hardware that mean that you cannot upgrade under any circumstances - specifically, Red Hat will support that, potentially at extra cost, to lock you in to one particular point release. At that point, they will custom fix software updates for you. That's the relevance of the maintenance periods and specialist extended update support for individual point releases.

The likelihood of significant numbers of Red Hat paying customers who have _never_ updated their machines and then called on support is small: the first question from support will likely be "Can you please connect the machine to a Red Hat Satellite server and bring it up to date - thanks". Pretty much every checklist of security guidance suggests keeping your OS patched and up to date and following your distributions notification channel for CVEs and consequences - in some industries it's mandated, but everybody reading this is smart and does it anyway, of course :)