Planning the CentOS 8 endgame [LWN.net]

By Jake Edge
July 14, 2021

CentOS 8 is reaching its end of life (EOL) at the end of 2021, though it was originally slated to be supported until 2029. That change was announced last December, but it may still come as a surprise to some, perhaps many, of the users of the distribution. While the systems running CentOS 8 will continue to do so, early next year they will stop getting security (and other) updates. The CentOS project sees CentOS Stream as a viable alternative, but users may not agree—should the project simply leave CentOS 8 systems as ticking time bombs in 2022 and beyond?

A discussion of the CentOS 8 EOL was kicked off by Rich Bowen in a post to the CentOS-devel mailing list. He noted that there will be more questions about the EOL process as that date approaches, so he wants "to make sure we have clear documentation, prominently displayed, that sets expectations". He outlined the process of archiving CentOS 8 to vault.centos.org and wondered if there were changes that should be made because this particular EOL event is rather different.

Alex Iribarren was concerned about "pulling the plug" right on December 31 and suggested that "an extra month or so would be nice, particularly given the holiday period". CentOS release manager Johnny Hughes said that the holiday will delay the switch a bit because people will not be working on those days, but that security updates will not happen past that point in any case. While the dates are still fluid, he described the plan in some detail:

[...] Our goal is, so long as RHEL 8.5 releases before 31 DEC 2021, we will get the files from 8.5 released before we remove CentOS Linux 8 from the mirrors. We will not be adding any updates from RHEL source code released after 31 DEC 2021 to CentOS Linux 8.
This release will be put into at least vault.centos.org/8.5.xxxx/ (where xxxx is the date). Of course, if the RHEL 8.5 release happens after 01 Jan 2022, we would not be doing that release in CentOS Linux 8.
New items (to CentOS Stream 8) will be being built and still going into CentOS Stream 8 after 01 JAN 2022 until CentOS Stream 8 EOLs 5 years after the RHEL 8 release (EOL is 31 May 2024).

But Carl George wondered if a more radical plan was in order. He believes that Stream is effectively the continuation of CentOS 8, so maybe "we should have mirrorlist.centos.org respond to requests for 8 repos with 8-stream repos, which effectively converts any remaining CentOS Linux 8 systems to CentOS Stream 8". That could either be done at EOL time or, perhaps, one to three months later. The third alternative he presented is the status quo—no more security updates—but he is worried that there are quite a number of people who are unaware of the EOL.

Multiple commenters in the thread seemed to agree that switching to CentOS Stream 8 is the right path forward, at least on a technical level, but there are some obvious concerns with that approach. Bowen put it this way:

While I agree with you, that it's an upgrade, I anticipate that moving people from CentOS Linux 8 to CentOS Stream 8 automatically will result in a lot of backlash from people who continue to believe that Stream is a alpha/beta/testing/buggy/unstable/[pick your favorite complaint] distribution. Y'know, once they noticed that it had happened.
Surprising users seldom goes well, even if it's an overall positive surprise.

Stephen John Smoogen thinks that auto-switching systems to CentOS Stream 8 "ends up with lawsuits and very very angry people". There may well be CentOS 8 systems controlling safety-critical infrastructure, even though that may not be a particularly smart choice for those types of systems. Given that, he does not see either of George's switching options working out "no matter how well it is messaged or done". Looking at the statistics shows lots of new CentOS systems since the announcement of the EOL change; the alternatives (e.g. AlmaLinux, Rocky Linux) are not even close to catching up, but:

While many of the current 450k CentOS 8 systems may function well on CentOS Stream, we don't know which ones are just web servers in some advertising farm and which ones are controlling the flow rates on a dam or petroleum flows at a refinery in Texas.

Fabian Arrotin split CentOS users into two categories; the first is paying attention to the announcement and making plans, while the second is not. For the former, neither of George's auto-switch options would affect them; they have presumably already arranged a switch: to Stream, one of the alternative, compatible distributions mentioned above, or to some other distribution entirely. But the latter group, whose systems will have more and more known vulnerabilities over time, should just be forced into CentOS Stream 8, he said. There are, after all, users still running even older EOL versions of CentOS:

I admit that if you deploy CentOS, a good sysadmin would be up2date with what happens in the distro land *but* also by looking at the number of mirrorlist requests even for CentOS 5, I can tell you for sure that some don't ..... (ouch).

Julien Pivotto said that there are two good reasons to auto-switch users to CentOS Stream 8:

Providing stream as a continuity of 8 is the best thing to do, to show that we are confident and that the whole "stream fits most of the CentOS use case". It also has the side effect of better protecting the internet.
[...] I think that it will not backfire if we announce it soon.

Safety of the internet was also on Leif Madsen's mind. He suggested adding a "security updates only" mode that CentOS 8 systems would switch to at EOL. It is "what a good netizen would do", rather than "leave 450k+ systems idling on the internet just waiting to be scooped up into a bot net".

But there are practical considerations with switching a bunch of systems to CentOS Stream 8 at once, as Phil Perry pointed out. "Whilst I completely understand the desire to take this approach", users voluntarily switching are already creating an extra support load, in part because they are finding that kernel module packages created for the CentOS 8 kernel no longer work on CentOS Stream 8. "If you switch all C8 users en mass at EOL, you run the risk of creating a potentially huge support burden that we are simply not in a position to manage."

Josh Boyer agreed:

There are too many situations like the kernel, or internal policy compliance, or other reasons make automated migration problematic. We should encourage and advocate for people to switch to CentOS Stream, and focus on making tools and documentation for that easy to use and easily accessible, but we should not be doing that kind of migration unilaterally.

It may in fact be better for some users to move on from CentOS if they are not interested in what CentOS Stream is offering, Smoogen said. CentOS was a drop-in replacement for Red Hat Enterprise Linux (RHEL), but CentOS Stream is not that:

CentOS Stream is about building a co-operative relationship between the consumers and the distribution where what the distribution builds is evaluated and feedback is given. If a consumer is expecting never to have problems, to never have to file/track a bugzilla or ever even check to see what is being delivered.. then CentOS Stream is not the distribution for them. Because even if it doesn't look like it, there is a very very large gap between 'never' and 'once in a while' that might happen with Stream. There is a need for co-operation between the consumer of stream and the makers to get things right. If you do not have time, energy, or want to do that, then Stream is going to be a constant thorn. The consumer in that case would be better off with another rebuild.

But George disagreed strongly with that characterization of CentOS Stream; multiple people have told him that they switched and never noticed any difference. While it is recommended that users participate in the process, he said, it is not required by any means:

CS8 hasn't been a constant thorn for them. I'm not claiming it's been perfect, there have certainly been regressions, but they are fixed faster than they ever were in CL8 (or previous major versions) and I strongly feel that most users will be best served by getting switched to CS8 at or just after the CL8 EOL.

But Leon Fauster said that when he tried switching some systems over to CentOS Stream 8, he encountered problems. Applications from third-party repositories (e.g. RPM Fusion) stopped working because their dependencies were not available. Installations that are more complex run the risk of failing to switch to CentOS Stream successfully. For his purposes, complex simply means that the system "uses more [than] just CentOS artifacts (ISV/proprietary software, 3rd/custom repos, etc.)".

Whatever the technical (and internet-safety) merits of forcibly switching systems to CentOS Stream 8, it is a little hard to see a company like Red Hat taking that risk. As unfortunate as it may be for some inattentive users (and, perhaps, the rest of us on the internet at large), it is much safer to simply point to the EOL announcement as a defense against any claims of fault for insecure CentOS 8 systems in 2022—and beyond. It is worrisome that these systems will be out there spamming (and worse), but it does not seem any more so than systems still looking for updates to CentOS 5, which was released in 2007 and stopped getting updates in 2017.

to post comments

Planning the CentOS 8 endgame

Posted Jul 14, 2021 22:48 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

> But Carl George wondered if a more radical plan was in order. He believes that Stream is effectively the continuation of CentOS 8

Wow, the corporate is strong in this one.

Planning the CentOS 8 endgame

Posted Jul 14, 2021 23:06 UTC (Wed) by mattdm (subscriber, #18) [Link] (4 responses)

LOL. Obviously, you've never met Carl.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 8:51 UTC (Thu) by smoogen (subscriber, #97) [Link] (3 responses)

I honestly have no idea what 'corporate' means to various people. It seems to be an easy to go vilification without having to think about it.

I disagree with Carl on this, but I also realize he is coming from a place where he is trying to make the best of what he was told to get done. Carl is a very dedicated and gung-ho person who will try to do the best for the most people with what he has available. He was gung-ho for years to make IUS as best a third-party repository of software that he could. And now he is 120% dedicated to CentOS Stream.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 9:49 UTC (Thu) by Wol (subscriber, #4433) [Link] (2 responses)

Especially here, but typically, "corporate" is usually used to mean the hell of responsibility without authority.

It comes up elsewhere - people who eg cannot move off RHEL 6 !!! because of decisions made by someone else which they have no power to change.

Cheers,
Wol

Planning the CentOS 8 endgame

Posted Jul 15, 2021 14:17 UTC (Thu) by mattdm (subscriber, #18) [Link] (1 responses)

Off topic, but oh yeah RHEL 6!

We see about a million EL 6 systems looking for EPEL updates every day -- that's twice the number of EL 8 systems (and half of EL 7).

Planning the CentOS 8 endgame

Posted Jul 19, 2021 1:25 UTC (Mon) by raven667 (subscriber, #5198) [Link]

I mean, I can relate, we replaced our main application and shared shell account server running RHEL5 on a 10yr old Dell R910 in 2020 to an RHEL7 host because we were finally forced to, the old one worked just fine and did everything we needed out of it, so the replacement which had been on the radar for 5 years finally became a priority. Our userbase is a little old school though, our main tools were developed in perl, ksh, make, RCS in Solaris long ago and tools like git or JSON are new and foreign :-)

Planning the CentOS 8 endgame

Posted Jul 15, 2021 2:27 UTC (Thu) by dowdle (subscriber, #659) [Link] (8 responses)

They should just move everything to vault and when updates quit working and give errors, one would hope admins would notice and start down the path of learning the reality of the EOL. Then they can decide what to do. If they don't notice broken updates, then there isn't much you can do to wake them up.

We see people stick to whatever version they originally installed and never do any updates... and then ask for help installing something on a dot release that is horribly out of date. Again, not much you can do for those folks. They seem to feel put out just doing updates in the first place.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 3:07 UTC (Thu) by cpanceac (guest, #80967) [Link] (4 responses)

The system (${yum/dnf update/upgrade/etc}) should reply with a meaningful message, similar to this: "Starting with ${Some Date} we are no longer providing updates to ${Current Centos 8 version}. You may choose to switch to Centos Stream instead to continue receiving updates. Find detailed info and instructions at ${This Web Page}"

Planning the CentOS 8 endgame

Posted Jul 15, 2021 13:34 UTC (Thu) by jcpunk (subscriber, #95796) [Link] (3 responses)

That feature doesn't currently exist in dnf.

It sounds neat though, could you open a feature request with those folks? I'd be a bit curious how they'd suggest securing it against a malicious mirror....

Planning the CentOS 8 endgame

Posted Jul 15, 2021 14:38 UTC (Thu) by mattdm (subscriber, #18) [Link]

This actually seems like a nice feature for EOM Fedora releases too.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 18:39 UTC (Thu) by mbunkus (subscriber, #87248) [Link] (1 responses)

Just require such messages to be signed with one of the GPG keys that dnf trusts for package installation. Index it via repodata, which itself is signed the same way, if I'm not mistaken.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 20:12 UTC (Thu) by mattdm (subscriber, #18) [Link]

Or simply put the message in a DNF plugin that is in a signed RPM. As a one-off, no need to develop an elaborate system.

Planning the CentOS 8 endgame

Posted Jul 17, 2021 2:42 UTC (Sat) by gdt (subscriber, #6284) [Link] (2 responses)

I once proposed that the default route be dropped (and require sysadmin action to restore) when an operating system version moved beyond its planned date of obsolescence. Whilst I still believe that would do a lot to reduce the level of misuse of the internet, it's not particularly useful in this case, where the planned date of obsolescence was moved earlier.

What is interesting to me is the lack of interest by government regulators in this proposed abandonment of security maintenance of a large installed base, for no reason other than corporate strategy. That's an all-too-frequent happening, and the sort of market failure which regulators exist to solve.

Planning the CentOS 8 endgame

Posted Jul 18, 2021 16:10 UTC (Sun) by pas (guest, #126462) [Link] (1 responses)

> lack of interest by government regulators [...] sort of market failure which regulators exist to solve.

For the free version of a product? That always came with absolutely no warranty or other guarantees? This is (and always was) a known risk. Picking CentOS was always the cheap way of dealing with vendor and long term support risk. Now folks are free to buy a subscription from RH/IBM.

If states/governments want to solve this market inefficiency, they are also free to contribute to https://www.cip-project.org/ .

Planning the CentOS 8 endgame

Posted Jul 22, 2021 8:34 UTC (Thu) by gdt (subscriber, #6284) [Link]

Yes, for a free product. Government regulates other free products for community safety, such as meals served by community food programs.

I don't understand your comment about a lack of warranty in the license. The public is not a party to that license, and so warranty or otherwise with another party is irrelevant to the public who bear the cost of future misuse of this unmaintained system.

At the present both regulation and regulators are not familiar with this particular form of pollution, and don't yet have a regulatory response. This episode is yet another example of the need for cyber security regulators to lift their eyes beyond government and military systems. As cyber security regulation stands today, IBM Red Hat will not even pay the costs of the negative externalities which will fall upon the non-contracting parties.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 4:35 UTC (Thu) by polyergic (guest, #153186) [Link] (7 responses)

I'm having a hard time believing they're even considering this given the huge backlash from the initial EOL announcement. It's unfortunate for security that users are bad at taking care of themselves, but part of the reason people chose CentOS is precisely to avoid the OS doing stuff like auto-updates and surprise migrations. I'm having trouble figuring out any way they could make this go over well.

Is there any (good) precedent for this?

Planning the CentOS 8 endgame

Posted Jul 15, 2021 10:38 UTC (Thu) by lsl (subscriber, #86508) [Link] (4 responses)

CentOS was always rolling between point releases. I get the feeling that, with some exceptions, most of the outraged users don't seem to understand the situation they're raging about.

If it's ok to provide rolling updates from 8.0 to 8.5, why is moving to Stream suddenly so outrageous?

I agree that it probably wouldn't end well but not for technical reasons.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 12:33 UTC (Thu) by hkario (subscriber, #94864) [Link] (3 responses)

The rolling that was happening in CentOS 8 was between a Red Hat QAd versions of packages and Red Hat QAd versions of packages. CentOS Stream packages didn't go through full Red Hat QA process.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 13:37 UTC (Thu) by jcpunk (subscriber, #95796) [Link] (2 responses)

> CentOS Stream packages didn't go through full Red Hat QA process.

Based on my understanding of "CentOS Stream CI: current state and future plans" from the May 2021 CentOS Dojo, I believe the Stream packages complete the RH Internal QA.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 16:05 UTC (Thu) by zdzichu (subscriber, #17118) [Link] (1 responses)

So why those packages aren't in RHEL? There must be something missing, or maybe a bugs presents. Centos freeloaders are not happy with packages "good enough for centos stream" but "not good enough for RHEL".

Planning the CentOS 8 endgame

Posted Jul 15, 2021 19:35 UTC (Thu) by jcpunk (subscriber, #95796) [Link]

Those packages are on the way into RHEL. RHEL patches are published monthly. Stream is published now.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 17:43 UTC (Thu) by rbowen (guest, #90479) [Link] (1 responses)

FWIW, in the board meeting yesterday we discussed this - automatically migrating people to CentOS Stream at the EOL - and we were unanimously agreed that this would be an unwelcome surprise to users. We won't be doing that.

(I'm the CentOS Community Manager, and chaired the meeting yesterday. Watch the centos-devel mailing list over the coming days for a draft of our detailed plan, including the above point.)

Planning the CentOS 8 endgame

Posted Jul 16, 2021 22:04 UTC (Fri) by polyergic (guest, #153186) [Link]

I'd say "good to hear" except I do sympathize a lot with the security and continuity angle and realize you guys are in a tough spot with existing community perception.

> Surprising users seldom goes well, even if it's an overall positive surprise.

Yeah. Just yeah.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 8:36 UTC (Thu) by taladar (subscriber, #68407) [Link] (1 responses)

> I admit that if you deploy CentOS, a good sysadmin would be up2date with what happens in the distro land *but* also by looking at the number of mirrorlist requests even for CentOS 5, I can tell you for sure that some don't ..... (ouch).

I think there are still two groups in there. Those that aren't aware that the version is EOL and those that are forced into this situation by others (customers, decision makers in their own company).

For the latter it would be useful if distros and the security community could provide more non-technical material (ideally translated into many common natural languages) to explain why running a version that is past EOL is a bad idea, giving examples of scenarios that can happen, their likelihood and the likelihood of being able to fix them while a system is still supported and while it isn't any more,...

Planning the CentOS 8 endgame

Posted Jul 15, 2021 9:44 UTC (Thu) by MortenSickel (subscriber, #3238) [Link]

Being a representative of the latter " those that are forced into this situation by others " - We are for some of our contract work dependent on some software developed by the contracter. That software is still stuck on CentOS 6, although we are promised that an update to CentOS 7 soon is to be released, so we might during the next months be able to be back on a supported version on all our machines.

We have tried to get a roadmap from the contractor of what they are planning to do when CentOS 7 is EOL, but so far we have not gotten and answer to that.

So we are at the moment stuck on CentOS 6 for some machines and hope that they find out how to move further on - if they manage to get on CentOS stream, jump to redhat or one of the other redhat clones or switch to another distribution entirely - in due time before CentOS 7 is EOL.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 9:14 UTC (Thu) by cortana (subscriber, #24596) [Link]

Maaaaybe Red Hat should have figured this out before announcing one day out of the blue that CentOS 8 would be discontinued 8 years early..!

Planning the CentOS 8 endgame

Posted Jul 15, 2021 10:19 UTC (Thu) by amacater (subscriber, #790) [Link] (16 responses)

The situation round CentOS 8 is interesting. I've been following events at CERN where it looks as if they have 44k systems
to sort out - pretty much all on CentOS.

I hope for the CentOS folks sake that, if Red Hat produce 8.5 in about November, that they'll at least be able to package that.
There appear to be at least two CentOS/clone communities out there: one of which assumes that CentOS successors will have to march to the beat of the Red Hat drum and provide updates for whatever is possible for however long is possible, the other of which asserts that they'll provide the traditional 10 year support even if Red Hat don't.

I'm listening out on IRC to the channels for each of Red Hat/CentOS/Almalinux/Rocky Linux: the same people are in several of them which is interesting. For myself, I think Red Hat fired the first footgun when they cancelled CentOS 8 December last, the second footgun has been the continuing lack of clarity round the continuty of the CentOS project within Red Hat and the personnel working on it [What exactly happens to the SIGs / what's the overlap with Fedora] and the third footgun will be the stability of Streams for those people who get caught out.

For me (as a long time Debian user at home and RHEL/CentOS user at work), I've always been amazed that there was no procedure for stable updates between major versions for CentOS and none for Red Hat until very recently: the advice was always "wipe and reinstall" On IRC, I was advised that there would be no way to move from CentOS Streams to RHEL.

For security's sake, I'd almost advocate moving CentOS 8 to vault.centos.org and then removing it from every mirror worldwide and putting a big blanket web announcement on the top of every webpage on every mirror or making the last yum update install a script to change /etc/issue or release version or whatever.

If I were a web hosting company / CERN, I'd be panicking like mad right now.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 14:43 UTC (Thu) by mattdm (subscriber, #18) [Link] (9 responses)

> continuing lack of clarity round the continuty of the CentOS project within Red Hat and the personnel working on it [What exactly happens to the SIGs / what's the overlap with Fedora]

Sooooo, I can't help with the rest very much, but I think I can probably provide clarity here (because the situation seems perfectly clear to me). What would you like to know?

Planning the CentOS 8 endgame

Posted Jul 15, 2021 15:12 UTC (Thu) by amacater (subscriber, #790) [Link] (8 responses)

Thanks mattdm, you're very kind.

2021-31-12: CentOS 8.5 ceases to be updated. There is no CentOS 9 or ongoing.

A core of CentOS people (primarily IBM/Red Hat employees) continue to work on CentOS Streams 8 / 9 ... which form part of RHEL 8.6 ..., RHEL 9 and so on.
[Rocky/Almalinux/Springdale take RHEL point releases and repackage as far as they can for as long as they can - one or more of which will probably vanish in due course as history repeats itself.]

There are today a bunch of CentOS SIGs - some of which appear to overlap with Fedora SIGs - all of which are presumably based on CentOS 8 non-Streams. What happens to those, proportionately, smaller communities?

CentOS Streams is a more open process for contribution to what becomes Red Hat but what's in it for the wider community
(and how do you keep RPMFusion / EPEL etc. able to work for everyone including Rocky/Almalinux - the world seems dependent on these third party repositories) ?
[Although Fedora folk kindly contribute EPEL, it's not designed to work with vanilla Fedora or with Streams - either too fast moving or volatile - the clue is in the name.]

Planning the CentOS 8 endgame

Posted Jul 15, 2021 16:01 UTC (Thu) by mattdm (subscriber, #18) [Link] (4 responses)

Okay, let's see what I can do. :)

> 2021-31-12: CentOS 8.5 ceases to be updated. There is no CentOS 9 or ongoing.

So, this seems pedantic, but it's the Official CentOS Terminology: "CentOS 8.5" is actually "CentOS Linux 8.5" -- CentOS being the project and CentOS Linux being the distro. (Just like it's "RHEL 8.5", not "Red Hat 8.5").

CentOS never actually maintained distinct CentOS Linux point releases -- in RHEL, you can get support for (some -- search for "RHEL EUS") minor point releases. In CentOS Linux, it's always been the case that once a RHEL minor update has been rebuilt, it gets rolled into the distro. This may also sound pedantic, but it's also kind of important, and it's better therefore to say "CentOS Linux 8 ceases to be updated" rather than specifying a point version.

There won't be a CentOS Linux 9, but there will be a CentOS Stream 9.

> A core of CentOS people (primarily IBM/Red Hat employees) continue to work on CentOS Streams 8 / 9 ... which form part of RHEL 8.6 ..., RHEL 9 and so on.

I'm going to break this one down into several parts....

1. Red Hat is continuing to operate as a separate company that happens to be owned by IBM. Like, previously there were shareholders and now there is IBM. So, "IBM/Red Hat employees" is really less clear than just "Red Hat employees", because non-RH IBMers aren't involved.

2. The folks who are paid to work on CentOS engineering by Red Hat are part of the "Community Platform Engineering" team. This is the same group of people who work as part of Red Hat's contribution to the community on Fedora infrastructure and release engineering. Notably, they're not part of packaging or software-that-goes-in-the-OS development teams. Those people will continue to work in that group on CentOS and Fedora infra and releng.

3. With previous versions of RHEL, once RHEL forked off from Fedora, development of RHEL happened all inside Red Hat with no transparency until beta release, then secret internal development again and then a final release. That development was done by hundreds of people working in RHEL engineering. Then, after the public release, maintenance and development work again happened all "inside". With CentOS Stream, those hundreds of people are now going to do that same development work as part of CentOS directly. (This plays into the SIG answer a bit below.)

4. Pedantic again, but: note "CentOS Stream" not "Streams". I didn't name it. :)

5. Anyway, yeah: package updates accepted for upcoming RHEL point releases will go into CentOS Stream as soon as they are ready rather than waiting for a point release. Then, RHEL point releases will be assembled as forks from CentOS Stream.

> [Rocky/Almalinux/Springdale take RHEL point releases and repackage as far as they can for as long as they can - one or more of which will probably vanish in due course as history repeats itself.]

Point releases and of course also the stream of errata that comes along.

> There are today a bunch of CentOS SIGs - some of which appear to overlap with Fedora SIGs - all of which are presumably based on CentOS 8 non-Streams. What happens to those, proportionately, smaller communities?

Generally, they'll continue. A lot of them were actually Red Hat's development teams taking off their Red Hats to get their work done in the public. With Stream, I think some of that will be more obvious but otherwise unchanged.

I'm very interested in, where there is overlap, building more cooperation between CentOS SIGs and equivalent Fedora groups. There's a new automotive IoT effort underway which I hope will provide a good model here.

> CentOS Streams is a more open process for contribution to what becomes Red Hat but what's in it for the wider community

It depends what you mean by "the wider community". I think more transparency around RHEL is generally great for the RHEL user community. And for many (I personally think MOST users, but we'll see) I think it's a straight-up improvement to have access to errata as they are produced rather than the CentOS Linux situation where updates basically pause for a month while a minor-update drop is figured out twice a year. For people who want a no-strings-attached free RHEL-like operating system, CentOS Stream can be a great option. And if you have a patch or something that might be appropriate for RHEL, you now have a path to collaborate with RHEL engineering that was previously only available to big customers via internal processes.

For people who really want to have direct influence into shaping the distro fundamentally, the answer is still Fedora.

> (and how do you keep RPMFusion / EPEL etc. able to work for everyone including Rocky/Almalinux - the world seems dependent on these third party repositories) ?
> [Although Fedora folk kindly contribute EPEL, it's not designed to work with vanilla Fedora or with Streams - either too fast moving or volatile - the clue is in the name.]

I think this problem is vastly over-worred-about. Everything that's going into CentOS Stream still has the same compatibility goals as RHEL itself, and the change in CentOS Stream won't be more than the change seen in minor RHEL releases (which, remember, are rolled into "classic" CentOS Linux).

In the old model, when a RHEL point release drops, there might be some minor incompatibilies with the previous point release necessitating updates in EPEL or third party repositories. But CentOS Linux doesn't update in lockstep, so what do you do? Wait until CentOS Linux updates, or rebuild immediately for the new RHEL minor release and now your packages won't install on CentOS Linux? This is the exact same problem as with CentOS Stream and RHEL except in reverse. The only real difference is that now we have people concerned about it so we're developing solutions (like EPEL Next).

...

Does any of this help? :)

Planning the CentOS 8 endgame

Posted Jul 15, 2021 17:31 UTC (Thu) by amacater (subscriber, #790) [Link]

Yes: many thanks indeed. Sorry for slips - thanks for your corrections in terminology: fingers outpacing brain, as ever. Very clear explanation which will benefit others too.

Planning the CentOS 8 endgame

Posted Jul 16, 2021 5:06 UTC (Fri) by pabs (subscriber, #43278) [Link] (2 responses)

> RHEL point releases will be assembled as forks from CentOS Stream.

I feel like I don't understand the word "fork" as it is being used here, could you elaborate? Is it just that the contents of the CentOS Stream repo are copied to the RHEL point release repo, and then the release happens immediately, or is there more development going on (as usually implied by the word "fork") after copying CentOS Stream to RHEL?

Planning the CentOS 8 endgame

Posted Jul 16, 2021 14:02 UTC (Fri) by mattdm (subscriber, #18) [Link] (1 responses)

> I feel like I don't understand the word "fork" as it is being used here, could you elaborate?

"Branch" would have been a better choice of word. These RHEL branches (or at least some of them) will be eligible for EUS, and so get "hotfixes" that don't go into Stream (just as current EUS branches aren't in CentOS Linux). The same issues will of course get fixed in Stream, although possibly in a different way. (Again same as EUS in the pre-Stream model.)

Planning the CentOS 8 endgame

Posted Jul 17, 2021 1:53 UTC (Sat) by pabs (subscriber, #43278) [Link]

OK, that makes sense, thanks.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 17:32 UTC (Thu) by davide125 (subscriber, #47485) [Link] (2 responses)

Not all SIGs target CentOS Linux 8. The Hyperscale SIG for example targeted CentOS Stream from the beginning: https://wiki.centos.org/SpecialInterestGroup/Hyperscale

Other SIGs are also building for Stream already, you can see the full list on CBS, which where the actual builds happen: https://cbs.centos.org/koji/search?terms=*8s*&type=ta...

EPEL as-is works out of the box on Stream for the vast majority of packages. For the ones where that's not the case, there's EPEL Next: https://fedoraproject.org/wiki/EPEL_Next

EPEL-next

Posted Jul 15, 2021 21:00 UTC (Thu) by corbet (editor, #1) [Link] (1 responses)

I wish things were so simple. I have a CS8 box that uses xapian-core-devel from EPEL; that box is now failing to update due to this issue. This is just the sort of thing that makes CS8 a bit scary for a lot of people.

EPEL-next

Posted Jul 16, 2021 2:15 UTC (Fri) by mattdm (subscriber, #18) [Link]

But ... that's exactly what I'm saying. That bug is about a change in RHEL. The same issue will affect EPEL with CentOS Linux all the same, and these kinds of things <i>have</i> affected EPEL all along.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 14:50 UTC (Thu) by BirAdam (guest, #132170) [Link]

I never actually noticed the lack of in-place upgrade in CentOS/RHEL. Having been the admin for thousands of machines that were varyingly Debian, CentOS, and (very few) FreeBSD, I also pushed those I worked with to stop using in-place upgrade with Debian. It was far less error prone to just partition the drives properly, and wipe the / partition and leave /home alone, and additionally to use blue/green deployment. Configs were always on backup servers anyway but blue/green meant that in a pinch you can just check the still live server’s configuration, and with most Debian upgrades a junior admin would end up calling (and sometimes waking) the seniors because he/she couldn’t figure out a configuration variable change between major version releases.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 15:24 UTC (Thu) by tbowling (guest, #113886) [Link] (3 responses)

> For me (as a long time Debian user at home and RHEL/CentOS user at work),
> I've always been amazed that there was no procedure for stable updates between
> major versions for CentOS and none for Red Hat until very recently: the advice
> was always "wipe and reinstall" On IRC, I was advised that there would be no
> way to move from CentOS Streams to RHEL.

This is inaccurate. We have had great success with RHEL upgrade tools 6->7->8 (8->9 planned)
- https://access.redhat.com/documentation/en-us/red_hat_ent...
- https://access.redhat.com/documentation/en-us/red_hat_ent...

A lot of effort has gone into identifying the many changes that occur between the deltas of the major releases. There is a lot that happens in the rpm scriptlets in upstream Fedora packages but gets lost as Fedora only tracks N-2. So this tooling does a LOT of clean up and fixes to accommodate for all of that. There is much involved beyond updating RPMS, and these tools take care of it.

For CentOS Stream -> RHEL, the issue is it introduces the possibility of package downgrades, which are known to be very problematic, regardless of the distribution or package manager. Again, package manager scriptlets are typically not idempotent or reversible. For lots of user space packages, it might not be an issue. But for some, such as a significant version upgrade, the installation scriptlets might make significant changes that are not accounted for during the downgrade. And with critical packages such glibc, rpmdb, or openssl, there is potential for system corrupting results. We have seen this in support cases when users have tried to downgrade RHEL minor releases to revert back and resulted in very painful experiences.

Because CentOS Stream is forward looking RHEL, it raises concern that for some packages, it could result in package downgrade scenarios similar to a RHEL 8.4->8.3 downgrade. So we cannot in good faith say, "here's a tool to convert it for you." We are exploring this and would like to do it in the future and have done some proof of concept testing, but we need to monitor this more closely to understand if this can be done in a safe, supportable manner. It is absolutely NOT because we are ignoring it. There are simply dragons lurking below.

Hope this helps!

Planning the CentOS 8 endgame

Posted Jul 15, 2021 17:37 UTC (Thu) by amacater (subscriber, #790) [Link]

Again: thanks very much that absolutely helps. I knew about the upgrade tools but also that a lot of people found difficulties: as ever, if you _only_ have software from Red Hat and no third party repositories, it's a good deal easier.

Thanks also for the clarification on downgrade being even more difficult - I think we're in ferocious agreement here :)

Planning the CentOS 8 endgame

Posted Jul 16, 2021 16:12 UTC (Fri) by Wizedom (guest, #153289) [Link] (1 responses)

> This is inaccurate. We have had great success with RHEL upgrade tools 6->7->8 (8->9 planned)
>- https://access.redhat.com/documentation/en-us/red_hat_ent...
> - https://access.redhat.com/documentation/en-us/red_hat_ent...
I think that what amacater meant to say is that unlike on Debian there isn't any tool that works flawlessly to upgrade from Centos 6 to Centos 7
And just like him, being a long time Debian user at home and RHEL/CentOS user at work, my experience leads me to concur with him.

Planning the CentOS 8 endgame

Posted Jul 16, 2021 16:54 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]

> I think that what amacater meant to say is that unlike on Debian there isn't any tool that works flawlessly to upgrade from Centos 6 to Centos 7

Upgrades have bugs that don't affect fresh installs regardless of the distribution. No such thing as flawless.

Planning the CentOS 8 endgame

Posted Jul 15, 2021 18:53 UTC (Thu) by rbowen (guest, #90479) [Link]

Yes, we will be building CentOS 8.5 after the release of RHEL 8.5.

I don't feel like there's any lack of clarity in what happens to the SIGs and where the overlap is with Fedora, but I would be glad to answer any questions you have about those topics.

Planning the CentOS 8 endgame

Posted Jul 21, 2021 19:10 UTC (Wed) by amarao (guest, #87073) [Link]

One of the projects I have a little connection with found a solution for centos 8 death. Upgrade to centos 7 and have few more years of not dealing with this problem.

I'm not joking. Centos 8 upgrade to centos 7 is the least troublesome process. In a few years a viable alternative gonna win. Until then centos 7 is a reasonable combination of longevity, stability and 'no surprises'.