Planning the CentOS 8 endgame

CentOS 8 is reaching its end of life (EOL) at the end of 2021, though it was originally slated to be supported until 2029. That change was announced last December, but it may still come as a surprise to some, perhaps many, of the users of the distribution. While the systems running CentOS 8 will continue to do so, early next year they will stop getting security (and other) updates. The CentOS project sees CentOS Stream as a viable alternative, but users may not agree—should the project simply leave CentOS 8 systems as ticking time bombs in 2022 and beyond?

A discussion of the CentOS 8 EOL was kicked off by Rich Bowen in a post to the CentOS-devel mailing list. He noted that there will be more questions about the EOL process as that date approaches, so he wants "to make sure we have clear documentation, prominently displayed, that sets expectations". He outlined the process of archiving CentOS 8 to vault.centos.org and wondered if there were changes that should be made because this particular EOL event is rather different.

Alex Iribarren was concerned about "pulling the plug" right on December 31 and suggested that "an extra month or so would be nice, particularly given the holiday period". CentOS release manager Johnny Hughes said that the holiday will delay the switch a bit because people will not be working on those days, but that security updates will not happen past that point in any case. While the dates are still fluid, he described the plan in some detail:

[...] Our goal is, so long as RHEL 8.5 releases before 31 DEC 2021, we will get the files from 8.5 released before we remove CentOS Linux 8 from the mirrors. We will not be adding any updates from RHEL source code released after 31 DEC 2021 to CentOS Linux 8.

This release will be put into at least vault.centos.org/8.5.xxxx/ (where xxxx is the date). Of course, if the RHEL 8.5 release happens after 01 Jan 2022, we would not be doing that release in CentOS Linux 8.

New items (to CentOS Stream 8) will be being built and still going into CentOS Stream 8 after 01 JAN 2022 until CentOS Stream 8 EOLs 5 years after the RHEL 8 release (EOL is 31 May 2024).

But Carl George wondered if a more radical plan was in order. He believes that Stream is effectively the continuation of CentOS 8, so maybe "we should have mirrorlist.centos.org respond to requests for 8 repos with 8-stream repos, which effectively converts any remaining CentOS Linux 8 systems to CentOS Stream 8". That could either be done at EOL time or, perhaps, one to three months later. The third alternative he presented is the status quo—no more security updates—but he is worried that there are quite a number of people who are unaware of the EOL.

Multiple commenters in the thread seemed to agree that switching to CentOS Stream 8 is the right path forward, at least on a technical level, but there are some obvious concerns with that approach. Bowen put it this way:

While I agree with you, that it's an upgrade, I anticipate that moving people from CentOS Linux 8 to CentOS Stream 8 automatically will result in a lot of backlash from people who continue to believe that Stream is a alpha/beta/testing/buggy/unstable/[pick your favorite complaint] distribution. Y'know, once they noticed that it had happened.

Surprising users seldom goes well, even if it's an overall positive surprise.

Stephen John Smoogen thinks that auto-switching systems to CentOS Stream 8 "ends up with lawsuits and very very angry people". There may well be CentOS 8 systems controlling safety-critical infrastructure, even though that may not be a particularly smart choice for those types of systems. Given that, he does not see either of George's switching options working out "no matter how well it is messaged or done". Looking at the statistics shows lots of new CentOS systems since the announcement of the EOL change; the alternatives (e.g. AlmaLinux, Rocky Linux) are not even close to catching up, but:

While many of the current 450k CentOS 8 systems may function well on CentOS Stream, we don't know which ones are just web servers in some advertising farm and which ones are controlling the flow rates on a dam or petroleum flows at a refinery in Texas.

Fabian Arrotin split CentOS users into two categories; the first is paying attention to the announcement and making plans, while the second is not. For the former, neither of George's auto-switch options would affect them; they have presumably already arranged a switch: to Stream, one of the alternative, compatible distributions mentioned above, or to some other distribution entirely. But the latter group, whose systems will have more and more known vulnerabilities over time, should just be forced into CentOS Stream 8, he said. There are, after all, users still running even older EOL versions of CentOS:

I admit that if you deploy CentOS, a good sysadmin would be up2date with what happens in the distro land *but* also by looking at the number of mirrorlist requests even for CentOS 5, I can tell you for sure that some don't ..... (ouch).

Julien Pivotto said that there are two good reasons to auto-switch users to CentOS Stream 8:

Providing stream as a continuity of 8 is the best thing to do, to show that we are confident and that the whole "stream fits most of the CentOS use case". It also has the side effect of better protecting the internet.

[...] I think that it will not backfire if we announce it soon.

Safety of the internet was also on Leif Madsen's mind. He suggested adding a "security updates only" mode that CentOS 8 systems would switch to at EOL. It is "what a good netizen would do", rather than "leave 450k+ systems idling on the internet just waiting to be scooped up into a bot net".

But there are practical considerations with switching a bunch of systems to CentOS Stream 8 at once, as Phil Perry pointed out. "Whilst I completely understand the desire to take this approach", users voluntarily switching are already creating an extra support load, in part because they are finding that kernel module packages created for the CentOS 8 kernel no longer work on CentOS Stream 8. "If you switch all C8 users en mass at EOL, you run the risk of creating a potentially huge support burden that we are simply not in a position to manage."

Josh Boyer agreed:

There are too many situations like the kernel, or internal policy compliance, or other reasons make automated migration problematic. We should encourage and advocate for people to switch to CentOS Stream, and focus on making tools and documentation for that easy to use and easily accessible, but we should not be doing that kind of migration unilaterally.

It may in fact be better for some users to move on from CentOS if they are not interested in what CentOS Stream is offering, Smoogen said. CentOS was a drop-in replacement for Red Hat Enterprise Linux (RHEL), but CentOS Stream is not that:

CentOS Stream is about building a co-operative relationship between the consumers and the distribution where what the distribution builds is evaluated and feedback is given. If a consumer is expecting never to have problems, to never have to file/track a bugzilla or ever even check to see what is being delivered.. then CentOS Stream is not the distribution for them. Because even if it doesn't look like it, there is a very very large gap between 'never' and 'once in a while' that might happen with Stream. There is a need for co-operation between the consumer of stream and the makers to get things right. If you do not have time, energy, or want to do that, then Stream is going to be a constant thorn. The consumer in that case would be better off with another rebuild.

But George disagreed strongly with that characterization of CentOS Stream; multiple people have told him that they switched and never noticed any difference. While it is recommended that users participate in the process, he said, it is not required by any means:

CS8 hasn't been a constant thorn for them. I'm not claiming it's been perfect, there have certainly been regressions, but they are fixed faster than they ever were in CL8 (or previous major versions) and I strongly feel that most users will be best served by getting switched to CS8 at or just after the CL8 EOL.

But Leon Fauster said that when he tried switching some systems over to CentOS Stream 8, he encountered problems. Applications from third-party repositories (e.g. RPM Fusion) stopped working because their dependencies were not available. Installations that are more complex run the risk of failing to switch to CentOS Stream successfully. For his purposes, complex simply means that the system "uses more [than] just CentOS artifacts (ISV/proprietary software, 3rd/custom repos, etc.)".

Whatever the technical (and internet-safety) merits of forcibly switching systems to CentOS Stream 8, it is a little hard to see a company like Red Hat taking that risk. As unfortunate as it may be for some inattentive users (and, perhaps, the rest of us on the internet at large), it is much safer to simply point to the EOL announcement as a defense against any claims of fault for insecure CentOS 8 systems in 2022—and beyond. It is worrisome that these systems will be out there spamming (and worse), but it does not seem any more so than systems still looking for updates to CentOS 5, which was released in 2007 and stopped getting updates in 2017.