Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model [LWN.net]

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 14:39 UTC (Sat) by pbonzini (subscriber, #60935)
In reply to: Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model by geofft
Parent article: Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Let's look at the three claims:

> These were (and are) not actually the RHEL source releases — rather, they appear to be primarily a testing ground for what might appear in RHEL later

The first part is true, the second is less true and a bit biased in my opinion.

The first part is true because packages can be branched from Stream to the next RHEL minor update at any time, depending on the developers' plans, so it is true that RHEL packages might have never been in CentOS Stream. However: 1) with the exception of embargoed issues, bugs will always be fixed first in Stream and then in RHEL, so you'll never have a situation where Red Hat hides bugfixes from the community 2) the most active packages generally are also the ones that tend to branch last, so for example there will almost always be a kernel in CentOS Stream that is identical to what ends up in a RHEL minor release.

I think it's biased to say that CentOS Stream is a "testing ground for what might appear in RHEL" because Red Hat neither wants nor needs unwilling beta testers. Outside contributions should be code that the community wants in the next RHEL, not reports of bugs that slipped in. Bugs should be fixed upstream and secondarily in Rawhide rather than in CentOS Stream, which aims to be a *preview* of what *will* appear in RHEL later and also a stable, usable rolling distribution in its own right.

> CentOS Stream is a rebuild of RHEL. It is a rebuild of exactly RHEL sources taken from the same git tree as RHEL builds take those sources. It is not a middle - it is RHEL

Almost completely wrong. CentOS Stream is upstream of RHEL and RHEL is periodically branched from CentOS Stream. CentOS Stream is not good is you want your CI to test against a released version of RHEL, because it is always *ahead* of the last RHEL release.

On the other hand CentOS Stream *is* an enterprise Linux distro and it's pretty damn close to RHEL. There will be a CentOS Stream commit for almost any package that is released in RHEL, because the RHEL minor release branches are relatively short-lived and especially no meaningful development happens on them. Compare to the situation until RHEL8 where the sources went internal after the branch from Fedora, and you had years of internal development with nothing but periodic code drops.

> For the 5% of packages, are those the ones at gitlab.com/redhat/centos-stream/src, including a public kernel repo?

No, these "srcgit" repos are turned into regular RPMs; they are the heirs of the internal repos that were used to manage RPMs with thousands of patches applied, the first of which was the kernel. That old tooling dating back to the late 2000s has been cleaned up and is now used in these GitLab repos. See https://docs.centos.org/en-US/stream-contrib/contributing... for more info.

The packages that require manual work for a given minor release are those that: 1) are not available in the UBI (which still has to have public SRPMs); 2) are updated in the first place (many basic packages, say "sed", are not); and 3) are rebased to new upstream releases (if they aren't, all changes are visible in Stream); and 4) had some patches applied after branching from Stream and after stream was rebased.

There is a bigger change which affects packages not in the UBI, one of which is the kernel. This corresponds to just conditions 1 and 2 above, and I also missed it when I first commented here on LWN. After RHEL 9.1 was released, git.centos.org used to receive the same updates as RHEL 9.1 until 9.2 was released, and so on. See for example https://git.centos.org/rpms/kernel/commits/c9. To be honest I have no idea why this was still happening since there was no CentOS Linux 9; but anyway, these stable updates will not be available anymore. In most cases, new tooling will be needed to fetch the updated SRPMs from UBI and that will be it, but for some other packages like the kernel it will be harder and it's possible that the only solution there is clean room reverse engineering or giving up the bug-compatibility to some extent.

More in general, Alma and Rocky have a wide choice of ways they can build their own enterprise-ready Linux distro based on CentOS Stream, just like Facebook does internally. Whether they will stay faithful to their goal of being bug-compatible with RHEL, or perhaps only try to be ABI-compatible, depends on how much effort that want to put in the task; same for how they'll tackle the problem of stable updates They certainly will have to make some tough decisions, and cannot anymore just blindly rebuild whatever Red Hat served them on git.centos.org. ¯⁠\⁠_⁠(⁠ツ⁠)⁠_⁠/⁠¯

Disclaimer: some of the above might not be entirely accurate, but should not be completely wrong either. Corrections are welcome.

to post comments

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 15:54 UTC (Sat) by mcatanzaro (subscriber, #93033) [Link] (14 responses)

"1) with the exception of embargoed issues, bugs will always be fixed first in Stream and then in RHEL, so you'll never have a situation where Red Hat hides bugfixes from the community"

This part is not true. We are not allowed to fix higher-severity security issues in CentOS Stream until the fix has shipped in RHEL. Very few such issues are embargoed.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 18:54 UTC (Sat) by pbonzini (subscriber, #60935) [Link]

Thanks for the correction.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 19:57 UTC (Sat) by passthejoe (guest, #156034) [Link] (12 responses)

This is where you run into problems. How do you "convince" users to run Stream if it's always behind RHEL when it comes to critical security patches?

How far (in time) is it behind? Days? Weeks? Months?

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 20:33 UTC (Sat) by pizza (subscriber, #46) [Link] (11 responses)

> How far (in time) is it behind? Days? Weeks? Months?

If you [1] cared about timely updates you'd have signed a service agreement with Red Hat.

If you're not explicitly paying someone for a specific level of service and responsiveness, any update you get is due to someone else's generosity and goodwill, and certainly not on any schedule that can be relied upon. Service guarantees cost money!

(Indeed, pre RH/Stream CentOS model routinely saw security updates held up for 1-2 months, twice a year, for the entirety of its existence while they iterated/rebased onto the latest RHEL point release.)

[1] the general you, not *you* specifically.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 27, 2023 8:25 UTC (Tue) by TRauMa (guest, #16483) [Link] (10 responses)

> If you cared about timely updates you'd have signed a service agreement with Red Hat.

Or you'd use a distro that doesn't hold back security fixes deliberately. Basically you're telling the original poster to get screwed. Which is funny because Red Hat points to stream as the solution for everyone who wants to live in the RH eco system and doesn't fit their commercial target.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 27, 2023 12:56 UTC (Tue) by pizza (subscriber, #46) [Link] (9 responses)

> Or you'd use a distro that doesn't hold back security fixes deliberately.

Oh, please. Who, exactly, is holding back security fixes? And how is that practice different from *any* other distribution, least of all the RHEL clones?

What you don't seem to understand here is that these security fixes are being created, or at least backported, by RH engineers, and that is what their customers are paying for. *everyone else* already had to wait until after the stuff went out to RHEL, and then create their own update incorporating the RHEL changes. (And, funnily enough, none of the rebuilders provided any support/updates for X.y after X.y+1 came out. So this hysteria about "holding back fixes" is talking about what has been the status quo *all along*.

> Basically you're telling the original poster to get screwed.

No, I'm telling the original poster that they can't expect to get what everything they want without having to pay something for it. RH owes its non-customers [1] *nothing*.

> Which is funny because Red Hat points to stream as the solution for everyone who wants to live in the RH eco system and doesn't fit their commercial target.

And that's ... wrong how?

[1] By "customers" I also include everyone who received binaries from RH.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 10:04 UTC (Thu) by madhatter (subscriber, #4665) [Link] (8 responses)

> Who, exactly, is holding back security fixes?

See mcatanzaro (who works for RH) above:

> We are not allowed to fix higher-severity security issues in CentOS Stream until the fix has shipped in RHEL.

A security fix has been prepared, but because it hasn't yet gone through whatever internal processes are required for shipping in RHEL, it can't be released for CentOS. That is pretty much my working definition of "holding back a security fix".

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 12:56 UTC (Thu) by pizza (subscriber, #46) [Link] (7 responses)

> A security fix has been prepared, but because it hasn't yet gone through whatever internal processes are required for shipping in RHEL, it can't be released for CentOS. That is pretty much my working definition of "holding back a security fix".

First, you realize this is for *embargoed* fixes, not "fixes" in general?

Secondly, rebuilders *always* had to wait until after a fix shipped into RHEL before they'd get the source to generate a build? And that RH *never* released sources to the rebuilders for anything other than the _current_ point release of RHEL?

And meanwhile, CentOS Stream still has the same processes and QA requirements as RHEL, so either way you were *never* going to get a fix for something until RH's internal processes were satisfied.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 13:38 UTC (Thu) by madhatter (subscriber, #4665) [Link] (6 responses)

> First, you realize this is for *embargoed* fixes, not "fixes" in general?

That is exactly not how I read what mcatanzaro said. My understanding was that they are not allowed to fix *any* higher-severity security issues in CentOS (until a RHEL fix has shipped) and very few such issues are embargoed. RH had already copped to the withholding in the case of embargoed issues, so I don't think his/her comment can be read any other way.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 13:56 UTC (Thu) by pizza (subscriber, #46) [Link] (5 responses)

> My understanding was that they are not allowed to fix *any* higher-severity security issues in CentOS (until a RHEL fix has shipped) and very few such issues are embargoed.

It was my understanding that very few issues are considered "high-severity" and even fewer of those are embargoed. So the vast majority of fixes land in CentOS *before* they end up in RHEL..

But even if you are completely correct, what exactly is the problem? Rapid access to serious updates is what RH's customers are paying for, after all! In the worst case it takes no longer to land publicly than it used to back in the old "we'll rebuild it after RHEL releases it" rebuilder paradigm, and everyone downstream was supposedly fine with that for the better part of two decades.

BTW, here's the full message, for context: https://lwn.net/ml/fedora-devel/D8JRWR.D2QFSVHOGZDV2@redh...

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 14:07 UTC (Thu) by madhatter (subscriber, #4665) [Link] (4 responses)

I'm not wading into the larger argument. I was simply responding to your question about who was holding back security fixes, which seemed to me to imply that you thought nobody was, when we'd just been told by an internal employee that they were.

You may be right that "very few issues are considered high-severity", but it's not just a numbers game: those are the very issues for which users most urgently need fixes, and for which there can be least justification for the withholding we are in fact assured takes place.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 14:24 UTC (Thu) by pizza (subscriber, #46) [Link] (3 responses)

Like many (most) things, this seems to be an argument over definition.

I do not consider fixing something in place A before you fix it in place B to be "withholding" it from place B. To me, withholding would be never offering that fix in place B to begin with.

Because by that same argument, RH is "withholding" *all* fixes from CentOS7 until after they are released in RHEL7. A practice that the rebuilders (and their users) were supposedly okay with for a couple of decades.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 14:40 UTC (Thu) by farnz (subscriber, #17727) [Link]

Note, too, that the fix may well be published upstream already (possibly even by Red Hat employees), and RH are simply applying it to the RHEL package.

In this circumstance, it's no more "withholding" a fix than Debian "withholds" fixes that are present in the latest upstream release but not in the most recent Debian package.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 14:51 UTC (Thu) by madhatter (subscriber, #4665) [Link] (1 responses)

> I do not consider fixing something in place A before you fix it in place B to be "withholding" it from place B

I do. I'm not saying such withholding is avoidable or reprehensible or surprising or unique to this case, but it *is* (to my mind, and apparently to mcatanzaro's mind) happening. If you don't want to think of it as such, that is of course up to you, but you might want to better define your terms before making general statements.

> Note, too, that the fix may well be published upstream already (possibly even by Red Hat employees), and RH are simply applying it to the RHEL package.

I don't see how it would have been published by RH employees, given that we've been told they are forbidden from working on it under those circumstances. If the upstream fix already exists from another source, and they're forbidden from pushing it through the CentOS pipe simply because it has not yet been pushed through the RHEL pipe, well then, that's withholding.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 15:55 UTC (Thu) by farnz (subscriber, #17727) [Link]

A fix may be published upstream first simply because at the time of fixing, the author didn't realise that they were fixing a security issue - this was just part of normal bug fixing when you take part in upstream development (and note that RH employs a lot of people who take part in upstream development). And the further behind the RHEL version of a package is, the more likely this is to be true, since you're less likely to remember all the development that's happened between the RHEL version being picked, and you making this fix.

And while it's a workable definition of "withholding", it does mean that any time you ship a released version and not the latest development tree, you're withholding fixes and features from your users - this isn't what most people think of when you say that things are being withheld.