Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model [LWN.net]

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 23, 2023 22:47 UTC (Fri) by geofft (subscriber, #59789)
Parent article: Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

There were a couple of comments in response to the original post (e.g., here on LWN, on Hacker News, and on the Fedora devel list) claiming that there is / will be no loss of complete corresponding source - ,just that they won't be on git.centos.org.

In particular, the claim in all of them is that in the past, Red Hat has produced SRPMs from internal trees, and CentOS would snapshot the published SRPMs and rebuild them and also check them into repos on git.centos.org. After Red Hat acquired CentOS, they kept maintaining git.centos.org. (I don't really follow how CentOS got the SRPMs or whether that stopped, but my impression is they used to be on public RPM mirrors and no longer are?) However, now, Red Hat produces SRPMs from public sources (specifically, the "c9s" i.e. "CentOS 9 Stream" branches of gitlab.com/redhat/centos-stream/rpms), and so there is no need for a separate git.centos.org mirror. It's true that there might be more development on CentOS Stream than actually made it into a release, but you can (probably?) straightforwardly figure out which version did make it in. There shouldn't be private branches, except temporarily for embargoed security releases; if the GitLab history isn't agreement with the RHEL product, it would make development confusing for actual Red Hat employees working on actual Red Hat.

In other words, the claim in the article, "These were (and are) not actually the RHEL source releases — rather, they appear to be primarily a testing ground for what might appear in RHEL later.", is at odds with the claim on devel@, "CentOS Stream is a rebuild of RHEL. It is a rebuild of exactly RHEL sources taken from the same git tree as RHEL builds take those sources. It is not a middle - it is RHEL." And the claim in the HN comment, "95% [...] of the platform will match [...] versions available in CentOS Stream and their RHEL counterparts, there are some packages that will not due to the way they are developed." isn't really in agreement with either of them. Which is it? (For the 5% of packages, are those the ones at gitlab.com/redhat/centos-stream/src, including a public kernel repo?)

I think Kuhn's points about how Red Hat dances on the border of GPL compliance are all very important, but I think it's also important to get clarity on whether we're actually losing CCS here, or whether we're effectively gaining Git history compared to the days when non-Red-Hatter CentOS folks were just snapshotting things into git.centos.org. (Every so often there's a claim on debian-devel@ that the "preferred form of modification," this century, includes VCS history and commit messages.)

... Separately, I just remembered that Red Hat has zero-cost zero-licensing container images running actual branded Red Hat, and they make CCS available (as they must for the GPL'd packages they distribute to the public!). Do these images cover enough of Red Hat that rebuilders can use them, either technically or legally?

to post comments

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 14:39 UTC (Sat) by pbonzini (subscriber, #60935) [Link] (15 responses)

Let's look at the three claims:

> These were (and are) not actually the RHEL source releases — rather, they appear to be primarily a testing ground for what might appear in RHEL later

The first part is true, the second is less true and a bit biased in my opinion.

The first part is true because packages can be branched from Stream to the next RHEL minor update at any time, depending on the developers' plans, so it is true that RHEL packages might have never been in CentOS Stream. However: 1) with the exception of embargoed issues, bugs will always be fixed first in Stream and then in RHEL, so you'll never have a situation where Red Hat hides bugfixes from the community 2) the most active packages generally are also the ones that tend to branch last, so for example there will almost always be a kernel in CentOS Stream that is identical to what ends up in a RHEL minor release.

I think it's biased to say that CentOS Stream is a "testing ground for what might appear in RHEL" because Red Hat neither wants nor needs unwilling beta testers. Outside contributions should be code that the community wants in the next RHEL, not reports of bugs that slipped in. Bugs should be fixed upstream and secondarily in Rawhide rather than in CentOS Stream, which aims to be a *preview* of what *will* appear in RHEL later and also a stable, usable rolling distribution in its own right.

> CentOS Stream is a rebuild of RHEL. It is a rebuild of exactly RHEL sources taken from the same git tree as RHEL builds take those sources. It is not a middle - it is RHEL

Almost completely wrong. CentOS Stream is upstream of RHEL and RHEL is periodically branched from CentOS Stream. CentOS Stream is not good is you want your CI to test against a released version of RHEL, because it is always *ahead* of the last RHEL release.

On the other hand CentOS Stream *is* an enterprise Linux distro and it's pretty damn close to RHEL. There will be a CentOS Stream commit for almost any package that is released in RHEL, because the RHEL minor release branches are relatively short-lived and especially no meaningful development happens on them. Compare to the situation until RHEL8 where the sources went internal after the branch from Fedora, and you had years of internal development with nothing but periodic code drops.

> For the 5% of packages, are those the ones at gitlab.com/redhat/centos-stream/src, including a public kernel repo?

No, these "srcgit" repos are turned into regular RPMs; they are the heirs of the internal repos that were used to manage RPMs with thousands of patches applied, the first of which was the kernel. That old tooling dating back to the late 2000s has been cleaned up and is now used in these GitLab repos. See https://docs.centos.org/en-US/stream-contrib/contributing... for more info.

The packages that require manual work for a given minor release are those that: 1) are not available in the UBI (which still has to have public SRPMs); 2) are updated in the first place (many basic packages, say "sed", are not); and 3) are rebased to new upstream releases (if they aren't, all changes are visible in Stream); and 4) had some patches applied after branching from Stream and after stream was rebased.

There is a bigger change which affects packages not in the UBI, one of which is the kernel. This corresponds to just conditions 1 and 2 above, and I also missed it when I first commented here on LWN. After RHEL 9.1 was released, git.centos.org used to receive the same updates as RHEL 9.1 until 9.2 was released, and so on. See for example https://git.centos.org/rpms/kernel/commits/c9. To be honest I have no idea why this was still happening since there was no CentOS Linux 9; but anyway, these stable updates will not be available anymore. In most cases, new tooling will be needed to fetch the updated SRPMs from UBI and that will be it, but for some other packages like the kernel it will be harder and it's possible that the only solution there is clean room reverse engineering or giving up the bug-compatibility to some extent.

More in general, Alma and Rocky have a wide choice of ways they can build their own enterprise-ready Linux distro based on CentOS Stream, just like Facebook does internally. Whether they will stay faithful to their goal of being bug-compatible with RHEL, or perhaps only try to be ABI-compatible, depends on how much effort that want to put in the task; same for how they'll tackle the problem of stable updates They certainly will have to make some tough decisions, and cannot anymore just blindly rebuild whatever Red Hat served them on git.centos.org. ¯⁠\⁠_⁠(⁠ツ⁠)⁠_⁠/⁠¯

Disclaimer: some of the above might not be entirely accurate, but should not be completely wrong either. Corrections are welcome.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 15:54 UTC (Sat) by mcatanzaro (subscriber, #93033) [Link] (14 responses)

"1) with the exception of embargoed issues, bugs will always be fixed first in Stream and then in RHEL, so you'll never have a situation where Red Hat hides bugfixes from the community"

This part is not true. We are not allowed to fix higher-severity security issues in CentOS Stream until the fix has shipped in RHEL. Very few such issues are embargoed.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 18:54 UTC (Sat) by pbonzini (subscriber, #60935) [Link]

Thanks for the correction.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 19:57 UTC (Sat) by passthejoe (guest, #156034) [Link] (12 responses)

This is where you run into problems. How do you "convince" users to run Stream if it's always behind RHEL when it comes to critical security patches?

How far (in time) is it behind? Days? Weeks? Months?

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 20:33 UTC (Sat) by pizza (subscriber, #46) [Link] (11 responses)

> How far (in time) is it behind? Days? Weeks? Months?

If you [1] cared about timely updates you'd have signed a service agreement with Red Hat.

If you're not explicitly paying someone for a specific level of service and responsiveness, any update you get is due to someone else's generosity and goodwill, and certainly not on any schedule that can be relied upon. Service guarantees cost money!

(Indeed, pre RH/Stream CentOS model routinely saw security updates held up for 1-2 months, twice a year, for the entirety of its existence while they iterated/rebased onto the latest RHEL point release.)

[1] the general you, not *you* specifically.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 27, 2023 8:25 UTC (Tue) by TRauMa (guest, #16483) [Link] (10 responses)

> If you cared about timely updates you'd have signed a service agreement with Red Hat.

Or you'd use a distro that doesn't hold back security fixes deliberately. Basically you're telling the original poster to get screwed. Which is funny because Red Hat points to stream as the solution for everyone who wants to live in the RH eco system and doesn't fit their commercial target.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 27, 2023 12:56 UTC (Tue) by pizza (subscriber, #46) [Link] (9 responses)

> Or you'd use a distro that doesn't hold back security fixes deliberately.

Oh, please. Who, exactly, is holding back security fixes? And how is that practice different from *any* other distribution, least of all the RHEL clones?

What you don't seem to understand here is that these security fixes are being created, or at least backported, by RH engineers, and that is what their customers are paying for. *everyone else* already had to wait until after the stuff went out to RHEL, and then create their own update incorporating the RHEL changes. (And, funnily enough, none of the rebuilders provided any support/updates for X.y after X.y+1 came out. So this hysteria about "holding back fixes" is talking about what has been the status quo *all along*.

> Basically you're telling the original poster to get screwed.

No, I'm telling the original poster that they can't expect to get what everything they want without having to pay something for it. RH owes its non-customers [1] *nothing*.

> Which is funny because Red Hat points to stream as the solution for everyone who wants to live in the RH eco system and doesn't fit their commercial target.

And that's ... wrong how?

[1] By "customers" I also include everyone who received binaries from RH.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 10:04 UTC (Thu) by madhatter (subscriber, #4665) [Link] (8 responses)

> Who, exactly, is holding back security fixes?

See mcatanzaro (who works for RH) above:

> We are not allowed to fix higher-severity security issues in CentOS Stream until the fix has shipped in RHEL.

A security fix has been prepared, but because it hasn't yet gone through whatever internal processes are required for shipping in RHEL, it can't be released for CentOS. That is pretty much my working definition of "holding back a security fix".

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 12:56 UTC (Thu) by pizza (subscriber, #46) [Link] (7 responses)

> A security fix has been prepared, but because it hasn't yet gone through whatever internal processes are required for shipping in RHEL, it can't be released for CentOS. That is pretty much my working definition of "holding back a security fix".

First, you realize this is for *embargoed* fixes, not "fixes" in general?

Secondly, rebuilders *always* had to wait until after a fix shipped into RHEL before they'd get the source to generate a build? And that RH *never* released sources to the rebuilders for anything other than the _current_ point release of RHEL?

And meanwhile, CentOS Stream still has the same processes and QA requirements as RHEL, so either way you were *never* going to get a fix for something until RH's internal processes were satisfied.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 13:38 UTC (Thu) by madhatter (subscriber, #4665) [Link] (6 responses)

> First, you realize this is for *embargoed* fixes, not "fixes" in general?

That is exactly not how I read what mcatanzaro said. My understanding was that they are not allowed to fix *any* higher-severity security issues in CentOS (until a RHEL fix has shipped) and very few such issues are embargoed. RH had already copped to the withholding in the case of embargoed issues, so I don't think his/her comment can be read any other way.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 13:56 UTC (Thu) by pizza (subscriber, #46) [Link] (5 responses)

> My understanding was that they are not allowed to fix *any* higher-severity security issues in CentOS (until a RHEL fix has shipped) and very few such issues are embargoed.

It was my understanding that very few issues are considered "high-severity" and even fewer of those are embargoed. So the vast majority of fixes land in CentOS *before* they end up in RHEL..

But even if you are completely correct, what exactly is the problem? Rapid access to serious updates is what RH's customers are paying for, after all! In the worst case it takes no longer to land publicly than it used to back in the old "we'll rebuild it after RHEL releases it" rebuilder paradigm, and everyone downstream was supposedly fine with that for the better part of two decades.

BTW, here's the full message, for context: https://lwn.net/ml/fedora-devel/D8JRWR.D2QFSVHOGZDV2@redh...

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 14:07 UTC (Thu) by madhatter (subscriber, #4665) [Link] (4 responses)

I'm not wading into the larger argument. I was simply responding to your question about who was holding back security fixes, which seemed to me to imply that you thought nobody was, when we'd just been told by an internal employee that they were.

You may be right that "very few issues are considered high-severity", but it's not just a numbers game: those are the very issues for which users most urgently need fixes, and for which there can be least justification for the withholding we are in fact assured takes place.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 14:24 UTC (Thu) by pizza (subscriber, #46) [Link] (3 responses)

Like many (most) things, this seems to be an argument over definition.

I do not consider fixing something in place A before you fix it in place B to be "withholding" it from place B. To me, withholding would be never offering that fix in place B to begin with.

Because by that same argument, RH is "withholding" *all* fixes from CentOS7 until after they are released in RHEL7. A practice that the rebuilders (and their users) were supposedly okay with for a couple of decades.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 14:40 UTC (Thu) by farnz (subscriber, #17727) [Link]

Note, too, that the fix may well be published upstream already (possibly even by Red Hat employees), and RH are simply applying it to the RHEL package.

In this circumstance, it's no more "withholding" a fix than Debian "withholds" fixes that are present in the latest upstream release but not in the most recent Debian package.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 14:51 UTC (Thu) by madhatter (subscriber, #4665) [Link] (1 responses)

> I do not consider fixing something in place A before you fix it in place B to be "withholding" it from place B

I do. I'm not saying such withholding is avoidable or reprehensible or surprising or unique to this case, but it *is* (to my mind, and apparently to mcatanzaro's mind) happening. If you don't want to think of it as such, that is of course up to you, but you might want to better define your terms before making general statements.

> Note, too, that the fix may well be published upstream already (possibly even by Red Hat employees), and RH are simply applying it to the RHEL package.

I don't see how it would have been published by RH employees, given that we've been told they are forbidden from working on it under those circumstances. If the upstream fix already exists from another source, and they're forbidden from pushing it through the CentOS pipe simply because it has not yet been pushed through the RHEL pipe, well then, that's withholding.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 15:55 UTC (Thu) by farnz (subscriber, #17727) [Link]

A fix may be published upstream first simply because at the time of fixing, the author didn't realise that they were fixing a security issue - this was just part of normal bug fixing when you take part in upstream development (and note that RH employs a lot of people who take part in upstream development). And the further behind the RHEL version of a package is, the more likely this is to be true, since you're less likely to remember all the development that's happened between the RHEL version being picked, and you making this fix.

And while it's a workable definition of "withholding", it does mean that any time you ship a released version and not the latest development tree, you're withholding fixes and features from your users - this isn't what most people think of when you say that things are being withheld.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 15:49 UTC (Sat) by mcatanzaro (subscriber, #93033) [Link] (3 responses)

I work for Red Hat. No, there are in fact private branches. RHEL is not released from CentOS Stream. Think of CentOS Stream as like Fedora rawhide, but for RHEL. Stable Fedoras branch from rawhide just like stable RHELs branch from CentOS Stream.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 16:19 UTC (Sat) by spmfox (subscriber, #125241) [Link] (2 responses)

In your opinion, is stream "stable" for production-like environments? Or, because it's similar to rawhide, is it possible for some serious bugs to make it in - even if they are short lived?

An example would be the recent XFS corruption bug. Is it possible something like this could make it into stream even for a few days?

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 25, 2023 7:29 UTC (Sun) by pbonzini (subscriber, #60935) [Link] (1 responses)

It's pretty good, I would have no problem using it for a small server. The kernel backports lag about 3 months and that usually gives enough time to find bugs and fix them. There's comprehensive CI and manual testing done before every commit (that was a multi-year effort prior to the announcement of CentOS Stream, and a requirement to make it viable). Especially rebases aren't done lightheartedly, even though they don't require all the red tape that was needed in the RHEL5/6 days.

The main issue of Stream is that it tells you what will be in RHEL, not what is on your *current* production machines that hypothetically run RHEL. That ironically makes it a better OS for production environments than for testing environments, in my opinion.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 25, 2023 12:27 UTC (Sun) by pizza (subscriber, #46) [Link]

> The main issue of Stream is that it tells you what will be in RHEL, not what is on your *current* production machines that hypothetically run RHEL. That ironically makes it a better OS for production environments than for testing environments, in my opinion.

Exactly. Despite all of the bits spilled about this, Stream is genuinely a better production environment for every use case [1] that doesn't *explicitly* require a specific RHEL point release. Those that have such strict requirements are brittle to the point where you won't/can't update beyond said point release without vendor approval, which makes even the old CentOS model (and what the current "free rebuilds" are still doing) a non-starter as once version N.M is released, updates for N.M-1 ceased.

(If you genuinely need bug-for-bug compatibility with a specific RHEL point release, the incremental cost for that license is going to be a rounding error on your other SW licensing/support costs. Please, prove me wrong.)

[1] Anectdotally, this applied to every environment I've deployed a CentOS installation, going back well over a decade.