Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model [LWN.net]

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 15:54 UTC (Sat) by mcatanzaro (subscriber, #93033)
In reply to: Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model by pbonzini
Parent article: Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

"1) with the exception of embargoed issues, bugs will always be fixed first in Stream and then in RHEL, so you'll never have a situation where Red Hat hides bugfixes from the community"

This part is not true. We are not allowed to fix higher-severity security issues in CentOS Stream until the fix has shipped in RHEL. Very few such issues are embargoed.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 19:57 UTC (Sat) by passthejoe (guest, #156034) [Link] (12 responses)

This is where you run into problems. How do you "convince" users to run Stream if it's always behind RHEL when it comes to critical security patches?

How far (in time) is it behind? Days? Weeks? Months?

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 20:33 UTC (Sat) by pizza (subscriber, #46) [Link] (11 responses)

> How far (in time) is it behind? Days? Weeks? Months?

If you [1] cared about timely updates you'd have signed a service agreement with Red Hat.

If you're not explicitly paying someone for a specific level of service and responsiveness, any update you get is due to someone else's generosity and goodwill, and certainly not on any schedule that can be relied upon. Service guarantees cost money!

(Indeed, pre RH/Stream CentOS model routinely saw security updates held up for 1-2 months, twice a year, for the entirety of its existence while they iterated/rebased onto the latest RHEL point release.)

[1] the general you, not *you* specifically.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 27, 2023 8:25 UTC (Tue) by TRauMa (guest, #16483) [Link] (10 responses)

> If you cared about timely updates you'd have signed a service agreement with Red Hat.

Or you'd use a distro that doesn't hold back security fixes deliberately. Basically you're telling the original poster to get screwed. Which is funny because Red Hat points to stream as the solution for everyone who wants to live in the RH eco system and doesn't fit their commercial target.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 27, 2023 12:56 UTC (Tue) by pizza (subscriber, #46) [Link] (9 responses)

> Or you'd use a distro that doesn't hold back security fixes deliberately.

Oh, please. Who, exactly, is holding back security fixes? And how is that practice different from *any* other distribution, least of all the RHEL clones?

What you don't seem to understand here is that these security fixes are being created, or at least backported, by RH engineers, and that is what their customers are paying for. *everyone else* already had to wait until after the stuff went out to RHEL, and then create their own update incorporating the RHEL changes. (And, funnily enough, none of the rebuilders provided any support/updates for X.y after X.y+1 came out. So this hysteria about "holding back fixes" is talking about what has been the status quo *all along*.

> Basically you're telling the original poster to get screwed.

No, I'm telling the original poster that they can't expect to get what everything they want without having to pay something for it. RH owes its non-customers [1] *nothing*.

> Which is funny because Red Hat points to stream as the solution for everyone who wants to live in the RH eco system and doesn't fit their commercial target.

And that's ... wrong how?

[1] By "customers" I also include everyone who received binaries from RH.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 10:04 UTC (Thu) by madhatter (subscriber, #4665) [Link] (8 responses)

> Who, exactly, is holding back security fixes?

See mcatanzaro (who works for RH) above:

> We are not allowed to fix higher-severity security issues in CentOS Stream until the fix has shipped in RHEL.

A security fix has been prepared, but because it hasn't yet gone through whatever internal processes are required for shipping in RHEL, it can't be released for CentOS. That is pretty much my working definition of "holding back a security fix".

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 12:56 UTC (Thu) by pizza (subscriber, #46) [Link] (7 responses)

> A security fix has been prepared, but because it hasn't yet gone through whatever internal processes are required for shipping in RHEL, it can't be released for CentOS. That is pretty much my working definition of "holding back a security fix".

First, you realize this is for *embargoed* fixes, not "fixes" in general?

Secondly, rebuilders *always* had to wait until after a fix shipped into RHEL before they'd get the source to generate a build? And that RH *never* released sources to the rebuilders for anything other than the _current_ point release of RHEL?

And meanwhile, CentOS Stream still has the same processes and QA requirements as RHEL, so either way you were *never* going to get a fix for something until RH's internal processes were satisfied.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 13:38 UTC (Thu) by madhatter (subscriber, #4665) [Link] (6 responses)

> First, you realize this is for *embargoed* fixes, not "fixes" in general?

That is exactly not how I read what mcatanzaro said. My understanding was that they are not allowed to fix *any* higher-severity security issues in CentOS (until a RHEL fix has shipped) and very few such issues are embargoed. RH had already copped to the withholding in the case of embargoed issues, so I don't think his/her comment can be read any other way.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 13:56 UTC (Thu) by pizza (subscriber, #46) [Link] (5 responses)

> My understanding was that they are not allowed to fix *any* higher-severity security issues in CentOS (until a RHEL fix has shipped) and very few such issues are embargoed.

It was my understanding that very few issues are considered "high-severity" and even fewer of those are embargoed. So the vast majority of fixes land in CentOS *before* they end up in RHEL..

But even if you are completely correct, what exactly is the problem? Rapid access to serious updates is what RH's customers are paying for, after all! In the worst case it takes no longer to land publicly than it used to back in the old "we'll rebuild it after RHEL releases it" rebuilder paradigm, and everyone downstream was supposedly fine with that for the better part of two decades.

BTW, here's the full message, for context: https://lwn.net/ml/fedora-devel/D8JRWR.D2QFSVHOGZDV2@redh...

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 14:07 UTC (Thu) by madhatter (subscriber, #4665) [Link] (4 responses)

I'm not wading into the larger argument. I was simply responding to your question about who was holding back security fixes, which seemed to me to imply that you thought nobody was, when we'd just been told by an internal employee that they were.

You may be right that "very few issues are considered high-severity", but it's not just a numbers game: those are the very issues for which users most urgently need fixes, and for which there can be least justification for the withholding we are in fact assured takes place.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 14:24 UTC (Thu) by pizza (subscriber, #46) [Link] (3 responses)

Like many (most) things, this seems to be an argument over definition.

I do not consider fixing something in place A before you fix it in place B to be "withholding" it from place B. To me, withholding would be never offering that fix in place B to begin with.

Because by that same argument, RH is "withholding" *all* fixes from CentOS7 until after they are released in RHEL7. A practice that the rebuilders (and their users) were supposedly okay with for a couple of decades.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 14:40 UTC (Thu) by farnz (subscriber, #17727) [Link]

Note, too, that the fix may well be published upstream already (possibly even by Red Hat employees), and RH are simply applying it to the RHEL package.

In this circumstance, it's no more "withholding" a fix than Debian "withholds" fixes that are present in the latest upstream release but not in the most recent Debian package.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 14:51 UTC (Thu) by madhatter (subscriber, #4665) [Link] (1 responses)

> I do not consider fixing something in place A before you fix it in place B to be "withholding" it from place B

I do. I'm not saying such withholding is avoidable or reprehensible or surprising or unique to this case, but it *is* (to my mind, and apparently to mcatanzaro's mind) happening. If you don't want to think of it as such, that is of course up to you, but you might want to better define your terms before making general statements.

> Note, too, that the fix may well be published upstream already (possibly even by Red Hat employees), and RH are simply applying it to the RHEL package.

I don't see how it would have been published by RH employees, given that we've been told they are forbidden from working on it under those circumstances. If the upstream fix already exists from another source, and they're forbidden from pushing it through the CentOS pipe simply because it has not yet been pushed through the RHEL pipe, well then, that's withholding.

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 29, 2023 15:55 UTC (Thu) by farnz (subscriber, #17727) [Link]

A fix may be published upstream first simply because at the time of fixing, the author didn't realise that they were fixing a security issue - this was just part of normal bug fixing when you take part in upstream development (and note that RH employs a lot of people who take part in upstream development). And the further behind the RHEL version of a package is, the more likely this is to be true, since you're less likely to remember all the development that's happened between the RHEL version being picked, and you making this fix.

And while it's a workable definition of "withholding", it does mean that any time you ship a released version and not the latest development tree, you're withholding fixes and features from your users - this isn't what most people think of when you say that things are being withheld.