Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model [LWN.net]

Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

Posted Jun 24, 2023 23:47 UTC (Sat) by pizza (subscriber, #46)
In reply to: Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model by NZheretic
Parent article: Kuhn: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

> As I pointed out that is the one of the exact argument used by the Telecom industry for the shift from Network Neutrality.

One of the many differences here is that the telecom industry has *never* provided anything free of charge. They also have a long, public history of double or even triple-dipping and acting in very bad faith.

So, please enlighten us as to why you or I or anyone other than Red Hat's paying customers are entitled to, in perpetuity and at zero cost, the software and services that Red Hat provides. And then explain how that doesn't apply to the software you have written.

Please.

to post comments

Why

Posted Jun 25, 2023 1:38 UTC (Sun) by NZheretic (guest, #409) [Link] (19 responses)

First of all, I fully acknowledge the effort the Red Hat, along with hardware vendors, has put into creating a stable Linux kernel & the distribution upon that platform. Red Hat & even IBM has been a major contributor to the open source & free licensed software community.

However ANY business decision that when implemented would significantly impact the market as a whole rightly deserves the full attention of Antitrust authorities, especially if the effect is world wide. ( The EU needs to look into this ASAP ).

CentOS ( released 14 May 2004 ) & Scientific Linux were created by government agencies, under the terms of the GPL & other open source licences, not based on Debian sources or other Linux distributions but based upon Red Hat Enterprise Linux sources because of the major issues of hardware comparability with off the shelf server hardware.

As I have pointed out "The main problem is that OEMs test & even validate server/workstation/desktop/laptop hardware for both Microsoft & RedHat OSs on the OEM provided hardware, under agreements which MAY have caveats that effect competition."
The issue is ongoing as new processors & hardware is being released constantly & will become an even more significant problem with the increasing deployment of newer APUs & other application acceleration hardware rolled into upcoming CPUs, motherboards & IO cards.

For the same hardware compatibility reason CentOS became the de facto distribution in countless VPS, cloud services, businesses & governmental organisations, who choose to support their deployments either internally or through third parties.
Even direct business competitors, such as Oracle's Linux distribution, chose to base off of CentOS exactly for the same reason.
There is now a significant market based upon both providing software & services upon that platform.
IBM's decision to limit the distribution of the sources by customer licences DIRECTLY impacts that existing market.

There has been a distinct advantage to Red Hat from the ubiquity of CentOS widespread deployment. It has remained the leading paid provider of services for the Linux platform & also collects and deploys many patches for fixes to its own Red Hat Enterprise distribution contributed by other companies & users of CentOS based distributions. These advantages were so significant that when the original CentOS maintainers began having difficulty keeping up with Red Hat patches, they folded the project into Red Hat itself.

Later Red Hat's subsequent removal of the stable distribution of CentOS rebadging it as CentOS Stream led to the inevitable creation of forks of the original CentOS kept tracked to Red Hat's Enterprise Linux Release.

https://itheresies.blogspot.com/2005_04_01_archive.html
"No vendor or open source software developer can block development for any substantial period of time without the risk of the development being taken over by a descendant of the same project -- it's called evolution."
Which, IMHO is a bigger risk to IBM's bottom line future profits from Red Hat. If the users of CentOS clones cannot access the same source then they will be forced to join together with upstream open source developers, vendors of all sorts & even antitrust authorities to force open equitable hardware vendor information access & create another single source distribution outside of the control of Red Hat & IBM.

As for the software I have written over the last three decades on behalf of my former employers, almost all of it has just been used internally although plenty of patches I have have added were contributed back to upstream open source projects under my former employers discretion.
Upstreaming is almost always easier than internal fork maintaining.

Why

Posted Jun 25, 2023 10:04 UTC (Sun) by tuna (guest, #44480) [Link] (4 responses)

"Which, IMHO is a bigger risk to IBM's bottom line future profits from Red Hat. If the users of CentOS clones cannot access the same source then they will be forced to join together with upstream open source developers, vendors of all sorts & even antitrust authorities to force open equitable hardware vendor information access & create another single source distribution outside of the control of Red Hat & IBM."

Or they could just use CentOS Stream.

Why

Posted Jun 25, 2023 10:20 UTC (Sun) by NZheretic (guest, #409) [Link] (3 responses)

"Or they could just use CentOS Stream." - Not for the roles in which the CentOS clones has been deployed as a stable distribution.

Why

Posted Jun 25, 2023 13:50 UTC (Sun) by pbonzini (subscriber, #60935) [Link] (2 responses)

CentOS Stream is pretty stable. Have you ever tried it?

Why

Posted Jun 25, 2023 14:31 UTC (Sun) by NZheretic (guest, #409) [Link] (1 responses)

Are you talking about stability in terms of reliability/uptime or the unchanging consistency of the ABIs/APIs of the packaged RPMs Libraries/RPCs etc?
Since it is both that has been offered by the finally released Red Hat Enterprise Linux & the CentOS ( pre-stream ) clones.

Why

Posted Jun 25, 2023 16:55 UTC (Sun) by pbonzini (subscriber, #60935) [Link]

Both. ABI goes without saying actually because an ABI change would not make it past CI for those libraries where RHEL guarantees stability.

Why

Posted Jun 25, 2023 10:36 UTC (Sun) by nim-nim (subscriber, #34454) [Link] (9 responses)

> Upstreaming is almost always easier than internal fork maintaining.

First, when you upstream, you will often work with someone whose salary is paid by Red Hat or one of its partners from money earned in the RHEL ecosystem.

Second, you severely undervalue the difficulty of maintaining something like RHEL. You need people able and willing to fix all the bits that go in RHEL, all year round, for a decade or so. It’s not a scratch your itch dev endeavour, where you write things on your own time, then go on holidays, then move to another project or bit of code because you’re fed up with the original one. You can not depend on specific deadlines like in an internal project, every single hour of the year is always critical for one of customers of RHEL. Getting that amount of sustained unglamorous work done takes big money (because people will accept doing unglamorous work for compensation).

Third,

> CentOS & Scientific Linux were created by government agencies

What those agencies did was take Red Hat work, and pay people to add and maintain a few components not present in RHEL, with a few fixes right and left. They never actually replicated the bulk of what Red Hat did. And they got fed up by this little effort soon enough, leading to CentOS & Scientific Linux demise.

In *theory*, it would save a lot of money to government agencies, big medium and small businesses, to finance a foundation charged with maintaining an alternative to RHEL. Hell in *theory* ISVs and IHVs could finance such a consortium themselves for a lot less money than they pay Red Hat, Google or Microsoft (but ISVs and IHVs have no interest in maintaining anything long term, they need the platform to exist for product launch and not much longer, and ISVs and IHVs compete with one another, so they are perpetually anxious about financing something that may benefit competition more than themselves).

In *practice* getting an awful lot of people and entities to agree among themselves in the name of paying less means BIG BUREAUCRACY. As in inefficiencies, infighting, high risk of getting stuck at some point because someone at some stage tried to avoid paying his dues. It took *years* for those people to agree on a way to pay *one* person, Linus Torvalds, a modest salary.

If it was *that* easy to make people work together no controversy would ever happen in Debian (haha), Canonical would not exist (people would have politely told Shuttleworth he was not needed) and RHEL’s marketshare would be zero. And Debian members love this stuff, they’re not a beancounter-ridden agency or business who would not care less about the software side.

Agencies and big business (who practice this kind of stuff all year round in the UN for example) are well aware of this difficulty. Which is why when they bitch about RHEL charges they’re not arguing about setting up a foundation they are arbitrating between giving their money to Red Hat, Microsoft, Amazon or Google (and sometimes get a few years of free ride before being forced to pay someone else some money).

Trying to set up a RHEL alternative is ridiculously easy, take the public Fedora or Centos stream sources, rebuild (hell you can even pay a one-time RHEL subscription if you want). Sustaining this effort without Red Hat doing most of the work for you, and convincing ISVs and IHVs to bet on your long term existence and performance, is ridiculously hard.

Why

Posted Jun 25, 2023 10:39 UTC (Sun) by nim-nim (subscriber, #34454) [Link]

> Trying to set up a RHEL alternative is ridiculously easy, take the public Fedora or Centos stream sources, rebuild (hell you can even pay a one-time RHEL subscription if you want).

Which is why BTW appeal to antitrust laws has no legal legs to stand on. The barrier to competitor entry is effectively nil. There are just no competitor willing to bet the amount of money necessary to make the whole thing sustainable and self-financing.

Why

Posted Jun 25, 2023 14:04 UTC (Sun) by NZheretic (guest, #409) [Link] (7 responses)

> First, when you upstream, you will often work with someone whose salary is paid by Red Hat or one of its partners from money earned in the RHEL ecosystem.
> What those agencies did was take Red Hat work, and pay people to add and maintain a few components not present in RHEL, with a few fixes right and left.

Those statements wouldn't be so galling if Red Hat was the sole or even majority contribute to the open source base Red Hat includes in distribution.
However Red Hat & IBM are not even the majority contributors to the Linux Kernel itself.
https://lwn.net/Articles/923410/
https://www.linuxfoundation.org/resources/publications/li...
https://project.linuxfoundation.org/hubfs/Reports/2020_ke...
"From 2007 to 2019" Red Hat supplied only 8.9% of commits & IBM 3.79%

It's like you have not even bothered to read the original article
Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model by Bradley M. Kuhn
https://sfconservancy.org/blog/2023/jun/23/rhel-gpl-analy...

Also you utterly fail to address the point I have make repeatedly
As I have pointed out "The main problem is that OEMs test & even validate server/workstation/desktop/laptop hardware for both Microsoft & RedHat OSs on the OEM provided hardware, under agreements which MAY have caveats that effect competition."
The issue is ongoing as new processors & hardware is being released constantly & will become an even more significant problem with the increasing deployment of newer APUs & other application acceleration hardware rolled into upcoming CPUs, motherboards & IO cards.

Red Hat has a long history of not sending all the Red Hat Enterprise Kernel patches it applies the stock Linux kernel to Linus to incorporate into the stock kernel code.
https://www.theregister.com/2011/03/04/red_hat_twarts_ora...
So despite Red Hat is not the major contributor to the Linux Kernel project, its OEM & hardware vendor relationship agreements mean that some particular patches it applies to the Stock Kernel are necessary for competing Linux distributions to interoperate with said hardware and operate in the market.

Red Hat itself has forked upstream projects like MySQL & CUPS for inclusion in its distribution when the upstream developer has changed the project licence to impose further restrictions. Shouldn't the rest of us have the same right to do the same with code developed under GPL by the community?

Why

Posted Jun 25, 2023 16:27 UTC (Sun) by pizza (subscriber, #46) [Link] (5 responses)

> However Red Hat & IBM are not even the majority contributors to the Linux Kernel itself.
> "From 2007 to 2019" Red Hat supplied only 8.9% of commits & IBM 3.79%

By that metric there are *zero* "majority" contributors to the Linux kernel.

Meanwhile, for every kernel version since approximately forever, LWN has broken down the top companies contributing to the Linux kernel. Red Hat has been at or near the top of that list for a *long* time, and RHEL subscriptions have funded all of that.

> The issue is ongoing as new processors & hardware is being released constantly & will become an even more significant problem with the increasing deployment of newer APUs & other application acceleration hardware rolled into upcoming CPUs, motherboards & IO cards.

Huh? All of this work is being done in the upstream kernels, not within the likes of RHEL. If you truly want bleeding-edge stuff you're going to have to use bleeding-edge software and distributions, not "stable enterprise" stuff that's obsolete before it ever ships.

> Red Hat has a long history of not sending all the Red Hat Enterprise Kernel patches it applies the stock Linux kernel to Linus to incorporate into the stock kernel code.

First, that article is a decade old. Second, that article doesn't say what you claim -- It talks about RHEL kernel sources combining all patches into a single file, not that that there are parts of it "not sent to Linus", or establish any sort of "history" of bad behavior. Red Hat has *always* shipped the complete corresponding source code for all GPL components to their customers and everyone else who receives binaries from them.

(Incidentally, *every* major distribution out there has out-of-tree patches in it. Linus and his lieutenants are infamously picky in what they accept, and sometimes you need to ship fixes or features before they are in a state mainline will accept. Red Hat also has a strict upstream-first policy, which means all new features are developed upstream first, and they won't ship (ie backport and support) any feature that hasn't first landed upstream.

> Red Hat itself has forked upstream projects like MySQL & CUPS for inclusion in its distribution when the upstream developer has changed the project licence to impose further restrictions.

Um, the original MySQL developer forked MySQL, not Red Hat, and pretty much everyone switched over to the new fork. CUPS was forked by OpenPrinting/Linux Foundation, not becaue of licensing issues, but because upstream (ie Apple) has long stopped caring about supporting anything other than current MacOS platforms and stripped out a lot of code that the Linux printing stack still needed. (Also, CUPS didn't "impose further restrictions" with their license change; they switched from GPLv2-only to Apache licensing, which strictly speaking is *incompatible* with GPLv2-only stuff. GPLv3 is compatible with ASL2, so that (and GPLv2+ stuff) wasn't affected)

Why

Posted Jun 26, 2023 4:58 UTC (Mon) by joib (subscriber, #8541) [Link] (4 responses)

CUPS is licensed under Apache 2.0 with the "LLVM exception" https://spdx.org/licenses/LLVM-exception.html which should make it compatible with the GPL 2.0.

Why

Posted Jun 26, 2023 13:40 UTC (Mon) by pizza (subscriber, #46) [Link] (3 responses)

> CUPS is licensed under Apache 2.0 with the "LLVM exception" https://spdx.org/licenses/LLVM-exception.html which should make it compatible with the GPL 2.0.

Do you have a citation for that?

Because this isn't documented in the LICENSE file of either CUPS repository [1], nor is it mentioned in the handful of source files I just looked at, which all seem (only) say: 'Licensed under Apache License v2.0 See the file "LICENSE" for mode information.

[1] Apple CUPS or OpenPrinting CUPS

Why

Posted Jun 26, 2023 13:47 UTC (Mon) by joib (subscriber, #8541) [Link] (2 responses)

Quoting from https://github.com/OpenPrinting/cups/blob/master/NOTICE :

>
> -- CUPS Exceptions to the Apache 2.0 License --
>
> As an exception, if, as a result of your compiling your source code, portions
> of this Software are embedded into an Object form of such source code, you
> may redistribute such embedded portions in such Object form without complying
> with the conditions of Sections 4(a), 4(b) and 4(d) of the License.

> In addition, if you combine or link compiled forms of this Software with
> software that is licensed under the GPLv2 ("Combined Software") and if a
> court of competent jurisdiction determines that the patent provision (Section
> 3), the indemnity provision (Section 9) or other Section of the License
> conflicts with the conditions of the GPLv2, you may retroactively and
> prospectively choose to deem waived or otherwise exclude such Section(s) of
> the License, but only in their entirety and only with respect to the Combined
> Software.

Why

Posted Jun 26, 2023 13:59 UTC (Mon) by pizza (subscriber, #46) [Link] (1 responses)

> Quoting from https://github.com/OpenPrinting/cups/blob/master/NOTICE :

Thank you.

But it's not clear which files, if any, this NOTICE applies to -- Because all of the source files still only mention ASLv2 and the LICENSE, with no mention of the exception.

(Also, looking at the git history, the exception was added in 2019, nearly two years after the ASL2 relicensing...)

Why

Posted Jun 26, 2023 14:07 UTC (Mon) by joib (subscriber, #8541) [Link]

That's something you'll have to ask OpenPrinting; I'm not affiliated with that project in any way.

Might actually be a good reason for a bug report, I agree with you the situation is unclear.

Why

Posted Jun 26, 2023 6:09 UTC (Mon) by nim-nim (subscriber, #34454) [Link]

> As I have pointed out "The main problem is that OEMs test & even validate server/workstation/desktop/laptop hardware for both Microsoft & RedHat OSs on the OEM provided hardware, under agreements which MAY have caveats" that effect competition.

And has anyone ever seen such an agreement Linux-side ? And how would such an agreement stand given that the software parts are all eventually published (nvidia aside which is an nvidia decision Red Hat disagrees with) so it would be mightily hard to hide anything special. The worst that you can say about new Red Hat arrangements is that in some cases sources are not published before Red Hat publishes the corresponding binaries, which is above and beyond what most OSS licensing requires (the whole of RHEL is treated as a GPL product secret building sauce included even though it embarks a lot of components with more lenient licensing).

OEM work with Red Hat because they are not a charity, if working with a few big players is sufficient to address the market why should they expend money working with small fishes ? Their time is valuable too you know.

Customers buy Red Hat because they know it will be there to fix things where their business-critical systems would go titsup (home owners have no such requirements which is why the Linux desktop never flourished, people do not see the point of paying someone for reliable youtube viewing).

> Those statements wouldn't be so galling if Red Hat was the sole or even majority contribute to the open source base Red Hat includes in distribution.

That’s irrelevant. You will pay a plumber an harm and a leg on week ends even though the parts he uses have been conceived by someone else, are readily available in hardware supermarkets, there are lots of online tutorials etc. You will pay an arm and a leg because you want things fixed *now*, he is available, you have no trust in your ability to fix things quickly yourself, and the über scientists that calculated the best materials and gaskets to use are not there (and are quite happy not to be there and deal with customer moods during their day off). And some people will blame the plumber for extortionate galling prices even though the hardware supermarket is this way and everyone else can see they could easily take up plumbing to correct the situation.

Why

Posted Jun 25, 2023 12:13 UTC (Sun) by kleptog (subscriber, #1183) [Link] (1 responses)

> However ANY business decision that when implemented would significantly impact the market as a whole rightly deserves the full attention of Antitrust authorities, especially if the effect is world wide. ( The EU needs to look into this ASAP ).

Let's not get carried away here. The threshold is that it has to lead sufficient market distortion that pricing mechanisms no longer work correctly. RedHat is working in a highly competitive market, with low vendor lock-in, anyone can cancel their subscription at any time without significant consequences. No-one is going out of business by this decision, Rocky & Alma Linux by their own admission will be continuing just fine, even if the product they produce is slightly different.

My reading of Kuhn article is that it's primarily concerned that it's no longer possible to verify that RedHat is actually GPL compliant. I can understand from SFC's view, this is a bad thing, but I'm really failing to see what harm this is causing anyone else (other than more work for Alma & Rocky Linux).

> If the users of CentOS clones cannot access the same source then they will be forced to join together with upstream open source developers, vendors of all sorts & even antitrust authorities to force open equitable hardware vendor information access & create another single source distribution outside of the control of Red Hat & IBM.

How is this different from any other Linux distribution? No-one's arm is being twisted to force them to use RedHat.

Why

Posted Jun 25, 2023 14:06 UTC (Sun) by NZheretic (guest, #409) [Link]

See
https://lwn.net/Articles/936274/

Why

Posted Jun 29, 2023 11:28 UTC (Thu) by madhatter (subscriber, #4665) [Link] (1 responses)

> CentOS [was] created by government agencies

I'd not heard that before, and don't believe it to be true. Could you elaborate?

Why

Posted Jul 4, 2023 6:18 UTC (Tue) by ceplm (subscriber, #41334) [Link]

I think the author confuses CentOS and Scientific Linux. The latter truly was created mostly by CERN and Fermilab.