|
|
Log in / Subscribe / Register

McGrath: Red Hat’s commitment to open source

[Posted June 26, 2023 by corbet]

Red Hat's Mike McGrath responds to the many criticisms aimed at the company since it changed its policy regarding RHEL source code.

Ultimately, we do not find value in a RHEL rebuild and we are not under any obligation to make things easier for rebuilders; this is our call to make. That brings me to CentOS Stream, of which there is immense confusion. I acknowledge that this is a change in a longstanding tradition where we went above and beyond, and change like this can cause some confusion. That confusion manifested as accusations about us going closed-source and about alleged GPL violations. There is CentOS Stream the binary deliverable, and CentOS Stream the source repository. The CentOS Stream gitlab source is where we build RHEL releases, in the open for all to see. To call RHEL “closed source” is categorically untrue and inaccurate. CentOS Stream moves faster than RHEL, so it might not be on HEAD, but the code is there. If you can’t find it, it’s a bug – please let us know.

to post comments

A note to commenters

Posted Jun 26, 2023 19:38 UTC (Mon) by corbet (editor, #1) [Link]

This topic has generated a lot of discussion recently; I suspect that will be the case for this posting as well. To make life easier for everybody, can we please try to avoid rehashing other recent debates and only post here if we truly have something new to add?

Thank you.

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 20:12 UTC (Mon) by mb (subscriber, #50428) [Link] (8 responses)

> I feel that much of the anger from our recent decision around the downstream
> sources comes from either those who do not want to pay for the time, effort and
> resources going into RHEL

I guess that means RH is now going to pay all upstream developers for the time, effort and ressources going into it, right?

> Simply rebuilding code, without adding value or changing it in any way, represents
> a real threat to open source companies everywhere.

Maybe. But it is fully compliant to the licenses that upstream decided to apply to their software.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 6:13 UTC (Tue) by oldtomas (guest, #72579) [Link] (7 responses)

> I guess that means RH is now going to pay all upstream developers for the time, effort and ressources going into it, right?

They have been doing it all along. By employing contributors to key projects. By sponsoring important initatives (e.g sourceware.org). If all commercial companies involved in free software behaved like RedHat, we'd be in a better place.

No, I don't work for RedHat. I don't even use their product (I'm more of a Debian type). But I do acknowledge that they have made my life better.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 8:12 UTC (Tue) by TRauMa (guest, #16483) [Link] (5 responses)

I wrote software that's included in Red Hat and I'm still waiting for my cut. Or any contribution. So?

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 8:28 UTC (Tue) by paulj (subscriber, #341) [Link] (4 responses)

Ditto.

I even applied for a job, and the manager who interviewed me it became quickly apparent the only reason they'd arranged the interview was to ask me about the background / politics of a fork of code I was working on. Pretty shit experience.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 14:12 UTC (Tue) by clump (subscriber, #27801) [Link] (3 responses)

Red Hat's hiring process has always been a bit "challenging". It's generally a disorganized process, with a pretty big disconnect between hiring managers and recruiters. When I interviewed people, I learned to first ask "Was the position explained to you?" and "Do you know the role you're applying for?"

What ends up happening is that referrals move to the front of the line because everyone knows each other and it bypasses the typical funnel. This seems pretty consistent with other tech companies but is still disappointing if it significantly disadvantages other applicants.

I don't know about your particular case however I heard frequent complaints about the hiring process from applicants.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 14:22 UTC (Tue) by paulj (subscriber, #341) [Link] (2 responses)

Mine was kind of weird. I was applying for a role that had nothing to do with the free software project I'd worked on - indeed, I was happy to get away from that project. The role was in the same general area of computing as the project, but not related otherwise.

It was clear to me afterwards the manager went into the interview with 0 intention of considering me for the role, but he just did it cause he wanted to ask me about issues with that (unrelated to the role) project. Something I didn't really want to discuss much - certainly not in that setting. Perfunctory questions to query my knowledge, no discussion about the role, and he concluded the interview telling me already had someone for it and sorry - but then the job req was left open for something like 12 months after.

He got my hopes up by scheduling an interview for a role I would have really liked, tbut wasted my time, and - as the months went by and I could see the ad still open - made me feel like he'd treated me like a fool. Just to satisfy his curiosity.

Still a bit annoyed by it.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 14:36 UTC (Tue) by paulj (subscriber, #341) [Link] (1 responses)

Oh, and in telling me he had filled the role, he said my experience and CV were great and it was a shame. So... whatever the actual reason - whether it was just his curiousity, or cause RedHat have some loyalties related to that project (which I wanted to get away from anyway), I felt treated like a fool.

To get back to the higher level point:

If your code is shipped by RedHat in RHEL, to paying customers, as some of mine was for years and years and is to this day I think, but that doesn't mean RedHat will ever support you.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 15:24 UTC (Tue) by paulj (subscriber, #341) [Link]

And to add a higher-level point again:

"Just buy a $WHATEVER_ENTERPRISE_LINUX licence - that's a good way to support open-source developers"

is not per se true.

These "enterprise Linux" corporations do not generally go and find some way to support all the developers of all the software they ship and support, where those developers do not already have some other support (e.g. another employer, or their own consulting). They employ leading developers on high-profile projects that they care about; and they employ maintenance engineers to support all the other stuff, often engineers from outside any of the communities of said software.

And a developer on some project, who really could use some kind of security in terms of employment, that was at least friendly to some OOH or even 10% time on non-core-job free software project maintenance, may be excluded from consideration by enterprise Linux corporations for any number of reasons - from where they live (e.g., not somewhere the corp hires), to industry politics (dev has issues with some other industry corporates, and said Linux corporate has some level or relations with those other industry corporates).

So that idea above, that $WHATEVER_ENTERPRISE_LINUX corps are good ways to help fund FOSS, is not one that I could agree with at all.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 16:37 UTC (Tue) by mb (subscriber, #50428) [Link]

>> I guess that means RH is now going to pay all upstream developers for the time, effort and ressources going into it, right?

>They have been doing it all along.

No. They haven't been paying all developers. By far. And that is perfectly acceptable and fine. It adheres to the licenses that the developers chose for their code.

>By employing contributors to key projects. By sponsoring important initatives (e.g sourceware.org).
>If all commercial companies involved in free software behaved like RedHat, we'd be in a better place.

Yes, I completely agree.
Red Hat is one of the biggest single contributors to Free Software.
But it's also far from paying all of the work that goes into their product.
How much of the work do they pay? 1%, 2%? 10%? Probably not more.

And that is perfectly acceptable and fine. It adheres to the licenses that the developers chose for their code.

What is not Ok is saying things like this:

>> I feel that much of the anger from our recent decision around the downstream
>> sources comes from either those who do not want to pay for the time, effort and
>> resources going into RHEL

because these people are just doing what the licenses permit.
These people are exploiting the *same* rights that Red Hat exploits to run their business. That is the basic concept and foundation of Free Software. These rights are at the base of Red Hat's business. It's what makes their business possible. If GNU/Linux was proprietary software, then they would have to pay for every single change set going into it.

But Free Software allows them to not pay all these bills. And that is fine.

Companies are using my Open Source and Free Software to make millions of Dollars. From where do I know? Because these companies frequently contact me for free support.

And that is perfectly acceptable and fine. It adheres to the licenses that I chose for my code. I knew that from the beginning and chose to go that route.

What would not be Ok is if these companies would put further restrictions on *my* code. E.g. by applying a threat model to stop business with their customers upon redistribution.

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 20:28 UTC (Mon) by davisr (guest, #154260) [Link] (58 responses)

> Simply rebuilding code, without adding value or changing it in any way, represents
> a real threat to open source companies everywhere.

My opinion, as an author who sells FLOSS, is that fewer people today are willing to pay for any software at all. Corporations are using mega-scale techniques to (a) sell user data to anyone willing to pay, and (b) entice users with gratis features, but lock them via proprietary/SaaS and slowly make them pay.

Users are coming to expect that software should be gratis. I make a frank appeal to my customers: either they pay for the software, or there will be no software. Pay for RedHat, or there will be no Rocky or Alma Linux. Out of all the companies I've worked for, they all paid next to nothing for their software, and they all eventually got what they paid for.

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 20:53 UTC (Mon) by Depereo (guest, #104565) [Link] (1 responses)

Extremely large technology companies are built on the back of unpaid and unappreciated open source software. Linux runs everything; on it sits open source databases, providing query responses to open source web application frameworks frontended by haproxy. The network equipment that connects this all runs on linux or BSD. Communication between all of this relies on OpenSSL, a thing that like, two people were maintaining for a long time without much if any investment.

Exploitation of open source / the commons from large organizations is so publicly and obviously profitable that I'm not surprised no one wants to pay for code. Not paying for labour is how people consolidate wealth, and lots of people want to be wealthy.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 0:41 UTC (Tue) by Paf (subscriber, #91811) [Link]

I’m not going to engage with the second part, but for the first part:

A huge part of the point and benefit of open source software is that it doesn’t have to be paid for. This is the source of enormous benefit to the world at large, and has dramatically reduced duplication of effort and certain kinds of concentration of power. Microsoft is a far less powerful company because of open source software.

This also creates a tragedy of the commons style difficulty getting development paid for. Well, so be it. It is a necessary corollary of that freedom/openness.

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 21:06 UTC (Mon) by kleptog (subscriber, #1183) [Link] (55 responses)

To be honest, this bugs me too. At my work we use a lot of Linux, mostly Debian but we also use all sorts for projects like Rust, Python, Django, VueJS, etc... You can download it all for free and build products on it, but it feels a bit like freeloading. I've tried to suggest that we have budget that we annually distribute as donations to open source projects we use a lot, but it doesn't work with management. Invoices they understand, donation are complex. All they hear "we would need to make lots of little payments to random people/organisations without PO numbers, that sounds like work and nothing goes wrong if we don't do it".

I've managed to arrange that if we find bugs, that we get time to figure the root cause, write a patch and submit it upstream, and that's a sort of payment. But sometimes it just doesn't feel like enough.

If someone has suggestions how to get a big organisation to contribute financially to some of the smaller open source projects they depend on, I'm all ears. I'm sure the mechanical process of payment can probably be solved, but the mindset issue is much harder.

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 21:30 UTC (Mon) by NYKevin (subscriber, #129325) [Link] (4 responses)

1. Don't characterize it as a donation. Characterize it as a "sponsorship."
2. Don't try to give money to every FOSS project you use. Sponsor things that management is willing to sponsor, or could be persuaded to sponsor (because the alternative is giving no money at all). For example, if you give substantial money or other assistance to Debian, they will (subject to various formalities) put your logo on https://www.debian.org/partners/ and write mildly nice things about you - that might be an easier sell than another project that doesn't have such a page.
3. Don't (just) sponsor projects directly. Sponsor conferences, events, organizations, and anything else that could reasonably help a project thrive.
4. If you can't get them to sponsor an event, at least try to get them to expense entry fees for any interested employees.
5. Talk to marketing and find out whether developers are a cohort they want to target. If so, they may be supportive of sponsorships.
6. If a project looks seriously underfunded, show management https://xkcd.com/2347/ and ask them what they plan to do when that one developer in Nebraska gets bored or retires. Write down their answer, and characterize it as an official contingency plan. If they refuse to answer, then document the non-existence of a plan, as overtly as possible (without looking like a smart ass).
7. If management wants to purchase a support contract, try to steer them towards whoever is actually doing the work, not just whoever has the lowest rate. Remember, the people who wrote the code are often best equipped to do something about bugs and other issues. A third-party support contract is much less valuable, because they may be unable or unwilling to do much more than file exactly the same bug report you would've filed yourselves.
8. Accept that a corporation is a profit-maximizing machine, and they won't always do what is right, or even what is in their own long-term interest. Do the best that you can, and don't worry too much about things you can't change.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 10:44 UTC (Tue) by tialaramex (subscriber, #21167) [Link] (1 responses)

Right. "Our name on a hat" is something companies can understand. A mid-size firm probably has made such arrangements already, just not for Free Software. Does a local kids sports team play in kit with your company name? Maybe that local theatre used by the town amateur dramatics society has seats with your company's name etched onto a little label?

Red Bull pays Grandpoobear to play Mario https://www.redbull.com/gb-en/athlete/david-grandpoobear-... while wearing their hat (with a Red Bull logo) and (obviously) not saying this is a terrible beverage, nobody should drink it. Is Mario an extreme sport? Sure, if you're as good as he is.

Sponsorship is good because companies understand what they're not getting for their money. Your logo on the Debian web site does not entitle you to bug fixes, or to decide what is or is not packaged.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 18:16 UTC (Tue) by HenrikH (subscriber, #31152) [Link]

It's also much easier to get money out of the PR department than having to force the software developer side create a budget for the expenses.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 9:44 UTC (Wed) by davidgerard (guest, #100304) [Link] (1 responses)

> 6. If a project looks seriously underfunded, show management https://xkcd.com/2347/ and ask them what they plan to do when that one developer in Nebraska gets bored or retires. Write down their answer, and characterize it as an official contingency plan. If they refuse to answer, then document the non-existence of a plan, as overtly as possible (without looking like a smart ass).

this is brilliant

McGrath: Red Hat’s commitment to open source

Posted Jun 30, 2023 19:19 UTC (Fri) by kpfleming (subscriber, #23250) [Link]

In the large-company world this is called "Business Continuity Planning" and is 100% appropriate. Many more companies should be doing this, for each and every piece of software they rely on to operate their business.

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 21:52 UTC (Mon) by Wol (subscriber, #4433) [Link] (2 responses)

This is why organisations like Red Hat, SUSE, Canonical, are important. Companies can take out a support contrtact, and then these can dole out money to the appropriate foundations looking after the "software of importance". Although, of course, what is seen as important, and what is important, are often not the same.

Cheers,
Wol

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 3:43 UTC (Tue) by Matt_G (subscriber, #112824) [Link] (1 responses)

I'll second the paid support is extremely important to large organizations. I work in an Industrial Plant (a large smelter) we have a lot of older system running custom apps (C/C++ code) for things like HMI's which are extremely critical. These mostly run HPUX or AIX, there are newer systems running Linux (Red Hat - I think) - we have had incidents in the past where companies have flown engineers to our plant to fix issues. Sometimes they will have written custom patches etc. for specific issues. The deployed life of these assets tend to be a long time - so long term support is crucial i.e. more than 2 to 3 years.

So I think this is one area where there is potentially a lot to be made in open source if you can offer this type of support there is potentially big money in the contracts and companies can and do pay for these services.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 17:05 UTC (Tue) by lyubomyrk (guest, #165831) [Link]

I work at a IMU design and manufacturing company and recently was asked to automate wafer inspections on 20 year old microscopes that are complete black boxes running MSDOS. No one remembered where the source code was and we got no support, running it for decades hoping nothing breaks otherwise we would have to spend $100,000 on new equipment. I still have nightmares when I opened up that Visual C++ project.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 9:03 UTC (Tue) by pabs (subscriber, #43278) [Link]

File bugs asking for them to send you an invoice you can send to your boss?

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 21:10 UTC (Wed) by jzb (editor, #7867) [Link] (45 responses)

The mechanical part of payment is *really hard* -- especially since many projects simply are not set up to receive money the way that corporations are set up to distribute it.

In a perfect or even better world all companies would follow your lead on bug reporting/patching. That would go a long way.

What would be even more amazing is if companies of certain sizes hired open source "janitors" or utility players to do that full time. Or if you can identify a project that is really important to you, hire a person to work on that project full time.

That could make vendor neutral projects much more sustainable and provide more work for developers who want to contribute to open source outside the vendor model.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 22:55 UTC (Wed) by pizza (subscriber, #46) [Link] (44 responses)

> The mechanical part of payment is *really hard* -- especially since many projects simply are not set up to receive money the way that corporations are set up to distribute it.

s/many/nearly all/

And even if you do have a way to receive money, the odds of you receiving enough money to do anything other than buy a pizza once in a while is effectively zero.

One project I'm heavily involved with receives (on average) enough to cover about 1/3 of its monthly hosting fees, to say nothing of compensating anyone for their actual time/work. And that's by far the most successful of any of 'em.

> That could make vendor neutral projects much more sustainable and provide more work for developers who want to contribute to open source outside the vendor model.

Even this "Vendor neutral" thing requires a project to be at least of a certain (large!) size to be viable. For every one of those there's probably 1000 (or more, looking at "modern" language ecosystems) that are entirely dependent on one or two volunteers.

</grump>

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 9:30 UTC (Thu) by kleptog (subscriber, #1183) [Link] (43 responses)

It remains one of the great missed opportunities of the internet that no functional widely available micropayments mechanism took hold. I know everything financial suddenly becomes complicated, but sometimes it's just too complicated.

But it can be done. If some YouTuber does a live-stream you can during the live-stream, with a few taps, donate some money. That works because one company, in this case Google, made it happen. I think maybe PayPal comes closest, but it's just a bit too clunky for micropayments. There's GitHub sponsors, but then I have to pick projects specifically.

What I'm kind of thinking of is that websites, projects, etc could include a header or something giving their donation details, a PayPal link or something. Then once a month a plugin could examine your browser history, summarise all this and calculate relative usages and present me with a list. I can then type in how much I want to donate this month and it will in the background donate to all the sites on the list in one go in proportion to usage (or some other metric).

This would take out much of the pain, since now I only need to do the payment steps once a month, rather than than for every site individually. I'd like to be able to donate €100 per month split across dozens of projects without the pain. Looking around it seems like all the individual parts are there, but it's all splintered.

One can dream.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 10:23 UTC (Thu) by excors (subscriber, #95769) [Link] (1 responses)

On Twitch (and I assume YouTube live-streaming is similar), I think the main motivation for donations is the social (or parasocial) aspect. You donate $5 and your username pops up on the screen for every viewer to see, and the streamer says "thanks [your username]", and you get some minor perks (like no Twitch ads on that channel for a month). Or you donate a lot of money and you get a big flashy animation popping up the screen, and the streamer is like "wow, that's so amazing, you're the best, let's have some poggers in the chat for [your username]" and lots of the other viewers spam emojis in response to your generosity, and you become a well-known member of their community, and get permanently displayed on a donation leaderboard, and maybe the streamer does a little dance for you or something.

I don't know if there's any data to back this up, but it often seems that most of the donated amount comes from a very small number of users (the whales) who spend hundreds or thousands of dollars a month on a single streamer - sometimes because they're very rich and like showing off, but sometimes they can't really afford it and they're trapped in this exploitative relationship with the streamer.

I'm not sure you could successfully replicate that donation model in other contexts (like for open source projects), without the associated community and real-time feedback to make people feel highly rewarded for their donations. The mechanism for frictionless payments is important, but the motivation for payment is even more important and seems like the harder problem.

McGrath: Red Hat’s commitment to open source

Posted Jun 30, 2023 10:33 UTC (Fri) by milesrout (subscriber, #126894) [Link]

Another question is: do you really want to replicate that model? As you rightly point out, the so-called "donations" (they're not donations, they're tips) are largely coming from a small number of users. Most of them are not rich. They're people stuck in parasocial relationships with streamers (and other members of those streamers' communities) because frankly they don't have any other social relationships. It's pretty exploitative. I think a big part of it is that it's a bit like the old "one-armed bandit" slot machine: there's a somewhat random reward mechanism involved. If you tip an amount, it might get a big reaction or a small reaction depending on what's happening on the stream, what time of day it is, how many people are there, random chance, etc. As far as I understand, it's been established that randomised rewards are more effective at creating Pavlovian conditioning than consistent rewards. It's essentially gambling money for social "reward".

There are programmers that livestream programming "content" on Twitch and YouTube, such as Jonathan Blow, Casey Muratori and others. I can only presume that they make a reasonable amount of money doing it, or they likely wouldn't. They definitely have parasocial communities. That aside, even if it were possible outside of livestreaming, is it really a good thing? "Free software funding" is an end, and perhaps an admirable one. But that doesn't necessarily justify the means. Isn't this just exploiting people? What are people actually getting from these tips? Transient applause? Whales are basically gambling addicts. To use an analogy, slot machines make pubs a lot of money, but - to put it mildly - they're a bit ethically questionable.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 10:51 UTC (Thu) by paulj (subscriber, #341) [Link] (40 responses)

There are websites that exist to make crowd-funding FOSS easier, e.g. Snowdrift.coop (https://lwn.net/Articles/625051/), LiberaPay, BountySource and such. Snowdrift wiki has a good page with a comparison table of various such sites:

https://wiki.snowdrift.coop/market-research/other-crowdfu...

As for micropayments, well, some might say the Internet has seen some distributed, open-source, payments systems become wildly successful since the above LWN article was written. E.g., Bitcoin and - more suitable for true micro-payments - Monero (added advantage of providing very good privacy, anonymous tipping).

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 11:59 UTC (Thu) by anselm (subscriber, #2796) [Link] (39 responses)

I would be very reluctant to mention the terms “Bitcoin”, “payment system”, and “very successful” in the same sentence. If the last decade has shown us anything, it is that Bitcoin is completely – and unfixably – unsuitable as a payment system at scale. (With transaction fees currently at US$2.70 or so, that goes especially for micropayments!) It is barely scraping by as a type of investment, and even that is likely to go down the drain for good eventually, now that the SEC has the big crypto exchanges in its sights.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 12:32 UTC (Thu) by paulj (subscriber, #341) [Link] (37 responses)

Have you looked at fees for international bank transfers? E.g., SWIFT? Have you looked at the fees that card payment processors charge?

Bitcoin is indeed more expensive than it should be, but it's comparable to the above. Anyway Monero fixes most of Bitcoin's issues. Fees on Monero transfers are sub-cent as a result. (Monero isn't perfect - not good for instant - but pretty good).

Bitcoin is _so_ much easier for transferring value to arbitrary others than the banking system - even just within Europe. I had to pay something fairly large recently, on behalf of a company, and it took _days_ to get funds consolidated into 1 account. And then it took 2 days just to transfer money _within the same bank_ - along with most of a morning spent on the phone with the bank, to figure out limits and bugs in their online banking (one limit being "sorry, can't do anything about that - you have to physically go to a branch").

Separately, I have been waiting for _2 weeks_ and counting for a bank transfer to me via SWIFT. There are large chunks of the world that are stuck with either slow-slow SWIFT or else exploitative money-transfer companies.

The whole pump-and-dump investment bros around crypto are a good reason to dislike and be sceptical (esp as an investment), but as a global clearing system this stuff works /way/ better than the banking system.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 21:13 UTC (Thu) by kleptog (subscriber, #1183) [Link] (6 responses)

> Have you looked at fees for international bank transfers? E.g., SWIFT? Have you looked at the fees that card payment processors charge?

That's the depressing part: there no technical reason for the charges to be so high or it to take so long. It's just laziness and greed. Once you're used to SEPA Instant with free (or nominal cost) payments in under 10 seconds to anywhere in the Euro area you suddenly see how weird it is that most of the world lives with shitty payment systems. And it's not that banks suddenly figured they'd do this to improve customer service. No, the EU simply regulated that it should happen, and while the banks fought tooth and nail all the way, it did happen in the end. Same with the credit cards fees being capped at 0.3%.

(I get not everywhere in Europe has SEPA Instant, but it demonstrates that there's no technical limitations here.)

So while Bitcoin could be competitive in places with truly awful banking systems, that says more about how bad most banking systems are than how good Bitcoin is.

Of course, currency exchange is a special problem that still affects some SEPA transactions but again, there's no technical reason for the costs, only greed and laziness.

McGrath: Red Hat’s commitment to open source

Posted Jun 30, 2023 9:50 UTC (Fri) by paulj (subscriber, #341) [Link] (5 responses)

The domestic banks in my country do not support SEPA Instant. It's very frustrating.

Also, SEPA is only free for retail users. Business customers pay fees for SEPA transfers. Those fees are generally significantly higher - 3+ orders of decimal magnitude - than, say, Monero transaction fees. And card acquiring fees of 0.2% are still damn high.

McGrath: Red Hat’s commitment to open source

Posted Jul 1, 2023 2:53 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

I don't believe SEPA has different rules for B2B transactions, at least we're not paying any unusual fees as a company. Perhaps it's your bank's issue?

McGrath: Red Hat’s commitment to open source

Posted Jul 1, 2023 14:13 UTC (Sat) by kleptog (subscriber, #1183) [Link] (2 responses)

SEPA only says that international has to cost the same as domestic transfers, so it depends on your bank. For a business account here we're paying I think €0.10/txn, which is a lot higher than (IIUC) Monero's ~$0.003/txn. OTOH, you can't pay your rent with Monero. And for us, that 10c is really nothing compared to other costs of business.

SEPA rules only apply for Euro payments though, so if you're in one of the non-Eurozone countries you're working in a completely different system with different rules.

McGrath: Red Hat’s commitment to open source

Posted Jul 1, 2023 16:32 UTC (Sat) by anselm (subscriber, #2796) [Link]

SEPA rules only apply for Euro payments though, so if you're in one of the non-Eurozone countries you're working in a completely different system with different rules.

That's not entirely correct. The United Kingdom, for example, participates in SEPA but was never part of the Euro zone. (In effect, this means that, e.g., giro transfers from and to the UK – in EUR – are cheap but you still need to bear the cost of conversion between EUR and GBP.)

McGrath: Red Hat’s commitment to open source

Posted Jul 6, 2023 16:10 UTC (Thu) by paulj (subscriber, #341) [Link]

In the context of the micro-payments in this discussion, 10c is a lot.

McGrath: Red Hat’s commitment to open source

Posted Jul 3, 2023 12:41 UTC (Mon) by paulj (subscriber, #341) [Link]

Not sure if it's SEPA or not, but there are rules here and there on fees for transfers of retail accounts. Or perhaps they are just expectations.

B2B banking often has higher fees. In return you get "better" interfaces and support.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 23:41 UTC (Thu) by anselm (subscriber, #2796) [Link] (4 responses)

Bitcoin is indeed more expensive than it should be, but it's comparable to the above.

Jet liners sometimes crash, but that doesn't imply that flying carpets are a better alternative.

In fact, the current US$ 2.70 average transaction fees are near the low end for Bitcoin, and only because hardly anyone uses it for payments these days (most of the transaction volume is wash trading inside exchanges). Back during the Bitcoin hype of 2020, average transaction fees of US$ 25 were not unheard of, with even higher spikes – which is OK if you're paying for a Tesla; less so for buying a cup of coffee or a newspaper, let alone making a “micropayment”. Of course with Bitcoin, the transaction fee is basically a bribe to entice Bitcoin miners to consider your transaction at all, so if you post a transaction fee that is too low, your transaction will never be picked up for the blockchain. At least with SWIFT, for all its hassle and obvious faults, a transaction will eventually go through; with Bitcoin, there is no such guarantee.

I haven't paid a lot of attention to Monero, but since it, like Bitcoin, is based on a proof-of-work blockchain it is likely to share most of Bitcoin's main disadvantages, i.e., it is potentially a grotesque waste of energy and won't scale to usable size. The additional issue with Monero is that it is a common ingredient of malware and viruses and therefore virus scanners will pick up on Monero miners. Its enhanced privacy means that it is becoming the cryptocurrency of choice for ransomware, and it is also popular on darknet markets, which means that many crypto exchagnes won't touch it. Some alternative.

McGrath: Red Hat’s commitment to open source

Posted Jun 30, 2023 10:09 UTC (Fri) by paulj (subscriber, #341) [Link]

That most transactions are on exchanges is... utterly irrelevant. This is true for Euro and Dollars too. While this might be part of a point about the investment/speculation side of currency markets - points I might agree with you on largely - it's not really relevant to a discussion about utility as a clearing system. It's like comparing a currency exchange to SWIFT. Different thing.

Bitcoin fees are generally no worse than, and usually much better, than SWIFT. And... I wouldn't even prefer Bitcoin, I'd go with Monero for clearing.

Energy use: The cheapest energy for mining is energy that can not otherwise be sold. E.g., hydro-power in remote areas (dams may be built for water management, with a turbine... just cause - e.g. to power the dam; there may be little local demand), hydro-power and other renewable power in off-peak times, gas flaring in hydro-carbon extraction (yes, we should greatly reduce HC dependence, but till we do, this will be happening - before it was just burned, now some oil companies get some work out of that heat through mining). A large chunk of BTC mining is from such energy sources - it is dishonest to estimate the energy of BTC mining and then just consider /all/ that energy to be a) from hydro-carbons b) energy that would not have been used if not for BTC mining.

I myself run a miner in the winter. I am going to use electricity to heat water, to heat the air. Instead I use some electricity to do computation and the waste heat heats the air. Essentially, I get a (very small) discount on my electricity bill - energy I was going to use anyway. Rather than just put the energy into a dumb heating element and radiator to heat a portion of my house, my computer gets to help play a small part in a global financial clearing system to heat a portion of my house.

Finally, the idea that humanity could somehow use less energy is ridiculous. Energy is quality of life. Energy is modern, western, life. A very large chunk of the world still aspires to achieving that. And they are not going to stop till they get it. And the privileged minority enjoying that life do NOT get to tell those other people to stay in their shacks and "think of the environment!".

We _are_ going to use _more_ energy in the future. And we _must_ figure out how to generate _much more_ energy, without HCs. Sustainable, CO2-neutral and *abundant* energy is required for the future. The answer is pretty obvious, but - sadly - fellow greens insist on denying it (which just ensures we continue with HCs, sigh).

As for privacy in financial transactions, I think it's a good thing. And I think we should have it. That some people use privacy for bad things doesn't mean we should ban privacy. The flip-side, of having governments able to view and control /all/ financial transactions between people (a future governments are actively building towards), has very very bad potential too.

Our editor isn't going to be happy if we continue though, probably. You can figure out my email address I think if you want to continue.

McGrath: Red Hat’s commitment to open source

Posted Jun 30, 2023 10:14 UTC (Fri) by paulj (subscriber, #341) [Link] (2 responses)

Also, I have never had a transaction on Bitcoin take anywhere as long as SWIFT transactions do. I have had BTC txes take days, in rare cases, which is still quicker than SWIFT.

I have had more than 1 case where bank transfers just got lost, and it took weeks to prod the banks into doing something of substance to go find it. The sender would say "We definitely sent it" - receiver "We definitely didn't", and that was it for /weeks/. A SEPA transaction no less.

SWIFT transfers typically take at least 3 days, usually 5, and sometimes 2 weeks.

The banking system is _terrible_. It's horrendously inefficient compared to Monero.

McGrath: Red Hat’s commitment to open source

Posted Jul 3, 2023 12:48 UTC (Mon) by gfernandes (subscriber, #119910) [Link] (1 responses)

I don't know when you last used swift. But I've done repeated international transfers with negligible fees and instantly (confirmed receipt on the other end, in one hour or less).

All through the "traditional" banking system.

Yes, direct bank to bank.

McGrath: Red Hat’s commitment to open source

Posted Jul 3, 2023 13:39 UTC (Mon) by pizza (subscriber, #46) [Link]

Yeah, it's been my experience that "real" bank-bank transfers happen quickly; it's the "consumer-facing" stuff that's batched and delayed and otherwise enshittified to maximize the opportunities for arbitrage fee/interest skimming.

McGrath: Red Hat’s commitment to open source

Posted Jul 3, 2023 20:13 UTC (Mon) by abufrejoval (guest, #100159) [Link]

> Have you looked at fees for international bank transfers? E.g., SWIFT? Have you looked at the fees that card payment processors charge?

I work for one and I can assure you, that we're a lot cheaper, at least within a relatively local range (say within the EU).
Cheaper than cash generally, because people tend to forget that cash has significant cost and risks (and feathers a more comfortable stuffing).

You can certainly buy a sandwidch for €2.50 anywhere within the Eurozone, even with VISA or MC involved without everybody losing money on the transaction, I see kids buying sweets for €0,50 using plastic. And they don't have to wait until the candy has melted before the transaction is validated, either.

We've come a long way...

In the good old day, yes we got 2% and we bought a new IBM mainframe every other year. But at current interchange fees we try to save money with CentOS.

You see, we can't really afford five minutes of down-time, let alone hours. By the time any human Redhat support person has even answered the phone, we have blood running along our office corridors and mutiny in the shops.

There is some value in Redhat code, but to us no value in their support.

We paid for RHEL, but ran CentOS, simply because it avoided having to deal with the whole business workflow of subscriptions. And we ran for a long time on OpenVZ, which Redhat stupidly ignored, pushing for CXL, VMs or Podman.

They can be an incredibly arrogant and pig headed bunch of people and it's rather unfortunate that those types are now ruining a company that has or at least used to have people of very different mindset.

McGrath: Red Hat’s commitment to open source

Posted Jul 4, 2023 10:38 UTC (Tue) by farnz (subscriber, #17727) [Link] (23 responses)

Have you looked at fees for international bank transfers? E.g., SWIFT? Have you looked at the fees that card payment processors charge?

I do fee-free transfers internationally with my banking setup; I also get a decent exchange rate in the process. Just for kicks, I've looked at what would happen if I tried to do the transfer by buying Monero with one currency, and selling it for the other, and I'd get about 97% of what I get by doing international bank transfers by the time I'd paid the various exchange fees converting USD to Monero, and Monero to local currency.

I also looked at what I'd end up getting if I did local currency to Monero to local currency, avoiding the credit card processors - the total fees for that route end up being around 4% to 5% (as in, I start with $100, I end up with $95 to $96 out), where I can get under 3% from credit card processors (where I get $97 to $97.20 out from $100 in). Further, with the card processors, I'm guaranteed compliant with my local anti-money-laundering and know-your-customer regulations (since the card companies take care of that); Monero exchanges require me to take that risk on myself. This risk is an unlimited fine and up to 15 years in jail for failing to perform adequate checks on the sources of funds.

I have heard complaints like yours from people transferring money in and out of Poland and Hungary, where the local banking system isn't yet as well-regulated as elsewhere, but I would be wary about generalising from those countries.

McGrath: Red Hat’s commitment to open source

Posted Jul 6, 2023 16:09 UTC (Thu) by paulj (subscriber, #341) [Link] (22 responses)

The context here is micro-payments. Can you make and accept micro-payments with the banking system? Globally? Or at least, between the major developed economies? It's great you can do SEPA within Europe, but... how would an EU project accept micro-payments from a US resident? And vice versa?

Yes, buying some Monero, to have to hand to distribute micro-payments to whatever X number of projects you want; and a project taking X micro-payments to an exchange to convert to the local currency might cost 4%, but if the alternative is: No micro-payments, then Monero wins. Does it not?

That said, I was talking more about the technology, not the value of the underlying token. The value and utility of the token recorded in a distributed ledger is a social issue, not technological.

Given Monero is designed to have much lower inflation (in the long-term) than any fiat currency you would have used to buy it with, one could argue "Why convert back to fiat? Just hold what you don't immediately need to convert - it will (slowly) appreciate". But that's not a technology argument, and not really on-topic for LWN. ;)

AML: Centralised Exchanges are required to comply with AML laws. Further, if you're dealing just in small amounts, AML laws may not apply to you in your jurisdiction. There is a whole other debate here about the social wisdom of governments stripping away privacy in transactions between citizens, with an evident goal of acquiring complete state insight into /all/ financial transactions - as described in working documents from various central banks and EU and others. But again, I guess not on topic for LWN. (I don't think it is wise, and I think privacy is a fundamental requirement to have some robustness against state oppression in the long run, and we should have that privacy even IF that privacy also enables some criminal use of money - cause you're not going to stop criminals finding things of value to exchange anyway; but hey).

McGrath: Red Hat’s commitment to open source

Posted Jul 6, 2023 16:55 UTC (Thu) by farnz (subscriber, #17727) [Link] (21 responses)

From my perspective, Monero involves me paying about 2.5% fees to turn local currency into Monero, then a transaction fee to get the Monero to the recipient, who has to pay about 2.5% to turn the Monero back into their currency. Compare that to credit cards, where the recipient can pay about 3% fees in total - if credit card payments aren't suitable for micropayments due to fees, then Monero, with its higher fees is definitely not suitable.

And that's the problem - Monero doesn't actually solve any of the issues that affect micropayments. To use it as an actual payment mechanism requires either someone to bootstrap it to the point where I can deal solely in Monero (e.g. paying my taxes, my food bills, my housing bills etc in Monero), without introducing a significant crime problem (since a significant crime problem means that I can't use Monero for other transactions).

Otherwise, all I see is higher fees than credit cards (around 5% for Monero, to at most 3% for cards, and at most 0.5% for international bank transfers with the accounts I have), just split into three buckets instead of being a single monolithic fee.

McGrath: Red Hat’s commitment to open source

Posted Jul 7, 2023 8:44 UTC (Fri) by paulj (subscriber, #341) [Link] (20 responses)

The use case here, the context of this sub-thread, is global Internet micro-payments. That's a use-case *not* served by the (not many) regional fast+free regional banking clearing systems, which are restricted to deal in the same currency (UK Faster Payments, SEPA Instant - not sure about US).

Outside those exceptions, if you want micro-payments to anywhere, Monero is probably the best and most accessible (for sure if you value privacy; and if we're talking global use cases, donations to projects that may displease one's local authoritarian regime are certainly one).

We are simply /never/ going to see any trad-banking-model, no-currency-exchange (i.e. single currency), global micro-payments system. So I appreciate that in your particular region, for your particular currency, you may have a local banking system that can meet that need /slightly/ cheaper. And go use that for donations to your project for others in the same region! But, that's not going to work globally, and it never will!

You're in the UK I think. How do I make a donation of, say, 5 Euro cents, to you for some project of yours via the banking system?

McGrath: Red Hat’s commitment to open source

Posted Jul 7, 2023 9:48 UTC (Fri) by farnz (subscriber, #17727) [Link] (7 responses)

You ask me for my Euro bank details, and send me a SEPA payment - I have an account with Wise that has banking details in multiple regions, including the Eurozone, USA, Canada, Australia, Singapore and other places, and I can get accounts with other providers who will give me account details in a foreign banking system.

And the other way I could take a micropayment is via the credit card system - I can get a card handling setup that has fees of 2%, minimum withdrawal £50, if I want to take micropayments. Compare that to the best I can do with Monero - 2.5% fees, minimum withdrawal also £50. If credit cards are a non-starter, Monero is also a non-starter, because the fees are higher.

This is the fundamental problem with Monero as a solution - you take my total 2% fee for credit cards, and you split it into a 2.5% fee for buying into the system, a transaction fee, and a 2.5% fee for buying out of the system. If you did the same split for credit cards, it's 0% to buy into the system, no transaction fee, and a 2% fee for buying out of the credit card system.

McGrath: Red Hat’s commitment to open source

Posted Jul 7, 2023 9:56 UTC (Fri) by paulj (subscriber, #341) [Link]

Ok, reply here with your bank details (name and IBAN), and I'll send you 5c.

I will quite happily give a Monero address here.

McGrath: Red Hat’s commitment to open source

Posted Jul 7, 2023 9:58 UTC (Fri) by paulj (subscriber, #341) [Link] (5 responses)

Also, Wise is kind of cheating a little. It's 1 company, not a clearing system. And there's no guarantee a) Wise will have you as a customer, b) Wise will keep you as a customer (e.g., if the balance of currency flows from your account doesn't fit in with their business, they may kick you).

McGrath: Red Hat’s commitment to open source

Posted Jul 7, 2023 10:19 UTC (Fri) by farnz (subscriber, #17727) [Link] (4 responses)

It's one of many alternatives - I use Wise, but I could have the same from Starling Bank, Revolut, or HSBC (to my knowledge - I chose Wise simply because of the four I looked at, they were the first to get back to me with confirmation that they could give me the specific letters I needed to get through the processes a US sender of currency wanted me to follow to get money from them).

The clearing system will be SEPA, for Euro payments to any of those three.

And I have related problems with Monero - there's no guarantee that any of the exchanges will let me convert my Monero into useful currency, nor is there a guarantee that if I use an exchange to turn Monero into real money that I won't then be held personally liable for someone else's financial crimes. At least with traditional banking, I'm guaranteed (by regulation) that the banks are the people liable, not me.

Note, too, that even if I go for the expensive option, the fees are still lower than Monero (0% to buy in as compared to Monero's 2.5%, zero transfer fee as compared to Monero's fraction of a cent, and 2% buy out fee, as compared to Monero's 2.5%). It's just that Monero splits the fees up so that you're hyper-focused on the thing that's free to me in the credit card system, and pointing out that that component of the fees is smaller than the total fees in the card system.

McGrath: Red Hat’s commitment to open source

Posted Jul 7, 2023 11:02 UTC (Fri) by paulj (subscriber, #341) [Link] (1 responses)

Ok, if Wise + SEPA is so much better and so much more suitable for public solicitation of micro-payments/tips, please give me your details so I can send 5c.

I am quite happy to give you my details for my preferred system: 87774rpgLdmjCFLqyV3BYN6VwBzdvaVbccVUF2K3NHGEFyoQKxCTqcxeDcPHpQPixqitthXhYK5uGbYuFExff24ACiaAUkH

McGrath: Red Hat’s commitment to open source

Posted Jul 7, 2023 11:06 UTC (Fri) by farnz (subscriber, #17727) [Link]

I've attempted to send you money. Tell me how much I sent when you get it.

McGrath: Red Hat’s commitment to open source

Posted Jul 7, 2023 11:04 UTC (Fri) by paulj (subscriber, #341) [Link] (1 responses)

Also, Wise is not free. Wise cost you money to buy-in too.

McGrath: Red Hat’s commitment to open source

Posted Jul 7, 2023 11:07 UTC (Fri) by paulj (subscriber, #341) [Link]

Least, Wise definitely charge money for certain accounts, e.g. business. And also certain kinds of transfers on top of that.

McGrath: Red Hat’s commitment to open source

Posted Jul 7, 2023 9:53 UTC (Fri) by paulj (subscriber, #341) [Link] (11 responses)

And I'm serious - post your name and IBAN and I'll see if I can send 5c.

And bear in mind that once you publish such details it's possible for others to setup direct debits. ;) (Inc giving those details to another company to setup a DD to pay for something).

McGrath: Red Hat’s commitment to open source

Posted Jul 7, 2023 16:17 UTC (Fri) by Wol (subscriber, #4433) [Link] (10 responses)

> And bear in mind that once you publish such details it's possible for others to setup direct debits.

If the recipient is sensible, an attacker CAN'T set up direct debits.

Most accounts are fully part of the banking system - sort codes, bank account numbers, IBAN etc. Not all, however, are current (checking) accounts.

If I publish bank account details, I always use a savings account, where much of this functionality just doesn't work. RECEIVING money always does, though. I would have no worries whatsoever making this info available on the Internet.

Cheers,
Wol

McGrath: Red Hat’s commitment to open source

Posted Jul 10, 2023 13:36 UTC (Mon) by glondu (subscriber, #70274) [Link] (9 responses)

I don't know about other countries, but in France, only the owner of a savings accounts (or his/her legal representatives) can wire funds to it. So it is not suitable for micro-donations. While setting up an IBAN only for receiving micro-donations (via SEPA) seems possible in theory, I would say it's not convenient in practice (and I am not aware of such commercial offering in France).

McGrath: Red Hat’s commitment to open source

Posted Jul 10, 2023 15:33 UTC (Mon) by geert (subscriber, #98403) [Link] (8 responses)

Interesting. Anyone can wire funds to my (Belgian) savings account, but only the account (mandate) holder can withdraw or wire funds from it.

Hence for receiving money (e.g. as a gift suggestion on a wedding invitation) people typically hand out a savings account number, which works also for children who are too young to have a checking account (which does not mean they can get married, though ;-).

McGrath: Red Hat’s commitment to open source

Posted Jul 10, 2023 15:47 UTC (Mon) by zdzichu (subscriber, #17118) [Link] (7 responses)

Maybe this is the France being special, or maybe some misunderstanding? Here in Poland we have savings accounts and settlement accounts (or more common hybrid savings-settlement accounts). Saving accounts typically have higher interest rate, but various limitations – like the number of free wire transfers per month or transfer only to predefined destinations.

Settlement accounts are more flexible. It is typical to give the account number to people to receive money (like when splitting the receipt). It is totally safe. There is a possibility of setting up direct debit for the account, but it requires account owner agreement.
I have DD setup with my mobile phone company, it works smoothly. I'm not aware if there is a possibility to limit direct debit amount. But I'm sure if the telecom ordered a transfer bigger than usual 10 EUR (that's monthly for 2 phones), I would have complaint processed quickly either by my bank or by the telecom itself.

McGrath: Red Hat’s commitment to open source

Posted Jul 10, 2023 16:29 UTC (Mon) by paulj (subscriber, #341) [Link] (6 responses)

If someone sets up a DD you did not authorise, you should be able to get it cancelled easily. Getting it refunded can still eat some of your time, phoning and emailing the bank and/or the business concerned.

A company could maliciously setup a DD or - more the case - someone who has your details may misrepresent themselves to a company and have a DD setup to your bank account (possibly obtaining some service or goods from said company, as part of a fraud). You should get your money back, but it can still cost a bit of a time.

Guess how I know.

McGrath: Red Hat’s commitment to open source

Posted Jul 10, 2023 16:31 UTC (Mon) by paulj (subscriber, #341) [Link]

Oh, and in the UK Jeremy Clarkson famously published his bank account details, as part of his claims in his column that some other data privacy breaches of bank account details were not important. Someone then disabused him of this by getting a DD setup out of his account.

http://news.bbc.co.uk/2/hi/7174760.stm

McGrath: Red Hat’s commitment to open source

Posted Jul 10, 2023 16:39 UTC (Mon) by paulj (subscriber, #341) [Link]

In my case, every time I cancelled the DD the company doing so would simply set it up again. The bank could not tell me exactly which company. I had to make an educated guess, based on the name. Some kind of payday loan company. However, ringing up the company a) Cost money (another country) b) They either had no record of me as a customer, under my name - or at least, that's all they would tell me given I couldn't give them any customer account / reference number.

In the end my bank had to completely remove the ability to setup DDs on that account to stop this DD reappearing again and again - they didn't have a function to refuse /specific/ DDs.

McGrath: Red Hat’s commitment to open source

Posted Jul 10, 2023 17:16 UTC (Mon) by zdzichu (subscriber, #17118) [Link] (3 responses)

How someone could setup DD without my authorisation? They would have to forge my signature.

McGrath: Red Hat’s commitment to open source

Posted Jul 10, 2023 20:42 UTC (Mon) by Wol (subscriber, #4433) [Link]

You clearly haven't set up a DD in the UK (and possibly Europe). It's incredibly easy to set up a fraudulent DD. It doesn't happen often, because (a) it's normally easily reversed, and (b) the banks are legally on the hook.

It's just a right pain if you're the victim, which is why I tend to treat my current account details as seriously private.

Cheers,
Wol

McGrath: Red Hat’s commitment to open source

Posted Jul 11, 2023 8:45 UTC (Tue) by paulj (subscriber, #341) [Link]

No signature is required for DDs in a number of countries, AFAIK. If a company wants to set up a DD, they just need your bank details to do so. And someone can misrepresent themselves to a company and give your bank details. I guess the company has to swallow the cost when it's discovered, but it will still cost you one or more mornings of phone calls to get it sorted out. See my sibling comment.

McGrath: Red Hat’s commitment to open source

Posted Jul 11, 2023 11:20 UTC (Tue) by kleptog (subscriber, #1183) [Link]

The bank does not physically check the signatures. The organisation setting up the direct debit asserts to the bank that they have received appropriate permission (a signature isn't the only way). In exchange, the end-user gets prior notification of the direct debit and can cancel it anytime and even undo any direct debits for the last 13 months. And the bank will charge the organisation setting up the DD a hefty charge-back fee. Do it too often and they'll kick you off altogether.

For this reason my apartment management refuses to do DD for the monthly service costs, because of shenanigans with people selling their apartment and then reversing all the fees for the previous year. Sure, you can prove that you had the right to do DD, even a signature, but you'll have to send it to debt collection or even the courts to get your money.

So yes, someone could publish their IBAN here and I could (attempt to) setup a direct debit for it, but it would be fraud. Really big organisations like energy and insurance companies use DD because people undoing direct debits without cause is just another one of the many things that can go wrong and it's just a cost of doing business.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 12:40 UTC (Thu) by paulj (subscriber, #341) [Link]

tl;dr: Monero would be quite suitable (technically) for projects to accept micro-payments.
- cheap, suitable for sending cents.
- anonymous by default (sender of a tx can always reveal themselves, of course, and prove it)
- open-source
- good community around it

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 20:54 UTC (Mon) by mgulick (subscriber, #63735) [Link] (5 responses)

The gitlab.com repository seems pretty hard to navigate. Most of the repositories in the CentOS Stream RPMs group don't have tags, just 'c8s' and 'c9s' branches. How do I find the corresponding source for a given package release? It looks like I have to browse the history and looks at each of their diffs to find when the package version and release changed in the spec file.

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 23:56 UTC (Mon) by jgg (subscriber, #55211) [Link] (1 responses)

I'm very interested in this as well, can someone post a git link to the commit of a RHEL kernel in Centos stream? eg one with the security backports after RHEL x.y is released?

If we think about a security commit to the kernel there will be at least three versions of it
- One in Linus's tree
- One backported to Centos Stream HEAD (eg for the next RHEL release)
- One backported to a released RHEL kernel x.y.z update

The way I read the blog post makes it sound like the first two are public, but the last one is not? Is that right?

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 5:00 UTC (Tue) by pbonzini (subscriber, #60935) [Link]

Yes, exactly. For example https://gitlab.com/redhat/centos-stream/src/kernel/centos... is a commit that might plausibly end up in 9.2 (I haven't checked, I am on the phone).

For the kernel and 20-30 other packages you have the entire "exploded" source tree on GitLab, with git history including merge commits; the rpm doesn't include individual kernel patches because they're just too many.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 23:19 UTC (Wed) by mmcgrath (guest, #44906) [Link] (2 responses)

> The gitlab.com repository seems pretty hard to navigate

I'm not trying to be facetious, but that's life in the big city. To get from that code to a RHEL distribution is a *lot* of work. Providing debranded source rpms in git.centos.org saved rebuilders a TON of effort. They're welcome to do the work we're doing from CentOS Stream (or straight from upstream if they're brave enough).

I really do think it's a double standard for anyone to think Red Hat should have to do that work, but asking the rebuilders to do it is too much.

McGrath: Red Hat’s commitment to open source

Posted Jun 30, 2023 19:50 UTC (Fri) by apple4ever (guest, #164280) [Link] (1 responses)

> I really do think it's a double standard for anyone to think Red Hat should have to do that work, but asking the rebuilders to do it is too much.

Why should they duplicate the work though? Especially since RH had already been doing it and providing it.

I think the decision is very shortsighted, and only will end up hurting RH in the long run.

McGrath: Red Hat’s commitment to open source

Posted Jul 1, 2023 13:51 UTC (Sat) by farnz (subscriber, #17727) [Link]

Why shouldn't other people do the work, and let Red Hat take their changes instead some of the time?

Red Hat does the work because its customers are paying for it to happen, and they want to retain those customers who care. CIQ are offering paid support for Rocky Linux, and could also do the work from the funds they get from their customers.

What Red Hat does not feel obligated to do is provide this service to CIQ for free, while CIQ is charging its customers for support; if CIQ step up to the mark and publish supported security upgrades for free, then Red Hat can take as well as give.

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 21:31 UTC (Mon) by thebluesgnr (guest, #37963) [Link] (31 responses)

> Simply rebuilding code, without adding value or changing it in any way, represents
> a real threat to open source companies everywhere.

Ehh, this is exactly what a Linux distribution is.

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 22:07 UTC (Mon) by quintesse (guest, #14569) [Link]

Creating a Linux distro is a HUGE amount of work! Without them you'd be forced to go to the sources of each and every component you want to install on your system (check the installed packages on your system, there are thousands), figure out what dependencies they need and what versions of those dependencies work together, then compile them (have you thought about how to compile those sources when you don't have a running system yet?), install them, configure them in the correct way and then hope your system will boot.

So distros definitely don't just "rebuild code without adding value", nobody would be running Linux if it wasn't for them. Heck even Linus would probably still be running Minix if it wasn't for distros.

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 22:13 UTC (Mon) by NightMonkey (subscriber, #23051) [Link] (1 responses)

Does this mean that every time I use Gentoo's Portage to install a new package I am a "real threat to open source companies everywhere"?

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 22:43 UTC (Mon) by dullfire (guest, #111432) [Link]

Shh. I don't think you are supposed to point that out. Or the fact reproducible builds are a thing. Or any other fallacies that may or may not have been in that blurb.

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 23:55 UTC (Mon) by flussence (guest, #85566) [Link] (6 responses)

Oof. That that line ever got published is a huge failure on the part of IBM's PR and legal teams.

Are they saying RHEL *is* this threat, or that it *isn't* this - are they confessing that their product differentiator is in modifying other people's codebases and then being obstinate and hostile to upstream in the same vein as Apple or grsecurity? Are they reversing their endorsement of things like Flatpak?

It's hard to see this reaction as anything but dismissive contempt for all of the people who actually built most of the OS they repackage into RPMs and slap a price tag on. It reminds me a lot of elementaryOS at this point.

There was never a question as to whether behaviour like this is legal (it is - just like what Oracle does to them is), but they're not coming back from below the trust thermocline with this weasel talk.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 0:55 UTC (Tue) by tuna (guest, #44480) [Link]

"Are they saying RHEL *is* this threat, or that it *isn't* this - are they confessing that their product differentiator is in modifying other people's codebases and then being obstinate and hostile to upstream in the same vein as Apple or grsecurity? Are they reversing their endorsement of things like Flatpak?"

RH is probably the best company in the world in pushing things upstream. According to the blog post that won't change.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 8:10 UTC (Thu) by milesrout (subscriber, #126894) [Link] (4 responses)

> Are they saying RHEL *is* this threat, or that it *isn't* this - are they confessing that their product differentiator is in modifying other people's codebases and then being obstinate and hostile to upstream in the same vein as Apple or grsecurity? Are they reversing their endorsement of things like Flatpak?

Why are you commenting if you actually haven't got any idea about the issue at all? They contribute a lot upstream. But you can't contribute "packaging" upstream, because application and library developers don't *want* to be distributions, and quite reasonably. RHEL does not upstream its integration work, and it would make no sense to. No distribution does.

What RH is no longer interested in doing is *downstreaming* its integration work to freeloading rebuilders.

McGrath: Red Hat’s commitment to open source

Posted Jun 30, 2023 19:55 UTC (Fri) by apple4ever (guest, #164280) [Link] (2 responses)

> What RH is no longer interested in doing is *downstreaming* its integration work to freeloading rebuilders.

Except they are not freeloading at all, but providing a valuable service. I suppose if that's what RH wants to do, that's up to them. But let's not pretend that this won't hurt RH in the long run.

McGrath: Red Hat’s commitment to open source

Posted Jul 3, 2023 13:17 UTC (Mon) by draco (subscriber, #1792) [Link] (1 responses)

> Except they are not freeloading at all, but providing a valuable service.

What valuable service do the downstream copycats provide?

McGrath: Red Hat’s commitment to open source

Posted Jul 3, 2023 13:35 UTC (Mon) by pizza (subscriber, #46) [Link]

> What valuable service do the downstream copycats provide?

From the rebuilders' users' perspective, they save 'em $350-800/system/year [1], so it's pretty "valuable".

From upstream/RH's perspective, though... according to McGrath, the rebuilders (as a whole) provide negative value, selling their own service/support offerings while still depending entirely [2] on RH's engineering.

[1] ie it's RHEL, maintained for at least five years at zero cost.
[2] To be fair, Oracle does at least provide & support their own kernel, and presumably better/easier integration with their other SW offerings, but the rest of it is a straight wart-for-wart RHEL rebuild.

McGrath: Red Hat’s commitment to open source

Posted Jul 2, 2023 12:29 UTC (Sun) by flussence (guest, #85566) [Link]

> you actually haven't got any idea about the issue at all

I (and evidently most people on the planet aware of this fiasco) do understand what's going on, thanks. It requires a bit more thinking than "Line go up exponentially forever, time to act like the Apache OpenOffice Crusader". IBM really struggled in the goodwill department after the SCO litigation, didn't it.

> RHEL does not upstream its integration work, and it would make no sense to. No distribution does.

Grow a little imagination. Ubuntu sends fixes back to Debian all the time. Debian itself is advancing the state of reproducible builds. Gentoo developers regularly reach out to upstreams everywhere to fix broken build systems in ways that benefit everyone. Without Alpine's extensive work to make using containers not suck you'd all still be suffering from the original pkg-config implementation, and containers themselves would be as much of a bloated insecure garbage fire as modern webdev. Rising tide lifts all boats, but not U-boats.

> What RH is no longer interested in doing is *downstreaming* its integration work to freeloading rebuilders.

Then would RH be interested in receiving and paying invoices for all the front line support those distros have been shielding it from thus far? No? How about adequate compensation for the accessible training onramps they provide into the RH money factory? Hey, maybe we could use sloccount to produce an itemised list of how much they owe the tens of thousands of individual upstream authors as a starting point. Let's make everything here brutally transactional and see how long IBM survives in the arena when it can't nakedly exploit people via externalities. Spoiler: it won't.

Is anyone under 40 proactively seeking out and paying for the RHEL certification from a cold start these days or is it all incumbent legacy experts with a time-limited window of productivity? Why would anyone smart buy into a Linux distro full of stale software careening toward terminal mass brain drain while its owning company is visibly flailing around to claw pennies out of random targets that *do not want or need* its only core competency of *providing active support*?

RHEL's only unique selling point, and the only thing that's kept it relevant to the outside world ever since the internet reduced the costs of software distribution to epsilon, is the service of being *a Misery Sponge* for corporate suits to yell at when their important computers break — everything else they're trying to nickel-and-dime here is available from a hundred other sources for *free*. Often better quality too, because *other* distros don't habitually butcher things like perl in the base install and then leave the outside community to carry the tech support burden they caused. So much for getting what you paid for!

Do you understand yet, what happens when you try to extract all the slack out of an already stressed system? It snaps. Things stop working. And at that point you are swamped in so much debt - technical, social - that must be paid down just to get it to start moving again. We do not live in a spreadsheet and woe upon anyone who tries to play that game with real lives.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 0:20 UTC (Tue) by jepler (subscriber, #105975) [Link] (17 responses)

I'm curious if it's possible to estimate what proportion of the work of having software in my favorite distro can be attributed to the project authors vs the packagers.

The most blunt instrument would be to look at the overall size of a source package vs the packaging. I tried comparing raw disk usage (du -s), non-empty lines (find -exec grep . {} + | wc -l) and sloccount for debian ffmpeg 4.3.5. The first two gave a similar figure of around 0.9% of ffmpeg being in the packaging; the latter gave 0.03%! This is because sloccount treats most files in debian/ as not "source code".

A second package, ntp 4.2.8p12+dfsg, is 1.76% packaging by du, 0.45% by wc, and 0.08% by sloccount.

As a gut check, "packaging software is 1% of overall software effort" doesn't seem outrageous.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 1:17 UTC (Tue) by pizza (subscriber, #46) [Link] (14 responses)

> As a gut check, "packaging software is 1% of overall software effort" doesn't seem outrageous.

This doesn't account for integration & subsequent testing.

One can quibble over the portion, but it's more than zero.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 3:11 UTC (Tue) by wtarreau (subscriber, #51152) [Link] (11 responses)

Indeed, building and testing costs a huge amount of developers time. Just figuring correct build options that end up as a single line in a spec file or any distro-specific build recipe can sometimes take a full day. Anyone having played with ./configure and build the same stuff 50 times to try to figure the correct paths etc so that everything assembles nicely knows it. Other distros know it pretty well, including the large free ones (debian, gentoo etc).

Here one difference is that a lot of developers' time is made available by a single company that collects the money to pay their salaries via sales. That's an important part of the ecosystem. Also one must not forget their significant representation among various critically important projects such as the kernel itself.

It continues to be important to me that this company (and a few other ones like Canonical) continue to make profit and pay as many developers as they can because ultimately they are the ones doing the free work we all want to rely on. Many people love to see Linux as the week-end hobbyist project it used to be, that's no longer the case. Just look at the amount of messages on LKML on week-ends, it's super low. The reality is that the projects we love are entirely run by people being paid to do that work, and that's what makes it scale that large.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 9:14 UTC (Tue) by pabs (subscriber, #43278) [Link] (6 responses)

Personally I think it is a huge problem that such a large percentage of paid upstream FOSS development depends on and is reliant on and directed by RedHat and other large companies. The funding for FOSS needs to be way more diversely sourced and distributed. This is especially concerning because of the types of projects that get funded and the audiences and activities that those projects enable.

Apart from that, it is definitely important that the existing funding RedHat represents doesn't dry up.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 13:32 UTC (Tue) by wtarreau (subscriber, #51152) [Link]

It can be seen as a problem or as an opportunity. I'm used to say that many opensource projects last only 3 years, i.e. while the creator is a student with lots of spare time. Once the creator gets a job and a family, the project dies by lack of time. The only sustainable model is when the author(s) get paid for their work so that they can do that *instead* of doing something else. Small companies are having a hard time achieving this except in a few situations, because they're always struggling to get money to pay their existing employees. As a result it's essentially well established large companies that can afford to pay a (sometimes large) team to spend time on opensource, and the community benefits from the regularity of their commitment. This obviously has an impact in their operating costs and products/services prices, but it's important to think about them when one wants to participate financially to the whole ecosystem. With that said, if you only have a little bit of money, think about the small companies first, because for them it's even more important given that their efforts put them in a much more risky situation. And self-funded volunteers definitely are heroes if they can do that for a living.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 11:23 UTC (Wed) by timofonic (guest, #165836) [Link] (1 responses)

> The funding for FOSS needs to be way more diversely sourced and distributed. This is especially concerning because of the types of projects that get funded and the audiences and activities that those projects enable.

I tought the same many years ago, even more after knew about IBM buying RedHat and now this.

My following opinion might be controversial: I consider GPLv4 must be made and designed in a clear way to avoid toxic behaviors like RedHat/IBM one.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 8:11 UTC (Thu) by milesrout (subscriber, #126894) [Link]

RedHat is doing precisely what the GPL is intended to allow! That's the entire POINT of the GPL.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 11:56 UTC (Wed) by paulj (subscriber, #341) [Link] (2 responses)

In some utopian world we'd have some kind of "FOSS clearing house" - akin to the way there are royalty clearing houses for music in some places.

There'd be some kind of representative sampling of computers running FOSS, to see what FOSS got used - voluntary/opt-in. With some kind of relatively rigorous correction for sampling biases. Then companies and individuals could donate to the clearing org, who could distribute proceeds onward to FOSS developers in some more-or-less fairish way.

Obviously, a lot of difficult and subjective things there.

We do have FOSS steward non-profits I guess, but they can be selective. Your project may not get taken in, and/or trying to do so may be an extremely long and opaque process. Some such stewards also take donations from large corporates - large corporates who have different interests to small FOSS developers - which may cause conflicts of interest in the stewarding org, particularly if the stewarding organisation (or officers thereof) have other activities around FOSS (licence enforcement, licence development, FOSS corporate outreach, etc.).

That's not meant to disparage such orgs, just it's a difficult world for FOSS.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 13:11 UTC (Thu) by nim-nim (subscriber, #34454) [Link] (1 responses)

> In some utopian world we'd have some kind of "FOSS clearing house" - akin to the way there are royalty clearing houses for music in some places.

That ’s a bad analogy, it implies the basic problem is insuring every dev is paid, and a distribution is just the sum of the work of those devs. Integrating a pile of source code into a reliable system that keeps working year after year is a *huge* amount of work. When you take into account deployment in the field, non-dev work involved into deploying the code the dev wrote, can easily get into several orders of magnitude more than the code writing itself.

As long as devs refuse to see they are just one part of the ecosystem they will be unable to imagine a payment structure that works in the real world.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 13:15 UTC (Thu) by paulj (subscriber, #341) [Link]

Oh, quite agree. Some kind of "good" sustainable model for FOSS development and *maintenance* is something we still don't have great answers for.

I'm sure LWN has had at least some articles on the challenges wrt maintaining FOSS, e.g.: https://lwn.net/Articles/712215/ - I thought there was another more recent than that, but can't find.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 9:16 UTC (Tue) by dottedmag (subscriber, #18590) [Link] (3 responses)

Having to wrestle with build systems is such a waste of everyone's time, and largely an self-inflicted wound. Moreover, the idea of hundreds of optional dependencies and configurations is also quite troublesome.

Having spent last several years profesionally in an ecosystem where there is no ./configure wrangling and no guess-the-dependency-version game, I'd suspect that streamlining it for C and C++ would release immense amount of effort to be applicable to something more useful than just building stuff (testing it, for example).

Alas, I couldn't see any way to coordinate improvements: in a Red Queen race one cannot stop and think how to change the race completely. New build systems for C and C++ improve ergonomics, but they do not question the way software is built.

McGrath: Red Hat’s commitment to open source

Posted Jul 2, 2023 23:23 UTC (Sun) by mathstuf (subscriber, #69389) [Link]

> New build systems for C and C++ improve ergonomics, but they do not question the way software is built.

I'd say that the Blaze-style build systems *do* that (bazel, buck, etc.). However, they come with the "you're going to vendor *everything*, right?" solution that…doesn't work for general FOSS projects.

McGrath: Red Hat’s commitment to open source

Posted Jul 4, 2023 18:56 UTC (Tue) by ssmith32 (subscriber, #72404) [Link] (1 responses)

What ecosystem is this?

I've worked in several different ones, from PHP, to C/C++, Java, and JavaScript.. they all have these issues, in some form.

McGrath: Red Hat’s commitment to open source

Posted Jul 4, 2023 21:23 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

I think this is more about the oodles of `--enable-*` flags that end up existing in C and C++ projects over time. Do you want network support? How about MPI? Sanitizers? Oh, you are missing some deps, that's OK, we'll just "helpfully" silently disable some features for you. Oh, we found some obscure dep on your machine, let's "helpfully" enable some broken part of our code.

Python, Rust, etc. have much more narrow "how is this built again?" style questions that tend to accumulate in the C and C++ ecosystem.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 18:16 UTC (Wed) by jepler (subscriber, #105975) [Link] (1 responses)

it's sure possible that "integration & subsequent testing" cost a lot of effort without yielding a lot of lines of code, and are done more by packagers than by upstream maintainers.

another major item I didn't account for is packager work that is (whether immediately or later) accepted upstream, which wouldn't be shown when counting lines in the distribution packaging.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 19:29 UTC (Wed) by mb (subscriber, #50428) [Link]

>it's sure possible that "integration & subsequent testing" cost a lot of effort without yielding a lot of lines of code, and are done more by packagers than by upstream maintainers.

Fortunately the GPL does not talk about development effort, packaging effort, maintenance effort, support cost and lines of code at all.

If somebody decides to distributes somebody else's GPL software, they can't further restrict it.
If somebody decides to distribute their own software under GPL license, why would they complain? Just distribute it under non-GPL (just like SuSE did with YaST), if you don't want other people to exploit GPL freedoms. But distributing GPL software and than complaining that customers exploit their right is just ridiculous.

It is as simple as that. If a company does not want their distribution to have GPL freedoms as a whole, just put a core component under a more restrictive license. If the company doesn't want to do that, because they want to be all-free software, then stop complaining, please, if people exploit their freedoms.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 5:48 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

> As a gut check, "packaging software is 1% of overall software effort" doesn't seem outrageous.

Then what's the big deal? Just get a one-time copy of RHEL source code and maintain it yourself.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 8:08 UTC (Thu) by milesrout (subscriber, #126894) [Link]

And what are the percentages for the vast majority of packages that are smaller than large packages like ffmpeg?

What about the time and effort that goes into it? Everyone knows "lines of code" is a terrible way to measure difficulty. Some lines of code are a lot harder to write and maintain than others.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 11:42 UTC (Tue) by nim-nim (subscriber, #34454) [Link] (1 responses)

>> Simply rebuilding code, without adding value or changing it in any way, represents
>> a real threat to open source companies everywhere.

>Ehh, this is exactly what a Linux distribution is.

Nope, otherwise competitors would have no problem restarting from upstream sources, instead of complaining bitterly Red Hat makes it harder to copy all the stuff it added over upstream sources to make them build and run reliably.

Why bother cloning something if it adds no value over public upstream ?

Distributions add a hell of a lot of value over raw upstream sources in the form of finding the build settings that actually work, finding the versions combinations that actually work, testing the result at scale on multiple hardware architectures and wrapping it all in an uniform deployment format so end users are not exposed to differences of dev opinion on how stuff should build.

(I intentionally omit all the bug fixing and other dev work distributions also perform, because even if upstream code was perfect and without a bug there would still be a huge value on the other non dev stuff performed distro-side).

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 14:12 UTC (Tue) by farnz (subscriber, #17727) [Link]

As an aside, note that much of Red Hat's stuff added over upstream sources is available in https://gitlab.com/redhat/centos-stream/ - while this isn't everything in RHEL, the difference only matters if you're either not keeping on top of security-relevant changes upstream yourself, or if it really really matters that you're on the same package version as RHEL N.M, and not on the version that's going to be part of RHEL N.M+1 or RHEL N+1.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 19:39 UTC (Tue) by mfuzzey (subscriber, #57966) [Link]

Not really.

A Linux distribution does far more than "rebuilding code", even if 100% of the source they use is available in the various upstreams.

They select components to use, figure out the dependencies, package it all as a coheseive whole, handle security updates and bug fixes and often add quite a bit of their own code. This is true for all distributions both commercial and community. And all of that is a *lot* of work.

I agree with Red Hat that *pure rebuilding* of an already created distribution without changing or adding anything doesn't add any value. On the other hand I don't agree that it is necesarilly a "threat to open source companies everywhere". A company that just does support wouldn't be threatened by rebuilders - their money comes from support contracts that are paid by those that want them. The problem for Red Hat is that they don't want to be (just) a support company but effectively sell a distribution. Rebuilders do threaten that model by reducing the number of customers.

McGrath: Red Hat’s commitment to open source

Posted Jun 26, 2023 23:44 UTC (Mon) by sadoon (subscriber, #155365) [Link] (28 responses)

Man's got a point. Developers need to live after all.
I don't care for the attitude of the cloners. They're not forks, they're not offering anything different, what's the point of their existence other than getting RHEL for free?

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 0:49 UTC (Tue) by Paf (subscriber, #91811) [Link]

Nothing. And, yeah - that’s open source for you. It’s open, especially the free software sort. That makes it hard to sustainably charge money for, because if it’s important and you charge enough, people can do this sort of thing. It’s an unavoidable corollary of the openness. (The thought behind my words is here is that impugning their motivations, etc, won’t work - it will just keep happening as long as RHEL is open source and what they charge for access is high enough.)

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 1:40 UTC (Tue) by pabs (subscriber, #43278) [Link] (26 responses)

In open source, you don't get a monopoly over commercial exploitation of your work, including the trademark-replacing clones:

https://drewdevault.com/2021/01/20/FOSS-is-to-surrender-y...

RedHat itself started out as just a clone of other people's source, the RHEL clones too will do extra QA, report bugs upstream, send patches etc. This is inevitable because it is how the culture of open source works; share everything, contribute back. Also because not contributing back means more work in the long run rebasing patches etc.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 4:52 UTC (Tue) by geofft (subscriber, #59789) [Link] (24 responses)

I think the point being made is that Red Hat is absolutely going to contribute things upstream and absolutely will make any fixes/enhancements they develop upstream - they just aren't going to make it particularly easy to be a bug-for-bug clone of Red Hat. And I'm a little bit sympathetic to that because the only people who really need a bug-for-bug clone of Red Hat are people who are running software (FOSS or proprietary) from some other commercial vendor who certifies it as running on literally Red Hat and nothing else, or people who want to run 10 Red Hat machines and 90 not-Red-Hat machines and report bugs to Red Hat regardless of which machine they came from.

The broader FOSS ecosystem doesn't really benefit from this. I don't think I've ever seen an upstream developer say "Oh you ran into this on Fedora, but can you reproduce it on Red Hat or a Red Hat rebuild?" It wouldn't be useful to the broader ecosystem if they said that, because how could you expect to use that software on Debian or Arch or anything? How could that software participate in the FOSS community as a whole instead of just Red Hat and its clones?

The broader FOSS ecosystem absolutely does benefit from Red Hat sending patches and bug reports upstream, and as a backstop for when Red Hat doesn't engage upstream, the broader ecosystem absolutely does benefit from Red Hat's patches and customizations being published to the world. Which are the things that Red Hat is committing to doing. If they find a bug in OpenLDAP and they patch it for their customers, that patch - and its commit message, and the discussion on the merge request, and a whole lot of other things that aren't in the SRPM! - is on GitLab.com for anyone else to pick up. The only thing they're no longer committing to is saying whether that patch was applied in openldap-2.6-20.el9.x86_64.rpm or openldap-2.6-21.el9.x86_64.rpm. That piece of information is wholly irrelevant to the culture of open source; you don't need it unless you're specifically trying not to have any patches of your own. It's only interesting if your interest in Red Hat publishing sources, and being open source at all, stops once you have an equivalent binary.

(Or, admittedly, unless you're Software Freedom Conservancy trying to verify that in fact they did push all the patches to GitLab that they claimed they were going to. That's a valid use case and I hope that gets resolved. I'm concerned by comments on the question I asked yesterday https://lwn.net/Articles/936138/ about how there are private branches, because that means it's easy to forget to push things publicly. Even a commitment to automation would be meaningful.)

All of Drew's examples there are about people adding value to your work in some way - incorporating it into their own work, becoming an expert and consulting/teaching/writing about the software, packaging it nicely, etc. Every single one of these use cases could be accomplished just as well with CentOS Stream as with Red Hat, if not better. If I wanted to, say, repackage 389-ds and run a little LDAP company, I would actively not want to be bound to Red Hat's release cycle and their discretion of what bugs are important! I would want to see their decisions about what patches go into stable releases and potentially make my own depending on what issues my customers are having. I wouldn't object if the base I'm working from is a little bit newer than Red Hat's commercial product; in fact I would find that useful.

The only use case where unthinking replication of Red Hat's decisions and renunciation of the ability to make your own decisions are valuable is when your product's value is entirely in it being the same as Red Hat's product but cheaper. It's true that Red Hat is also surrendering their ability to prevent this by redistributing FOSS, but it is telling that this isn't a use case where you can write a convincing story about how it's a good thing for the world.

(And, in turn, the money that Red Hat charges goes to fund their ability to continue participating in the ecosystem. If you look at e.g. the development stats article right below this one https://lwn.net/Articles/929582/ , a lot of the major contributors are hardware vendors who want to sell you their hardware, or companies like Google or Facebook that have their own internal use cases and their own agendas. The argument being made in the article is that a world where Red Hat's business model is not commercially viable is actually worse for FOSS than a world where Red Hat survives by doing things that just barely comply with the GPL.)

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 7:28 UTC (Tue) by taladar (subscriber, #68407) [Link] (18 responses)

RHEL clones are mainly useful to build and test software meant for RHEL. As such they make supporting RHEL easier for people who do not have a big investment in RHEL, adding value to the RedHat ecosystem. And no, many people do not want to deal with some sort of license, even a free one, to support that use case.

It is bad enough, frankly, that 10 year support distros like RHEL often force us to make software work with current and 10+ year old software at the same time at all.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 9:51 UTC (Tue) by tuna (guest, #44480) [Link]

"It is bad enough, frankly, that 10 year support distros like RHEL often force us to make software work with current and 10+ year old software at the same time at all."

You are not forced to write software for certain platforms. You might have a marketing incentive to write software for certain platforms, but no one is forcing you to write software.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 12:37 UTC (Tue) by pizza (subscriber, #46) [Link] (13 responses)

> RHEL clones are mainly useful to build and test software meant for RHEL.

No; developer use of the clones is a very tiny minority of their user base.

I'd put money down that over 95% of the deployments of the clones are from folks that want a "LTS" linux distribution they don't have to pay for, deploying anywhere from snowflake production systems to hundreds of servers (and/or workstations running Very Expen$ive Software) to tens of thousands (or more) of cattle compute nodes.

(and for the record, I've done or worked at companies that do all of these. The latter even had a handful of RHEL licenses they used for "official" support if their compute clusters ran into problems and it was reproducible on genuine RHEL).

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 18:52 UTC (Tue) by wtarreau (subscriber, #51152) [Link] (12 responses)

I totally agree, I've seen the same. Consultants proposing CentOS because "the customer cannot afford a single RHEL", despite paying $5K worth of consulting time. Many times I said them "that's not correct, you're hurting the ecosystem, this is a wrong excuse".

I really think Red Hat should propose a free version themselves that can be upgraded to a supported one without having to reinstall anything. Just download the ISO for free like you do with Ubuntu, and if one day you decide that your server has become sensitive enough to warrant paying, just do it. At least there would be *their* distro in field for all users, free and paid. The problem they're facing now is that for absurd reasons of pricing (or even the perceived difficulty of reselling when you only expected to deliver service to a customer), they end up forcing their potential future customers to deploy a competing solution instead of theirs. And *this* is hard to upgrade later, as the customer certainly doesn't want to touch their running system when it becomes critical.

They'd basically just need an motd reminding the registration link so that those logging in via SSH to inspect something are greeted with "Experiencing some trouble? Need some help? May be it's time to subscribe. ->link". It would completely fix the consultant problem above: nothing is sold, it's only installed and the customer may decide later to pay.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 14:28 UTC (Wed) by ewan (guest, #5533) [Link] (10 responses)

> I really think Red Hat should propose a free version themselves that can be upgraded to a supported one without having to reinstall anything.

That's *exactly* what Red Hat Linux was.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 15:25 UTC (Wed) by zdzichu (subscriber, #17118) [Link] (9 responses)

And RHEL still is? You can download and use it on 16 servers for free. It's not "grab an iso and install" easy, but creating a red hat account is simple enough.

McGrath: Red Hat’s commitment to open source

Posted Jul 4, 2023 6:19 UTC (Tue) by ceplm (subscriber, #41334) [Link] (8 responses)

Officially it is the development version only for development.

McGrath: Red Hat’s commitment to open source

Posted Jul 4, 2023 8:20 UTC (Tue) by anselm (subscriber, #2796) [Link] (7 responses)

Officially it is the development version only for development.

According to Red Hat, “the no-cost Red Hat Developer Subscription for Individuals may be used for demos, prototyping, QA, small production uses, and cloud access” (emphasis mine).

McGrath: Red Hat’s commitment to open source

Posted Jul 5, 2023 7:47 UTC (Wed) by ceplm (subscriber, #41334) [Link] (6 responses)

Yes, and you would base your company on this side comment somewhere in the documentation. Congratulations, you are the most courageous (or reckless) entrepreneur of the year!

McGrath: Red Hat’s commitment to open source

Posted Jul 5, 2023 7:49 UTC (Wed) by anselm (subscriber, #2796) [Link]

No, I wouldn't. I prefer Debian.

McGrath: Red Hat’s commitment to open source

Posted Jul 6, 2023 17:20 UTC (Thu) by cortana (subscriber, #24596) [Link] (4 responses)

No, the terms are in the Red Hat Developer Subscription for Individuals subscription agreement:

> “Individual Production Use” means any use other than for Individual Development Use including, but not limited to, using the Software (a) in a production environment, (b) with live data and/or applications and/or (c) for backup instances.

McGrath: Red Hat’s commitment to open source

Posted Jul 13, 2023 16:24 UTC (Thu) by ceplm (subscriber, #41334) [Link] (2 responses)

Exactly:

> “Individual Production Use” means any use other than for Individual Development Use including, but not limited to, using the Software (a) in a production environment, (b) with live data and/or applications and/or (c) for backup instances.

So, Red Hat can sue you any time if you just use packages of their repositories (without any of your own development) in the production environment? Meaning, this is so confusing and ambiguous, that I would immediately run to openSUSE or Debian.

McGrath: Red Hat’s commitment to open source

Posted Jul 13, 2023 16:44 UTC (Thu) by pizza (subscriber, #46) [Link]

> So, Red Hat can sue you any time if you just use packages of their repositories (without any of your own development) in the production environment? Meaning, this is so confusing and ambiguous, that I would immediately run to openSUSE or Debian.

...Debian and OpenSUSE are not equivalent to, or substitutes for, RHEL (or even CentOS Stream), especially where the support window is concerned.

(Unless of course you never actually needed RHEL to begin with, and/or were just using one of its rebuilds because you didn't have to pay anything for it)

Seriously, please, there are a lot of options, but don't delude yourself into thinking that the grass is necessarily greener everywhere else -- Each of those options represents a different culture, warts, and compromises.

McGrath: Red Hat’s commitment to open source

Posted Jul 13, 2023 19:27 UTC (Thu) by kleptog (subscriber, #1183) [Link]

People should really read the whole text[1], it's not long. The passage quoted above is merely the definition of Individual Production Use, the rest of the subscription doesn't distinguish between production and non-production. The distinction is probably for some legal purpose not relevant to us.

And suing? Really? Yes, in theory RedHat can sue anybody, but as an individual the law really limits what they can do to cancelling your subscription. They're not going to sue you unless you happen to be a million dollar business pretending to be an individual developer, because then consumer rights won't protect you.

It feels like some people here want to find a reason to hate RedHat. I don't use RedHat, never have, but I don't think they're evil or anything.

[1] https://developers.redhat.com/terms-and-conditions

McGrath: Red Hat’s commitment to open source

Posted Jul 25, 2023 14:21 UTC (Tue) by MattJD (subscriber, #91390) [Link]

Also from the terms:

> The Individual Developer Subscriptions allow you (as an individual, natural person) to use certain Red Hat Subscription Services in connection with Red Hat Software for Individual Development Use and for Individual Production Use subject to these Program Terms at no cost.

I don't know why they are making a distinction, but they explicitly allow both uses.

McGrath: Red Hat’s commitment to open source

Posted Jul 3, 2023 16:46 UTC (Mon) by jccleaver (guest, #127418) [Link]

>I totally agree, I've seen the same. Consultants proposing CentOS because "the customer cannot afford a single RHEL", despite paying $5K worth of consulting time. Many times I said them "that's not correct, you're hurting the ecosystem, this is a wrong excuse".

Seen this plenty of times as well. The primary issue is that RHEL is pricing as if they're selling a fully proprietary solution instead of selling support services. For some companies' business models, an OS license is a trivial addition. In many revenue cases it can even be passed directly on to the client/customer as a distinct line item! For farm and compute node customers, many of whom have plenty of Linux Systems Engineers on staff, this is not a justifiable cost.

RHEL should have provided reasonable site licenses, or sold individual support instances, or allowed CentOS users to receive support if they donated to the CentOS Foundation to help with finances for centos infrastructure... *something* . This is especially the case when they adopted CentOS as an *internal* project solution, thereby killing off the other group -- Scientific Linux -- and thus reliability diversity in the EL ecosystem. This almost counts as an "embrace, extend, extinguish" for rebuilds, and probably would if CentOS were having problems with project and donation management at the time.

This wasn't fair play by Red Hat, even if it complies with the letter of the various OSS licenses. It's a turnabout that it's still difficult to see the justification for, given that the relationship between RHEL and things like WBEL goes back almost 20 years, and CentOS has been part of RH officially for quite a while. Rocky didn't steal revenue from RHEL because NASA wasn't going to pay for support it didn't need. At that point, Rocky is offering consultant services, since they're not doing anything about upstream bugs other than... report them to upstream. That's the entire point of a rebuild.

EL-rebuilds value is in removing community support costs from RHELs bottom line, while upstreaming the collective experience and bug reports as necessary. THAT is the value, and RH needs to look beyond per-CPU OS licensing, or free as in beer replacement.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 12:43 UTC (Tue) by anselm (subscriber, #2796) [Link]

RHEL clones are mainly useful to build and test software meant for RHEL. As such they make supporting RHEL easier for people who do not have a big investment in RHEL, adding value to the RedHat ecosystem.

Red Hat is already going out of its way by providing free RHEL developer licenses to support this.

And no, many people do not want to deal with some sort of license, even a free one, to support that use case.

These people get actual RHEL for free already. How much farther do they want Red Hat to bend over backwards on their behalf? Would they like an engineer from Red Hat to come to their site, for free, to install RHEL for them? Perhaps write their code for them while they're drinking beer by the pool? This is entitlement, pure and simple.

Anyway, why would you even futz around with some RHEL clone in the first place? The main reason to deal with RHEL at all is that you have paying customers who run RHEL, and who would prefer you to test your software on genuine RHEL and not just some clone, even if the clone purports to be just the same as RHEL – but who is prepared to guarantee that? The clone maker? You?

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 20:55 UTC (Tue) by amacater (subscriber, #790) [Link] (1 responses)

TL;DR - If you want to move from Red Hat, do it now but make it a reasoned decision in your best interests. Support from any one particular distro is always a cost/benefit decision: they are all equally bad when your bug can't be fixed.

Full disclosure: I currently have a free RHEL developer subscription. I use it to try small things for people, set up machines and VMs to gain expertise for myself. I *can't* force anyone to pay for Red Hat so I sometimes prototype on RHEL and use Rocky. I've a distro mirror next door - I built it on RHEL because I need to show it to RHEL-native sysadmins. Because I can't force the RHEL natives to take out another RHEL subscription, I took the scripts and remade it on Rocky.

Self support on the free tier makes it *really hard* to file a bug for RHEL. Two show stopper bugs took forever to be raised. I could demonstrate that they only occurred with RHEL signing certificates / Red Hat subscription manager. The response made me less enamoured of RHEL. I'd quite like to raise a bug that RHEL 9 doesn't run on my HP Microserver but no-one will listen so the Microserver now runs on Rocky 8.

But RHEL developers |= RHEL system architects and customer supporters |= RHEL senior management. Former RHEL |= current RHEL and current RHEL == IBM.
Red Hat (was) a large company with differences of opinion - hating one RHEL person for what they may or may not say in a press release is perhaps not useful.

If the problem really is Oracle Linux (and not Rocky/Alma) can we just get some popcorn
and watch as IBM take on Oracle? Both have licence compliance mavens but IBM has over a century's worth of attack dog lawyers and a significantly longer corporate memory.

McGrath: Red Hat’s commitment to open source

Posted Jul 6, 2023 17:24 UTC (Thu) by cortana (subscriber, #24596) [Link]

> Self support on the free tier makes it *really hard* to file a bug for RHEL.

Hm. I've filed a few bugs in Bugzilla and some have been fixed, others I've corresponded with developers and are not, I think, in a state where they'll rot forever.

subscription-manager is always a pain to deal with though even when it's working. And it's so s...l...o...w...!

McGrath: Red Hat’s commitment to open source

Posted Jul 2, 2023 11:06 UTC (Sun) by sadoon (subscriber, #155365) [Link] (4 responses)

Thank you for this, very informative. I still don't use any of RedHat's distros (big Debian guy, although I'm toying around Fedora Kinoite on an older laptop) but a lot of the software I depend on on a daily basis was either written from scratch or is heavily maintained and supported by RH (libvirt comes to mind, couldn't live without it).

The Venn diagram of FOSS enthusiasts and anti-corporate hipsters is a little less than a circle. It's unfortunate but I understand why it is and I agree with their view most of the time. Most big corporations are evil by default, but that's more of an incentive to support a corporation that's doing genuinely good things for the community as a whole, even if you don't agree with some of their decisions; they've enabled your freedom, and (good) software is hard work that doesn't come from unicorn farts (hobbyist work).

If no one supports RH with money, they'll find another way to make profit and I'll bet that it won't be something to help FOSS.

McGrath: Red Hat’s commitment to open source

Posted Jul 2, 2023 15:41 UTC (Sun) by Wol (subscriber, #4433) [Link] (3 responses)

> Most big corporations are evil by default,

Most big corps are evil BY LAW. In order to comply with the law, companies find it difficult to be compassionate, and are forced into psychopathic behaviour.

Don't blame the company, blame the environment they live in. (Think eg the US DoD spending $100/hammer, so they can prove they aren't being ripped off ...)

Cheers,
Wol

McGrath: Red Hat’s commitment to open source

Posted Jul 2, 2023 16:04 UTC (Sun) by pizza (subscriber, #46) [Link] (2 responses)

> Most big corps are evil BY LAW.

Not quite. Corporations (of all types) must act *in their shareholders' interest* Typically, this means means maximizing (short term!) financial returns. which inevitably seems to lead to sociopathic evil-ness. However, if the shareholders make it clear they prioritize other things, that's perfectly fine in the eyes of the law!

Generally speaking, corporations that don't keep their shareholders happy tend to find their senior management removed and replaced with ones that promise to do whatever the shareholders want -- which, unfortunately, tends to be "maximum short term financial return. No-so-coincidentally, that's also easy to measure, so of course it becomes an (the?) important metric.

McGrath: Red Hat’s commitment to open source

Posted Jul 3, 2023 12:21 UTC (Mon) by kleptog (subscriber, #1183) [Link] (1 responses)

> Not quite. Corporations (of all types) must act *in their shareholders' interest*

Even that's not strictly true. They must first abide by the goals set out in the Articles of Association that created the business. If the shareholders don't like that, they can be changed, but that may be a much higher bar than a simple majority.

That's how you get social businesses, whose goal at incorporation may be e.g. "ensure affordable housing for people in area X" or even "provide safe drinking water for the people in area X". Handing out large dividends would run counter to that goal. Mostly these kinds of organisation are associations or cooperatives, but sometimes an LLC is chosen.

(Articles of Association are supposed to be on the public record, but can be surprisingly difficult to locate if you don't know where to look.)

McGrath: Red Hat’s commitment to open source

Posted Jul 3, 2023 14:04 UTC (Mon) by Wol (subscriber, #4433) [Link]

> That's how you get social businesses, whose goal at incorporation may be e.g. "ensure affordable housing for people in area X" or even "provide safe drinking water for the people in area X". Handing out large dividends would run counter to that goal. Mostly these kinds of organisation are associations or cooperatives, but sometimes an LLC is chosen.

I wish that described the situation here in the UK ... they went from mismanaged nationalised industries to - reasonably decent - privately owned companies, to subsidiaries of profit-mad international conglomerates.

They also have the big problem that - like an awful lot of public infrastructure - the government loves dumping responsibilities on them, then taking away their ability to pay for those responsibilities. Then the government gets all upset when everything goes pear-shaped.

Cheers,
Wol

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 8:50 UTC (Wed) by jzb (editor, #7867) [Link]

“RedHat itself started out as just a clone of other people's source”

Did it? And for how long?

As I understand it, the first Red Hat Linux release was based on SLS. Not a clone, a fork with additional work added. And, I think, SLS had actually ceased distribution by then. It’s a far cry from a fair comparison.

If the discussion was about Red Hat trying to prevent a project or company from building something novel off RHEL sources instead of just xeroxing whole RHEL releases, it’d be very different.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 1:36 UTC (Tue) by pabs (subscriber, #43278) [Link] (8 responses)

I have already seen some projects dropping support for RHEL, I wonder if there will be any projects rejecting upstream contributions from RHEL folks because of this.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 3:26 UTC (Tue) by adam820 (subscriber, #101353) [Link] (7 responses)

> some projects

Which ones?

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 9:20 UTC (Tue) by pabs (subscriber, #43278) [Link] (6 responses)

At least Jeff Geerling's Ansible related and other projects:

https://www.jeffgeerling.com/blog/2023/removing-official-...
https://www.jeffgeerling.com/blog/2023/im-done-red-hat-en...

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 12:27 UTC (Tue) by pizza (subscriber, #46) [Link] (4 responses)

He's not dropping "support" for RHEL, he's cropping "official support", while continuing to "officially" support Alma, Rocky, and CentOS Stream.

Which... is a problem how? RH should be the front line of support for everything RHEL. It is, after all, what its users are paying for.

That said, his definition of "no official support" is pre-emptively removing all mention of RHEL in what he distributes, and actively closing any bug tickets from RHEL users as "not reproducible" (when it's 99.9% certain that any RHEL bug would of course manifest in Alma or Rocky and thus be relevant to what he does "officially" support) That's his right of course, but it's one thing to not do any effort, it's another thing entirely to put active effort into ... not doing effort -- it all comes off as pretty petulant, and that's the sort of thing that leads to more (*ahem*) inclusive forks happening.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 13:02 UTC (Tue) by jafd (subscriber, #129642) [Link] (3 responses)

> it all comes off as pretty petulant

I'm reading it more like an "abusive ex situation" rather than "petulant" and choose to interpret it like that. It's his right to cut ties with an entity he thinks did him/others wrong. Which means that if I'm using his stuff with RHEL and encounter a bug, it stands to reason that it's on me to reproduce it elsewhere and then report it.

Here's another angle: there exist fixed bugs in macOS and Macs which have been reported indirectly by the hackintosh community. But they had to go an extra mile to reproduce those bugs on real Macs, or research them enough to convince Apple this was a problem with fully legit usage before they would report them. The difference was that hackintoshes would run into those problems every time you looked at them funny, and real Macs could have them once in a blue moon. But you could not just approach Apple with a "Your system exhibits problems on my non-Apple hardware" and expect them to listen to you (or not take a legal action depending on who you were).

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 13:50 UTC (Tue) by pizza (subscriber, #46) [Link] (2 responses)

> I'm reading it more like an "abusive ex situation" rather than "petulant" and choose to interpret it like that. It's his right to cut ties with an entity he thinks did him/others wrong. Which means that if I'm using his stuff with RHEL and encounter a bug, it stands to reason that it's on me to reproduce it elsewhere and then report it.

Again, it's not that he's ceasing to expend effort to support RHEL users, it's that he's actively *removing* already completed RHEL integration, thus making his software objectively *worse* along the way. Which means that downstream users (possibly including RH themselves) will now have to start maintaining their own forks, or utilize something different.

History has repeatedly shown that anti-inclusive attitudes like this are detrimental to the long-term health of a project.

> Here's another angle: there exist fixed bugs in macOS and Macs

Ugh, MacOS represents the overwhelming majority of the user-support requests for one of the F/OSS projects I'm involved with. Coupled with Apple's unofficial policy of breaking something we need with each new MacOS release and their tools removing support for older OS versions, we're _very_ close to officially throwing in the towel entirely, instead of responding to most prolems with "none of the active developers have _access_ to a Mac, much less the technical knowhow to fix Mac-specific problems and generate new builds."

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 15:28 UTC (Tue) by jafd (subscriber, #129642) [Link] (1 responses)

> it's that he's actively *removing* already completed RHEL integration

Only wording about it (if he's supporting Rocky/Alma, it's in essence THE SAME THING, it's just that you shouldn't be counting on explicit support if you are on bona fide RHEL — this is not like some active harmful activity subject to being run on RHEL). And then, if you look at his repos as of today, the words like “RHEL” or “Red Hat” are still there — search/replace is quite a hassle, and Jeff does look like a person with more interesting things to do.

Now if you're wondering who is being petulant here, you should probably take a look in the mirror.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 15:32 UTC (Tue) by pizza (subscriber, #46) [Link]

And I quote:

"Today: Removing official support for RHEL in Ansible role metadata, so users searching for roles to work with RHEL will not find my roles (I would rather they find roles that are actually tested against RHEL, because I cannot guarantee my roles will work for these users).

"Ongoing: As issues crop up on various roles against RHEL, I will decide on a case-by-case basis whether to strip all RHEL-like support (effectively making the project only run on Fedora, Arch, Ubuntu, Debian, or other distros), or attempting to fix the problem so existing users of Rocky Linux, AlmaLinux, and CentOS Stream may still benefit from the fix.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 8:18 UTC (Thu) by milesrout (subscriber, #126894) [Link]

Jeff Geerling is just being a drama queen. His immediate reaction to the post we are commenting on was to post lies on Mastodon. He's just acting out to get attention from his followers.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 2:08 UTC (Tue) by ayay (guest, #144420) [Link] (21 responses)

I work for a small non-profit.

We started out small, and used CentOS. As we didn't have resources, we would report bugs, submit patches, documentation etc. as a way of giving back. That also came to include our workstations, running Fedora.

I already knew Red Hat wasn't eager to deal with small potatoes when I first wanted to get my employet to pay for some certifications. The price was too steep for me to even mention.

And then, when the first CentOS debacle came to be, we decided it was time to look into paying for RHEL.

Their pricing structure makes it crystal clear they do not want to deal with small or medium organizations... it's bonkers. We would be better off moving all of our infrastructure to Ubuntu and paying for THAT instead, and it would be no small job to go forward with the migration, but goes to show how much cheaper - and a lot more straightforward - their pricing and licensing is.

Rocky and Alma came to the rescue, so we carried forward on the RH ecosystem. That has been an issue with our new hires - young. talented developers who grew up on a cloud-first environment. They are all in on Ubuntu. They become familiar with the RHEL ecosystem through us.

So we are stuck now: it's clear Red Hat thinks "the juice is not worth the squeeze" for small and medium sized companies, and yet, we can't reason - nor afford - paying them what they're charging. The only thing we can conclude is that they really don't want us using their product.

I am still dumbfounded, though: even if you consider we're not worth that much to Red Hat as customers, if nothing else, we are training grounds for some amazing developers that come to learn and like RHEL's ecosystem through us, and take that with them when they move to higher-paying jobs in bigger companies that can pay what they deserve, after they learn and take off their training wheels.

Then I recall that, in the 90s, Microsoft seemed very eager to cut some sweetheart deals with schools, a move that Google and even Apple also followed eventually.

It strikes me as an extremely short sighted decision. Now, seeing that I am not rich nor a shareholder, that may be why I don't understand it...

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 5:04 UTC (Tue) by pbonzini (subscriber, #60935) [Link] (5 responses)

You can use CentOS Stream too.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 8:18 UTC (Tue) by TRauMa (guest, #16483) [Link] (3 responses)

I couldn't recommend a distro where security fixes are deliberately held back to anyone. And I'm not talking about embargoes, all the fixes. It's one of the ways Red Hat "adds value" to RHEL.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 12:41 UTC (Tue) by pizza (subscriber, #46) [Link] (2 responses)

> I couldn't recommend a distro where security fixes are deliberately held back to anyone. And I'm not talking about embargoes, all the fixes.

Then you'd be happy with CentOS stream, which lands non-embargoed fixes *before* they go out into RHEL, meaning at worst it's no slower than the pre-Stream rebuild flow.

> It's one of the ways Red Hat "adds value" to RHEL.

Um, you act like "adding value for your paying customers" is a bad thing.

McGrath: Red Hat’s commitment to open source

Posted Jun 30, 2023 19:31 UTC (Fri) by kpfleming (subscriber, #23250) [Link] (1 responses)

Just for clarity, repeating what has been posted on one of the other LWN article threads:

This is not true. Fixes for 'low' and 'moderate' severity CVEs are generally made 'in the open' in CentOS Stream and then appear in RHEL when the next batch of updates for that RHEL stream are published.

Fixes for 'important' and 'critical' CVEs (embargoed or not) are made in RHEL first, in private repositories, shipped to RHEL customers (and generally do not wait for batch updates but are shipped as soon as they are ready), and are then made in CentOS Stream as the RHEL developer gets time to push the changes there. This could be minutes, or hours, or days, but wouldn't often be more than a few days.

McGrath: Red Hat’s commitment to open source

Posted Jul 1, 2023 2:56 UTC (Sat) by passthejoe (guest, #156034) [Link]

This is very good to hear, and is something Red Hat and CentOS people should be talking about non-stop.

If CentOS Stream is worth speaking up for, a whole lot of people should be doing it.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 2:21 UTC (Wed) by ayay (guest, #144420) [Link]

Heh, the machines I converted to Stream almost got me fired. Then I decided to do a "from-scratch" install to prove a point, did not help, quickly walked it back in shame.

n = 1 and all that, but I've had less issues with Fedora Server. The only unfortunate thing is that it requires frequent rebooting, as it updates often - of course, that's exactly what it says on the tin, so although it's not surprising, it's not ideal for our use case.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 5:08 UTC (Tue) by geofft (subscriber, #59789) [Link] (4 responses)

Serious question - why not run CentOS Stream? You don't need 100% drop-in compatibility with Red Hat, do you? It feels like you are exactly their target audience - you want to run something that's essentially Red Hat, but you were never going to be in a position to pay for actual Red Hat.

(Second serious and practical question, but less relevant to the discussion: are you considering Debian? It seems like one of the better choices for an "enterprise" distro, especially since they seem to have quietly switched to time-based releases, and your new hires familiar with Ubuntu will be very comfortable with it. We've been happy with it at my day job for about a decade.)

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 6:55 UTC (Tue) by maniax (subscriber, #4509) [Link]

I'd love to understand that, too.

I think that people feel like CentOS stream is like ... "Debian unstable", all new stuff goes in there and when it's decided to be stable, a release is made that's actually the stable one. I feel like a lot of this is perception, not the truth, but nobody seems to want to use it.

And as for stability, in the last month I had two separate kernel bugs, one in RHEL7 and one in RHEL8, that if ran on KVM hosts led to VM hangs/stalls, and at some point having something move fast enough so you get your bugs fixed quickly might be the better choice than depending on something that releases every 3 months and is supposed to be stable.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 10:14 UTC (Tue) by jafd (subscriber, #129642) [Link] (1 responses)

> Serious question - why not run CentOS Stream?

Right now, as I'm writing this, I'm trying to build ~200 packages with mock using CentOS Stream 9. The process is failing as AppStream's repodata checksums are mismatched. Now they match, but other packages are not yet mirrored and thus I'm left to dry until they all sync or something. There's a bug report in Pagure about this about twice a year, it seems, from internal people too.

My project is a perfect use-case for CentOS Stream: I want some LTS on par with Ubuntu so I don't do the time-consuming "upgrade to the next major version" busywork dance every year, thank you very much. I don't need more guarantees, certifications, and most other fluff RHEL target audience may find useful. But I also want to plug package and container building into a CI/CD pipeline, and if their mirrors keep barfing once in a while without a reliable indication when it happens and how to plan around it, it's quite a sad state of affairs.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 22:27 UTC (Tue) by geofft (subscriber, #59789) [Link]

This seems like a decent use case for someone to be a CentOS Stream rebuilder - start with what's in Git alone, build your own RPMs from scratch and host your own repo, and be willing to patch for major security issues.

(Honestly this would get you eligibility to be on the closed distros list, so you could patch for embargoed vulnerabilities at the same time as RHEL! You still wouldn't be guaranteed to be binary-identical but it would address the concern of unpatched vulnerabilities in Stream, which seems like it would be plenty for people who need a Red-Hat-ish enterprise OS but don't need to 100% mimic RHEL.)

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 16:02 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

+1 to Debian. In my informal hobbyist experience, it is an excellent distribution for making systems that "just work" for years on end with very little fuss or effort.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 20:47 UTC (Tue) by jzb (editor, #7867) [Link] (9 responses)

"The only thing we can conclude is that they really don't want us using their product."

I understand why you think that, but I think it's more accurate to say "we are not their target audience."

Rightly or wrongly, Red Hat (and other businesses) make choices about personas and product fit for specific use cases, pricing, etc. You're right that Red Hat isn't eager to deal with small potatoes. It is, after all, called "Enterprise" Linux for a reason.

You cite Microsoft - but that's not apples to apples. Microsoft's annual *marketing* budget is several times RHEL's revenue overall. Probably Red Hat's revenue overall. Microsoft can afford to push into schools and such because 1) its revenue and budgets absolutely dwarf Red Hat's and 2) they're trying to own client computing, backend computing, etc.

It may be a shortsighted decision, I haven't made up my mind just yet and I also don't have the information in front of me that McGrath and the other execs at Red Hat have to inform their decision. They know how many people have renewed RHEL subs, how many are still on RHEL 7 and haven't moved to RHEL 8 or 9, and so forth. This may be a reaction to any number of things that the community doesn't have knowledge of -- and whether it's the right one or not, who knows?

I know in 2003 people threw the same arguments at Red Hat about ending Red Hat Linux and closing off access to Red Hat Advanced Server, etc. With 20 years of perspective it seems that Red Hat didn't implode.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 20:55 UTC (Tue) by pizza (subscriber, #46) [Link] (8 responses)

> I also don't have the information in front of me that McGrath and the other execs at Red Hat have to inform their decision.

To quote McGrath:

"The generally accepted position that these free rebuilds are just funnels churning out RHEL experts and turning into sales just isn’t reality. I wish we lived in that world, but it’s not how it actually plays out. Instead, we’ve found a group of users, many of whom belong to large or very large IT organizations, that want the stability, lifecycle and hardware ecosystem of RHEL without having to actually support the maintainers, engineers, writers, and many more roles that create it. These users also have decided not to use one of the many other Linux distributions."

That lines up with my experience, at both small (<20 employee) companies, at large-ish companies (3000-5000 people) and very large highly-regulated companies (>60,000 employees).

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 13:35 UTC (Wed) by pbonzini (subscriber, #60935) [Link] (1 responses)

Also quoting him from Reddit:

"The problem of rebuilders has been around forever. Things heated up a couple of months ago when we detected what we think was a continued bad-faith action from one of the rebuilders, not on the code/engineering side but on the commercial/money making side of their house. That's as far as I'll go publicly. After that it was just a matter of discussion on what to do about it, so we landed on the announcements I made last week."

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 14:35 UTC (Wed) by pizza (subscriber, #46) [Link]

In case anyone else is interested, here's all of McGrath's posts:

https://old.reddit.com/user/mmcgrath

And another pertinent comment:

"People keep saying that but that doesn't make it true. You don't want free as in freedom. You have that. The code is all out there today and will be in the future, all of it. You want the promise red hat applies to that code, that our engineers will continue to support it for 10 more years. And even though we will push all that code upstream, and put it in CentOS Stream first, that's not good enough. No, you want that service free as in beer.

"RHEL is product, not a project. We aren't preventing rebuilders from building a rebuild, but we aren't going out of our way to make it easy for them any more.

McGrath: Red Hat’s commitment to open source

Posted Jun 30, 2023 20:02 UTC (Fri) by apple4ever (guest, #164280) [Link] (5 responses)

Given that sales are up 17% in 2022, I think he is entirely wrong. Also, there are a lot of IT organizations, large and small, that want to support the RH community, but RH themselves makes it difficult and cost prohibitive to do. The solution to the problem is there, they just don't want to do it, and now are cutting off their nose to spite their face. This will kill RH in the long run.

McGrath: Red Hat’s commitment to open source

Posted Jul 3, 2023 19:05 UTC (Mon) by jccleaver (guest, #127418) [Link] (4 responses)

> Given that sales are up 17% in 2022, I think he is entirely wrong. Also, there are a lot of IT organizations, large and small, that want to support the RH community, but RH themselves makes it difficult and cost prohibitive to do. The solution to the problem is there, they just don't want to do it, and now are cutting off their nose to spite their face. This will kill RH in the long run.

This 1000%. Everyone remotely associated with the OSS community is aware of the perils of the free-rider problem. There are solutions; Red Hat in fact grew into a $34B business by understanding and in many cases creating those solutions. There aren't solutions if Red Hat thinks it's selling Windows Server instead of a hardened version of Free Software and the salaries of the developers and support staff who make it happen.

More than willing to find ways to make EL-hardening a true "Community Enterprise OS" while still paying for upstream support at a reasonable site license rate that reflects the reality, which is that most of the things we're going to reach out to RH for will be bugs that RHEL has a justification for fixing entirely on its own. Community bug reports are a Good Thing here, and it baffles me that RH is so myopic as to think that it's moot.

McGrath: Red Hat’s commitment to open source

Posted Jul 3, 2023 19:30 UTC (Mon) by pizza (subscriber, #46) [Link] (3 responses)

> Community bug reports are a Good Thing here, and it baffles me that RH is so myopic as to think that it's moot.

I don't think anyone, least of RH, would disagree with the importance of "community bug reports", and indeed, that's one of their stated reasons behind the CentOS Stream model.

But you need to stop looking at this in terms of absolutes. Does the [now-]former rebuilder model offer RH some benefits? Undoubtedly! But do the benefits outweigh the price? According to RH, not any more.

I can't be the only one who thinks that a decision of this magnitude doesn't have some hard data backing it up. It's not hard to imagine multiple now-former customers telling their now-former RH reps "We're not renewing our seven-digit RHEL licenses unless you match Oracle's pricing" or even "We're going to just use Rocky as it gives us everything we care about and we don't have to pay anything at all to get it."

And that's what it all comes down to in the end -- "We lost $$$$$$ worth of license renewals to rebuilders, and our sales folks say only $$$ came in from folks that decided to go from a rebuilder to RHEL. Why exactly should we keep paying our engineers and hosting providers $$$$ to enable these rebuilders to easily take business from us?"

McGrath: Red Hat’s commitment to open source

Posted Jul 3, 2023 19:37 UTC (Mon) by pizza (subscriber, #46) [Link]

> I can't be the only one who thinks that a decision of this magnitude doesn't have some hard data backing it up

Ugh, s/doesn't have/has/

McGrath: Red Hat’s commitment to open source

Posted Jul 3, 2023 22:31 UTC (Mon) by jccleaver (guest, #127418) [Link]

The problem here (for Red Hat) is that adopting an antagonistic posture toward rebuilds won't force a long-term change here. The underlying code is open source; the patches are open source. SRPMs can be re-assembled, and OSS collaborators can simply work harder to try to get it back to the same level that CentOS Linux had been in the EL6 and EL7 era. At best it delays things, and the cost was a huge amount of goodwill.

Furthermore, this recent action really can't be taken alone, separately from the decision two years ago to clobber EL8 rebuilds halfway through the RHEL8 lifecycle. McGrath at that point clearly demonstrated on the CentOS list where he seemed to dig even deeper with every possible opportunity: eg, https://www.spinics.net/lists/centos-devel/msg19597.html

Oracle and Amazon can throw whatever amount of money they want at reverse-engineering EL. If that's their concern, then they haven't figured out yet that it's better to co-opt your enemies than fight them. Demonstrate what RedHat's value is and cash in some of that goodwill. Instead, they've burned their goodwill and Oracle and Amazon will do whatever it is they want to do anyway, and CloudLinux, Rocky/Alma, and NASA or whichever lab decides to work on Scientific Linux 2: Electric Boogaloo will find a way to formalize support for things that they now actually have a larger claim to have placed real value into, based solely on the amount of (re-)engineering effort they're now having to put in.

McGrath: Red Hat’s commitment to open source

Posted Jul 4, 2023 8:36 UTC (Tue) by paulj (subscriber, #341) [Link]

"I can't be the only one who thinks that a decision of this magnitude [has] some hard data backing it up."

You may think so, maybe others, but anyone with experience of suits at big corporations also know decisions like this often get made without hard data. Indeed, anyone with experience will know such suits often have a very strong presumption /against/ things like goodwill and intangible value being of benefit to the corporation's bottom line.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 2:12 UTC (Tue) by cjcox (guest, #60378) [Link] (3 responses)

Insane. Red Hat distributes binaries, they cannot without the source, and even if you think they can hold the source for ransom (again, they cannot), they are actually forcing you to get a support subscription. And that's very much a violation of the GPL.

GPL in summary: You distribute binaries, you supply source. It's that simple. Mic drop.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 3:29 UTC (Tue) by adam820 (subscriber, #101353) [Link]

You can't get access to the binaries without the subscription, ergo you have access to the source.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 22:39 UTC (Wed) by mmcgrath (guest, #44906) [Link] (1 responses)

> GPL in summary: You distribute binaries, you supply source. It's that simple. Mic drop.

We're still committed to provide the source for any binaries we have distributed in compliance with the GPL. But the GPL does not force us to continue a business relationship with anyone - so it's possible that access to future binaries will go away.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 22:33 UTC (Thu) by ben0x539 (guest, #119600) [Link]

Surely this is not what anyone who ever published code under the GPL was hoping to achieve.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 2:26 UTC (Tue) by pabs (subscriber, #43278) [Link] (5 responses)

One of the reactions to this RH post:

https://www.jeffgeerling.com/blog/2023/im-done-red-hat-en... https://news.ycombinator.com/item?id=36479882

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 2:44 UTC (Tue) by pabs (subscriber, #43278) [Link]

Related video by the author of that post:

https://www.youtube.com/watch?v=kF5pyVUQBH8 https://news.ycombinator.com/item?id=36488156

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 23:08 UTC (Wed) by mmcgrath (guest, #44906) [Link] (3 responses)

A more recent tweet: https://twitter.com/geerlingguy/status/1673860031501017091

I've had some interactions with Jeff during this process. Seems like a smart guy and while I don't want to speak for him. I think he started last week with an anger level dialed up to 11, and now he's had a bit more time to process, has decided he's still angry, but probably at a level 4.

The confusion around this event has been really frustrating, if people lose trust over Red Hat, or are angry at Red Hat, it should really be because of something we did or a choice we made - not what the headline grabbers are saying we did.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 23:21 UTC (Wed) by Wol (subscriber, #4433) [Link] (1 responses)

> it should really be because of something we did or a choice we made - not what the headline grabbers are saying we did.

I've noticed a lot of the commenter subscriber numbers are about 50,000. I guess it's rent-a-mob. They obviously came here some while ago, went away, and this was an excuse to come back :-(

Cheers,
Wol

Subscriber numbers

Posted Jun 29, 2023 7:56 UTC (Thu) by pbonzini (subscriber, #60935) [Link]

I'm #60000 and I subscribed like in 2009. #50000 are fairly old mavens!

McGrath: Red Hat’s commitment to open source

Posted Jun 30, 2023 19:53 UTC (Fri) by apple4ever (guest, #164280) [Link]

> The confusion around this event has been really frustrating, if people lose trust over Red Hat, or are angry at Red Hat, it should really be because of something we did or a choice we made - not what the headline grabbers are saying we did.

And they are losing trust and angry at RH for something you did and the choice you made. Pretending its just because of the "headline grabbers" isn't going to make that better.

And there is no confusion, you just don't like the pushback to a bad decision.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 11:21 UTC (Tue) by rgilton (subscriber, #31330) [Link] (3 responses)

What I'm not getting from McGrath's post is anything about why it has suddenly become a problem for them... Redhat has been operating as a profitable company for years.

My *guess* is that the shareholders are demanding "growth", and sadly they only mean in financial terms. They'll cash-in on the RHEL name for a few years, before no-one is interested in it any more.

He appears to be trying to reframe the idea of FOSS (note: only referred to as "open source" in his post) as just submitting patches upstream. Nothing about avoiding lock-in, etc.

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 12:09 UTC (Tue) by nim-nim (subscriber, #34454) [Link]

> What I'm not getting from McGrath's post is anything about why it has suddenly become a problem for them...

Because changes in the business market take a *loong* time, and we’ve just reached the point where big companies, having dumped proprietary unixes and switched to Linux, were feeling certain enough about Centos’ continued existence, to stop paying RHEL for a *huge* part of their IT systems, deploying no-cost Centos instead.

(And some other companies were starting to sell Centos “support”, as in pay me and I’ll start harassing Red Hat and upstreams in your stead when you have a problem. Mass support ticket opening as a service, no commitment to actually fix anything included)

McGrath: Red Hat’s commitment to open source

Posted Jun 27, 2023 18:50 UTC (Tue) by HenrikH (subscriber, #31152) [Link]

Who says that it was sudden? Most likely they have seen this as a problem for years but "let's not deal with it right now since it's a mine field".

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 18:07 UTC (Wed) by jzb (editor, #7867) [Link]

I wouldn't generally look to any company to say "here's exactly the competitive landscape as we see it and the adverse conditions we are trying to navigate around" except as required in public statements. And even then they're going to be as opaque as possible while giving investors enough information that they can't be accused of withholding material information.

Since RHEL is not enough revenue to be broken out separately in IBM's results I don't know if they'd call RHEL subscription numbers as a risk or specifically the amount of annual renewal, etc. IBM's Q2 2023 results should land around July 19, so that might shed some light on this?

Some of the things that you might consider as reasons, note that these are examples and may or may not be accurate.

- Channel sales historically were important for RHEL. As cloud usage goes up there's more difficulty in making RHEL the default.
- I'm not sure what percentage of users/organizations are on EL7, EL8, etc. Red Hat may be looking at a cliff where RHEL 7 is approaching EOL and customers have to migrate to something, but what? Red Hat may want to do as much as possible to cut off other options.
- They're seeing CIQ bragging about winning business from NASA
- Related to the above - one too many "Rocky Linux / AlmaLinux" in the "loss" field for win/loss data from sales.
- Alternately, any uncertainty about the future of Rocky or Alma before this may have fed into higher renewals/subscriptions and they decided to step on the gas.

It's not sufficient for Red Hat to be "profitable" -- they have to be maintaining renewals at a certain percentage, projecting continued renewals and new sales at certain percentages.

That's a long way of saying that "growth" is a fair guess as to "why this and why now?" but I wouldn't expect the company to actually spell that out, especially approaching the end of a quarter.

"He appears to be trying to reframe the idea of FOSS (note: only referred to as "open source" in his post) as just submitting patches upstream. Nothing about avoiding lock-in, etc."

Defining FOSS is sort of like defining punk rock. There's a few bands we'd all agree on as punk, but disagree on others. (Insert your favorite genre of music if punk isn't your thing...)

Everybody argues for their pet definition of open source or FOSS. The only (mostly) agreed-on definition is the open source definition and that only speaks to licenses and not the surrounding practices. So it's not a re-frame so much as Red Hat's definition.

Let's also acknowledge that Red Hat is doing more, even today, than just submitting patches. They're driving new features and doing a big chunk of project maintenance that flows into other distributions. Even if you have never used RHEL, if you've used Linux you benefit from Red Hat's work in a significant way.

retiring but retaining expertise

Posted Jun 28, 2023 13:28 UTC (Wed) by mtaht (guest, #11087) [Link]

I would personally like to find a way to find another mere 2k/mo in income so I could semi-retire, and mostly work on writing down and passing along stuff that I know that not enough folk seem to know, and help make a few improvements there or there to slow start, cake, wifi, etc.

I am not up to major projects anymore, and do wish fervently that somehow more companies realize that expertise is vanishing as the initial wave of linux developers begin to hang it up. But they don't know what they don´t know, and have paved paradise to put up a parking lot.

LibreQos.io, which is my primary project now, I hope succeeds wildly one day for at least some cloudy applications, but donations remain pretty low, the ongoing cost of the initial development, high, and while I like to think longterm a redhat would benefit, taking the time to package it for them seems less worthwhile by the day. That said, without redhat´s investment into eBPF, libreqos would not exist today, so there´s that.

The FLOSS future really seems uncertain so long as the real costs of software development are not better cared for. One of the ironies I keep watching is the whole "made in america" set of arguments made by the current US administration, without a hint that software, at least compiled in america, would help a lot to control a bunch of national security issues introduced by the long tail of ancient software and lagging support.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 17:39 UTC (Wed) by abufrejoval (guest, #100159) [Link] (4 responses)

McGrath writes:
>There was a time, not too long ago, that Red Hat found value in the work done by rebuilders like CentOS. We pushed our >SRPMs out to git.centos.org in a neat package that made them easy to rebuild; we even de-branded it for them. More >recently, we have determined that there isn’t value in having a downstream rebuilder.
and
>This demand for RHEL code is disingenuous.

I believe this decision to be a) stupid, b) disingenouous, c) unargued

and believe Redhat should have held an open discussion right at the beginning of this path: that could have kept them from shooting themselves into the foot.

c) unargued and b) disingenouous
He admits, their opinion around CentOS has changed, but doesn't really argue the "value", which ones he means, how they were assessed, and how they developed.

If lack of direct remuneration is the issue (which seems his major complaint), that hasn't changed as CentOS was never a money-maker.

If the value assessment was much more complex, there must be data, trends and arguments. Leaving them out completely is being disingenouous, in their original stance (what was the value originally?), their current stance (why is it gone?) and in the fact that the change isn't explained.

"This is our call to make" puts the Enterprise Linux ecosystem at their whim. That has always been legally so, and whoever did not see this coming, evidently misplaced their trust.

But now that it's crystal clear Redhat is as decided about community enterprise Linux as Putin about controlling the Ukraine, the community will react.

And that brings me to

a) stupid

He (and all those, who made that call) seems to believe, that if someone benefits from Redhat's work, that should also benefit Redhat, because otherwise they are just parasites.

They also seem to believe that denying any such benefit to parasites will benefit Redhat.

Well, we might have to wait a bit until this plays out, but I believe the reason why Microsoft Windows and Office continue to be available for relatively easy pirating or at under-the-counter prices is because Microsoft learned long ago, that enforcing full licensing compliance would have killed their babies: they understand that it's only because few consumers pay full license cost for M$ that they can continue to charge the bigger prices to corporate and government users. It's because consumers walk into offices and then vote with their feet to support habits and convenience.

Likewise Redhat achieved its current domination only, because there were so many users runing CentOS. And I am pretty sure that a lot of feet will now drag themselves to other Enterprise capable Linux variants which will deprive Redhat of the community user base, which enabled their domination.

Redhat's position in the Enterprise space suffers from evaporation into the clouds, not from the CentOS fanbase. The only thing they can achieve by annoying those who decided not to pay for RHEL after being offered the choice, is faster evaporation to the side as well.

The damage this decision will inflict on people who are not direct Redhat customers will be giant, not because there are huge untapped budgets, but because the cumulative effect on millions of users will add up. But while is no Redhat revenue potential, the pain it inflicts on the community is very real and its appreciation of greed very low.

The eventual damage to Redhat could kill RHEL and I don't know how much money Redhat hopes to make on OpenShift without all those feet in the door.

If I had any shares, I'd sell or sue. And let's please hold onto those bonuses until the story played out.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 19:20 UTC (Wed) by pizza (subscriber, #46) [Link] (3 responses)

> and believe Redhat should have held an open discussion right at the beginning of this path: that could have kept them from shooting themselves into the foot.

What would RH have got out of this, other than folks saying "Hey, keep providing everything we want from you (ie RHEL), for free, forever?" Seriously asking here.

> He admits, their opinion around CentOS has changed, but doesn't really argue the "value", which ones he means, how they were assessed, and how they developed.

This "value" is from RH's perspective; ie the benefit they get back from investing their resources. While they're under no obligation to share the details of their analysis or their raw data, I don't think it's unreasonable to take RH at their word that their "investment" in making things nearly trivially easy to rebuild RHEL was actively harming them -- due to certain rebuilders selling their own commercial support offerings.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 20:16 UTC (Wed) by abufrejoval (guest, #100159) [Link] (2 responses)

The rebuilders were reselling commercial offerings, long before they took on CentOS. Money evidently wasn't a concern then, because they never talked or complained about it.

When they did take CentOS on, they said it would not change things. And people who had reasons not to use RHEL under contract, therefore trusted that to remain the case. Still there were no complaints about rebuilders.

Then they redefined CentOS as upstream, but waxed about everything except the impact on rebuilders. People were sceptical about the benefits of upstream, but Redhat felt compelled to emphasize that it wasn't about the rebuilders but agility benefits all around.

Now they cut off the repo access, wax on again about this and that, but not about rebuilders and only after being pushed hard, they admit that it was all about the rebuilders after all and Redhat not having any direct benefits from them.

They weren't honest about intention and motives for years. And now they pulled the plug without warning, because "it's our call" and they do not care about collateral damage.

Yes, they were under no legal obligation to share, but they were hiding their intent, did not give warning and as a consequence a lot of people are getting harmed.

And when people are being burned, they aren't very understanding (Maslov), they just won't forgive whoever is responsible.

It doesn't matter if McGrath then explains that you shouldn't have touched the pan without a RHEL support glove, or that it's unfair to expect the hot pan to be cool on the handle without paying for the protection.

Pain destroys trust and followings very radically and there is plenty of data out there on how human networks amplify churn e.g. in TelCos when pain and anger are involved.

And they show just how incredibly expensive it would be to turn such a tide.

IMHO the bean counters didn't have the right numbers for that risk on their balance sheet.

Let's not argue, but revisit in a year or two: by then we might know if they shot themselves a bullet in the foot or vitamins in the arm.

P.S. If anyone had actually invested into making it easier to clone RHEL, then that person should now be sued. I can't believe that would have been an 'honest mistake'.

McGrath: Red Hat’s commitment to open source

Posted Jun 28, 2023 21:09 UTC (Wed) by pizza (subscriber, #46) [Link] (1 responses)

> The rebuilders were reselling commercial offerings, long before they took on CentOS. Money evidently wasn't a concern then, because they never talked or complained about it.

No, the rebuilders (other than perhaps Oracle) were not previously "reselling" anything.

Is it really so dificult to comprehend that something might have grown from a non-problem in the past to something much more serious in the present? (And given the reaction to when they are talking about it now, are you seriously saying the response would have been any different if they'd done this sooner?)

> When they did take CentOS on, they said it would not change things.

No, they announced their plans for the CentOS "brand" almost from the very outset, and the status quo of today is pretty close to what they laid out. (Note that CentOS itself was barely alive by the time they acqu-hired the folks doing all the work!)

> And people who had reasons not to use RHEL under contract, therefore trusted that to remain the case.

Without a contract, you didn't, and still don't, have the right to use "RHEL."

> Then they redefined CentOS as upstream, but waxed about everything except the impact on rebuilders. People were sceptical about the benefits of upstream, but Redhat felt compelled to emphasize that it wasn't about the rebuilders but agility benefits all around.

Yes... and by every measure, CentOS Stream is a massive improvement for folks that don't actually require RHEL, especially when compared to the prior CentOS model that saw updates considerably delayed from when RHEL customers received them.

> Now they cut off the repo access, wax on again about this and that, but not about rebuilders and only after being pushed hard, they admit that it was all about the rebuilders after all and Redhat not having any direct benefits from them.

First, the only point of the repos was to make things easier for rebuilders. Why should RH continue to do work they not only don't benefit from, but actively harms them instead?

> They weren't honest about intention and motives for years. And now they pulled the plug without warning, because "it's our call" and they do not care about collateral damage.

Be careful, you're claiming to know and speak to their intentions and motives.

> Yes, they were under no legal obligation to share, but they were hiding their intent, did not give warning and as a consequence a lot of people are getting harmed.

Who, exactly, is "getting harmed" by this? Everything Red Hat has ever released remains available. It's all Free Software, after all. Or are you seriously claiming that non-RedHat-customers are somehow entitled full access to Red Hat's future work, for free, in perpetuity?

> Let's not argue, but revisit in a year or two: by then we might know if they shot themselves a bullet in the foot or vitamins in the arm.

Yep, time will tell. But I should point out that this is the third or fourth time I've seen this argument spelling doom and gloom over something that Red Hat has done.

> P.S. If anyone had actually invested into making it easier to clone RHEL, then that person should now be sued. I can't believe that would have been an 'honest mistake'.

Huh? I'm not able to follow your logic here. This is about RH no longer handing rebuilders everything they need on a silver platter; the software is _all_ F/OSS, so other parties are completely free to make up the (considerable) difference if they think the effort is worth it.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 10:03 UTC (Thu) by farnz (subscriber, #17727) [Link]

> Let's not argue, but revisit in a year or two: by then we might know if they shot themselves a bullet in the foot or vitamins in the arm.

Yep, time will tell. But I should point out that this is the third or fourth time I've seen this argument spelling doom and gloom over something that Red Hat has done.

One thing I have noted each time this has come round (and I've been watching this since Red Hat Linux 5.2, so saw the creation of Fedora and RHEL) is that the "doom and gloom" for Red Hat (RH) is based on the argument that the change RH is making will cost RH goodwill, and that loss of goodwill is going to lead to a long term decline for RH.

That's always been a very weak argument about any company - goodwill is notoriously hard to value - and RH in any case also generates goodwill by working upstream, such that any change like this is a blip downwards in a constantly regenerating stream of goodwill.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 8:01 UTC (Thu) by zeroheure (guest, #165859) [Link] (9 responses)

> Ultimately, we do not find value in a RHEL rebuild

Greg Kurtzer provided an interesting answer to that in 2021 in an interview for thenewstack.io:

"Linux is a community endeavor, and it is freely available. There are thousands of separate projects that are all built together in a Linux distribution, but the distribution is simply the culmination of these packages. While there is value in assembling it, it is still comprised exclusively of freely available software. For that reason, there always needs to be freely available options that are not controlled by corporate needs to ensure proper mitigation of inherent conflicts of interest."

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 8:24 UTC (Thu) by milesrout (subscriber, #126894) [Link] (1 responses)

Good thing there's Arch, Debian, and every other Linux distribution out there, then, eh? What significant Linux distributions are "controlled by corporate needs" other than RHEL and Ubuntu?

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 10:07 UTC (Thu) by aragilar (subscriber, #122569) [Link]

SUSE? I don't think alpine is, but I'm not sure who employs the maintainers (which may or may not change opinions). There are also the cloud provider distros, which may also change the balance between community and "corporate".

I think if you're looking at "LTS" distros though, Debian is the only independent community one as far as I know, and there are no independent RPM-based ones?

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 12:33 UTC (Thu) by kleptog (subscriber, #1183) [Link] (6 responses)

I think that seriously underestimates the effort involved in integration. The effort it takes to go from "works on my machine" and "works on everybodies machine" is about a factor of three, at least. And distributions do a large part of that work. The idea that the author of a software project is going to, without help, make it work on a dozen different computer architectures is farcical.

(Maybe the fact that the number of distinct architectures is much reduced is related to the reduced perceived value of distributions. My impression is that a lot of the 32->64bit transition was done with heavy involvement of distributions.)

Ironically, the argument "Alma & Rocky Linux are harmed by the decision" automatically concedes that what RedHat does adds value. If RedHat did not add value there would not be any harm.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 13:17 UTC (Thu) by pizza (subscriber, #46) [Link] (2 responses)

> Ironically, the argument "Alma & Rocky Linux are harmed by the decision" automatically concedes that what RedHat does adds value. If RedHat did not add value there would not be any harm.

Yeah, the cognitive dissonance around that point is quite astounding.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 15:16 UTC (Thu) by Wol (subscriber, #4433) [Link] (1 responses)

I think the cognitive dissonance everywhere is quite astounding - "Wah wah wah why won't Red Hat do all this worthless work for me for free ..."

Cheers,
Wol

McGrath: Red Hat’s commitment to open source

Posted Jun 30, 2023 3:31 UTC (Fri) by ayay (guest, #144420) [Link]

"I think the cognitive dissonance everywhere is quite astounding - "Wah wah wah why won't Red Hat do all this worthless work for me for free ...""

It's more like:
1) they don't *want* to deal with an organization that's so small as to not be worth their time - fine.
2) we proceed to use CentOS, as allowed by the rules they have laid out.
3) things change for them, the rules change - again, that is fine! - but then they proceed to throw the word "freeloaders'" around, as if we were not following the rules they've laid out by their own free will.

Why is it that, when it benefits *them* (market share, patch submissions, "goodwill" or whatever) it's all charitable and noble, and yet, when it doesn't, everybody is just too willing to bleed our most magnificent benefactor dry, such treacherous snakes that we have now suddenly become?

To be fair, other than that, I'd say McGrath was fairly clear. As much as I wish they`d see the value in dealing with others that are not huge corpos eye-to-eye, at least I can appreciate he being refreshingly forthcoming.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 16:53 UTC (Thu) by mb (subscriber, #50428) [Link] (2 responses)

>I think that seriously underestimates the effort involved in integration.
>The effort it takes to go from "works on my machine" and "works on everybodies machine"
>is about a factor of three, at least.

Did you even think about that before posting it here?

The step "works on my machine" includes developing the whole application, designing the program architecture, designing the code structure, writing the code, writing the documentation, writing the unit tests, writing the system tests, etc, etc.

The step "works on everybodies machines" includes fixing portability issues and packing the application, running some system tests, maybe write some distro specific docs. That's basically it.

McGrath: Red Hat’s commitment to open source

Posted Jun 29, 2023 18:06 UTC (Thu) by pizza (subscriber, #46) [Link] (1 responses)

> The step "works on everybodies machines" includes fixing portability issues and packing the application, running some system tests, maybe write some distro specific docs. That's basically it.

If writing the software represents 90% of the effort, "Works on everybodies machines" represents the second 90%.

It also represents ongoing effort, as it actually requires testing on different hardware (== cpu arch, graphics card, etc) and software combinations ("stock OS release, fully updated as of time X, and likely points in between. For each OS target!) and of course all of that has to be pre-emtively repeated each time you generate a new release or the OS(es) update a component or driver.

McGrath: Red Hat’s commitment to open source

Posted Jun 30, 2023 14:43 UTC (Fri) by jeleinweber (subscriber, #8326) [Link]

That second 90% is spot on.

In Fred Brook Jr's classic "The Mythical Man-Month" he estimated the difference in work between an internal-use-only project and a commercially supported and saleable one at roughly 9X.

To take a personal and trivial example, once upon a time a project I had developed and was supporting got a brilliant suggestion from an end-user. Development time to implement her request was 10 minutes. Project effort to notify two different organizations, get two sets of management approval, document the change, and train two different sets of end-users on the resulting new and additional procedure was 4 hours. I considered this 24X difference between development time and project time to be quite reasonable, and was thrilled that I managed same-day turnaround. The suggestion, as implemented, probably eliminated several weeks worth of busywork for the involved organizations just in its first year. Even at 24X non-development overhead, the ROI was amazing. Thinking that the development work - which is incredibly important - represents the bulk of the effort would be naive. This was paid work, not volunteer.


Copyright © 2023, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds