|
|
Subscribe / Log in / New account

The things nobody wants to pay for

The things nobody wants to pay for

Posted Jan 26, 2024 0:20 UTC (Fri) by geofft (subscriber, #59789)
Parent article: The things nobody wants to pay for

So here's a problem I've been thinking about a bit. I'm a fairly senior engineer at a sizable tech company (1000+ engineers and a whole lot of servers that run Linux). We understand and believe in open source. We have a workable open-source contribution process, and we employ core contributors to a couple of projects we care specifically about. We donate money to various foundations and sponsor conferences that fairly explicitly use their proceeds to fund their associated project. But we benefit a whole lot from, say, the Linux kernel existing, and we have very few opportunities to help with code (because it works so well!) and we aren't funding any regular work on upstream Linux. (We run upstream stable kernels on top of Debian, so there isn't an obvious commercial vendor or even a distro packager for us to buy a support contract from.)

What should we do, and how do I, as someone with some amount of influence but no actual spending authority, get us to do it?

The Linux kernel contribution maturity model (https://docs.kernel.org/process/contribution-maturity-mod...) does not seem particularly relevant to us. We're not a hardware vendor or otherwise in the business of writing kernel code. We're at level one - we can and do contribute patches to the Linux kernel, in practice every couple of years. Level two suddenly talks about people being expected to contribute as part of their job and having my contributions assessed in performance reviews. That's a drastic step up! I cannot figure out what an employee who has sent in the occasional patch (like me) would possibly do to work on the Linux kernel with even 10% of my time year-round. I'm honestly not sure LKML would really want me to dive in and say "Look guys, my boss told me to contribute, so I'm just going to keep refactoring this subsystem." And the people who do my performance reviews aren't really in a spot to evaluate my contributions beyond whether they happened or not.

Besides, this article is mostly about specific existing projects and specific existing contributors talking about whether they can do them, not the need for new blood in general. (There are other articles about that.)

So, okay, let's not think about contributing more code ourselves. Greg KH points out that there isn't enough testing because companies aren't funding the work. We'd love more testing, it would make us more confident in upgrades. Where do we send a check? (Again, I have contributed to the kernel a little bit, and I have no idea what I would be asking my company to do.) The Linux Foundation gets a good bit of corporate money, as I understand ($177M from their last form 990). Does that money go to this sort of work? If we send LF a small amount more money, would it fund this work?

More importantly, how do I convince my company that this is a good use of our money? I doubt I can convince my employer to sponsor even one full-time engineer at the wages we pay our full-time engineers; we have enough problems of our own to solve, and as important as kernel testing is, it's not as important as anything else we could put one full-time engineer on. (We run at a scale large enough to see kernel regressions that weren't caught by anyone else, so it would concretely help us, but we don't run at a scale large enough that it's regular: it happens about once every few years.) We certainly benefit from Linux in general, but we're not talking about paying to make sure we can keep using Linux at all, we're talking about making the development process a little faster and a little more reliable. But the argument in the article about tooling paying off compounding rewards is an interesting one. Is there a compelling argument that it really is a good use of our money to fund the kernel by an entire engineer's salary?

Or let's say that we want to contribute a fraction of an engineer's salary. How do I convince the people who sign checks that this money is going to go somewhere useful? It won't make any of the existing contributors have more hours in the day (or would it? is part of their income coming from Linux-irrelevant private consulting work that they could stop doing?). I suppose we and many other companies like us could collectively fund one engineer, but how do we get that to happen?

(Is this a question for us the corporate users to answer? Like should we be proactively reaching out to other companies in our industry and saying, hey, all of us get value from Linux being good, let's collectively fund someone?)

Or maybe funding development time isn't the right thing to focus on. I see in the thread some talk about KernelCI. Would contributing money to KernelCI to buy more machines or more public cloud time help? If I go to the KernelCI home page, there's a "contributions welcome!" button ... that links to GitHub. A little bit below, it says, "Learn how your organization can contribute to the project now," but the way to learn is to send someone an email. What does it mean to join? And then there are logos of a couple of companies below it, at least a few of which are much larger than we are and care much more than we do about Linux development. Are they contributing so much that we'd be a drop in the bucket? I have no idea. Nothing on the website answers the question about how much money KernelCI is spending and whether they could productively spend more (but it does have some brand guidelines, great).

Or maybe I'm totally off base here, and when Greg KH and others are talking about "companies," they are actually talking about gold/platinum members of the Linux Foundation who are already contributing a lot but could still justifiably employ several more individuals full time on upstream kernel work, and non-huge companies like mine are just not part of the conversation, and articles like these aren't intended for people like me to try to get our employers to contribute more. So maybe a simpler question is - in an ideal world, where everyone in my employer's management was completely and accurately convinced of the need to fund Linux kernel development, would that accurate funding really be more than the almost-zero it is now? (One argument in this direction is we pay quite a bit of money to our hardware and cloud vendors, and a feature of their product is compatibility with upstream Linux, so we're already doing the thing that ought to fund kernel development. In that case, perhaps what I really should do is tell our account reps at those companies - all of which are already gold/platinum members of the LF with full-time kernel developers on staff - that we want to see them funding this work even more.)


to post comments

The things nobody wants to pay for

Posted Jan 26, 2024 5:25 UTC (Fri) by NYKevin (subscriber, #129325) [Link] (2 responses)

As someone who works at (and does *NOT* speak for) a FAANG, my opinion is that non-huge companies should focus on specific areas where they have the expertise, the motivation, and the practical ability to contribute developer time or other resources. It may be the case that your company does not stand to benefit from funding Linux any more than it is now. This is not a moral failing on the part of your company. It's just business.

At the same time, I must say that I find it rather disturbing that you were unable to get useful answers to your questions about certain specific areas of contribution. It is all well and good to complain that nobody wants to pay for certain things, but you can't put this sort of information at the bottom of a locked filing cabinet in a disused lavatory etc. etc., and then complain that nobody managed to find it and contribute.

The things nobody wants to pay for

Posted Jan 26, 2024 8:06 UTC (Fri) by lunaryorn (subscriber, #111088) [Link] (1 responses)

Moving someone else's money (eg a donation or recurring fund from a company) around on behalf of someone else (like an open source project which isn't even a legal entity on its own in most cases) comes with so many legal, taxation, billing, accounting, reporting, etc. issues that it's probably not surprising that many open source projects have no infrastructure and thus no documentation around this.

The things nobody wants to pay for

Posted Jan 27, 2024 1:05 UTC (Sat) by NYKevin (subscriber, #129325) [Link]

That is an understandable problem to have. But it does not change the fact that, if you don't have documentation telling people how to help you, then some people will not bother trying.

The things nobody wants to pay for

Posted Jan 26, 2024 7:46 UTC (Fri) by pbonzini (subscriber, #60935) [Link] (2 responses)

If you run the stable Debian kernel, checking for patches that didn't apply to long-term stable kernels (in an area that you care about) and backporting them yourself could be an option.

The things nobody wants to pay for

Posted Jan 26, 2024 8:26 UTC (Fri) by Wol (subscriber, #4433) [Link] (1 responses)

> If you run the stable Debian kernel, checking for patches that didn't apply to long-term stable kernels (in an area that you care about) and backporting them yourself could be an option.

If your setup is at all out-of-the-ordinary, see if you can set up your own CI test-bot, and hammer it (as part of the general kernel testing system), and then you can sell it as buying you extra confidence when you upgrade, you know your own setup has been put through the mill on your own test-bot.

Cheers,
Wol

The things nobody wants to pay for

Posted Jan 26, 2024 12:12 UTC (Fri) by spacefrogg (subscriber, #119608) [Link]

Yes, exactly this!

For many projects it would also be helpful to:
- diversify their build/test infrastructure and
- do the boring jobs (writing, maintaining documentation, translations, platform-independence, packaging)

People are motivated to build stuff that helps them(!), the aforementioned topics are chores that do not benefit the original authors and thus need monetary persuasion. This is where your company has leverage where private people have not.

Free Software is a common good like streets, utilities or parks and nature reserves. Your company, as any other company, usually does not directly invest into common good but rather pays taxes. Putting money into free software is charity, not investment (aside from rare cases in highly specialised fields). Thus, you should never try to sell it as that. So, as for convincing the bean counters? You won't. Either it is in your company's DNA to give back to common good or it is not, and it won't happen. There will be no outcome that can positively reflect in an Excel sheet. Even in those rare cases, the outcome will not be measurable and always under scrutiny of being scrubbed for short-term benefit. The will to pay has to come from outside of money making or it won't last. You might sell it as good marketing, or your boss being the cooler kid on the golf course etc. These are all human factors and not numbers.

The things nobody wants to pay for

Posted Jan 26, 2024 11:11 UTC (Fri) by dvrabel (subscriber, #9500) [Link]

The company you work for seems to be doing the right thing already. It's non-kernel contributions likely gives much more value to the ecosystem than any kernel-related contribution it could make.

The things nobody wants to pay for

Posted Jan 26, 2024 20:34 UTC (Fri) by mfuzzey (subscriber, #57966) [Link]

I'm not sure it's necessary, or useful, for most companies that use the Linux kernel to somehow force themselves to contribut to it.
I think the kernel is one of the better off OSS projects that already has quite a lot of funded developers.

But there are lots of other, often critical, OSS projects that have very little funding - maybe it would make more sense to contribute to those rather than the kernel.

The things nobody wants to pay for

Posted Jan 27, 2024 11:06 UTC (Sat) by amacater (subscriber, #790) [Link]

You run upstream kernels on Debian servers so you don't have a support contract to worry about - but you have 1000 servers. Give each one of your engineers 10% time to find some utility that they use that they want to see maintained.
Check that it actually *has* a maintainer and that it's not being maintained by the QA team in Debian because there isn't an actual maintainer somewhere. Allow them some time to look at bug reports for that utility, check whether they apply, submit fixes (or suggest closure because "can't reproduce here") If your firm can help reduce backlogs on fixes by contributing a fix/patch - do so.

If you have some in-house tooling/integration that works for you - _really_ document it well. Explain it - you'll benefit when those maintaining it move to another project. If it's specific to you, but the general knowledge you used to build it was hard-won because you had to look in 20 places - write up the experience and list the 20 places so the next person doesn't have to do that. Use that documentation experience to help upstream documentation / write technical blog posts / summarise where to find the vital information and share the 20 places. Write the missing manuals from bitter experience ...

if all else fails, Debian could do with clueful experts lurking on mailing lists and contributing real experience where it's meaningful :) Teach good practice in real-world sysadmin and security or contribute constructively to debate - but contribute.

The things nobody wants to pay for

Posted Jan 27, 2024 17:30 UTC (Sat) by ceplm (subscriber, #41334) [Link] (5 responses)

Isn't the most simple answer to your problem just “Go, and buy some licences for SLE/RHEL/Ubuntu”? It is easy, all management overhead is already included, you don’t have to organize anything, and most of your money will directly or indirectly benefit Linux.

The things nobody wants to pay for

Posted Jan 27, 2024 18:05 UTC (Sat) by geofft (subscriber, #59789) [Link] (4 responses)

Will it? The article talks about how Red Hat and SUSE and Canonical aren't paying for things. I doubt one additional customer who doesn't even use their product will change that.

Also, I would need to successfully convince my company to spend money on it. I can pretty easily get the people who manage the donations and conference sponsorship budget to give, say, $10,000 to a relevant open-source nonprofit. I absolutely cannot get the purchasing team to buy a $10,000 product we won't use.

And if I tell them the motivation is "Well, the company that sells this product is going through layoffs, but maybe they'll have fewer layoffs if we support them, which might cause them to resume spending resources on the infrastructure behind the shared commons behind the product we won't use, because we also use that shared commons, and by the way their competitive differentiation is things that aren't part of that same shared commons, because the shared commons is free and we've been using it for free for twenty years," I would probably not change their minds.

I do actually think these companies would be well placed to sell a "sponsor 25% of an upstream developer" product. We would get something measurable about what our money is accomplishing. They'd be in a good spot to put that money to use, because they could reallocate an existing employee's time or use their reputation and market position to hire someone new and find three other customers. But they don't seem interested in selling that.

The things nobody wants to pay for

Posted Jan 27, 2024 18:30 UTC (Sat) by ceplm (subscriber, #41334) [Link] (1 responses)

> Will it? The article talks about how Red Hat and SUSE and Canonical aren't paying for things.

We may not pay for the things (I work for SUSE, but of course I have zero influence on the corporate decisions of the company), but we spend zillion of money on upstream development. Yes, you may not be able to decide what exact part of the Linux universe you support, but comparing with supporting somebody who says they support Debian, it is the easiest and most secure way how to do it (OK, I see https://www.debian.org/donations, but I guess that goes completely on the Debian infrastructure, right? So, nothing goes upstream … that’s absolutely nothing against them, but if you want to support upstream development, this may not be the right channel).

Of course, I don’t know if you want to target your donations than you have to work your way through various pages like https://www.linuxfoundation.org/about/join, https://sfconservancy.org/, or many other organizations like that.

The things nobody wants to pay for

Posted Feb 1, 2024 13:36 UTC (Thu) by rhertzog (subscriber, #4671) [Link]

> I guess that goes completely on the Debian infrastructure, right? So, nothing goes upstream

Indeed money donated to Debian is not used to pay for the time of Debian developers/contributors. It might be used for hardware, hosting, travel reimbursement, debconf organization, and that kind of activities.

<shameless plug>
With Freexian we try to help fill that gap. Thanks to the various LTS offers, we are paying Debian developers to work on Debian. This is part of our mission and we are regularly reporting our activities:
https://www.freexian.com/about/
https://www.freexian.com/about/debian-contributions/
</shameless plug>

For now, it's the margin on the various services that make this possible but we are considering to add some broader offer that would also encompass the "please take my money to contribute to Debian and make it a bit more sustainable" use-case.

I'd be happy to discuss with companies that are looking to contribute to Debian in such a way to help refine our future offer.

The things nobody wants to pay for

Posted Jan 28, 2024 10:07 UTC (Sun) by jengelh (guest, #33263) [Link] (1 responses)

>Will it? The article talks about how Red Hat and SUSE and Canonical aren't paying for things [such as the desktop]. I doubt one additional customer who doesn't even use their product will change that.

Now, if RH/SUSE systems were running "popcon", corporate would at least have an idea that they should be spending so-and-so-many percent for the desktop.

The things nobody wants to pay for

Posted Jan 28, 2024 11:03 UTC (Sun) by rahulsundaram (subscriber, #21946) [Link]

> Now, if RH/SUSE systems were running "popcon", corporate would at least have an idea that they should be spending so-and-so-many percent for the desktop.

They have clear stats on which packages are being used because a commercial subscription already provides that data. I would bet they are already using this to drive decisions,

The things nobody wants to pay for

Posted Jan 28, 2024 3:53 UTC (Sun) by jmalcolm (subscriber, #8876) [Link]

At the moment at least, I feel like the Linux kernel is well taken care of. I would not worry about it. Companies with more specific needs for kernel dev are contributing on your behalf.

In my view, a good Open Source citizen helps promote what they use, files bug reports when they should, contributes what they can, and at least partially funds the "change" they need to see in the code ( either through contribution or cash ). If we stick to the original "Free Software" vision, just using the software is contributing to the change some people want to see in the world.

For smaller projects, the core devs may not be making a living. Those kinds of projects would most directly benefit from some cash infusion if you have the means. For projects that have already crossed a certain threshold though, I see no reason to feel bad about using software as-is without funding its development. I mean, if you are getting a bunch of direct commercial benefit from a specific project, it would be nice to share the wealth. I am not trying to dissuade anybody from funding Open Source by any means.

I guess what I am saying, as an example, is that most of us should not feel bad for using the Linux kernel "for free". My use of Linux these days places no demands on the kernel as a project. Nobody is having to spend any dev time directly on my behalf. If there were some deficiency I needed to see addressed, I would think about how to contribute to that. As you say though, for most of our use cases, the kernel is simply excellent these days already. Yes, I benefit from it getting even better but I am not the "primary" beneficiary. I am simply along for the ride and, in the case of Linux, I am a "free rider" but I am a "zero cost" one. Unlike say roads, I am not wearing the Linux kernel out by using it. I do not consume any of its resources. In fact, as above, my use of Linux "for free" can be thought of as a contribution of sorts.

Concentrate on the projects where you can and do add value. Don't worry about using Open Source for free. Just be thankful.

Although I guess I will make one request. Please do not turn around and insist that the fine people that do the work, that author all this amazing software for us, bow down to "your rights" when they do try to get some value for what they do. We have had some "community" back-lash moments over the last year that, in my view, show a massive lack of respect and appreciation for the people that write the code.

Enjoy what you use. Fund or contribute where you can but don't sweat it when you don't. At the very least, refrain from attacking the people doing the work.

The things nobody wants to pay for

Posted Jan 30, 2024 13:19 UTC (Tue) by metan (subscriber, #74107) [Link]

The kernel testing is actually a complicated landscape. There is numerous entities that work on a different parts. We do have a several upstream projects for a different kinds of tests, different projects for lab management software, project that work on processing and visualizing the results, and then different labs that actually run all of that and report results. So I suppose that it's not really easy figure out how to invest or contribute because it's hard to figure out where or how to do so. As far as I can tell there is no single entity that would take donations and distribute them between projects.

As for myself I've been maintaining the Linux Test Project for more than a decade and what we really need is contributors and maintainers who can help with the patch review. However I have no idea how to make this happen in practice, there is a more than handful of companies that use the testsuite for testing, but only a few allocate resources to contribute back or even invest into a developer becoming a maintainer.


Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds