The things nobody wants to pay for
The GNU project, as described by Richard Stallman in the 1985 GNU Manifesto, looked hopelessly ambitious at the time. To many of us, it seemed that only large companies could build operating systems, and that a group of volunteers would never be able to aspire to such a goal. The volunteers got surprisingly far, to the point that, less than ten years after the GNU Manifesto was published, running a system on only free software (or something close to that) was possible. It was an impressive achievement.
Even then, though, that software not entirely devoid of corporate contributions. The X Window System, for example, was the product of a commercial consortium that predated Linux. The development of GCC was pushed forward by companies like Cygnus Computing. When the Linux kernel arrived on the scene, there was indeed a substantial body of GNU software that could run on it, but there was a nontrivial amount of company-contributed software as well.
Linux in the 1990s still lagged far behind the proprietary Unix systems in many ways, even though it was better in others. That began to change with the arrival of corporate funding, which supercharged development on Linux and on free software in general. Without it, we would not have the system we take for granted today. Companies, working in their own interest, have built up our body of free software hugely; it is almost as if this capitalism thing actually works.
The problem, of course, is that these companies have a tendency to interpret their own self-interest rather narrowly. They will happily pay to develop a driver for a hardware product, but they are less interested in supporting the maintainership and review that are needed to integrate that driver, the development of the subsystem into which the driver fits, or the support the driver will need over the years. Companies pay for thousands of developers to work on the kernel, but none pays for a single technical writer to create documentation. Work that is not seen as contributing to short-term revenue tends not to get much attention in the corporate world.
There are, needless to say, numerous other pathologies exhibited by corporations in the open-source community. These include license violations, free-riding, throwing code over the wall, and more. Projects that are controlled by a single company are often particularly problematic. These difficulties have often been covered elsewhere; today the focus is on the failure to support parts of the community that we all depend on.
Consider some recent examples of how this behavior affects the development and user communities.
In a recent linux-kernel discussion, Kent Overstreet complained
about the lack of testing infrastructure for the kernel, calling it a
failure of leadership. Greg Kroah-Hartman, instead, said that
the problem lies elsewhere: "No, they fall into the 'no company
wants to pay someone to do the work' category, so it doesn't get done.
"
He pointed out that there is not much the leadership (such as it is in the
kernel community) can do in a situation like this. So testing
infrastructure for the kernel tends to languish.
Also recently, Konstantin Ryabitsev, the keeper of kernel.org and the
author of the b4
tool, apologized
for a lack of progress with b4, observing that he has not been able to get
the go-ahead to put time into that work. He later clarified
that post: "it just means that we haven't properly reallocated resources
to allow me to prioritize tooling work
". He has a lot of demands on
his time, and b4 has not been able to rise to the top of the list.
This matters; anybody who has been watching kernel development over the last few years has seen that b4 has transformed the maintainer's task in many ways. It must have paid back the effort that went into its development many times over. The kernel community has never put the effort into tooling that it needs; the advent of b4 was a change in that pattern and one that we would all like to see continue.
One can easily criticize the Linux Foundation (LF) for not supporting this work at the level that we would like. But the fact of the matter is that the LF is about the only entity that has supported this work at all; without that support, we would not have b4. So, while encouraging the LF to more strongly support b4 is a good thing to do, it is also worth asking why no other company has seen fit to support that work, despite the fact that they have all benefited from it. The incentives that drive companies (and their managers) simply leave little room for this kind of work, even though the benefits from it would be real and immediate.
In a different part of our community, a discussion on the upcoming openSUSE Leap 16 distribution has led to fears that this version of the distribution will not support packages like KDE or Thunderbird — fears that are arguably getting ahead of the game, since they have not yet been confirmed by SUSE. Such a move would seem similar to Red Hat's decision to drop LibreOffice from Red Hat Enterprise Linux. Support from companies for the Linux desktop, it seems, is threatened.
Once again, it comes down to corporate priority setting, though with a slightly different driving force. Companies like Red Hat and SUSE (and a number of others) have supported Linux desktop development heavily for many years. But that investment is not seen as paying off. As Neal Gompa put it in the Leap discussion:
Have you ever wondered why KDE was deprecated in RHEL? It wasn't just because KDE Plasma 5 was so big that they couldn't make the jump for RHEL 8, it was also because people were not paying for RHEL for the desktop, so they had no budget for more people. It happened again with the layoffs at Red Hat last year. SUSE is no different. And Ubuntu? Canonical already did their big layoff in 2017 where they laid off hundreds of staff related to the desktop.
In many parts of the system, the work that leads to some company making money just happens to benefit all of us, even if we are not directly paying for it. But, seemingly, that path toward making money is more elusive in the desktop realm; those of us who are using desktop Linux are not generally paying for the privilege, and neither is anybody else. So the resources going into desktop development are reduced, and Linux desktop users will feel the effects of that.
Then, consider this ongoing discussion in the Python community about funding for the PyPI repository. As "fungi" put it:
Companies are unlikely to fund open source communities that are critical to their own business because “someone else will do it,” so it ends up falling on a handful of volunteers and people donating their own time after hours because they’re already being paid to work full time on other stuff.
There are few projects in our community that do not contain this kind of neglected area.
Solutions to these problems are not easy to come by. Criticizing companies for a failure to support the ecosystem they depend on can have results, but only to a point. Organizations like the LF can organize resources toward the solution of common problems, but they have to please the same companies that are paying their bills in the end. Governments can help to fund areas that the market has passed over; that path gets harder in the absence of a functioning government, which is the situation to varying degrees in many parts of the world at the moment.
If we cannot find a solution, we are likely to be forced back to our roots, depending on volunteers to do the work in areas that companies decline to fund. That can be successful, but often at a high cost to the people who are doing that work. Depending on volunteers is not an inclusive approach; there are a lot of people who do not have the luxury of giving many hours of their time to this kind of project. Progress in that world will be slow.
The community that has accomplished so much over the last few decades
should be capable of doing better than that. We have solved many problems
getting to the point we are at now — problems that few thought we would be
able to overcome; we should be able to find a way around these difficulties
as well. It will be interesting to see what we come up with.
Posted Jan 25, 2024 16:31 UTC (Thu)
by bluca (subscriber, #118303)
[Link] (3 responses)
I think it's a winning model: tax these corporations, and use the money to redistribute it to FOSS projects they use. The voluntary-based donation scheme will never go far enough - it's a constant uphill battle, that relies on well-meaning employees trying to do the right thing and fight the bean counters budget after budget after budget.
Posted Jan 25, 2024 16:44 UTC (Thu)
by sdeetee (subscriber, #141041)
[Link] (1 responses)
Posted Feb 14, 2024 13:30 UTC (Wed)
by wiktor (guest, #132450)
[Link]
They are great initiatives indeed and I'm recommending them to open-source developers. But they are not ideal: STF targets large, known projects. NLNet is far more lightweight but still requires writing a proposal and getting through the selection process. Additionally NLNet requires to plan the work ahead so this requires some careful analysis on what would be delivered. STF is far younger and issues bigger grants than NLNet (which is capped to 50k EUR per mini-grant) but, at least in my opinion, is harder to get.
I hope it all gets better with time as more and more people understand the critical role open-source software plays in today's world.
Posted Jan 25, 2024 22:24 UTC (Thu)
by kleptog (subscriber, #1183)
[Link]
There's generally two ways a government can address a market failure: either try to fix it directly, or change the rules of the game so the market fixes it. The latter is generally better (this is what the CRA is attempting), but I honestly can't think of a way to setup the market rules such that companies would be interested in funding kernel documentation. So I guess we're stuck with prospective writers asking for funding. On the plus side, even under Europe's strict government subsidy rules, financing open-source projects is an fairly easy sell.
But a question which I couldn't find a quick answer for: LF funds many projects, but do they fund someone to work full time on kernel documentation? Could they? Would they? Or are there just so few people who want to do it?
Posted Jan 25, 2024 16:46 UTC (Thu)
by mricon (subscriber, #59252)
[Link] (1 responses)
Posted Jan 26, 2024 12:41 UTC (Fri)
by lisandropm (subscriber, #69317)
[Link]
Posted Jan 25, 2024 20:00 UTC (Thu)
by sdalley (subscriber, #18550)
[Link] (37 responses)
What an indictment.
I fondly remember, in the dim and distant past, shelves of beautiful manuals for the OpenVMS system hosting the cross-compiler toolchain used by our company at the time.
Posted Jan 25, 2024 21:53 UTC (Thu)
by ballombe (subscriber, #9523)
[Link] (4 responses)
<https://www.goodreads.com/series/143548-the-definitive-gu...>
I ended up writing my own window manager., because why not ?
Posted Jan 26, 2024 18:12 UTC (Fri)
by hubcapsc (subscriber, #98078)
[Link] (1 responses)
I wrote a program that would display the grayscale image of
My friend Gregg tried to compile and run it on his Sun workstation,
-Mike
Posted Jan 26, 2024 20:19 UTC (Fri)
by jem (subscriber, #24231)
[Link]
DECstation...? :-)
Posted Jan 28, 2024 0:25 UTC (Sun)
by jg (guest, #17537)
[Link]
I wrote much of the text for the library document, but he made it professional quality. Similarly for other parts of the system (protocol, toolkit, ICCCM), etc.
You never know if you don't ask what people will pay for.
The one part of the book that wasn't given away was the really high quality index that was generated. This is less an issue when online, but when using a book, was easily worth the price of admission...
Posted Jan 31, 2024 23:35 UTC (Wed)
by adam820 (subscriber, #101353)
[Link]
The company I was at previously did a HUGE "physical paper" cleanout, and there were so many manuals and stuff from this era. Books about UNIX, a _really_ nice set of these Xlib Programming manuals in a slip case that looked pristine like that hadn't really been used (probably left over from the Solaris days).
On the one hand, it kinda hurt me to pass up some of the books knowing they were headed for the dumpster, but also there's nothing I would do with them. But given their condition I should have rescued the X collection just for curio's/conversation's sake.
Posted Jan 25, 2024 22:06 UTC (Thu)
by Wol (subscriber, #4433)
[Link] (9 responses)
So much of the difficulty of learning all this new computing stuff comes from the LACK of manuals.
Cheers,
Posted Jan 25, 2024 23:07 UTC (Thu)
by b7j0c (guest, #27559)
[Link] (4 responses)
And so much of what we refer to as "tech debt" is simply stuff that is unknowable or very hard to know, a lot of which could be easily addressed with decent documentation.
Posted Jan 26, 2024 22:41 UTC (Fri)
by marcH (subscriber, #57642)
[Link] (3 responses)
I've heard used exclusively for short-term solutions hacks that save a little bit of time in the short term and cost a lot more time over years. Very much like the interest you have to pay for a loan, hence the name.
Posted Jan 29, 2024 22:03 UTC (Mon)
by gfernandes (subscriber, #119910)
[Link] (1 responses)
Posted Feb 10, 2024 22:47 UTC (Sat)
by marcH (subscriber, #57642)
[Link]
However I've don't remember the lack of documentation ever being named a "hack" and I've never seen it "laying the grounds" for more "hacks" either. So I still think it's a stretch. Everyone is of course free to use their own terminology; in fact we're constantly drowning under cryptonyms and other technical jargon. But the more "diverse" and fragmented terminology is and the less efficient is communication.
Wait... maybe we could call these language inconsistencies... technical debt? :-D Documentation is just a special case of communication after all.
Posted Feb 8, 2024 1:22 UTC (Thu)
by milesrout (subscriber, #126894)
[Link]
Posted Jan 26, 2024 19:58 UTC (Fri)
by laicos (guest, #169322)
[Link]
Posted Jan 29, 2024 15:23 UTC (Mon)
by bferrell (subscriber, #624)
[Link] (2 responses)
"Documentation?! If you're competent, what do you need THAT for?"
Posted Jan 30, 2024 17:30 UTC (Tue)
by jezuch (subscriber, #52988)
[Link] (1 responses)
This. Was. Hell.
Posted Feb 10, 2024 20:12 UTC (Sat)
by sammythesnake (guest, #17693)
[Link]
Really, what we need is for programming languages to work high enough level that "writing documentation" is a whole bunch less work because the programming language expresses not only what happens and how, but what's *supposed* to happen, and something of *why*. Unit tests/regression tests could also help a whole bunch on some of this by providing clarification of intended behaviour.
With the right expressiveness of programming languages and good unit/regression tests that refine the implicit "specification", clever automation could perhaps produce summaries from the code/tests - something vaguely akin to Javadoc etc. but *code* focused and able to summarise code (CodeGPT, maybe!)
With good languages/tools on this front, the amount of documentation that would need *manually* writing could be somewhat less and a much less daunting task - possibly less likely to be put off indefinitely(!)
Posted Jan 25, 2024 23:44 UTC (Thu)
by ejr (subscriber, #51652)
[Link] (21 responses)
Documentation is hard even initially. Maintaining documentation is nigh impossible.
Posted Jan 26, 2024 8:21 UTC (Fri)
by Wol (subscriber, #4433)
[Link] (20 responses)
I'm a jack-of-all-trades. Not a particularly good programmer, but then I can write decently. Not least, I appreciate documentation, therefore I try and write it.
But with something like FLOSS, which tends be written by lone programmers in basements (okay, I may exagerate a bit :-), who are the writers going to talk to, to learn what to write?
Plus, from what I've picked up elsewhere, the model that seems to work well is that programmers have to write some sort of documentation, and then the tech writers improve (edit, whatever) it. So much has been dumbed now - expensive workers are expected to be typists, etc etc, that the job of "technical writer" is seen as unnecessary, when in fact it's needed more than ever.
Cheers,
Posted Jan 26, 2024 12:01 UTC (Fri)
by fwiesweg (guest, #116364)
[Link] (13 responses)
I tend to politely disagree. It's admittedly an observable fact, but that does not mean it has to stay this way. Writing, like programming, is a learnable skill and they have more in common than one would generally believe.
They're both about organizing ideas, about finding the right words for those ideas, and being harsh on oneself with respect to certain rules: use the same name for the same thing, dont use names which you have not introduced yet, dont write overly long paragraphs (functions). There are tons more, just ask your local writer's school, but these I actually find to be of most use.
It's just that there is no compiler/linter enforcing those rules and that we as an industry do not reward those skills. As highly functional lazy people, we tend to then simply not put effort into it, so most programmers don't write well.
As a part of our employee integration, I force newcomers to apply some of these rules (by simply not merging anything which they cannot explain in exact terms), and they actually improve as time passes. Leadership can make quite a difference here.
Posted Jan 26, 2024 13:53 UTC (Fri)
by khim (subscriber, #9252)
[Link]
Sure. This depends on your goals, ultimately. There are only 24 hours in a day and 7 days in a week. And these may be spent on writing code or on writing documentation. Thus you may have good codes and awful documentation, or good documentation and awful code or so-so and so-so documentation. Or you may have great code, great documentation… only all these goodies would arrive late. Sometimes it's even right thing to do. Think Sure, but leadership couldn't squeeze more hours into a day or more days into a week.
Posted Jan 26, 2024 15:39 UTC (Fri)
by farnz (subscriber, #17727)
[Link]
My best experiences in terms of both codebase quality and also documentation quality have been in places that had dedicated tech writers who wrote English, instead of code. The workflow for all documentation was for developers to provide a quick overview, which the tech writers expanded into good documentation; however, if the tech writer was confused by your quick overview, they'd expect you to work with them to explain yourself.
As a consequence, you'd get feedback from the documentation process about the UX of the thing you were documenting; if a simple overview had the tech writers happy, then you'd get no request to explain yourself, but if you did something that was confusing, the tech writers would help you rearrange the API (or UI) into something that could be cleanly documented. For example, if I have a function that returns the number of bytes written, or -1 if the buffer supplied is too small, the tech writers would call me out on this and ask how the user is meant to find out how big the buffer should be.
The result was a better outcome via specialization; developers (my side) cared about the code while tech writers cared about the documentation, and we came together to make the documentation + code better in tandem.
Posted Jan 26, 2024 15:40 UTC (Fri)
by JohnMoon (subscriber, #157189)
[Link]
You may be interested in Vale which I learned about recently: https://vale.sh/. It's sort of a linter for prose that can check for some of the things you mention.
Posted Feb 22, 2024 5:46 UTC (Thu)
by fest3er (guest, #60379)
[Link] (9 responses)
"... Writing, like programming, is a learnable skill and they have more in common than one would generally believe. ..."
Writing is the *same* as programming. When one writes good software, she programs the system to execute it properly. When one writes good documentation, he programs the readers' neural nets to think correctly. It's all programming.
When coders, programmers, developers and engineers (and that entire industry, really) come around to this way of thinking, they will be able to write draft documentation that human communication experts (i.e., tech writers) can transform into manuals that the target audiences can read and easily understand, because those manuals will have programmed their neural networks to think correctly.
QEMU is nice, but their documentation lags by 5-15 years. Systemd may be nice, but the documentation is all but non-existent. Netfilter do try, but their documentation, somewhat more than occasionally, is lacking, and not because English is not their primary language. IProute2's HTB was poorly understood by those who valiantly tried to document it; once I dropped most of the fancy features and made it a single-level (non-hierarchy) with four priority levels (isochronous, high, normal, low; iso and high having limited bit rates), it enabled conns in all four priorities to transit data very efficiently. Many more examples can be found.
Often, the culprit is Creeping Featuritis. FLOSS coders, programmers, devs and engineers are often guilty of continually adding features yet never go back to document what they designed or implemented, and even more rarely go back to review and fix what they wrote. Often, they seem to be interested only in what they want; what others want is immaterial.
In my 47 years of experience in the computing industry, I think DEC's end-user user guides and reference manuals were, by far, the best ever produced. (And, IIRC, GTE Government Systems' printing department performed a good bit of DEC's typesetting using their 'photographic' Linotype system.) Interleaf (UNIX desktop publishing) also produced some very good documentation.
Back when I worked for Motorola, I often told co-workers that the three most important things engineers must do every day are to communicate, communicate, and communicate. If we do that, the rest will fall into place.
I believe we, the FLOSS community, can do better all around if we but try.
Posted Feb 22, 2024 10:12 UTC (Thu)
by farnz (subscriber, #17727)
[Link] (4 responses)
I've worked with ex-DEC technical writers, and the key insight into how DEC did such great manuals is that the technical writers were part of the process from design onwards; they did not document things after they were written, but instead were working on documentation in parallel to the code being written.
And a consequence of this is that there was a feedback path from the documentation into every stage of creating something at DEC, because the technical writers were part of the team working on any feature, and did spend time working with the developers to fix features that had evolved into a mess that wasn't possible to document nicely. This, in turn, improved the actual features - if you can't document it cleanly, then your feature is almost certainly a mess in real use, too.
Posted Feb 22, 2024 12:23 UTC (Thu)
by Wol (subscriber, #4433)
[Link]
When designing something, always think about ALL the possibilities, before fixing your problem. Let's say you have three yes/no choices, that's an 8-way truth table. If even ONE of those eight doesn't make sense, then your solution is broken! If you're only interested in no 4, just fix no 4. But if no 7 doesn't make sense, you've just prevented the next person from fixing that one, and possibly several others, *properly*.
So your no 4 fix is, of logical necessity, a dubious bodge. That's playing dirty :-(
Cheers,
Posted Feb 23, 2024 11:50 UTC (Fri)
by maniax (subscriber, #4509)
[Link] (2 responses)
Posted Feb 25, 2024 20:50 UTC (Sun)
by zaitseff (subscriber, #851)
[Link]
Posted Feb 26, 2024 10:56 UTC (Mon)
by farnz (subscriber, #17727)
[Link]
Maybe Guide to Creating Modular Procedures on VAX/VMS or VAX / VMS Install Utility Reference Manual, depending on what your team does.
Posted Feb 22, 2024 13:44 UTC (Thu)
by pizza (subscriber, #46)
[Link] (2 responses)
Seriously? If systemd's extensive, voluminous documentation [1] is "all but non-existent" I'd love to see what standard you are comparing it to.
[1] Both in the "reference manual" and "user guide" sense, along with many "design rationale" essays. And that's just the first-party stuff.
Posted Feb 22, 2024 20:44 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Ugh. The problem is that systemd's documentation is unstructured. It's really hard to discover features or to understand the full lifecycle of a service.
For example, I only recently discovered that systemd can keep file descriptor store for the application, so it can use them to persist data between restarts.
Posted Feb 23, 2024 8:42 UTC (Fri)
by atnot (subscriber, #124910)
[Link]
Posted Feb 22, 2024 17:23 UTC (Thu)
by kleptog (subscriber, #1183)
[Link]
It does raise the question of what the target is of documentation these days. Are we expecting people to actually read all the docs, or do they just want their questions answered? For the latter case I see a promising future for LLMs. The other style documentation I use often are walk-throughs that go step-by-step to set up some environment, because that often involves lots of related tools working together, something you'll not find out by reading the docs of the individual tools.
At $DAYJOB we're experimenting with LLMs to write documentation. Not that it produces perfect results, but it gets you 90% of the way and maintains a consistent style. And people appear to be happier tweaking the results compared to writing from scratch. As so often with coding, a lot of documentation is just obvious boilerplate in a slightly different format, so letting the computer generate that and letting the expert fill-in the non-obvious looks to be a great time-saver (and we actually get docs, instead of nothing which what we have now).
Posted Jan 26, 2024 14:57 UTC (Fri)
by corbet (editor, #1)
[Link] (5 responses)
Posted Jan 26, 2024 16:34 UTC (Fri)
by anselm (subscriber, #2796)
[Link] (1 responses)
Posted Jan 26, 2024 22:45 UTC (Fri)
by marcH (subscriber, #57642)
[Link]
It gives you the required distance.
Posted Jan 26, 2024 18:33 UTC (Fri)
by ejr (subscriber, #51652)
[Link]
Posted Jan 26, 2024 23:40 UTC (Fri)
by atnot (subscriber, #124910)
[Link] (1 responses)
I would personally love to spend a lot of my time doing technical writing, but not enough to justify investing the time to get good enough at it for something that I will largely not be taken seriously for. I know that if I showed up to my next job with a bunch of writing on my resume, I would immediately get taken less seriously as an engineer and just get asked to take notes in meetings and write company blog posts all day even far beyond the baseline women levels. It's just not an attractive prospect to specialize in in this environment and everyone is worse off for it.
[1] It needs to be pointed out that these are all currently primarily feminine-coded, unlike programming.
Posted Jan 26, 2024 23:51 UTC (Fri)
by atnot (subscriber, #124910)
[Link]
Posted Jan 26, 2024 0:20 UTC (Fri)
by geofft (subscriber, #59789)
[Link] (17 responses)
What should we do, and how do I, as someone with some amount of influence but no actual spending authority, get us to do it?
The Linux kernel contribution maturity model (https://docs.kernel.org/process/contribution-maturity-mod...) does not seem particularly relevant to us. We're not a hardware vendor or otherwise in the business of writing kernel code. We're at level one - we can and do contribute patches to the Linux kernel, in practice every couple of years. Level two suddenly talks about people being expected to contribute as part of their job and having my contributions assessed in performance reviews. That's a drastic step up! I cannot figure out what an employee who has sent in the occasional patch (like me) would possibly do to work on the Linux kernel with even 10% of my time year-round. I'm honestly not sure LKML would really want me to dive in and say "Look guys, my boss told me to contribute, so I'm just going to keep refactoring this subsystem." And the people who do my performance reviews aren't really in a spot to evaluate my contributions beyond whether they happened or not.
Besides, this article is mostly about specific existing projects and specific existing contributors talking about whether they can do them, not the need for new blood in general. (There are other articles about that.)
So, okay, let's not think about contributing more code ourselves. Greg KH points out that there isn't enough testing because companies aren't funding the work. We'd love more testing, it would make us more confident in upgrades. Where do we send a check? (Again, I have contributed to the kernel a little bit, and I have no idea what I would be asking my company to do.) The Linux Foundation gets a good bit of corporate money, as I understand ($177M from their last form 990). Does that money go to this sort of work? If we send LF a small amount more money, would it fund this work?
More importantly, how do I convince my company that this is a good use of our money? I doubt I can convince my employer to sponsor even one full-time engineer at the wages we pay our full-time engineers; we have enough problems of our own to solve, and as important as kernel testing is, it's not as important as anything else we could put one full-time engineer on. (We run at a scale large enough to see kernel regressions that weren't caught by anyone else, so it would concretely help us, but we don't run at a scale large enough that it's regular: it happens about once every few years.) We certainly benefit from Linux in general, but we're not talking about paying to make sure we can keep using Linux at all, we're talking about making the development process a little faster and a little more reliable. But the argument in the article about tooling paying off compounding rewards is an interesting one. Is there a compelling argument that it really is a good use of our money to fund the kernel by an entire engineer's salary?
Or let's say that we want to contribute a fraction of an engineer's salary. How do I convince the people who sign checks that this money is going to go somewhere useful? It won't make any of the existing contributors have more hours in the day (or would it? is part of their income coming from Linux-irrelevant private consulting work that they could stop doing?). I suppose we and many other companies like us could collectively fund one engineer, but how do we get that to happen?
(Is this a question for us the corporate users to answer? Like should we be proactively reaching out to other companies in our industry and saying, hey, all of us get value from Linux being good, let's collectively fund someone?)
Or maybe funding development time isn't the right thing to focus on. I see in the thread some talk about KernelCI. Would contributing money to KernelCI to buy more machines or more public cloud time help? If I go to the KernelCI home page, there's a "contributions welcome!" button ... that links to GitHub. A little bit below, it says, "Learn how your organization can contribute to the project now," but the way to learn is to send someone an email. What does it mean to join? And then there are logos of a couple of companies below it, at least a few of which are much larger than we are and care much more than we do about Linux development. Are they contributing so much that we'd be a drop in the bucket? I have no idea. Nothing on the website answers the question about how much money KernelCI is spending and whether they could productively spend more (but it does have some brand guidelines, great).
Or maybe I'm totally off base here, and when Greg KH and others are talking about "companies," they are actually talking about gold/platinum members of the Linux Foundation who are already contributing a lot but could still justifiably employ several more individuals full time on upstream kernel work, and non-huge companies like mine are just not part of the conversation, and articles like these aren't intended for people like me to try to get our employers to contribute more. So maybe a simpler question is - in an ideal world, where everyone in my employer's management was completely and accurately convinced of the need to fund Linux kernel development, would that accurate funding really be more than the almost-zero it is now? (One argument in this direction is we pay quite a bit of money to our hardware and cloud vendors, and a feature of their product is compatibility with upstream Linux, so we're already doing the thing that ought to fund kernel development. In that case, perhaps what I really should do is tell our account reps at those companies - all of which are already gold/platinum members of the LF with full-time kernel developers on staff - that we want to see them funding this work even more.)
Posted Jan 26, 2024 5:25 UTC (Fri)
by NYKevin (subscriber, #129325)
[Link] (2 responses)
At the same time, I must say that I find it rather disturbing that you were unable to get useful answers to your questions about certain specific areas of contribution. It is all well and good to complain that nobody wants to pay for certain things, but you can't put this sort of information at the bottom of a locked filing cabinet in a disused lavatory etc. etc., and then complain that nobody managed to find it and contribute.
Posted Jan 26, 2024 8:06 UTC (Fri)
by lunaryorn (subscriber, #111088)
[Link] (1 responses)
Posted Jan 27, 2024 1:05 UTC (Sat)
by NYKevin (subscriber, #129325)
[Link]
Posted Jan 26, 2024 7:46 UTC (Fri)
by pbonzini (subscriber, #60935)
[Link] (2 responses)
Posted Jan 26, 2024 8:26 UTC (Fri)
by Wol (subscriber, #4433)
[Link] (1 responses)
If your setup is at all out-of-the-ordinary, see if you can set up your own CI test-bot, and hammer it (as part of the general kernel testing system), and then you can sell it as buying you extra confidence when you upgrade, you know your own setup has been put through the mill on your own test-bot.
Cheers,
Posted Jan 26, 2024 12:12 UTC (Fri)
by spacefrogg (subscriber, #119608)
[Link]
For many projects it would also be helpful to:
People are motivated to build stuff that helps them(!), the aforementioned topics are chores that do not benefit the original authors and thus need monetary persuasion. This is where your company has leverage where private people have not.
Free Software is a common good like streets, utilities or parks and nature reserves. Your company, as any other company, usually does not directly invest into common good but rather pays taxes. Putting money into free software is charity, not investment (aside from rare cases in highly specialised fields). Thus, you should never try to sell it as that. So, as for convincing the bean counters? You won't. Either it is in your company's DNA to give back to common good or it is not, and it won't happen. There will be no outcome that can positively reflect in an Excel sheet. Even in those rare cases, the outcome will not be measurable and always under scrutiny of being scrubbed for short-term benefit. The will to pay has to come from outside of money making or it won't last. You might sell it as good marketing, or your boss being the cooler kid on the golf course etc. These are all human factors and not numbers.
Posted Jan 26, 2024 11:11 UTC (Fri)
by dvrabel (subscriber, #9500)
[Link]
Posted Jan 26, 2024 20:34 UTC (Fri)
by mfuzzey (subscriber, #57966)
[Link]
But there are lots of other, often critical, OSS projects that have very little funding - maybe it would make more sense to contribute to those rather than the kernel.
Posted Jan 27, 2024 11:06 UTC (Sat)
by amacater (subscriber, #790)
[Link]
If you have some in-house tooling/integration that works for you - _really_ document it well. Explain it - you'll benefit when those maintaining it move to another project. If it's specific to you, but the general knowledge you used to build it was hard-won because you had to look in 20 places - write up the experience and list the 20 places so the next person doesn't have to do that. Use that documentation experience to help upstream documentation / write technical blog posts / summarise where to find the vital information and share the 20 places. Write the missing manuals from bitter experience ...
if all else fails, Debian could do with clueful experts lurking on mailing lists and contributing real experience where it's meaningful :) Teach good practice in real-world sysadmin and security or contribute constructively to debate - but contribute.
Posted Jan 27, 2024 17:30 UTC (Sat)
by ceplm (subscriber, #41334)
[Link] (5 responses)
Posted Jan 27, 2024 18:05 UTC (Sat)
by geofft (subscriber, #59789)
[Link] (4 responses)
Also, I would need to successfully convince my company to spend money on it. I can pretty easily get the people who manage the donations and conference sponsorship budget to give, say, $10,000 to a relevant open-source nonprofit. I absolutely cannot get the purchasing team to buy a $10,000 product we won't use.
And if I tell them the motivation is "Well, the company that sells this product is going through layoffs, but maybe they'll have fewer layoffs if we support them, which might cause them to resume spending resources on the infrastructure behind the shared commons behind the product we won't use, because we also use that shared commons, and by the way their competitive differentiation is things that aren't part of that same shared commons, because the shared commons is free and we've been using it for free for twenty years," I would probably not change their minds.
I do actually think these companies would be well placed to sell a "sponsor 25% of an upstream developer" product. We would get something measurable about what our money is accomplishing. They'd be in a good spot to put that money to use, because they could reallocate an existing employee's time or use their reputation and market position to hire someone new and find three other customers. But they don't seem interested in selling that.
Posted Jan 27, 2024 18:30 UTC (Sat)
by ceplm (subscriber, #41334)
[Link] (1 responses)
We may not pay for the things (I work for SUSE, but of course I have zero influence on the corporate decisions of the company), but we spend zillion of money on upstream development. Yes, you may not be able to decide what exact part of the Linux universe you support, but comparing with supporting somebody who says they support Debian, it is the easiest and most secure way how to do it (OK, I see https://www.debian.org/donations, but I guess that goes completely on the Debian infrastructure, right? So, nothing goes upstream … that’s absolutely nothing against them, but if you want to support upstream development, this may not be the right channel).
Of course, I don’t know if you want to target your donations than you have to work your way through various pages like https://www.linuxfoundation.org/about/join, https://sfconservancy.org/, or many other organizations like that.
Posted Feb 1, 2024 13:36 UTC (Thu)
by rhertzog (subscriber, #4671)
[Link]
Indeed money donated to Debian is not used to pay for the time of Debian developers/contributors. It might be used for hardware, hosting, travel reimbursement, debconf organization, and that kind of activities.
<shameless plug>
For now, it's the margin on the various services that make this possible but we are considering to add some broader offer that would also encompass the "please take my money to contribute to Debian and make it a bit more sustainable" use-case.
I'd be happy to discuss with companies that are looking to contribute to Debian in such a way to help refine our future offer.
Posted Jan 28, 2024 10:07 UTC (Sun)
by jengelh (guest, #33263)
[Link] (1 responses)
Now, if RH/SUSE systems were running "popcon", corporate would at least have an idea that they should be spending so-and-so-many percent for the desktop.
Posted Jan 28, 2024 11:03 UTC (Sun)
by rahulsundaram (subscriber, #21946)
[Link]
They have clear stats on which packages are being used because a commercial subscription already provides that data. I would bet they are already using this to drive decisions,
Posted Jan 28, 2024 3:53 UTC (Sun)
by jmalcolm (subscriber, #8876)
[Link]
In my view, a good Open Source citizen helps promote what they use, files bug reports when they should, contributes what they can, and at least partially funds the "change" they need to see in the code ( either through contribution or cash ). If we stick to the original "Free Software" vision, just using the software is contributing to the change some people want to see in the world.
For smaller projects, the core devs may not be making a living. Those kinds of projects would most directly benefit from some cash infusion if you have the means. For projects that have already crossed a certain threshold though, I see no reason to feel bad about using software as-is without funding its development. I mean, if you are getting a bunch of direct commercial benefit from a specific project, it would be nice to share the wealth. I am not trying to dissuade anybody from funding Open Source by any means.
I guess what I am saying, as an example, is that most of us should not feel bad for using the Linux kernel "for free". My use of Linux these days places no demands on the kernel as a project. Nobody is having to spend any dev time directly on my behalf. If there were some deficiency I needed to see addressed, I would think about how to contribute to that. As you say though, for most of our use cases, the kernel is simply excellent these days already. Yes, I benefit from it getting even better but I am not the "primary" beneficiary. I am simply along for the ride and, in the case of Linux, I am a "free rider" but I am a "zero cost" one. Unlike say roads, I am not wearing the Linux kernel out by using it. I do not consume any of its resources. In fact, as above, my use of Linux "for free" can be thought of as a contribution of sorts.
Concentrate on the projects where you can and do add value. Don't worry about using Open Source for free. Just be thankful.
Although I guess I will make one request. Please do not turn around and insist that the fine people that do the work, that author all this amazing software for us, bow down to "your rights" when they do try to get some value for what they do. We have had some "community" back-lash moments over the last year that, in my view, show a massive lack of respect and appreciation for the people that write the code.
Enjoy what you use. Fund or contribute where you can but don't sweat it when you don't. At the very least, refrain from attacking the people doing the work.
Posted Jan 30, 2024 13:19 UTC (Tue)
by metan (subscriber, #74107)
[Link]
As for myself I've been maintaining the Linux Test Project for more than a decade and what we really need is contributors and maintainers who can help with the patch review. However I have no idea how to make this happen in practice, there is a more than handful of companies that use the testsuite for testing, but only a few allocate resources to contribute back or even invest into a developer becoming a maintainer.
Posted Jan 26, 2024 9:58 UTC (Fri)
by meven-collabora (subscriber, #168883)
[Link] (2 responses)
Thunderbird project has recently made big strides in that matter, receiving millions in donation from users. KDE community has launched a fundraiser recently achieving its goal.
Perhaps we can have a similar "small donation" or "recurrent donation" model from many smaller parties/companies rather than just focusing on big companies, a crowdsourcing kind of financing. An instance of this idea is Github sponsors.
Posted Jan 26, 2024 11:58 UTC (Fri)
by luna (guest, #166424)
[Link] (1 responses)
Posted Jan 26, 2024 12:21 UTC (Fri)
by bluca (subscriber, #118303)
[Link]
Posted Jan 26, 2024 10:34 UTC (Fri)
by rwky (subscriber, #119998)
[Link]
Posted Jan 26, 2024 23:16 UTC (Fri)
by marcH (subscriber, #57642)
[Link] (2 responses)
Market forces can do wonders but _quality_ is another example where they simply don't work. Computing products are incredibly complex, so customers simply cannot gauge their quality - or not before their system has crashed or been hacked and they lost everything. You need regulations to enforce a minimum level - period. Bruce Schneier's blog has explained this very well for security which is "just" a particular aspect of quality.
If all corporations had no choice but to raise the bar, then they would all need a better testing infrastructure. Then some companies would certainly prefer to cooperate rather than reinvent that wheel too. We're not really there yet but I'm cautiously optimistic.
This is the same with any other complex products like... transportation. Even extreme libertarians hopefully agree that it's better to pay some taxes and have some proper, independent government agency inspecting how planes are designed and manufactured rather than waiting for them to go down[*] and let the market decide which brand is best. This especially doesn't work when you have only two manufacturers left and the backlog of the other one is 10 years long - because very companies can make complex products and they just keep buying their competitors.
In fact a totally free market does not even work with... food. Because it's not a simple product either.
[*] or doors to blow in mid air, or "many" bolts to go loose, or wheels to detach on the runway,... "Downfall: The Case Against B." on Netflix + any news channel.
Posted Jan 27, 2024 9:39 UTC (Sat)
by Wol (subscriber, #4433)
[Link] (1 responses)
They DO work, but only in the absence of a dominant vendor, and in the presence of strong trademark-style enforcement.
Given that a "dominant vendor" only needs about 20% market share, that's not that big a barrier in an established market.
Cheers,
Posted Jan 27, 2024 17:30 UTC (Sat)
by rra (subscriber, #99804)
[Link]
More fundamentally, they only work when (a) customers have a choice (as you point out), (b) customers are quality-sensitive, and (c) customers are able to judge quality before purchasing from a vendor. Those sound like easy criteria to meet, but in practice they usually are not. They work best with recurring purchases of low-cost items from a large market where people can easily switch vendors.
A classic example of market failure around quality is stoves, from Don Norman's The Design of Everyday Things. Many stoves are purchased by developers, managers of multifamily dwellings, or other entities that will not be using the stoves to cook, not by the people who will be cooking on them. Those purchasers are price-sensitive but not particularly quality-sensitive, except on the high end where they're intentionally trying to construct a very expensive living environment. The stove will probably not make the difference in whether someone buys a new house, and even when renting an apartment it will likely only make a difference if it's obviously horrible. So they buy cheap stoves that look okay.
Another classic example is air travel. Everyone complains about low quality of the flying experience, and yet we've seen time and time again from market studies that people who purchase airline tickets are incredibly price-sensitive and almost entirely insensitive to quality. So poor quality experiences persist, because it's simply not a common purchase criteria, even though everyone complains.
Free software is a relatively quality-sensitive market because the cost is uniform (free), so the primary criteria that people use to pick free software is quality, although it still has the problem that it's often difficult to judge quality before "purchasing." But it's also a weird market because a lot of people choosing your free software usually drains you of resources rather than provides you with more resources, since all your "customers" don't pay but ask for support. If a piece of free software becomes somewhat popular but not stunningly popular, the quality often DROPS because the maintainer becomes overwhelmed but cannot convert the interest into monetary resources or time.
Commercial software suffers from multiple quality market failures: not purchased by the people who will be using the software, price prioritized over quality, lack of choice in many fields, and inability to judge quality before purchasing.
Posted Jan 28, 2024 10:04 UTC (Sun)
by nim-nim (subscriber, #34454)
[Link] (1 responses)
The basic problem is that engineers are expensive, very few entities can or want to contribute full engineering time, and when you try to allocate money not engineering time most of this money will be eaten away by commercial structures with little to show as a practical result.
Therefore the main purpose of the Linux foundation should be to collect money and allocate it wisely with as little structure loss as possible. Something that all successful charities have managed to do.
Of course that is a *very* different proposition than playing at the software editor with unicorn-level executive salaries (I’m also looking at you, Mozilla).
Posted Jan 31, 2024 11:59 UTC (Wed)
by paulj (subscriber, #341)
[Link]
I know of US 501.3cs, ostensibly setup to fund maintenance of open-source projects, where - when the public declarations came out X years later - it turned out that 80% of the donated funds had gone to the FTE salary of the FT manager/cheerleader/fundraiser executive, and 20% to the actual engineer funded to do the open-source maintenance.
The executive was getting a decent salary, even for where they lived, while the engineer had worked for ~minimum wage (in their country) because it was their understanding the foundation hadn't yet gotten funding.
Posted Jan 31, 2024 8:43 UTC (Wed)
by vegard (subscriber, #52330)
[Link]
In other words, it's not companies' fault that their employees choose to spend their "20% time"-equivalent on those "things nobody wants to pay for" -- it would arguably be worse for individual employees if the employer mandated that they work on (say) documentation over letting them have the choice.
I think the question we need to ask instead is, why do people choose to work on code rather than documentation?
The answer -- my answer -- is a mixture of things: Most people are in kernel development in the first place because of the code, not because of the documentation. It was the code that they ran, that they looked at, that drew them in -- not the documentation. Writing code is fun, getting something new to work is exciting, writing new features will designate you as an expert on an area of the code, maybe even get you some (community) news coverage. Useful code ultimately makes somebody money, making developers and maintainers of said code valuable to companies. Code may get your next job (or at least a bonus); documentation, probably not so much.
I believe I've said similar things about code review before: Code review is simply less glamorous than writing said code. If you had the choice -- write new code or review somebody else's code -- one of them has a very obvious and short-term personal payoff, the other one has long-term payoffs for the community at large.
We need to find better ways to incentivize each of these "things that nobody wants to pay for".
Posted Feb 2, 2024 11:39 UTC (Fri)
by smitty_one_each (subscriber, #28989)
[Link]
Something akin to an LWN with a nominal subscription rate (maybe it's a posh distro?) that explicitly includes a kickstarter percentage to go toward defraying infrastructure costs?
A somewhat related request would be for some better onboarding resources to get people started in kernel development. Here I'm lookind for a "DuoLingo for the kernel". I have no idea how that would even work. Maybe an extension of the mighty linuxfromscratch.org ?
The gap is between insert_kid_here and motivated_developer. How to bridge? I have a 12yo son who would benefit from tech mentoring. Is there even a coding summer camp worth attending?
Posted Feb 5, 2024 4:52 UTC (Mon)
by apollock (guest, #14629)
[Link] (3 responses)
For example, if there's no technical writers for the Linux Kernel, is this because (for example) the Linux Foundation has elected not to fund any, or no technical writers are available (i.e. they're a scarce commodity in the first place)?
Posted Feb 5, 2024 12:13 UTC (Mon)
by mathstuf (subscriber, #69389)
[Link]
Posted Feb 5, 2024 15:21 UTC (Mon)
by sdalley (subscriber, #18550)
[Link] (1 responses)
And being dismayed, a few months later, to read in LWN that this same developer was being "let go" (what a charming expression) again because of financial constraints something something.
Maybe somebody can recall the specifics.
Posted Feb 14, 2024 23:11 UTC (Wed)
by plougher (guest, #21620)
[Link]
You're probably remembering Rob Landley, to quote from his online resume/CV (https://www.landley.net/resume.html).
"2007 Position: Fellowship from the Linux Foundation
Project: Linux Kernel Documentation. Six month fellowship to create and maintain a central kernel documentation website, improve and maintain existing documentation and related tools, and write new documentation. Started "http://kernel.org/doc" directory (see source control repository at "http://landley.net/hg/kdocs", and May-Oct 2007 in the mailing list archive at "http://marc.info/?l=linux-doc")."
Also see https://www.linux.com/news/ols-kernel-documentation-and-s...
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
The thing nobody wants to pay for - documentation
> but none pays for a single technical writer to create documentation.
The thing nobody wants to pay for - documentation
Xlib Programming Manual for Version 11
The thing nobody wants to pay for - documentation
DEC equipment, I forget what they called their workstations...
myself I'd gotten as a novelty at a usenix thing I'd recently been
to.
but it wouldn't go, so he ribbed me about my code not being
portable...
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
>
> <https://www.goodreads.com/series/143548-the-definitive-gu...>
The thing nobody wants to pay for - documentation
Wol
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
Wol
The thing nobody wants to pay for - documentation
> Writing, like programming, is a learnable skill and they have more in common than one would generally believe.
The thing nobody wants to pay for - documentation
TeX and METAFONT. But most of the time that combo arrives too late to make a difference.Tech writing as a separate skill
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
- User guide: "This is generally how you use the program, with lucid examples."
- Reference manual: "This is everything you can do with this program with examples and limits of
every option and switch and how they interact."
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
Wol
The thing nobody wants to pay for - documentation
DEC documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
The thing nobody wants to pay for - documentation
I have to disagree with that statement; there are programmers out there who are exceptional writers as well. As a rule, though, people who can do both tend to prefer programming; that's part of why we have such a hard time finding the kind of writers we need.
Writers and programmers
“Besides a mathematical inclination, an exceptionally good mastery of one's native tongue is the most vital asset of a competent programmer.” Writers and programmers
– Edsger W. Dijkstra
Writers and programmers
Writers and programmers
Writers and programmers
Writers and programmers
The things nobody wants to pay for
The things nobody wants to pay for
Moving someone else's money (eg a donation or recurring fund from a company) around on behalf of someone else (like an open source project which isn't even a legal entity on its own in most cases) comes with so many legal, taxation, billing, accounting, reporting, etc. issues that it's probably not surprising that many open source projects have no infrastructure and thus no documentation around this.
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
Wol
The things nobody wants to pay for
- diversify their build/test infrastructure and
- do the boring jobs (writing, maintaining documentation, translations, platform-independence, packaging)
The things nobody wants to pay for
The things nobody wants to pay for
I think the kernel is one of the better off OSS projects that already has quite a lot of funded developers.
The things nobody wants to pay for
Check that it actually *has* a maintainer and that it's not being maintained by the QA team in Debian because there isn't an actual maintainer somewhere. Allow them some time to look at bug reports for that utility, check whether they apply, submit fixes (or suggest closure because "can't reproduce here") If your firm can help reduce backlogs on fixes by contributing a fix/patch - do so.
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
With Freexian we try to help fill that gap. Thanks to the various LTS offers, we are paying Debian developers to work on Debian. This is part of our mission and we are regularly reporting our activities:
https://www.freexian.com/about/
https://www.freexian.com/about/debian-contributions/
</shameless plug>
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
Other more niche application already spearheaded this model, krita, inkscape, obs or kdenlive.
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for: quality
The things nobody wants to pay for: quality
Wol
The things nobody wants to pay for: quality
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
The things nobody wants to pay for
The kernel once had a paid documenter
The kernel once had a paid documenter