Creating a healthy kernel subsystem community

Creating welcoming communities within open-source projects is a recurring topic at conferences; those projects rely on contributions from others, so making them welcome is important. The kernel has, rather infamously over the years, been an oft-cited example of an unwelcoming project, though there have been (and are) multiple efforts to change that with varying degrees of success. Hans de Goede talked about such efforts within his corner of the kernel project in a talk (YouTube video) at Open Source Summit Europe.

De Goede introduced himself as a Red Hat employee working mostly on Linux hardware enablement, "so mostly driver-related stuff"; since the talk, he has announced that he is leaving the company after 17 years. Recently, he has been focused on laptops and MIPI cameras. He has been a maintainer for the platform-drivers-x86 (pdx86) kernel subsystem since 2020.

Friendly mailing list

He started with some suggestions on making the mailing list a "friendly and welcoming place". Kernel mailing lists have a bad reputation, where "sometimes mails can be rude or unfriendly; that has been changing lately, changing for the better". But the reputation remains and it is often seen as scary to post something to the list.

As a kernel maintainer, he wants to lead by example, so he always tries to stay professional and friendly with his responses to mailing-list posts. A question may seem stupid, but the person likely just does not have the background and knowledge that De Goede does; he tries to put himself in others' shoes and suggested that responders remember back to when they were posting their first patch. Posting a grumbling response because you are busy and stressed does not help anything; patience is a virtue, "take a deep breath, count to ten, be patient". Both the contributor and the maintainer are trying to improve the kernel, so a maintainer should assume good intentions—even after having to explain something multiple times.

The language barrier can be a major source of problems. English is not difficult for him, as the Netherlands does not dub TV and movies so he has been exposed to it all his life; "English is pretty close to Dutch, actually, in a way". He has, however, noticed many other participants struggling with English, so he has come up with some rules that he uses to help overcome that barrier.

First off, De Goede tries to write short, clear sentences; he has noticed that engineers often write long sentences, with multiple sub-sentences for clarification, which should be avoided in his experience. Another mistake is to have a "wall of text" rather than lots of clear, concise paragraphs; for further clarification, often a footnote can be used. Maintainers will sometimes need to ask for multiple changes to a patch, or for several different kinds of debugging information, so making that really clear, using a numbered list that has white space between each item, can help. "These are just really simple tricks, but they help to overcome the language barrier", he said.

When a maintainer is having a bad day for some reason, it is especially important that they try to follow his suggestions. Sometimes, though, miscommunication will happen even when maintainers are patient, friendly, and trying to communicate clearly. Language barriers can play a role, but email is also a less-than-perfect communication mechanism; being unable to see the other person in order to judge their attitude and emotion makes it prone to miscommunication. Because of the inherent delays in responses, it can take time to even recognize that a miscommunication has happened; it is often the case that a message initially perceived as disagreeing actually turns out to be agreeing—or at least not disagreeing as strongly as it first appeared.

He has come up with a four-step process for dealing with miscommunication, starting with identifying that one has occurred in a response. Clearly laying out the original interpretation of the other person's message, followed by a description of the new understanding of the person's meaning, are the next two steps. Finally, suggest how to move forward with the discussion and/or patches. It is important not to get caught up in questions of blame—was the original message unclear or was it just read poorly—because it doesn't matter. "Miscommunication is just something which happens, it happens all the time"; it is easy to diagnose and fix when talking face-to-face, email makes miscommunication take longer to unwind.

An audience member asked about repetitive replies: when does it make sense to turn those replies into documentation and is sending a link to the documentation considered a friendly reply? De Goede said that he was bad at turning replies into documentation but that he did have some canned answers that he can copy-and-paste into a reply along with a note that it is from a template. There is already so much documentation on submitting kernel patches that it is daunting; he thought that maybe it should be split into "Submitting patches 101" and a "201 with all the details".

Another question concerned finding the time to be able to be friendly and help newcomers. De Goede said that he was lucky to be paid for being a subsystem maintainer, "so I could use some of my boss's time for it". In addition, the subsystem he maintains is not that large; the number of postings to the mailing list, especially in the early going, was small.

Unfortunately, helping newcomers may not produce a payback, so it is "a bit of a scattershot approach". He had the good fortune to have two community members who he was able to turn into reviewers for parts of the subsystem. He noted that they were working in specific areas and asked them directly to help out; "it was a combination of being open and friendly" coupled with directly seeking out their aid when he was too busy.

The question played into several other topics that he planned to talk about, De Goede said: community growth and maintainer burnout. Obviously, spending time to grow the community can help relieve some of the burden on the existing maintainers, but it is not guaranteed, so it is a tricky balancing act. It would be nice if all maintainers could be paid for their work, but that is not the case; he was able to work on his subsystem around one day a week, which was generally enough time to deal with the most pressing things.

Unpaid maintainers may not feel like they have sufficient time to bring up newcomers, which is understandable, but may well result in burnout down the line. He suggested managing expectations with newcomers to give them a realistic idea of what level of assistance the maintainer can provide—and when. If the maintainer is simply too swamped, it may make sense to encourage a newcomer to seek out another mentor to help with their patch.

Growth

A welcoming mailing list is only one of the things needed to grow a community, he said. Newcomers' patches should probably not be overly nitpicked about minor stylistic problems (e.g. a forgotten comma at the end of the last structure initializer); the maintainer should just do some small cleanups if needed and merge the result. For seasoned developers, nitpicking may make sense, but for newcomers, the goal should be to give them "that little boost of 'yes I got my first patch upstream'; you want to hook them". Pointing out the changes needed in the merged patch may help the newcomer learn without having to go through multiple revisions of the patch for stylistic issues.

As noted, the time spent hand-holding newcomers may not pay itself back, but the investment needs to be made to add people to the project. There are not enough maintainers—who are paid for that work—to keep up with all of the newcomer mentoring that is needed. That message needs to be spread more widely, he said. He referred to the curl keynote from earlier in the day by noting that none of the car companies using the tool are contributing back to it. One way for big companies, some of which are making billions relying on open source, to give back to the community, would be to hire maintainers, he said.

Appreciating contributions and contributors is an important part of being a maintainer, both for existing community members and, especially, for newcomers. He tries to always start his patch review emails with "thank you for your patch"; it is a simple thing to do and is "so underrated" in making a difference. He also thanks people who review his patches and, importantly, thanks those he sees reviewing other people's patches.

Another way to help grow the community is to delegate tasks and to ask for help. Noting community members who are reviewing patches in a particular area, then asking them directly for help on a review or a bug, can work well to get more assistance. It does not always work, of course, since other people can also be too busy, but it is worth trying.

Two questions were asked about the level of nitpicking De Goede was talking about. He clarified that if a patch has actual problems in the logic or implementation, he will ask for changes, but that simple typo kinds of fixes are best just dealt with directly. He knows that other maintainers differ, but in his opinion it saves the maintainer time to not have to context switch and go through the review again (and, perhaps, again).

Maintainer burnout

The root cause of burnout is that kernel maintainers are "often being overasked"; they are also generally the type of people who will push themselves further than they should in pursuit of perfection. He sees that in himself, but perfect is the enemy of good; "you want to aim for 'good enough', perfect does not exist". A healthy community with assistance from trusted members helps, but is not the ultimate solution; his is a small subsystem, but can be overwhelming, and there are much larger subsystems out there.

For subsystems where there is a "firehose of patches", there is a need for a second developer who is being paid for doing review and other tasks to help reduce that burden. As an example, he pointed to the media subsystem, which he thinks is lacking in reviewer capacity for its enormous patch load. There is an effort to move to a multi-committer model, which may help some, but what is really needed, he thinks, is for some seasoned kernel developer to be paid by some interested company to do patch review. They do not even need to be familiar with the media subsystem; if it is known they are being paid to work on it for two or three days a week, it is worth the time needed to bring them up to speed.

He had a "mini-burnout" himself a while back and noticed that it was not directly caused by the kernel-maintainer role, or by his technical work at Red Hat on top, but by other factors. It came from the feeling that he was not in control of how he spent his time. "Do you feel productive at the end of your work day? Or do you have the feeling you were only spinning your wheels on useless administrative tasks?" For him, the answers to those questions were major factors.

De Goede said that he "had the unlucky and lucky experience" that someone close to him "had a pretty hefty burnout and they are only now fully recovering". So he knew the symptoms, what to look for, and what not to do: "go on go on go on until you crash; don't do that". It is important to recognize the symptoms "and pull that emergency brake early".

In March 2023, he recognized some of those symptoms in himself; armed with the knowledge of what happened with his friend, he knew he did not want to suffer the same fate. He is not a psychologist, but the symptoms he saw were things like "having a very short fuse, being tired all the time, not having energy", and getting annoyed quickly by small things. "Causes for me were feeling loss of control, not feeling appreciated, not feeling very productive, [and] not feeling heard." Those were all work-related, but there were also personal factors, such as aging parents needing care. It is always a combination of things, he said; not being able to spend the time that he felt he needed to on the kernel maintainership just added onto the pile.

Once he realized what was happening, he immediately reported the problem to his employer; "my work is making me sick, I'm calling in sick". He is lucky, he said, to be "from a European country where we have proper worker rights and labor protection". He asked for some changes at work and his management chain agreed, "which was really great". He only ended up being on full sick-leave for three weeks, and then slowly built back up to his regular work hours over a seven-week period.

He recommended that any attendees who recognize burnout symptoms in themselves intervene quickly to ask for help from professionals, someone at work, a friend, or someone else. "Do something, don't keep on the treadmill." The belief that mental illness is somehow not the same as physical illness is problematic and untrue; for one thing, eventually mental illness will lead to physical symptoms. It can be tricky to do, depending on the laws of the country you live in, but he suggested getting your employer involved if the problem is work related; "try to work with them to fix things, to change things, to make your work fun again".

His love of Linux and open-source software makes him "give it my all" every day. That makes it important that he get some positive energy back out of that effort every day, so he can do it again the next day. His manager suggested that he should not try to give 100% every day, and to keep some buffers in reserve. "I'm still working on that."

Handing off

After his mini-burnout, he was working with his manager to make some changes on the maintainer side of his job. He also would like to reduce the number of subsystems with a "bus factor" of one, including pdx86. Even though the subsystem is fairly small, it made sense to him to share the load. He noticed that many of the patches were coming from Intel, so he asked the company if it could provide a seasoned developer to co-maintain the subsystem.

The company gave Ilpo Järvinen time to help out and he has been co-maintaining pdx86 with De Goede since September 2023. In the beginning, they would swap roles for every kernel-development cycle; one would do the bug fixing for the cycle, while the other did the review and patch merging for the subsystem. Since then, De Goede has been getting more involved in the media subsystem and the MIPI camera work, in part due to the lack of reviewers for that part of the kernel, so he asked Järvinen to be the primary pdx86 maintainer in May 2025. De Goede is still part of the team, and can fill in when needed, which also helps the bus factor for pdx86.

The slides for the talk are also available.

[I would like to thank the Linux Foundation, LWN's travel sponsor, for supporting my trip to Amsterdam for Open Source Summit Europe.]