LinuxCon Japan: Advice for new kernel hackers

The final two sessions at LinuxCon Japan were targeted at helping Asian developers become more involved in kernel development. James Bottomley talked about how the social aspects of kernel engineering were as important as its technical aspects. He then moderated a panel of kernel developers that focused on the problems faced by international developers, adding advice to assist new or aspiring kernel hackers in clearing those hurdles.

Some history

Bottomley began with a bit of history, noting that the kernel started off as a tiny "few thousand line" program. In the early days, features were needed so there were scant reviews and very little feature justification required. It was "almost the wild west of patches" and he is somewhat embarrassed to think back on some of the patches he got into the kernel. The only requirement was that a developer could write the code to implement a needed feature.

As examples, he cited the error handling in both the SCSI and IDE layers, which were done at interrupt level and could sometimes lock up the machine. The block layer had a single lock for all devices, while the TTY layer had a static array of TTYs.

The era that Bottomley and other kernel developers grew up in was an era of feature development, but "that era is over", which is a good thing. That "anything goes" style produced a full-featured kernel very quickly, but left behind lots of problems. In particular, robustness and scalability suffered under that scheme.

Ten years later, around 2002 or so, things started changing. Kernel development was moving from "hot coders" trying to make a functional Unix system to a community that cared more about robustness. Various subsystems were rewritten, including SCSI, the block layer, and USB (several times). There was also a "five year plan that took ten years" to move the kernel to more fine-grained locking (and eliminate the big kernel lock).

The attitude today is all about code quality, Bottomley said, and that is something that new kernel developers run into immediately. The kernel development process is more about ensuring that new code doesn't affect older code and that there aren't regressions in kernel functionality. Patches get much more review before being merged, and can often get held up waiting for reviewers. Today's kernel is "incredibly feature rich", but that also means that is complex and thus more fragile. Any new feature that adds complexity gets a lot of scrutiny to see if it really is needed.

There is an elaborate process as well as tools to help enforce the process. It is "exactly like ISO 9001, but worse", Bottomley said to a round of laughter. He pointed out that the Linux development process is compliant with the much-maligned quality management "standardization" effort from the early 2000s. The "Signed-off-by" line in patches, as well as the coding style that is enforced kernel-wide, are elements of a well-established process that seeks to ensure the quality of kernel submissions.

Longtime kernel developers have grown up with these processes, so they make sense to those people, but may be less clear to new contributors. For the most part, they came about because of clear problems that were seen and then addressed. For example, the "Signed-off-by" requirement came about because of the SCO case, but it is useful in its own right; allowing people to track back on the origin of various parts of the kernel.

Getting patches into the kernel

The easiest way to get your patches into the kernel is by making a bug fix, he said. The processes are much easier for a bug fix than they are for a new feature. But, don't label a new feature as a bug fix; it's been done before and it won't be successful. For a bug fix, the patch just needs to describe the bug and how the changes fix that bug.

Trying to get a new feature in is much harder; there is a need to "socialize" it first. That means getting a group of users and others interested in the feature, so that they can help advocate its inclusion. Conferences are a great way to socialize a new feature by talking to other kernel developers about it. Meeting face to face to discuss the feature, perhaps over "wine or whisky", can be beneficial as kernel developers are more inclined to listen in those settings, Bottomley said.

When posting patches for review, there is a natural human desire to show your best work, but that is actually somewhat counter-productive. The patch does not need to be perfect, and the discussion about the patch may help build a community of people interested in the feature. In addition, an imperfect patch makes people think they can still give input.

One thing to remember is that a patch posting will get input of various sorts that will require the poster to be the advocate for their patch. That means that one needs to be prepared to argue on the mailing list, which can be difficult, especially for Asian developers. The key is to only respond to the technical complaints and to ignore any personal attacks that might get posted.

It is also important to identify which of the comments to focus on. There are some people that "lurk" on the mailing lists, trying to make life harder for contributors. Ignoring those comments, while responding to and working with the more influential developers is the right strategy. Developers who get lots of patches accepted (of a technical, rather than janitorial, nature) are likely to be influential, as are those whose feedback is listened to and quoted by others in the discussion. Typically, those people will come up with sensible and constructive suggestions, but they won't always be right, Bottomley said.

For that reason, argument is an essential part of getting any patch accepted. For those who are uncomfortable doing so he suggested practicing defending the patch with friends and colleagues, ideally in English. Starting out by posting to the proper subject mailing list, rather than linux-kernel, will also help as the signal to noise ratio tends to be much better on those lists. In addition, becoming better known on the mailing lists will help with subsequent postings as people will recognize your name, which will make them more favorably disposed toward new postings. It can also help to meet some of the more influential developers at conferences, he said.

If you are getting arguments from influential developers, it may be because they don't understand what the feature is—or why it is needed. It is important to be able to explain the feature to those folks, but don't waste time arguing with the wrong people. One should definitely accept feedback from the right people and make the appropriate changes; there are often multiple iterations to a patch set before it gets accepted.

Bottomley rounded out his talk with some quick hit suggestions that will help smooth the path for a patch. A good changelog message is essential, one that doesn't say what the code does ("almost all maintainers can read C") but why it is needed. Splitting up patches into manageable pieces is important to simplify the review process. One way to do that is to describe what the patch does to colleagues and split up the patch in a way that corresponds to that description. It is also important to follow the established rules, as they have evolved over a long time and there is a reason for each of them.

Kernel panel

After that, several other kernel developers joined him on the stage: Arnd Bergmann of IBM/Linaro, Chris Mason of Oracle (at that time), Herbert Xu of Red Hat, and Masami Hiramatsu of Hitachi,. After they introduced themselves, Bottomley asked each to describe what obstacles they had faced and overcome in becoming a kernel developer. Hiramatsu mentioned "English, of course" as one of the hurdles, but also noted that the time difference between Japan and the rest of kernel developers can be a barrier. Sometimes he has stayed up late to be able to be a part of the discussion that would otherwise start without him.

Xu talked about some of the problems he faced as a Debian kernel maintainer. There were numerous patches carried by distributions that he and others tried to push upstream, but they were rejected. The distribution developers had a very different view of what was needed in the kernel than the subsystem maintainers and other mainline developers had. That problem has been largely overcome now, he said, by knowing what the kernel as a whole needs and adapting the distribution patches appropriately.

Mason pointed out that the time zone problem can actually be an advantage. One way to tamp the flames in a mailing list discussion is to reply slowly, which gives all of the participants some time to reconsider their words and thoughts. He "did everything wrong" in his first kernel project by choosing an enormous feature that he had no idea how to implement, then worked on it by himself for months without asking any questions. His suggestion was that new contributors not follow his lead and instead start with smaller bug fixes and get to know the community before moving on to bigger projects.

Bergmann said that his first patch was to support some new hardware that he sent to a maintainer who disappeared shortly thereafter. His next effort displayed his lack of C knowledge and had three bugs in a single line patch. Those problems were pointed out by a good maintainer, which led him to go back and study C and programming some more before his next submission. Bottomley pointed out that Bergmann's experience is an example that normally the criticism is aimed at the code, and not at the person.

Problems specific to developers in Asia (beyond the time zone and language issues) was next up. Hiramatsu noted that his teachers always wanted him to make things perfect in his code, which is kind of the opposite of the approach that works best for the Linux kernel. While we are ultimately aiming for the perfect kernel, we get there by having an open discussion on patches that are imperfect, he said.

Don't think, just do it

Bottomley asked specifically what he recommended for Japanese developers so that they were willing to submit imperfect patches. "Don't think, just do it", Hiramatsu—who may have a future in sports gear marketing—said. The key is to not worry about it too much before sending the patch, he said.

An area that Asian developers could improve, Xu said, is in participating in the kernel as more than just their work duty. In the west, many developers see their efforts in the kernel community as independent of their work, so that even if they started working in an unrelated field, "they would still be part of the community". Engaging the community takes time, and people are very busy with work and family, so they don't have time to make a long-term commitment to one area of the kernel. But it is very useful to do so, he said, and something that developers in Asia could do better.

Both IBM and Linaro have done private reviews for new developers, Bergmann said. The idea is to help them describe and defend the patch by discussing it on an internal mailing list. Essentially it helps train developers in how to argue on mailing lists, so that it will be easier when it needs to be done on linux-kernel.

Mason said that you can only create the perfect patch indirectly. The most important thing to do is to "prove that you understand the problem". That is done with a description of the feature or bug that shows that understanding. What code actually ends up in the mainline is dependent on the maintainers, so the patch should be seen as a "sample solution" that may or may not end up in the mainline. Bottomley pointed to the "Andrew Morton bug fix solution", which is to post a bad patch as a challenge to the rest of the kernel developers, as a similar idea.

Attracting and keeping new kernel developers was another topic covered. Mason said that it is important to encourage people that offer to help, by suggesting that they study the problem they are having. If they pick a feature or bug fix that they need or really care about, they will be more successful. Bergmann suggested that people be encouraged to become maintainers of smaller pieces of the kernel, rather than take on a bigger piece. He has done that and said it has "worked surprisingly well". It makes for "better maintainers than I am", he said.

As LinuxCon Japan wound to a close, Bottomley asked for "inspirational statements" from the panel. Mason said "Linux is yours, treat it like it's yours", while Xu extended that thought: "Treat it like a hobby", he said. Hiramatsu had the final word by reprising his earlier "Don't think, just do it" message. Suitably inspired, the assembled masses then made their way to the post-conference party.

---

(Log in to post comments)