The final two sessions at LinuxCon Japan were targeted at helping Asian
developers become more involved in kernel development. James Bottomley
talked about how the social aspects of kernel engineering were as important
as its technical aspects. He then moderated a panel of kernel developers
that focused on the problems faced by international
developers, adding advice to assist new or aspiring kernel hackers
in clearing those hurdles.
Bottomley began with a bit of history, noting that the kernel started off
as a tiny "few thousand line" program. In the early days, features were
needed so there were scant reviews and very little feature justification
required. It was "almost the wild west of patches" and he is somewhat
embarrassed to think back on some of the patches he got into the kernel.
The only requirement was that a developer could write the code to implement
a needed feature.
As examples, he cited the error handling in both the SCSI and IDE layers,
which were done at interrupt level and could sometimes lock up the
machine. The block layer had a single lock for all devices, while the TTY
layer had a static array of TTYs.
The era that Bottomley and other kernel developers grew up in was an era of
feature development, but "that era is over", which is a good thing. That
"anything goes" style produced a full-featured kernel very quickly, but
left behind lots of problems. In particular, robustness and scalability
suffered under that scheme.
Ten years later, around 2002 or so, things started changing. Kernel
development was moving from "hot coders" trying to make a functional Unix
system to a community that cared more about robustness. Various subsystems
were rewritten, including SCSI, the block layer, and USB (several times).
There was also a "five year plan that took ten years" to move the kernel to more
fine-grained locking (and eliminate the big kernel lock).
The attitude today is all about code quality, Bottomley said, and that is
something that new kernel developers run into immediately. The kernel
development process is more about ensuring that new code doesn't affect
older code and that there aren't regressions in kernel functionality.
Patches get much more review before being merged, and can often get held up
waiting for reviewers.
Today's kernel is "incredibly feature rich", but that also means that is
complex and thus more fragile. Any new feature that adds complexity gets a
lot of scrutiny to see if it really is needed.
There is an elaborate process as well as tools to help enforce the
process. It is "exactly like ISO 9001, but worse", Bottomley said to a
round of laughter. He pointed out that the Linux development process is
compliant with the much-maligned quality management "standardization"
effort from the early
2000s. The "Signed-off-by" line in patches, as well as the coding style
that is enforced kernel-wide, are elements of a well-established process
that seeks to ensure the quality of kernel submissions.
developers have grown up with these processes, so they make sense to those
but may be less clear to new contributors. For the most part, they came
about because of clear problems that were seen and then addressed. For
"Signed-off-by" requirement came about because of the SCO case, but it is
useful in its own right; allowing people to track back on the origin of
various parts of the kernel.
Getting patches into the kernel
The easiest way to get your patches into the kernel is by making a bug
fix, he said. The processes are much easier for a bug fix than they are
for a new
feature. But, don't label a new feature as a bug fix; it's been done before
and it won't be successful. For a bug fix, the patch just needs to
describe the bug and how the changes fix that bug.
Trying to get a new feature in is much harder; there is a need to
"socialize" it first. That means getting a group of users and others
interested in the feature, so that they can help advocate its inclusion.
Conferences are a great way to socialize a new feature by talking to other
kernel developers about it. Meeting face to face to discuss the feature,
perhaps over "wine or whisky", can be beneficial as kernel developers are
more inclined to listen in those settings, Bottomley said.
When posting patches for review, there is a natural human desire to show
your best work, but that is actually somewhat counter-productive. The
patch does not need to be perfect, and the discussion about the patch may
help build a community of people interested in the feature. In addition,
an imperfect patch makes people think they can still give input.
One thing to remember is that a patch posting will get input of various
sorts that will require the poster to be the advocate for their patch.
That means that one needs to be prepared to argue on the mailing list,
which can be difficult, especially for Asian developers. The key is to
only respond to the technical complaints and to ignore any personal attacks
that might get posted.
It is also important to identify which of the comments to focus on. There
are some people that "lurk" on the mailing lists, trying to make life
harder for contributors. Ignoring those comments, while responding to and
working with the
more influential developers is the right strategy. Developers who get lots
of patches accepted (of a technical, rather than janitorial, nature) are
likely to be influential, as are those whose feedback is listened to and
quoted by others in the discussion. Typically, those people will come up
with sensible and constructive suggestions, but they won't always be right,
For that reason, argument is an essential part of getting any patch
accepted. For those who are uncomfortable doing so he suggested practicing
defending the patch with friends and colleagues, ideally in English.
Starting out by posting to the proper subject mailing list, rather than
linux-kernel, will also help as the signal to noise ratio tends to be much
better on those lists. In addition, becoming better known on the mailing
lists will help with subsequent postings as people will recognize your
name, which will make them more favorably disposed toward new postings. It
can also help to meet some of the more influential developers at
conferences, he said.
If you are getting arguments from influential developers, it may be because
they don't understand what the feature is—or why it is needed. It is
important to be able to explain the feature to those folks, but don't waste
time arguing with the wrong people. One should definitely accept feedback
from the right people and make the appropriate changes; there are often
multiple iterations to a patch set before it gets accepted.
Bottomley rounded out his talk with some quick hit suggestions that will
help smooth the path for a patch. A good changelog message is essential,
one that doesn't say what the code does ("almost all maintainers can read
C") but why it is needed. Splitting up patches into manageable pieces is
important to simplify the review process. One way to do that is to
describe what the patch does to colleagues and split up the patch in a way
that corresponds to that description. It is also important to follow the
established rules, as they have evolved over a long time and there is a
reason for each of them.
After that, several other kernel developers joined him on the stage: Arnd
Bergmann of IBM/Linaro, Chris Mason of Oracle (at that time), Herbert Xu of Red
Hat, and Masami Hiramatsu of Hitachi,. After they introduced themselves,
asked each to describe what obstacles they had faced and overcome in
becoming a kernel developer. Hiramatsu mentioned "English, of course" as
one of the hurdles, but also noted that the time difference between Japan
and the rest of kernel developers can be a barrier. Sometimes he has
stayed up late to be able to be a part of the discussion that would
otherwise start without him.
Xu talked about some of the problems he faced as a Debian kernel
maintainer. There were numerous patches carried by distributions that he
and others tried to push upstream, but they were rejected. The
distribution developers had a very different view of what was needed in the
kernel than the subsystem maintainers and other mainline developers had.
That problem has been largely overcome now, he said, by knowing what the
kernel as a whole needs and adapting the distribution patches appropriately.
Mason pointed out that the time zone problem can
actually be an advantage. One way to tamp the flames in a mailing list
to reply slowly, which gives all of the participants some time to
reconsider their words and thoughts. He
"did everything wrong" in his first kernel project by choosing an enormous
feature that he had no idea
how to implement, then worked on it by himself for months without asking
any questions. His suggestion was that new contributors not follow his
lead and instead start with smaller bug fixes and get to know the community
before moving on to bigger projects.
Bergmann said that his first patch was to support some new hardware that he
sent to a maintainer who disappeared shortly thereafter. His next effort
displayed his lack of C knowledge and had three bugs in a single line
patch. Those problems were pointed out by a good maintainer, which led him
to go back and study C and programming some more before his next
submission. Bottomley pointed out that Bergmann's experience is an example
that normally the criticism is aimed at the code, and not at the person.
Problems specific to developers in Asia (beyond the time zone and language
issues) was next up. Hiramatsu noted that his teachers always wanted him
to make things perfect in his code, which is kind of the opposite of the
approach that works best for the Linux kernel. While we are ultimately aiming for the
perfect kernel, we get there by having an open discussion on
patches that are imperfect, he said.
Don't think, just do it
Bottomley asked specifically what he recommended for Japanese
developers so that they were willing to submit imperfect patches. "Don't
think, just do it", Hiramatsu—who may have a future in sports gear
marketing—said. The key is to not worry about it too much before
sending the patch, he said.
An area that Asian developers could improve, Xu said, is in participating
in the kernel as more than just their work duty. In the west, many
developers see their efforts in the kernel community as independent of
their work, so that even if they started working in an unrelated field,
still be part of the community". Engaging the community takes time, and
people are very busy with work and family, so they don't have time to make
a long-term commitment to one area of the kernel. But it is very useful to
do so, he said, and something that developers in Asia could do better.
Both IBM and Linaro have done private reviews for new developers, Bergmann
said. The idea is to help them describe and defend the patch by discussing
it on an internal mailing list. Essentially it helps train developers in
how to argue on mailing lists, so that it will be easier when it needs to
be done on linux-kernel.
Mason said that you can only create the perfect patch indirectly. The most
important thing to do is to "prove that you understand the problem". That
is done with a description of the feature or bug that shows that
code actually ends up in the mainline is dependent on the maintainers, so
the patch should be seen as a "sample solution" that may or may not end up
in the mainline. Bottomley pointed to the "Andrew Morton bug fix
solution", which is to post a bad patch as a challenge to the rest of the
kernel developers, as a similar idea.
Attracting and keeping new kernel developers was another topic covered.
Mason said that it is important to encourage people that offer to help, by
suggesting that they study the problem they are having. If they pick
a feature or bug fix that they need or really care about, they will be more
successful. Bergmann suggested that people be encouraged to become
maintainers of smaller pieces of the kernel, rather than take on a bigger
piece. He has done that and said it has "worked surprisingly well". It
makes for "better maintainers than I am", he said.
As LinuxCon Japan wound to a close, Bottomley asked for "inspirational
statements" from the panel. Mason said "Linux is yours, treat it like it's
yours", while Xu extended that thought: "Treat it like a hobby", he said.
Hiramatsu had the final word by reprising his earlier "Don't think, just do
it" message. Suitably inspired, the assembled masses then made their way
to the post-conference party.
to post comments)