The two opening sessions at the 2010 Linux Kernel Summit had to do with the
newcomers. It has occasionally be noted that the core kernel development
community is getting older. Kernel maintainers often seem to be quite
comfortable in their roles; there does not often seem to be a lot of
turnover. Are we doing enough to bring new developers into the community?
Jiri Kosina did a bit of research in the git repository; he identified a
"major contributor" as somebody who produced at least 30 patches in one
kernel development cycle and significant numbers in the following couple of
cycles. By this measure, we had added about 25 new major developers over
three cycles, with the rate decreasing over that time. That is, he says,
not an entirely great result. He observed that getting into kernel
development at the level of fixing bugs or writing drivers is relatively
easy, but getting into longer-term core work is harder. Also, the
development community has a bit of a reputation for being unfriendly,
though opinions vary on that; unsurprisingly, outsiders see the community
as being less friendly than insiders do.
It has been noted by a number of people that the development
community is not particularly friendly to developers who show up and want
to have a task handed to them. Some attempts to organize information about
tasks seeking developers have been made in the past, but they have not
generally worked very well; the KernelNewbies projects
page is essentially dead, for example. Many subsystem maintainers have
"to do" lists, but those are not organized in any standard way and can be
hard to find. Meanwhile, incoming developers seeking advice often hear
contradictory things, which makes their lives harder.
Whether all of this is a problem is not clear to everybody. Those who are
truly self-motivated will find their way into the community and figure out
how to work within it. Whether we want to be concerned about the rest is
Back to "to do" lists: all maintainers have them, or at least should
have them. They are well scattered; some are maintained on the net
somewhere, others are in the kernel tree, but most just live in the
maintainers' heads and are generally inaccessible. Organizing these lists
might make it easier for others to find work which needs to be done. They
could all be put onto a wiki somewhere, or distributed with pointers in the
MAINTAINERS file, but there are concerns about keeping such resources
maintained over time. So Jiri suggested that putting them into the kernel
tree would be best.
Al Viro added that, rather than distributing lists throughout the code,
developers should put them under the documentation directory. That would
make it easy to find them all. James Bottomley, instead, would rather
keep these lists outside of the kernel tree entirely; these lists tend to
be maintained in a way which is different from everything else in the tree
and don't fit into the normal development cycle. Steve Rostedt suggested
specially-formatted comments in the code itself which could be pulled
together with grep. Ted Ts'o noted that there are many different
types of tasks; some of them are large and tricky and not well suited to be
annotated in a specific part of the code. Also, he's not really sure that
he wants new developers to attempt some of those tasks, it can be a real
pain when they try.
Thomas Gleixner rained on the parade somewhat by claiming that "to do"
lists will not be helpful in bringing in new developers. The people who
are making significant contributions to the kernel now did not come in by
way of a "to do" list. They showed up because they had an interest in
hacking on the code. Steve responded that this pattern holds because we
don't have good lists, but Thomas held up the "feature removal" list
as an example. That list has been in the kernel and well maintained for
years, but it still doesn't attract developers to do the work. Part of the
problem, it seems, is that these lists tend not to be "sexy"; maintainers
tend to keep the more interesting tasks for themselves.
Ben Herrenschmidt made a claim which many agreed to: people who are looking
for somebody else to provide them with a task tend not to be the best
developers. Linus jumped in at this point to say that, if you have to ask
what you should be working on, he doesn't want you to work on anything at
all. We have a lot of one-patch developers and don't need more of them;
what we want is more maintainers. These people generally already know what
they want to do. Self motivation, Linus says, is an important attribute
for anybody who is still going to be around three years from now. Ben
added that "todo-driven" developers tend to require a lot of hand-holding,
which he tends not to have the time to do.
But, it was asked, what is the real issue here? James Bottomley suggested
that the real problem is that we're not getting enough good people.
Herbert Xu added that we're doing fine at getting new people involved, but
we can have trouble keeping them engaged in the development process. Christoph
Hellwig asked if there is actually a problem - is stuff not getting done?
Linus suggested that perhaps what we really need to do is to get rid of
some people instead.
What it comes down to, according to some, is that we're not bringing in enough
core developers. Finding people to do simple stuff is not too hard,
but the complexity of the core kernel presents a higher bar. Grant Likely
said that developers can begin with drivers and move to more complex work,
but Thomas asserted that driver developers tend not to grow into core
work. Dave Airlie pointed out one problem: doing core work pretty much
requires a position that allows for full-time kernel development; Thomas
added that not many companies are willing to fund development at that
Another concern, according to Steve Rostedt, is that a lot of developers,
especially those in Asia, have a language barrier to overcome. Mel Gorman
suggested that language is not a huge problem; developers in those areas
help each other a lot and people have gotten used to working with
non-native English. Ted said that there are a lot of developers who are
not particularly visible in the community for a number of reasons; language
issues are only one of those reasons. Some developers don't like the
communication style on the linux-kernel, for example. Others are very
"mission-focused," once they get their stuff (a specific driver, perhaps)
working, they don't really care about broader applicability. There are
also a lot of developers "fixing enterprise bugs" who are not generally
visible. Different types of developers have different reasons for staying
off the lists, and not all of them are amenable to change.
Another class of potential developer, according to Kay Sievers, is made up
of user space programmers who end up maintaining significant amounts of
code to work around kernel shortcomings which can often be easily fixed.
These developers tend to be nervous about the kernel community and don't
necessarily want to be pulled into it, so they often just continue to
maintain their workarounds. Mel added that, when such a developer does
come forward with a kernel fix, they are often asked to take on much bigger
tasks to get their little patch in. The problem is that user-space
developers often cannot take into account all of the possible implications
of a change; they need help from people in the kernel community.
Sarah Sharp raised the issue of developers who may have gotten their feet
wet in kernel development and are now looking for a medium-level project.
Often these people are coming from an academic background; they have a hard
time finding the right kind of project. Mel noted that the right kind of
project is hard for a lot of academic people to find in general: an awful
lot of them will be told by their advisers that their good idea has been
"ruined" by virtue of being actually implemented in the kernel. James
added that systems-level academics are in a position where they often have
to publish more papers than their peers to be respected, and getting code
merged is not seen as "publication". In summary: academia raises a number
of obstacles to participation that cannot necessarily be fixed on the
Somewhere around this point the agenda moved into a related session on
"patch etiquette," led by Kay Sievers. The problem Kay is concerned about
is new developers who post patches and never get a response at all. Or, if
they do hear something, it's along the lines of "fix the whole subsystem to
get this change merged." Kay allowed that the community needs some sort of
filter against arbitrary developers, but, he thinks, it's a little too
severe now. Can we maybe do better?
Arnd Bergmann suggested that we could benefit from better documentation for
newcomers. Mel responded with the idea of special training for new
developers: proprietary operating systems have training courses, but they
don't exist in the same way for Linux. Christoph, instead, thinks that it
comes down to how a developer can get the attention of the community: fix a
real bug and "send reasonable stuff." That, he says, is more useful than a
bunch of "fluffy bullshit" that will get ignored.
Kay tried to bring the discussion back to the problem of patches not being
responded to, especially when they apply to subsystems which lack a
maintainer. Dirk Hohndel talked about how hard it is for many professional
developers to join the community and asked that it be made easier. Intel
has the resources to do in-house training on how to work with free software
communities, but a lot of companies are not so lucky. So, in many of these
companies, working with the community just seems too hard and not worth the
Ben responded that there are a lot of professional programmers out there
that we don't want at all. What we want are people with real interest and
with the pride needed to push things through; such people are not very
common in the corporate world. But Dirk insisted that cynical attitudes
toward corporate programmers can be harmful; there are a lot of people
trying to do good stuff. So, he said, we can either be arrogant and reject
them, or we can try to help the companies that pay programmers to be
successful. He noted that everybody in the room was paid to be there.
Thomas made the point that companies are, in general, getting better at
figuring out how to work with open source projects. He is currently
working with a company planning to add support for a new architecture; they
brought him in early to help them learn how to work with the community. It
all can be done; the problem just needs to be solved at the company manager
At this point, Linus came in with a brief speech. Most open source
projects, he said, would be "laughing their heads off" to hear this
discussion. The discussion started by talking about bringing in "only" 25
new major contributors; most projects never get to where they have that
many. We have something like 1-200 core developers and well over
1000 regular contributors. That is a lot of developers; we should be
thinking about where the scalability limit is. Do we want to bring in all
of China, or do we want to be arrogant and insist on only the best?
The kernel community, Linus said, cannot scale to five times its current
size and should not even try. There is always a need for new blood, but,
he said, the demographics of the community are fairly good. What we should
be concerned with is getting more outstanding maintainers - that's a
problem that he doesn't know how to solve. But, Linus said, we have "serious
quality management issues," and we will not fix them by adding more
The biggest problem with maintainership, according to Linus, is that the
organization is very flat. In most trees that he pulls, all of the commits
have been done by a single maintainer. They are missing a lower level of
maintainers who should be taking on some of the work. The networking tree,
he said, is an example of how things should be done: Dave Miller builds his
tree from repositories maintained by a number of other developers. The x86
architecture was once "a horrible mess," but it also works very well from
this point of view. In general, with multiple levels of maintainership,
either patches will be lost or the subsystem is too small to matter. He
said, only half jokingly, that "well maintained" means "maintained the way
he does it." There may be other ways, but, in the end, his process is
known to scale; he has not found anything else that does.
Ben pointed out that having submaintainers is one of the best ways to train
people to take over the higher-level maintainer positions when that becomes
necessary. And at that point, this lengthy session came to an end.
Next: ABI status for tracepoints.
to post comments)