KS2010: Welcoming newcomers

The two opening sessions at the 2010 Linux Kernel Summit had to do with the welcoming of newcomers. It has occasionally be noted that the core kernel development community is getting older. Kernel maintainers often seem to be quite comfortable in their roles; there does not often seem to be a lot of turnover. Are we doing enough to bring new developers into the community?

Jiri Kosina did a bit of research in the git repository; he identified a "major contributor" as somebody who produced at least 30 patches in one kernel development cycle and significant numbers in the following couple of cycles. By this measure, we had added about 25 new major developers over three cycles, with the rate decreasing over that time. That is, he says, not an entirely great result. He observed that getting into kernel development at the level of fixing bugs or writing drivers is relatively easy, but getting into longer-term core work is harder. Also, the development community has a bit of a reputation for being unfriendly, though opinions vary on that; unsurprisingly, outsiders see the community as being less friendly than insiders do.

It has been noted by a number of people that the development community is not particularly friendly to developers who show up and want to have a task handed to them. Some attempts to organize information about tasks seeking developers have been made in the past, but they have not generally worked very well; the KernelNewbies projects page is essentially dead, for example. Many subsystem maintainers have "to do" lists, but those are not organized in any standard way and can be hard to find. Meanwhile, incoming developers seeking advice often hear contradictory things, which makes their lives harder.

Whether all of this is a problem is not clear to everybody. Those who are truly self-motivated will find their way into the community and figure out how to work within it. Whether we want to be concerned about the rest is controversial.

Back to "to do" lists: all maintainers have them, or at least should have them. They are well scattered; some are maintained on the net somewhere, others are in the kernel tree, but most just live in the maintainers' heads and are generally inaccessible. Organizing these lists might make it easier for others to find work which needs to be done. They could all be put onto a wiki somewhere, or distributed with pointers in the MAINTAINERS file, but there are concerns about keeping such resources maintained over time. So Jiri suggested that putting them into the kernel tree would be best.

Al Viro added that, rather than distributing lists throughout the code, developers should put them under the documentation directory. That would make it easy to find them all. James Bottomley, instead, would rather keep these lists outside of the kernel tree entirely; these lists tend to be maintained in a way which is different from everything else in the tree and don't fit into the normal development cycle. Steve Rostedt suggested specially-formatted comments in the code itself which could be pulled together with grep. Ted Ts'o noted that there are many different types of tasks; some of them are large and tricky and not well suited to be annotated in a specific part of the code. Also, he's not really sure that he wants new developers to attempt some of those tasks, it can be a real pain when they try.

Thomas Gleixner rained on the parade somewhat by claiming that "to do" lists will not be helpful in bringing in new developers. The people who are making significant contributions to the kernel now did not come in by way of a "to do" list. They showed up because they had an interest in hacking on the code. Steve responded that this pattern holds because we don't have good lists, but Thomas held up the "feature removal" list as an example. That list has been in the kernel and well maintained for years, but it still doesn't attract developers to do the work. Part of the problem, it seems, is that these lists tend not to be "sexy"; maintainers tend to keep the more interesting tasks for themselves.

Ben Herrenschmidt made a claim which many agreed to: people who are looking for somebody else to provide them with a task tend not to be the best developers. Linus jumped in at this point to say that, if you have to ask what you should be working on, he doesn't want you to work on anything at all. We have a lot of one-patch developers and don't need more of them; what we want is more maintainers. These people generally already know what they want to do. Self motivation, Linus says, is an important attribute for anybody who is still going to be around three years from now. Ben added that "todo-driven" developers tend to require a lot of hand-holding, which he tends not to have the time to do.

But, it was asked, what is the real issue here? James Bottomley suggested that the real problem is that we're not getting enough good people. Herbert Xu added that we're doing fine at getting new people involved, but we can have trouble keeping them engaged in the development process. Christoph Hellwig asked if there is actually a problem - is stuff not getting done? Linus suggested that perhaps what we really need to do is to get rid of some people instead.

What it comes down to, according to some, is that we're not bringing in enough core developers. Finding people to do simple stuff is not too hard, but the complexity of the core kernel presents a higher bar. Grant Likely said that developers can begin with drivers and move to more complex work, but Thomas asserted that driver developers tend not to grow into core work. Dave Airlie pointed out one problem: doing core work pretty much requires a position that allows for full-time kernel development; Thomas added that not many companies are willing to fund development at that level.

Another concern, according to Steve Rostedt, is that a lot of developers, especially those in Asia, have a language barrier to overcome. Mel Gorman suggested that language is not a huge problem; developers in those areas help each other a lot and people have gotten used to working with non-native English. Ted said that there are a lot of developers who are not particularly visible in the community for a number of reasons; language issues are only one of those reasons. Some developers don't like the communication style on the linux-kernel, for example. Others are very "mission-focused," once they get their stuff (a specific driver, perhaps) working, they don't really care about broader applicability. There are also a lot of developers "fixing enterprise bugs" who are not generally visible. Different types of developers have different reasons for staying off the lists, and not all of them are amenable to change.

Another class of potential developer, according to Kay Sievers, is made up of user space programmers who end up maintaining significant amounts of code to work around kernel shortcomings which can often be easily fixed. These developers tend to be nervous about the kernel community and don't necessarily want to be pulled into it, so they often just continue to maintain their workarounds. Mel added that, when such a developer does come forward with a kernel fix, they are often asked to take on much bigger tasks to get their little patch in. The problem is that user-space developers often cannot take into account all of the possible implications of a change; they need help from people in the kernel community.

Sarah Sharp raised the issue of developers who may have gotten their feet wet in kernel development and are now looking for a medium-level project. Often these people are coming from an academic background; they have a hard time finding the right kind of project. Mel noted that the right kind of project is hard for a lot of academic people to find in general: an awful lot of them will be told by their advisers that their good idea has been "ruined" by virtue of being actually implemented in the kernel. James added that systems-level academics are in a position where they often have to publish more papers than their peers to be respected, and getting code merged is not seen as "publication". In summary: academia raises a number of obstacles to participation that cannot necessarily be fixed on the kernel side.

Somewhere around this point the agenda moved into a related session on "patch etiquette," led by Kay Sievers. The problem Kay is concerned about is new developers who post patches and never get a response at all. Or, if they do hear something, it's along the lines of "fix the whole subsystem to get this change merged." Kay allowed that the community needs some sort of filter against arbitrary developers, but, he thinks, it's a little too severe now. Can we maybe do better?

Arnd Bergmann suggested that we could benefit from better documentation for newcomers. Mel responded with the idea of special training for new developers: proprietary operating systems have training courses, but they don't exist in the same way for Linux. Christoph, instead, thinks that it comes down to how a developer can get the attention of the community: fix a real bug and "send reasonable stuff." That, he says, is more useful than a bunch of "fluffy bullshit" that will get ignored.

Kay tried to bring the discussion back to the problem of patches not being responded to, especially when they apply to subsystems which lack a maintainer. Dirk Hohndel talked about how hard it is for many professional developers to join the community and asked that it be made easier. Intel has the resources to do in-house training on how to work with free software communities, but a lot of companies are not so lucky. So, in many of these companies, working with the community just seems too hard and not worth the trouble.

Ben responded that there are a lot of professional programmers out there that we don't want at all. What we want are people with real interest and with the pride needed to push things through; such people are not very common in the corporate world. But Dirk insisted that cynical attitudes toward corporate programmers can be harmful; there are a lot of people trying to do good stuff. So, he said, we can either be arrogant and reject them, or we can try to help the companies that pay programmers to be successful. He noted that everybody in the room was paid to be there.

Thomas made the point that companies are, in general, getting better at figuring out how to work with open source projects. He is currently working with a company planning to add support for a new architecture; they brought him in early to help them learn how to work with the community. It all can be done; the problem just needs to be solved at the company manager level.

At this point, Linus came in with a brief speech. Most open source projects, he said, would be "laughing their heads off" to hear this discussion. The discussion started by talking about bringing in "only" 25 new major contributors; most projects never get to where they have that many. We have something like 1-200 core developers and well over 1000 regular contributors. That is a lot of developers; we should be thinking about where the scalability limit is. Do we want to bring in all of China, or do we want to be arrogant and insist on only the best?

The kernel community, Linus said, cannot scale to five times its current size and should not even try. There is always a need for new blood, but, he said, the demographics of the community are fairly good. What we should be concerned with is getting more outstanding maintainers - that's a problem that he doesn't know how to solve. But, Linus said, we have "serious quality management issues," and we will not fix them by adding more programmers.

The biggest problem with maintainership, according to Linus, is that the organization is very flat. In most trees that he pulls, all of the commits have been done by a single maintainer. They are missing a lower level of maintainers who should be taking on some of the work. The networking tree, he said, is an example of how things should be done: Dave Miller builds his tree from repositories maintained by a number of other developers. The x86 architecture was once "a horrible mess," but it also works very well from this point of view. In general, with multiple levels of maintainership, either patches will be lost or the subsystem is too small to matter. He said, only half jokingly, that "well maintained" means "maintained the way he does it." There may be other ways, but, in the end, his process is known to scale; he has not found anything else that does.

Ben pointed out that having submaintainers is one of the best ways to train people to take over the higher-level maintainer positions when that becomes necessary. And at that point, this lengthy session came to an end.

Next: ABI status for tracepoints.

---

(Log in to post comments)