Some notes from the Collaboration Summit

Graybeards

As has seemingly become traditional, your editor moderated a panel of kernel developers (James Bottomley, Christoph Hellwig, Greg Kroah-Hartman, and Andrew Morton, this year). We discussed a wide range of topics, but the subject that caught the attention of the wider world was the "graybeards" question. As a result, we've been treated to a number of lengthy discussions on just why the kernel is no longer attracting energetic young developers.

The only problem is: the kernel doesn't really have that problem. Every three-month development cycle involves well over 1000 developers, a substantial portion of whom are first-time contributors. Nearly 20% of these contributors are working on their own time, because they want to. There does not appear to be a problem attracting developers to the kernel project; indeed, if anything, the problem could be the reverse: the kernel is such an attractive project that it gets an order of magnitude more developers than just about anything else. Your editor has heard it said in the past that Linux as a whole might be better off if some of those developers were to devote a bit more time to user-space projects, most of which would be delighted to have them.

The question the panel discussed, instead, had to do with the graying of the ranks of the kernel's primary subsystem maintainers. Many of those developers wandered into kernel development in the 1990's, when there were opportunities everywhere. A decade or more later, those developers are still there; James Bottomley suggested they may stay in place until they start dying off. Entrenched graybeards at high levels could, someday, start to act as a brake on kernel development as a whole. That does not appear to be a problem now; it's just something to watch out for in the future.

Andrew Morton raised an interesting related issue: as the core developers gain experience and confidence, they are happy to put code of increasing complexity into the kernel. That could indeed make it harder for new developers to come up to speed; it might also not bode well for the long-term maintainability of the kernel in general.

Companies and Linux 1

Dan Frye's keynote reflecting on IBM's 10+ years of experience with Linux was easily one of the best of the day. IBM's experience has certainly not been 100% smooth sailing; there were a lot of mistakes made along the way. As Dan put it, it is relatively easy for a company to form a community around itself, but it's much harder - and more valuable - to join an established community under somebody else's control.

A number of lessons learned were offered, starting with an encouragement to get projects out into the community early and to avoid closed-door communications. IBM discovered the hard way that dumping large blocks of completed code into the kernel community was not going to be successful. The community must be involved earlier than that. To help in that direction, IBM prohibited the use of internal communications for many projects, forcing developers to have their discussions in public forums.

Beyond that, companies need to demonstrate an ongoing commitment to the community - no drive-by submissions. Developers need to remain immersed in the community in order to build the expertise and respect needed to get things done. Companies, Dan says, should manage their developers, but they should not attempt to manage the maintainers those developers are working with. In general, influence in the project should be earned through expertise, skills, and a history of engagement - and not through attempts to control things directly.

One interesting final point is that what matters is results, not whose code gets merged. To that end, IBM has reworked things internally to reward developers who have managed to push things forward, even if somebody else's code gets merged in the end. This is an important attitude which should become more widely adopted; it certainly leads to a more team-oriented and productive kernel community in the long term.

Companies and Linux 2

Chris DiBona's talk on Google and the community was always going to be a bit of a hard sell; Greg Kroah-Hartman's discussion of Android and the community had happened just two days before. Additionally, Chris was scheduled last in the day, immediately after Josh Berkus gave a high-energy version of his how to destroy your community talk. So perhaps Chris cannot be blamed for his decision to dump his own slides and, instead, give a talk to Josh's slides - running backward.

Chris did eventually move on to his real talk. There was discussion of Google's contributions to the community, including 915 projects which have been released so far and something like 200 people creating patches at any given time. There are over 300,000 projects on code.google.com now. Google has also supported the community by providing infrastructure to sites like kernel.org and, of course, the Summer of Code program.

In short: Chris doesn't think that Google has much to apologize for; indeed, the contrary is true. This extends to the Android situation, which, Chris says, he's not really unhappy with. The targeted users for Android are different, and, in any case, it always takes a long time to get a real community going. That said, more of the Android code should indeed get into the mainline, and Google should be doing a better job. Part of the problem, it seems, is finding "masochists" who are willing to do the work of trying to upstream the code.

The truth of the matter is that Chris's talk failed to satisfy a lot of people; much of it came off as "Google is doing good stuff, we know best, we're successful, and we're going to continue doing things the same way." Starting with his playing with Josh's slides, Chris gave the impression that he wasn't taking the community's concerns entirely seriously.

On the other hand, the announcement at the end of his talk that Google was giving a Nexus One phone to every attendee almost certainly served to mute the criticism somewhat.

Outside of the sessions, your editor had the opportunity to talk with some of Google's Android kernel developers; the folks actually doing the work have a bit of a different take on things. They are working flat-out to create some very nice Linux-based products, and they are being successful at it, but they are in the bind that is familiar to so many embedded systems developers: the product cycles are so short and the deadlines are so tight that there just isn't time to spend on getting code upstream. That said, they are trying and intend to try harder. We are starting to see some results; for example, the NVIDIA Tegra architecture code has just been posted for review and merging.

The Android developers seem to feel that they have been singled out for a level of criticism which few other embedded vendors - including those demonstrating much worse community behavior - have to deal with. It can be a dismaying and demotivating thing. Your editor would like to suggest that, at this point, the Android developers have heard and understood the message that the community has tried to communicate to them. There is a good chance that things will start to get better in the near future. Perhaps it's time to back off a little bit and focus on helping them to get their code merged when they get a chance to submit it.

In summary

As promised, this article has barely scratched the surface of what happened at the 2010 Collaboration Summit. In particular, the large presence of MeeGo (which had a separate, day-long session dedicated to it) has been passed over, though some of that will be made up for in a separate article. In general, it was a good collection of people who do not always get a chance to mingle in the same hallway, all backed by the Linux Foundation's seamless "it all just works" organization. Improved collaboration within our community should indeed be the result of this event.