Your editor has just returned from the Linux Foundation's annual
Collaboration Summit, held in San Francisco. LFCS is a unique event;
despite becoming more developer-heavy over the years, it still pulls
together an interesting combination of people from the wider Linux
ecosystem. The following article is not intended to be a comprehensive
report from the summit; it is, instead, a look at a few of the more
interesting thoughts that came from there.
As has seemingly become traditional, your editor moderated a panel of
kernel developers (James Bottomley, Christoph Hellwig, Greg Kroah-Hartman,
and Andrew Morton, this year). We discussed a wide range of topics, but
the subject that caught the attention of the wider world was the
"graybeards" question. As a result, we've been treated to a number of lengthy
discussions on just why the kernel is no longer attracting energetic
The only problem is: the kernel doesn't really have that problem. Every
three-month development cycle involves well over 1000 developers, a
substantial portion of whom are first-time contributors. Nearly 20% of
these contributors are working on their own time, because they want to.
There does not appear to be a problem attracting developers to the kernel
project; indeed, if anything, the problem could be the reverse: the kernel
is such an attractive project that it gets an order of magnitude more
developers than just about anything else. Your editor has heard it said in
the past that Linux as a whole might be better off if some of those
developers were to devote a bit more time to user-space projects, most of
which would be delighted to have them.
The question the panel discussed, instead, had to do with the graying of
the ranks of the kernel's primary subsystem maintainers. Many of those
developers wandered into kernel development in the 1990's, when there were
opportunities everywhere. A decade or more later, those developers are
still there; James Bottomley suggested they may stay in place until they
start dying off. Entrenched graybeards at high levels could, someday,
start to act as a brake on kernel development as a whole. That does not
appear to be a problem now; it's just something to watch out for in the
Andrew Morton raised an interesting related issue: as the core
developers gain experience and confidence, they are happy to put code of
increasing complexity into the kernel. That could indeed make it harder
for new developers to come up to speed; it might also not bode well for the
long-term maintainability of the kernel in general.
Companies and Linux 1
Dan Frye's keynote reflecting on IBM's 10+ years of experience with Linux
was easily one of the best of the day. IBM's experience has certainly not
been 100% smooth sailing; there were a lot of mistakes made along the way.
As Dan put it, it is relatively easy for a company to form a community
around itself, but it's much harder - and more valuable - to join an
established community under somebody else's control.
A number of lessons learned were offered, starting with an encouragement to
get projects out into the community early and to avoid closed-door
communications. IBM discovered the hard way that dumping large blocks of
completed code into the kernel community was not going to be successful.
The community must be involved earlier than that. To help in that
direction, IBM prohibited the use of internal communications for many
projects, forcing developers to have their discussions in public forums.
Beyond that, companies need to demonstrate an ongoing commitment to the
community - no drive-by submissions. Developers need to remain immersed in
the community in order to build the expertise and respect needed to get
things done. Companies, Dan says, should manage their developers, but they
should not attempt to manage the maintainers those developers are working
with. In general, influence in the project should be earned through
expertise, skills, and a history of engagement - and not through attempts
to control things directly.
One interesting final point is that what matters is results, not whose code
gets merged. To that end, IBM has reworked things internally to reward
developers who have managed to push things forward, even if somebody else's
code gets merged in the end. This is an important attitude which should
become more widely adopted; it certainly leads to a more team-oriented and
productive kernel community in the long term.
Companies and Linux 2
Chris DiBona's talk on Google and the community was always going to be a
bit of a hard sell; Greg Kroah-Hartman's discussion of Android and the
community had happened just two days before. Additionally, Chris was
scheduled last in the day, immediately after Josh Berkus gave a high-energy
version of his how to destroy
your community talk. So perhaps Chris cannot be blamed for his
decision to dump his own slides and, instead, give a talk to Josh's slides
- running backward.
Chris did eventually move on to his real talk. There was discussion of
Google's contributions to the community, including 915 projects which have
been released so far and something like 200 people creating patches at any
given time. There are over 300,000 projects on code.google.com now.
Google has also supported the community by providing infrastructure to
sites like kernel.org and, of course, the Summer of Code program.
In short: Chris doesn't think that Google has much to apologize for;
indeed, the contrary is true. This extends to the Android situation,
which, Chris says, he's not really unhappy with. The targeted users for
Android are different, and, in any case, it always takes a long time to get
a real community going. That said, more of the Android code should indeed
get into the mainline, and Google should be doing a better job. Part of
the problem, it seems, is finding "masochists" who are willing to do the
work of trying to upstream the code.
The truth of the matter is that Chris's talk failed to satisfy a lot of
people; much of it came off as "Google is doing good stuff, we know best,
we're successful, and we're going to continue doing things the same way."
Starting with his playing with Josh's slides, Chris gave the impression
that he wasn't taking the community's concerns entirely seriously.
On the other hand, the announcement at the end of his talk that Google was
giving a Nexus One phone to every attendee almost certainly served to mute
the criticism somewhat.
Outside of the sessions, your editor had the opportunity to talk with some
of Google's Android kernel developers; the folks actually doing the work
have a bit of a different take on things. They are working flat-out to
create some very nice Linux-based products, and they are being successful
at it, but they are in the bind that is familiar to so many embedded systems
developers: the product cycles are so short and the deadlines are so tight
that there just isn't time to spend on getting code upstream. That said,
they are trying and intend to try harder. We are starting to see some
results; for example, the NVIDIA
Tegra architecture code has just been posted for review and merging.
The Android developers seem to feel that they have been singled out for a
level of criticism which few other embedded vendors - including those
demonstrating much worse community behavior - have to deal with. It can be
a dismaying and demotivating thing. Your editor would like to suggest
that, at this point, the Android developers have heard and understood the
message that the community has tried to communicate to them. There is a
good chance that things will start to get better in the near future.
Perhaps it's time to back off a little bit and focus on helping them to get
their code merged when they get a chance to submit it.
As promised, this article has barely scratched the surface of what happened
at the 2010 Collaboration Summit. In particular, the large presence of
MeeGo (which had a separate, day-long session dedicated to it) has been
passed over, though some of that will be made up for in a separate
article. In general, it was a good collection of people who do not always
get a chance to mingle in the same hallway, all backed by the Linux
Foundation's seamless "it all just works" organization. Improved
collaboration within our community should indeed be the result of this
to post comments)