LWN.net Logo

Pointy-haired kernel hackers?

Don Marti attended Greg Kroah-Hartman's Linux Symposium talk on the kernel development process; he wrote an informative article (titled Linux contributor base broadens) about it. The article states:

In the latest kernel release, the most active 30 developers authored only 30% of the changes, while two years ago, the top 20 developers did 80% of the changes, he said. Kroah-Hartman himself is now doing more code reviewing than coding. "That's all I do, is read patches these days," he said.

An important part of this is that Greg presented this change as a good thing. The kernel has a far broader developer base than it once did, with patches for any given release coming from almost 1000 different people. We have a growing development community which is healthy and robust.

Seeing what the mainstream media makes of things can be great fun sometimes. This time around, ComputerWorld UK picked up Don's article, running the same text but giving it a new title: Are top Linux developers losing the will to code? Slashdot picked it up under that title, then Wired chimed in with Core Linux Developers Stuck In Middle Management Mode, complete with a picture of a necktie-wearing employee wielding a stapler and a telephone. The prize must go to the Jem Report's The coders and the talkers, though; this article breaks new ground completely:

Linus' [sic] job is leaning more towards spokesman than programmer. He's been a relatively effective manager up until now, but I think that effectiveness will begin to erode rapidly with time. The further you get away from the actual work, the less you are able to accurately judge the appropriateness of other people's work. You need to stay in the game -- you need to keep your skills in condition. If you don't, you might understand the theory pretty well, but you'll get further and further away from being in touch with its application. Linus has become more of a talker and less of a coder.

It seems we have trouble here. While we weren't looking, the Linux kernel drifted into a Dilbertesque realm and is now controlled by people who don't actually create software anymore. World Domination, it would seem, is now in grave doubt.

Or perhaps all of this is just silly nonsense, an extreme extrapolation taken from a couple of sentences spoken at a Linux developer conference.

If one wanted to investigate this subject further, a good starting place might be the 2.6.22 changelog; there one can see just how many patches our pointy-haired top-level maintainers have contributed over the latest development cycle. Here's a subset:

DeveloperPatches
Andi Kleen 70
Andrew Morton 79
David Miller 193
Greg Kroah-Hartman 14
James Bottomley 13
Linus Torvalds 20
Russell King 61

One could add more names to the list, but the end result would be about the same: the top-level kernel maintainers are not among the most prolific contributors (except maybe for David Miller - but, then, he's an exception in many regards), but neither are they absent from the game. They are still hacking on the code and cranking out the patches.

From some of the articles that have been posted, one might think that subsystem maintainers spend the rest of their time attending meetings and writing weekly reports. What they are actually doing is working with patches - lots of patches - and with developers. A subsystem maintainer must review code, decide whether it is appropriate for the kernel, and, if not, give the developer guidance on how to make it better. Incoming patch streams must be merged together, and decisions must be made on which ones are ready for any given development cycle. It is a task which requires the maintainers to keep their hands deep in the code. Subsystem maintainers who do not touch the code, and who do not maintain a deep understanding of how the kernel works are not likely to remain maintainers for very long.

So the broadening of the kernel development community - and the associated need for more work by subsystem maintainers - is not really costing us our best developers. They are not "losing the will to code." The things that cost us developers are elsewhere: a somewhat adversarial process which turns off some people, general burnout, or getting a job which takes their attention away from contributing to the community. All very normal stuff. In the Linux kernel community we may have our share of Dogberts, but we need not lose much sleep over the threat of pointy-haired subsystem maintainers bringing the process to a halt. Instead, they have helped the kernel development process to scale to a level beyond that of almost any other software development project anywhere; that is a good thing, not a sign of trouble.


(Log in to post comments)

office space

Posted Jul 12, 2007 1:44 UTC (Thu) by mattdm (subscriber, #18) [Link]

That's not just any old stock photo of a necktie-wearing employee wielding a stapler and a telephone, you know....

office space

Posted Jul 16, 2007 20:07 UTC (Mon) by jzbiciak (✭ supporter ✭, #5246) [Link]

Indeed... but it has Root! ;-)

Pointy-haired kernel hackers?

Posted Jul 12, 2007 7:27 UTC (Thu) by lipak (guest, #43911) [Link]

This movement towards code-review (and one might add, giving direction
to the various kernel sub-projects) by people as they grow older has
a natural counterpart in the academic world. As academicians get
more senior they take on projects with broader scope and do a lot of
"editorial work".

This leads some science-writers to conclude (erroneously) that the
best research is only done by young people. Tell that to Andrew Wiles
who put the keystone into the proof of Fermat's Last Theorem at the
age of 39/40.

I believe we can expect great stuff still to come from our ever-young
kernel hackers.

Kapil Paranjape.

Pointy-haired kernel hackers?

Posted Jul 12, 2007 9:59 UTC (Thu) by smitty_one_each (subscriber, #28989) [Link]

>If you don't, you might understand the theory pretty well, but you'll get further and further away from being in touch with its application. Linus has become more of a talker and less of a coder.

It's pretty obvious that, as soon as next year, Linus will have an office adjacent Andrew Tannebaum, as they theorize away. ;)

Pointy-haired kernel hackers?

Posted Jul 12, 2007 16:36 UTC (Thu) by kamil (subscriber, #3802) [Link]

I've heard that Andrew Tannenbaum is approaching retirement age, so Linus could even inherit his office :-).

Pointy-haired kernel hackers?

Posted Jul 12, 2007 17:40 UTC (Thu) by ortalo (subscriber, #4654) [Link]

It's not unsurprising in fact that external attendees simply do not understand the amazing *success* of that development project (as well as some others in our field btw). Please correct me if you think I'm wrong, but IMHO this kind of success has something to do with people capable and willing to use a modern direct and personal communication tool (the network) for their work (and certainly improving the tool while using it).
Imagine what kind of productivity and innovation improvement could emerge of *actual* application of the same stubborn individual advances to other areas (finance, law enforcement, public administration, manufacturing)... I am not so surprised that so many people do not want to understand what their (lack of?) role could be in such a future. (Note that I do not count among those that counts so I hope I am no biased; except by looking for more reasons to hope for my children future.)

Pointy-haired kernel hackers?

Posted Jul 12, 2007 19:29 UTC (Thu) by caitlinbestler (guest, #32532) [Link]

To take these criticisms to their logical extreme, Linux would be better off if *everybody* focused their energies on posting patches and nobody wasted any time reviewing them.

Pointy-haired kernel hackers?

Posted Jul 13, 2007 18:16 UTC (Fri) by vonbrand (subscriber, #4458) [Link]

... or integrating them into the kernel ;-)

Pointy-haired kernel hackers?

Posted Jul 14, 2007 8:20 UTC (Sat) by robbat2 (guest, #40105) [Link]

Missing here, is that the top-level maintainer also contribute to code outside of the kernel. Linus is putting a lot of work into Git still (such as the large contribution of submodules recently). IIRC gregkh is putting stuff into udev as well as the -stable series. Morton has -mm.

Copyright © 2007, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds