2.6.24 - some statistics

As of this writing, the 2.6.24 kernel is getting close to a release - though there is likely to be one more -rc version to look at first. The rate of change has slowed significantly, though, and the final regressions are being chased down. So it seems like a suitable time to look at the patches which went into this kernel and where they came from.

This is, in many ways, a record-breaking development cycle. Over 10,000 individual changesets have been merged this time around, with a net growth of almost 300,000 lines of code. 950 developers contributed this code; of those, 358 contributed just one patch. By comparison, the previous cycle (2.6.23) merged some 6200 patches from about 860 developers. Given that, it's not surprising that the 2.6.24 cycle has been a little longer than some of its predecessors.

Without further ado, here is the list of top contributors to this kernel:

Most active 2.6.24 developers

By changesets Thomas Gleixner 362 3.6% Bartlomiej Zolnierkiewicz 205 2.0% Adrian Bunk 190 1.9% Ralf Baechle 176 1.8% Pavel Emelyanov 146 1.5% Ingo Molnar 141 1.4% Tejun Heo 138 1.4% Paul Mundt 131 1.3% Johannes Berg 119 1.2% Al Viro 116 1.2% Takashi Iwai 115 1.1% Jeff Garzik 107 1.1% David S. Miller 102 1.0% Matthew Wilcox 97 1.0% Jens Axboe 89 0.9% Krzysztof Helt 89 0.9% Stephen Hemminger 86 0.9% Rusty Russell 86 0.9% Alan Cox 85 0.8% Herbert Xu 84 0.8%

By changed lines Thomas Gleixner 46358 5.9% Zhu Yi 35133 4.5% Auke Kok 25861 3.3% Michael Buesch 24480 3.1% Ivo van Doorn 22178 2.8% Matthew Wilcox 20416 2.6% Adrian Bunk 19050 2.4% Larry Finger 15003 1.9% David S. Miller 14315 1.8% Andy Gospodarek 13814 1.8% Nathanael Nerode 12821 1.6% Jeff Dike 11103 1.4% Johannes Berg 10118 1.3% Ralf Baechle 9555 1.2% Scott Wood 9328 1.2% Krzysztof Helt 8162 1.0% Kumar Gala 8002 1.0% Jeff Garzik 7689 1.0% David Gibson 7284 0.9% Michael Hennerich 7181 0.9%

By either method of counting, Thomas Gleixner comes out at the top of the list by virtue of his work on the i386/x86_64 architecture merger. Bringing those architectures together and making the result work well was a huge job; this effort will continue into future development cycles. (For the curious, simply renamed files were not counted as "changed lines" in the generation of these numbers). Note that many of these patches also carry a signoff by Ingo Molnar, but git only stores the name of a single "author" for a changeset.

Other contributors of large numbers of changesets in 2.6.24 include Bartlomiej Zolnierkiewicz (lots of IDE driver patches), Adrian Bunk (cleanups all over the kernel tree), Ralf Baechle (MIPS architecture work), Pavel Emelyanov (mostly network and PID namespaces), Tejun Heo (serial ATA and a number of sysfs cleanups), Johannes Berg (wireless networking), and Al Viro (mostly annotation patches and related fixes). If one looks at the number of changed lines, the list of developers changes almost completely: Zhu Yi (iwlwifi driver), Auke Kok (e1000 driver), Michael Buesch (wireless networking and the b43 driver), Ivo van Doorn (rt2x00 wireless driver), Matthew Wilcox (SCSI, especially advansys and sym53c8xx drivers), Adrian Bunk (cleanups and code deletions), Larry Finger (mainly addition of the b43 legacy driver), and David Miller (networking and SPARC64).

If one assigns developers' contributions to employers and totals the results, the following numbers emerge (note that these tables have been updated since initial publication to fix an error):

Most active 2.6.24 employers

By changesets (None) 1417 14.1% (Unknown) 1108 11.1% Red Hat 1045 10.4% IBM 819 8.2% Novell 680 6.8% Intel 446 4.5% linutronix 369 3.7% Oracle 240 2.4% SWsoft 212 2.1% CERN 205 2.0% Movial 190 1.9% Linux Foundation 190 1.9% MIPS Technologies 176 1.8% Renesas Technology 140 1.4% (Academia) 132 1.3% Freescale 126 1.3% MontaVista 122 1.2% Analog Devices 115 1.1% (Consultant) 112 1.1% NetApp 101 1.0%

By lines changed (None) 140730 18.0% (Unknown) 121511 15.5% Intel 114990 14.7% Red Hat 58858 7.5% IBM 51777 6.6% linutronix 47968 6.1% Novell 29856 3.8% Movial 19093 2.4% Freescale 15262 1.9% Analog Devices 14971 1.9% MIPS Technologies 11726 1.5% SWsoft 8331 1.1% Linux Foundation 7917 1.0% Oracle 7777 1.0% Atmel 7125 0.9% CERN 6618 0.8% Renesas Technology 6414 0.8% Google 6373 0.8% MontaVista 6026 0.8% NetApp 5620 0.7%

In many ways, these lists look similar to those posted for past kernels. But there are a few things which jump out this time around:

• Intel has made it to the top of the "by lines changed" list - and not just by a little bit. This happened by virtue of the work done by four of the top-20 developers, but also by dozens of others who contributed to the 2.6.24 kernel. Intel has a lot of people working on the kernel, many of whom spend little time in the limelight.

• Movial found its way onto the list for the first time as a result of having hired a very active developer.

• The amount of work done by people known to be hacking on their own time has grown a bit. This change is mostly a result of more complete information on our side - many developers have moved out of the "unknown" category. Quite a bit of the no-employer work this time around was done on the wireless networking tree; since much of the interesting work in this area currently involves reverse engineering, perhaps it is not surprising that relatively few companies are willing to sponsor it.

All told, some 130 distinct employers were identified for the contributors to 2.6.24. That is a lot of companies to be working on one body of code.

Looking at the Signed-off-by headers of patches is always interesting; if one removes the signoffs added by the authors themselves, what is left is a list of the gatekeepers - those who channel the code into the mainline. The people who signed off on the most patches which they did not write are:

Sign-offs in the 2.6.24 kernel

By developer Andrew Morton 1679 17.6% David S. Miller 894 9.4% Jeff Garzik 631 6.6% Ingo Molnar 626 6.6% John W. Linville 413 4.3% Mauro Carvalho Chehab 367 3.9% Greg Kroah-Hartman 337 3.5% Paul Mackerras 305 3.2% Jaroslav Kysela 284 3.0% James Bottomley 260 2.7% Linus Torvalds 250 2.6% Thomas Gleixner 216 2.3% Bryan Wu 166 1.7% Takashi Iwai 115 1.2% Jens Axboe 113 1.2% Len Brown 113 1.2% Avi Kivity 107 1.1% Roland Dreier 107 1.1% Ralf Baechle 96 1.0% Adrian Bunk 88 0.9%

By employer Red Hat 2935 30.2% Linux Foundation 1929 19.9% (None) 823 8.5% (Unknown) 736 7.6% Novell 636 6.6% IBM 584 6.0% Intel 318 3.3% linutronix 216 2.2% Analog Devices 175 1.8% SGI 141 1.5% Oracle 133 1.4% Cisco 107 1.1% Qumranet 107 1.1% NetApp 106 1.1% MIPS Technologies 96 1.0% Movial 88 0.9% (Consultant) 85 0.9% Renesas Technology 84 0.9% Cendio 43 0.4% CERN 40 0.4%

There are not a lot of changes here from previous development cycles. While quite a few developers add signoffs to code and pass it on, they work for a relatively small number of companies - 7 employers account for 70% of the non-author signoffs.

Finally, given that we are starting a new year, it is worth taking a quick look back at the entirety of 2007. In 2007, Linus merged just over 30,000 changesets (more than 80 per day, every day) from 1900 developers working for (at least) 200 companies. All told, they changed over 2 million lines of code, growing the kernel by more than 750,000 lines. The kernel developers are, in other words, touching over 5,000 lines of code every day - that is a high rate of change.

The top contributors over the course of the year (by changesets) were:

Top contributors in 2007

By developer Ralf Baechle 507 1.7% Thomas Gleixner 485 1.6% David S. Miller 468 1.6% Adrian Bunk 439 1.5% Tejun Heo 394 1.3% Ingo Molnar 351 1.2% Paul Mundt 351 1.2% Al Viro 337 1.1% Bartlomiej Zolnierkiewicz 330 1.1% Andrew Morton 319 1.1% Stephen Hemminger 302 1.0% Patrick McHardy 277 0.9% Alan Cox 270 0.9% Takashi Iwai 269 0.9% Trond Myklebust 256 0.9% David Brownell 254 0.8% Avi Kivity 229 0.8% Jeff Dike 227 0.8% Jeff Garzik 216 0.7% Jean Delvare 215 0.7%

By employer (None) 4881 16.2% Red Hat 3441 11.4% (Unknown) 2933 9.7% IBM 2379 7.9% Novell 2054 6.8% Intel 1060 3.5% Linux Foundation 784 2.6% Oracle 677 2.2% (Consultant) 631 2.1% MIPS Technologies 507 1.7% linutronix 507 1.7% Renesas Technology 394 1.3% (Academia) 392 1.3% SWsoft 384 1.3% SGI 368 1.2% MontaVista 342 1.1% CERN 330 1.1% Freescale 291 1.0% NetApp 279 0.9% Astaro 277 0.9%

It should be noted that the employer numbers are more approximate than usual. Some developers changed employers in 2007, but LWN, as a matter of policy, does not maintain a database of developers and their employers over time. Still, the picture is relatively constant - the same companies continue to contribute approximately the same percentage of the patches going into the kernel over relatively long periods of time.

Overall, the picture that results from all these numbers is one of a widespread and healthy development community. There appears to be no shortage of jobs for kernel developers, but also room for those who work outside of the office. The kernel truly is a common resource, with literally thousands of people working to improve it. And it shows no signs of slowing down anytime soon.

Your editor would like to profusely thank Greg Kroah-Hartman for his help in improving these statistics.

---

(Log in to post comments)

Andrew Morten has a Linux Foundation email address, but isn't he paid by Google? The
"Sign-offs in the 2.6.24 kernel" figure certainly seems to contradict that...

The Nokia attribution is also bogus, all of my changes should be attributed to Renesas
Technology instead of Nokia. There was a mismatch in earlier versions of the script, which
Greg's version still seems to suffer from. :-)

Thanks for this, Jonathan, very interesting.

I wonder if it is worthwhile to continue tracking changes to the kernel after it is released
into the wild? It should (AFAICS) be straightforward to track what patches are needed to what
code, for what reason (security, known exploit, race condition, etc.) and from what source.
That would make pretty interesting reading. And any given kernel has only quite a short life
until the next is ready. For myself I'd like the tracking to be as long-term as possible so as
to see how long a tail exists.

Dave

I was wondering how on earth it could happen that I came into the TOP20 at "Sign-offs in the
2.6.24 kernel".

The problem is that the following statement sn't completely true:
"Looking at the Signed-off-by headers of patches is always interesting; if one removes the
signoffs added by the authors themselves, what is left is a list of the gatekeepers - those
who channel the code into the mainline."

But this also happens when Andrew merges patches.

E.g. in current -mm the header of maps4-add-proc-kpagecount-interface.patch reads:

<--  snip  -->

From: Matt Mackall <mpm@selenic.com>

This makes physical page map counts available to userspace. Together
with /proc/pid/pagemap and /proc/pid/clear_refs, this can be used to
monitor memory usage on a per-page basis.

[bunk@stusta.de: make struct proc_kpagemap static]
Signed-off-by: Matt Mackall <mpm@selenic.com>
Cc: Jeremy Fitzhardinge <jeremy@goop.org>
Cc: David Rientjes <rientjes@google.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Cc: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

<--  snip  -->

If this patch will get into Linus' tree your statistics will wrongly list me as a gatekeeper
for this patch.