Some development statistics for 2.6.27 [LWN.net]

By Jonathan Corbet
October 7, 2008

It's that time of the development cycle again: the 2.6.27 kernel, if not yet released by the time you read this, will be shortly. Various other LWN articles have looked at features found in this release; here we will look at where that code came from.

As of 2.6.27-rc9, a total of 10,604 non-merge changesets had been added to the mainline for the 2.6.27 kernel; those patches added a total of 826,000 lines of code while removing 608,000, for a net growth of 217,000 lines. There were 1,109 developers who contributed to 2.6.27, representing over 150 employers. 376 of those developers contributed a single patch during this development cycle.

The most active developers for 2.6.27 were:

Most active 2.6.27 developers

By changesets

Ingo Molnar 238 2.2%

Bartlomiej Zolnierkiewicz 235 2.2%

Adrian Bunk 221 2.1%

David S. Miller 206 1.9%

Alan Cox 196 1.8%

Yinghai Lu 192 1.8%

Jeremy Fitzhardinge 162 1.5%

Tomas Winkler 128 1.2%

Ben Dooks 120 1.1%

Jean Delvare 113 1.1%

Steven Rostedt 108 1.0%

Harvey Harrison 105 1.0%

Pavel Emelyanov 103 1.0%

Thomas Gleixner 101 1.0%

Jean-Francois Moine 89 0.8%

Lennert Buytenhek 88 0.8%

Hans Verkuil 81 0.8%

Joerg Roedel 81 0.8%

Arnd Bergmann 76 0.7%

David Brownell 75 0.7%

By changed lines

Paul Mackerras 138374 12.1%

David Woodhouse 44759 3.9%

Jean-Francois Moine 41157 3.6%

Adrian Bunk 35160 3.1%

Artem Bityutskiy 34545 3.0%

Luis R. Rodriguez 31825 2.8%

Sam Ravnborg 27443 2.4%

Karsten Keil 24674 2.2%

Russell King 22861 2.0%

Eilon Greenstein 19470 1.7%

Alan Cox 16957 1.5%

Felipe Balbi 16287 1.4%

Kumar Gala 14490 1.3%

David Brownell 12551 1.1%

Ralf Baechle 11057 1.0%

Lennert Buytenhek 9735 0.9%

David S. Miller 8621 0.8%

Juergen Beisert 8516 0.7%

Steven Rostedt 8455 0.7%

Ben Dooks 8399 0.7%

On the changeset side, Ingo Molnar ended up on top by virtue of the creation of large numbers of mostly x86-related changes, including a big subarchitecture reorganization; Ingo's count also includes the addition of ftrace, though much of that code was written by others. Bartlomiej Zolnierkiewicz continues to rework the old IDE layer, and Adrian Bunk, as always, energetically cleans up code all over the tree. David Miller's total includes the multiqueue networking code and a lot of other changes; Alan Cox did a lot of TTY work and big kernel lock removal.

Your editor was disappointed to come in at #23, and, thus, off the bottom of the table. Time to send in some quick white space fixes. More seriously, though, it's worth noting that there are relatively few patches of the "trivial change" variety in the mix this time around.

If we look at changed lines, Paul Mackerras comes out on top as the result of a single patch removing the obsolete ppc architecture. David Woodhouse reworked the management of firmware throughout the driver tree. Jean-François Moine brought the GSPCA webcam drivers into the tree, then put vast amounts of effort into cleaning them up. Artem Bityutskiy added the UBIFS flash filesystem, and Luis Rodriguez merged the ath9k wireless driver.

If we look at the companies behind this work, we get the following results (note that, as always, these results are somewhat approximate):

Most active 2.6.27 employers

By changesets

(None) 1925 18.2%

Red Hat 1405 13.2%

(Unknown) 921 8.7%

IBM 791 7.5%

Intel 605 5.7%

Novell 586 5.5%

Movial 234 2.2%

SGI 197 1.9%

(Consultant) 193 1.8%

Sun 184 1.7%

XenSource 165 1.6%

Parallels 157 1.5%

Oracle 148 1.4%

Marvell 143 1.3%

Fujitsu 138 1.3%

AMD 129 1.2%

Renesas Technology 125 1.2%

linutronix 121 1.1%

Simtec 119 1.1%

(Academia) 108 1.0%

By lines changed

IBM 207215 18.1%

(None) 129998 11.4%

Red Hat 109970 9.6%

(Unknown) 108878 9.5%

Nokia 52022 4.5%

Novell 49944 4.4%

(Consultant) 46529 4.1%

Broadcom 43438 3.8%

Atheros 38212 3.3%

Movial 35439 3.1%

Intel 32887 2.9%

Freescale 25511 2.2%

SGI 23444 2.0%

Marvell 20967 1.8%

Renesas Technology 15723 1.4%

MIPS Technologies 15701 1.4%

Pengutronix 13334 1.2%

Atmel 10786 0.9%

Analog Devices 10725 0.9%

Sun 9176 0.8%

There are not too many surprises in this table - in particular, the list of companies at the top tends not to change very much. That said, a few things are worthy of note. One is that Sun Microsystems has made its first appearance on this list. People complain about this company, but Sun's engineers have been quietly fixing things all over the tree. Broadcom is another company with a mixed reputation in the Linux community, but Broadcom is happy to provide support for some of its network adapters. Nokia's strong showing in the lines-changed table results primarily from the contribution of the UBIFS filesystem.

The most welcome change, though, is the first appearance of Atheros on this list. Atheros is a company which has quickly moved from a position of complete non-cooperation to one of supporting all of its hardware in the mainline kernel. To say that this is an encouraging development would be an understatement.

All told, the 2.6.27 development cycle shows that the process continues at full pace in a seemingly healthy state. Developers from all over the industry are all working together to make the kernel better for all. The number of companies which see participation in the process as being in their interest is growing, as is the number of developers who contribute patches. The Linux kernel, it seems, is in good shape.

Index entries for this article
Kernel	Releases/2.6.27

to post comments

Some development statistics for 2.6.27

Posted Oct 9, 2008 6:41 UTC (Thu) by pabs (subscriber, #43278) [Link] (2 responses)

Quick correction, IIRC the Atheros AR6000 WiFI chip used in the OpenMoko FreeRunner does not yet have a chip in mainline. It also cannot have master or monitor mode added because it uses full-mac firmware that Atheros don't seem to be updating. Details here:

http://lists.openmoko.org/nabble.html#nabble-td1095866

Some development statistics for 2.6.27

Posted Oct 9, 2008 9:54 UTC (Thu) by nhippi (subscriber, #34640) [Link] (1 responses)

I woudn't blame only atheros, openmoko's track record of mainlining drivers is not very good outside wifi either. Which, I don't blame them for. Getting good battery life and stable reliable phonecall UI is more important than having everything nicely contributed to mainline.

Some development statistics for 2.6.27

Posted Oct 9, 2008 12:57 UTC (Thu) by TRS-80 (guest, #1804) [Link]

Their track record of battery life and stable reliable phonecall UI isn't that great either.

Some development statistics for 2.6.27

Posted Oct 9, 2008 10:05 UTC (Thu) by dwmw2 (subscriber, #2063) [Link] (3 responses)

dwmw2 changed lines: 44759
Intel changed lines: 32887

How's that work? Some other Intel employee did _negative_ lines of work? Even removing code doesn't work like that, as Paulus demonstrated.

(OK, so some of the firmware stuff was done while I was still at the other place or waiting for Intel HR to take ownership of my soul, but still...)

Negative work

Posted Oct 9, 2008 12:45 UTC (Thu) by corbet (editor, #1) [Link] (2 responses)

Our tool works from the date associated with the first git commit - that's about the only date available, really. I've not looked in detail, but I assume that much of the firmware work, by this measure, predates the transfer of your soul to its new owner, even though it hit the mainline afterward.

It could be that we have that date wrong by a bit (drop me a note if you would like it to be more precise).

Negative work

Posted Oct 9, 2008 12:57 UTC (Thu) by dwmw2 (subscriber, #2063) [Link]

About June 6th, I believe. But the precise date isn't massively important — I mostly just wanted to make sure you have it updated for future reference. Thanks.

Negative work

Posted Oct 9, 2008 15:38 UTC (Thu) by jengelh (subscriber, #33263) [Link]

I'd rather use a tag spec v2.6.26.. instead of doing it date-based. It might not be too accurate, but may give an interesting statistic nevertheless.

Some development statistics for 2.6.27

Posted Oct 9, 2008 11:17 UTC (Thu) by hppnq (guest, #14462) [Link] (9 responses)

Why the employers statistics? I can see that they are relevant to some people at some companies (they always are), but personally I would find it much more interesting to see geographical data, for instance, or a histogram of submit times.

If only because the employer statistics really don't seem to mean much more than the very obvious: X developers sent in Y accepted patches from Z mail addresses. It should then be presented as such, no?

Some development statistics for 2.6.27

Posted Oct 9, 2008 12:24 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (8 responses)

"Why the employers statistics? I can see that they are relevant to some people at some companies"

It is much more than that. Individuals can influence and promote the hardware vendors which contribute towards their operating system of choice compared to others who refuse to cooperate or actively work against it.

Customers who rely on support contracts from commercial distributions know who is likely to give them higher value by moving free software in the direction that they would like.

More importantly, given the increasingly high influence of commercial vendors in the world of free software, a better of understanding of the commercial ecosystem is critical to take advantage of our better strengths and address our weaknesses. It is statistics such as these that help us understand how Linux kernel is progressing in attracting the participation of embedded manufacturers which is an area of current interest. Overall, I and I am sure many others would as individuals find these stats important.

Some development statistics for 2.6.27

Posted Oct 9, 2008 21:26 UTC (Thu) by hppnq (guest, #14462) [Link] (7 responses)

Individuals can influence and promote the hardware vendors which contribute towards their operating system of choice compared to others who refuse to cooperate or actively work against it.

That is exactly what I meant. It's the kind of conclusion you cannot draw from these statistics. There are many reasons for that, but let's take a very obvious one: how about out-of-tree drivers?

Customers who rely on support contracts from commercial distributions know who is likely to give them higher value by moving free software in the direction that they would like.

Again this is a remarkably strong statement, not in any way supported by the stats. Last time I looked, Free Software was still moved by the developers and not by the customers, if only because the Free Software "ecosystem" -- which seems to have replaced the ancient "community" -- contains a lot of other users and proponents of Free Software. Note that this is actually reflected in the stats for (even) the Linux kernel, more than fifteen years after it was started.

I am not complaining about the involvement of vendors and customers, but it would be interesting to see how many kernel hackers switched jobs in the middle of a project, or who started a project before joining a certain company, or worse, who perhaps stopped working on it after joining a certain company.

More importantly, given the increasingly high influence of commercial vendors in the world of free software, a better of understanding of the commercial ecosystem is critical to take advantage of our better strengths and address our weaknesses.

I would say that the biggest strength of Free Software is its almost complete independence of the whims of vendors and to me it would be difficult to find a weakness in it, in the absence of a specific goal other than world domination. ;-)

Some development statistics for 2.6.27

Posted Oct 9, 2008 23:53 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (6 responses)

"There are many reasons for that, but let's take a very obvious one: how about out-of-tree drivers?"

Depends on why they are out-of-tree. Is it code quality? licensing? The vendor supported drivers with proper licensing and good quality code tends to be in upstream Linux kernel.

"Last time I looked, Free Software was still moved by the developers and not by the customers"

Then you need to look again. For the Linux kernel (and several other successful projects), a large amount of code is driven by vendors which are representing customer requirements. This is part of the value that hardware vendors and commercial Linux distributions are offering. Also customers in some instances are directly participating in the development as well.

"I would say that the biggest strength of Free Software is its almost complete independence of the whims of vendors and to me it would be difficult to find a weakness in it, in the absence of a specific goal other than world domination. ;-)"

The views of vendors started to matter long since when they became participants. Influence is different from lock-in however. Besides, world-domination is indeed a goal and participation of embedded vendors or lack-of has been covered in many articles in LWN as well as reaching out to the needs of some of the Asian vendors including cultural differences so it is quite clear that the Linux world has a good understanding of the mutual benefit from vendor participation and has been working to address any short comings.

Some development statistics for 2.6.27

Posted Oct 10, 2008 11:45 UTC (Fri) by hppnq (guest, #14462) [Link] (5 responses)

If you would like to conclude anything about the Free Software market from these numbers, please knock yourself out. In the end, of course, it is not the statistics that are actually going to achieve world domination. It will still be the developers. It is their code. The great thing is, I can take it, and I can use it freely.

But by your reasoning, if a certain torvalds, cox, viro, mingo or davem (to name just a few) decide to go work for whitehouse.gov because of the lovely climate in DC, Linux might all of a sudden be an OS sponsored by the US government!

That's why I find these statistics and their interpretations amusing, nothing more, nothing less. It is certainly not rock solid science. But people will try to sell it like that, of course.

If you would simply look at the statistics for the last couple of releases, you would see that there is a quite limited number of recurring companies that by and large seem to contribute quite a bit, and that we all know, and a huge contribution that cannot be categorized at all. Although undoubtedly all kinds of companies are nowadays much more involved in some way in Linux kernel development, which is great, the process is still very much a triumphant example of Free Software development, which is greater.

Some development statistics for 2.6.27

Posted Oct 10, 2008 13:20 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link] (4 responses)

U.S government is already sponsoring Linux development to some extend - SELinux for example. That's a good thing. It is not just LWN but also others Linux Foundation publishing such statistics. Obviously people are finding them useful. If you don't, well then, feel free to ignore it.

Some development statistics for 2.6.27

Posted Oct 10, 2008 14:00 UTC (Fri) by hppnq (guest, #14462) [Link] (3 responses)

U.S government is already sponsoring Linux development to some extend - SELinux for example.

I don't see US government in the list, do you? You seem to miss the point though.

It is not just LWN but also others Linux Foundation publishing such statistics.

Why else would I raise the question? You should mention Novell also, because LWN, Novell and Linux Foundation all published the same results some time back. Here is the press release. Interesting read.

Obviously people are finding them useful.

Obviously.

Some development statistics for 2.6.27

Posted Oct 10, 2008 16:22 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

"I don't see US government in the list, do you?"

Not in this kernel release but reports looking at metrics over a longer period of time having mentioned this. Greg KH's talked about NSA sponsoring Xorg development as well. Those patches are part of the same SELinux effort.

http://www.kroah.com/log/linux/lpc_2008_keynote.html

"You seem to miss the point though"

You agree that others find it useful so I don't understand your opposition.

Some development statistics for 2.6.27

Posted Oct 11, 2008 16:47 UTC (Sat) by hppnq (guest, #14462) [Link] (1 responses)

My original point about the US government was not terribly difficult: if developer X leaves company A and joins B, what does that mean in terms of sponsorship of the projects involved? A somewhat related question would be: how does this influence statistics like the one given here? The added value of a publication like LWN, of which I am a customer, is to say something clever about this. But as I said, personally I am more interested in submit time statistics.

My other point was about the noble art of handwaving and the eagerness with which these statistics will be used to "prove" all kinds of things, some of which are not even present in the data, and I think you have illustrated that quite nicely: for instance, the fact that SELinux development also happens outside the scope of Linux kernel development is a nice example of the relative insignificance of these statistics.

Some development statistics for 2.6.27

Posted Oct 11, 2008 22:27 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link]

If a different company steps forward to sponsor the developers, then it shows they are interested in the development and want to influence it. These metrics are presented because they are often quite insightful especially if you are able to see the bigger picture over a period of time.

The single kernel release statistics are themselves useful as well but for other reasons. It is a matter of how well you use the data. If everybody worries about how things can be potentially misinterpreted, nothing will get done. Everything is aware that metrics such as these have that potential. Nothing new can be said about that.

Some development statistics for 2.6.27

Posted Oct 9, 2008 18:14 UTC (Thu) by aegl (guest, #37581) [Link] (4 responses)

It looks like we did a lot more "post-rc1" merging/churn in this release (compared to the three preceeding releases ... I'm too lazy to look back any further right now).

Going from v2.6.23 to v2.6.24 we had:
$ git diff -C --stat --summary v2.6.23 v2.6.24-rc1 | grep "files changed"
8982 files changed, 653530 insertions(+), 370352 deletions(-)
$ git diff -C --stat --summary v2.6.24-rc1 v2.6.24 | grep "files changed"
2631 files changed, 52462 insertions(+), 42568 deletions(-)

post -rc1 insertions about 8% of those in the merge window.

Going from v2.6.24 to v2.6.25 we had:
$ git diff -C --stat --summary v2.6.24 v2.6.25-rc1 | grep "files changed"
8780 files changed, 700202 insertions(+), 358222 deletions(-)
$ git diff -C --stat --summary v2.6.25-rc1 v2.6.25 | grep "files changed"
2846 files changed, 75547 insertions(+), 45193 deletions(-)

post -rc1 insertions about 11% of those in the merge window.

Going from v2.6.25 to v2.6.26 we had:
$ git diff -C --stat --summary v2.6.25 v2.6.26-rc1 | grep "files changed"
7519 files changed, 491515 insertions(+), 350262 deletions(-)
$ git diff -C --stat --summary v2.6.26-rc1 v2.6.26 | grep "files changed"
2811 files changed, 73196 insertions(+), 35199 deletions(-)

post -rc1 insertions about 15% of those in the merge window.

And for this release
$ git diff -C --stat --summary v2.6.26 v2.6.27-rc1 | grep "files changed"
8613 files changed, 623699 insertions(+), 501755 deletions(-)
$ git diff -C --stat --summary v2.6.27-rc1 HEAD | grep "files changed"
7116 files changed, 289097 insertions(+), 192860 deletions(-)

post -rc1 insertions about 46% of those in the merge window.

This does not look to be a good trend!

Some development statistics for 2.6.27

Posted Oct 10, 2008 5:13 UTC (Fri) by iabervon (subscriber, #722) [Link] (2 responses)

There's no reason that most of the changes for 2.6.x should be in 2.6.x-rc1; rather, most of the changes should be submitted before 2.6.x-rc1 is released, but Linus is likely to want to sit on some of them, ask for some of them to be merged by their authors and reposted, etc. to make -rc1 less chaotic. I haven't come up with a way to get overall statistics, but:

$ git shortlog --no-merges --since=jul.28 v2.6.26..v2.6.27

is a start (looking at the changes where the commit that ended up in the official history was made after the end of the merge window).

Some development statistics for 2.6.27

Posted Oct 10, 2008 15:55 UTC (Fri) by aegl (guest, #37581) [Link] (1 responses)

"There's no reason that most of the changes for 2.6.x should be in 2.6.x-rc1"

The current kernel development model is that all new code for the next release be merged during the two week merge window. It should have been submitted before the merge window so that it see some test time in linux-next or -mm, and so that merge and integration issues are sorted out before being sent to Linus. Post-rc1 is supposed to be for bug-fixes and regression. I don't think that those should require 45% of the code changes that were needed for the original features (if they do, it would seem to indicate that the features were not really ready to be merged).

At this year's kernel summit Linus admitted to being a big softie when it comes to accepting merges after -rc1 ... it seems that developers are taking advantage of this more and more each release.

Some development statistics for 2.6.27

Posted Oct 10, 2008 21:26 UTC (Fri) by iabervon (subscriber, #722) [Link]

The process as of 2.6.27-rc1 was that code would be written and tested before the merge window, submitted during the merge window, and merged whenever Linus felt like it; there wasn't a process in place to submit changes intended for 2.6.27 before 2.6.26 came out. Plus, he was a big softie, some developers weren't clear on the process. Also, not just any bug fix is acceptable after the merge window; there ended up being push back on a ton of code post-merge-window for 2.6.27 that was fixing features introduced in 2.6.25 and 2.6.26.

IIRC, there was talk at the summit about going to a model where everything for 2.6.28 must be submitted before 2.6.27 came out, by way of -next. But Linus didn't want to have to deal with pull requests for 2.6.28 while he's trying to get 2.6.27 released.

Some development statistics for 2.6.27

Posted Oct 22, 2008 16:52 UTC (Wed) by Duncan (guest, #6647) [Link]

It's worth noting (as covered in previous LWN kernel pages) that a lot of
arch reorg (including that 12 percent mentioned in TFA) happened this
round, and that much of it went in between rc1 and rc2. My testing
happened to be affected by this, so I know. Anyway, much of the reorg
was pretty much maintenance, simply moving headers around, and it needed
to wait until all the real changes had gone in, IOW until after the rc1
merge window closed, in ordered not to screw them up. Well, it could
have gone in earlier, but it would have made merging and testing MUCH
more complicated than necessary. so there was good reason for Linus to
hold this for rc2. But most of this wasn't really additions, more
renames, tho AFAIK it would have shown up in the stats as additions.

But this was pretty much a single-shot deal. I'm not sure the degree to
which other archs will be reorganizing as x86 did, or if so how much of
that has been completed for .27 and how much remains, but certainly, the
degree to which the x86 code moved around this time shouldn't continue to
happen every time, so it's single-shot in that regard, and even if other
archs remain to be similarly reorged, things should settle down in a
release or two, when it's all done.

Duncan