Kernel development

Kernel release status

The current development kernel is 3.0-rc7, released on July 11. "I think I said -rc6 might be the last -rc. I lied. Things have been pretty quiet, but there's enough new stuff here that I wanted to do another -rc, and we still have some issues with the RCU changes causing problems when RCU events happen before the scheduler has been fully initialized etc. So -rc7 is out there." See the full changelog for all the details.

Stable updates: the 2.6.32.43 and 2.6.33.16 stable kernel updates were released on July 13; 2.6.39.3 came out on July 8. All three contain the usual list of important fixes - quite a few of them, in the 2.6.39.3 case.

Comments (none posted)

Quotes of the week

-- Andrew Morton

-- Linus Torvalds

Comments (1 posted)

What will that kernel be called?

When Linus announced the 3.0-rc1 kernel at the end of May, he noted that it was actually identifying itself as "3.0.0". There were enough scripts out there expecting that third digit to exist that he didn't want to break them from the outset. Still, he was hopeful that the final ".0" could go away:

Fast forward to 3.0-rc7, and the kernel is still identifying itself as 3.0.0. While hard information is relatively scarce, it seems that there are still some programs and scripts out there that will break without that final number. In a sense, the three-number versioning scheme has become a part of the kernel ABI over the years. One can argue that anything which breaks was written in a silly way to begin with, but the goal is still to avoid breaking user space. So, while no definitive answer has been given, it looks like we're going to have three-number mainline releases for some time. It would be most surprising to see the stable updates starting with anything other than 3.0.1, though.

There is one related question, though: what about programs that cannot cope with anything other than "2.6.x" as the kernel version number? Evidently such programs exist. Saying that kernel releases must start with "2.6" forevermore stretches even the firmest commitment to binary compatibility; it's not something that is going to happen. In an attempt to make life easier for people stuck with such programs, Andi Kleen has implemented a new system personality that pretends the 3.0 change never happened. If a program is run under that personality, it will think that the kernel version is 2.6.40; 3.1 will look like 2.6.41, etc.

Andi says "I know this is somewhat ugly, but I didn't find a better workaround, and compatibility to existing programs is important." As of this writing, though, the patch has not made it into the mainline.

Comments (12 posted)

Who wrote 3.0 - from two points of view

Linus Torvalds had hoped to release the 3.0 kernel after -rc6, but reality, as is its wont, intervened; thus, 3.0-rc7 was released on July 11. That probably is the last development release for 3.0, though. Tradition dictates that we take a look at the contributor statistics for this development cycle, which we will now do.

This kernel release inaugurates the beginning of the 3.x series of kernels. As has been mentioned many times here, there is nothing particularly special about the 3.0 release; it has been, in many ways, a relatively boring development cycle. But it still provides a good opportunity to look back over a longer period of time. But, before doing that, we'll start with this cycle, which has, as of 3.0-rc7, seen 9,007 changesets contributed by 1,110 developers. The kernel grew 113,000 lines in this development cycle - a relatively modest figure by contemporary standards.

The most active developers during this cycle were:

Most active 3.0 developers

By changesets K. Y. Srinivasan 343 3.8% David S. Miller 176 2.0% Dan Williams 149 1.7% Jonathan Cameron 119 1.3% Takashi Iwai 108 1.2% Mark Brown 91 1.0% Johannes Berg 84 0.9% Peter Zijlstra 80 0.9% Sage Weil 79 0.9% Tejun Heo 78 0.9% Joe Perches 77 0.9% Michał Mirosław 77 0.9% Konrad Rzeszutek Wilk 76 0.8% Jamie Iles 75 0.8% Alex Deucher 71 0.8% Artem Bityutskiy 69 0.8% Steven Rostedt 66 0.7% Mike Frysinger 63 0.7% Sujith Manoharan 62 0.7% Avi Kivity 58 0.6%

By changed lines Dan Williams 82466 9.1% Larry Finger 74643 8.3% Dmitry Kravkov 38960 4.3% Vasanthakumar Thiagarajan 33618 3.7% Mauro Carvalho Chehab 26815 3.0% Bing Zhao 25576 2.8% Ralph Metzler 19933 2.2% Takahiro Hirofuchi 19318 2.1% Chaoming Li 14743 1.6% Jonathan Cameron 14574 1.6% Chris Metcalf 12144 1.3% Luis R. Rodriguez 11443 1.3% Dave Jiang 11006 1.2% Wolfram Sang 9886 1.1% K. Y. Srinivasan 9709 1.1% Mark Brown 9127 1.0% Arend van Spriel 7667 0.8% Kenji Toyama 7528 0.8% Alan Cox 7449 0.8% Takashi Iwai 7410 0.8%

K. Y. Srinivasan topped the list of changeset contributors with a massive set of cleanups to the Microsoft HV driver in the staging tree; it's impressive to see how much cleanup less than 15,000 lines of code can require. David Miller made a lot of changes in the networking subsystem; some were warning fixes and such, while others were more substantial. Dan Williams contributed Intel's "isci" storage driver, merged in 3.0-rc6. Jonathan Cameron contributed a lot of work to rationalize the industrial I/O (iio) subsystem and prepare it for an eventual merge into the mainline. Takashi Iwai continues to do large amounts of work in the ALSA sound driver subsystem.

The isci driver put Dan Williams at the top of the "lines changed" column. Larry Finger's contribution is largely negative (in line counts - not in value): he removed the rt2860sta and rt2870sta drivers from the staging tree now that the mainline driver can replace them. Dmitry Kravkov appears due to a firmware update; the bnx2x driver is one of the few which still has firmware in the mainline kernel tree. Vasanthakumar Thiagarajan also removed a lot of code, mostly through the process of eliminating duplication between Atheros wireless drivers. Mauro Carvalho Chehab removed the obsolete Micronas drx397xD driver.

A total of 184 employers (that we were able to identify) participated in the 3.0 cycle; the most active among them were:

Most active 3.0 employers

By changesets (None) 1085 12.0% Red Hat 1000 11.1% Intel 839 9.3% (Unknown) 569 6.3% Novell 441 4.9% IBM 374 4.2% Microsoft 361 4.0% Atheros Communications 241 2.7% Texas Instruments 234 2.6% Broadcom 222 2.5% Oracle 187 2.1% AMD 162 1.8% Nokia 158 1.8% Fujitsu 154 1.7% Google 129 1.4% University of Cambridge 119 1.3% Analog Devices 118 1.3% (Consultant) 113 1.3% Samsung 103 1.1% Wolfson Microelectronics 103 1.1%

By lines changed Intel 163232 18.1% (None) 152840 16.9% Broadcom 61948 6.9% Red Hat 59079 6.5% Atheros Communications 53268 5.9% Marvell 31118 3.4% (Unknown) 29261 3.2% IBM 20587 2.3% Metzler Brothers Systementwicklung GbR 19933 2.2% Novell 19578 2.2% University of Cambridge 16969 1.9% Pengutronix 16207 1.8% Realsil Microelectronics 14876 1.6% Analog Devices 12998 1.4% Tilera 12257 1.4% Freescale 11637 1.3% Microsoft 11564 1.3% Texas Instruments 10802 1.2% Wolfson Microelectronics 10051 1.1% Samsung 9784 1.1%

There are few surprises here. Microsoft at 4% of the total changes is unusual; one assumes that presence will not be permanent: even the HV drivers can only need so much cleaning up. The percentage of changes from hobbyists continues to drop; whether that's a bad thing (the kernel is becoming increasingly unapproachable to volunteer developers) or a good thing (it's impossible for anybody who can hack the kernel to remain unemployed) is still not clear.

A longer-term look

The release of 3.0 provides as good an opportunity as any to look at the entire 2.6 series. Thanks to the BitKeeper history tree put together by Thomas Gleixner, it is possible to get detailed information back almost to the beginning of the 2.5 development cycle, which can be thought of as the set of -rc kernels leading up to 2.6.0. This information is far from complete, unfortunately. The 2.5.0 through 2.5.3 releases predate the BitKeeper transition, and thus appear as big patches from Linus. Even thereafter, a lot of early changes appear to have been contributed by the maintainer they passed through instead of the actual author; it took a while to establish the infrastructure to properly credit all work. Still, there is enough data there to work with.

The history from the beginning of the 2.5 development series covers about 9.5 years of development. During this time, some 291,664 changesets were contributed by 8,078 developers; those changes added 10.5 million lines of code. Here are the most active developers over that extended period:

Most active developers since 2.5.0

By changesets Andrew Morton 7638 2.6% David S. Miller 5203 1.8% Al Viro 3828 1.3% Greg Kroah-Hartman 3309 1.1% Russell King 3226 1.1% Alan Cox 2609 0.9% Ingo Molnar 2599 0.9% Stephen Hemminger 2535 0.9% Bartlomiej Zolnierkiewicz 2485 0.9% Linus Torvalds 2479 0.8% Christoph Hellwig 2429 0.8% Takashi Iwai 2414 0.8% Adrian Bunk 2306 0.8% Tejun Heo 2205 0.8% Thomas Gleixner 2205 0.8% Paul Mundt 2113 0.7% Dave Jones 2067 0.7% Randy Dunlap 1853 0.6% Ralf Baechle 1786 0.6% Johannes Berg 1770 0.6%

By changed lines Greg Kroah-Hartman 738134 2.3% Bartlomiej Zolnierkiewicz 553077 1.7% Andrew Morton 537737 1.7% Alan Cox 432023 1.4% Jaroslav Kysela 387649 1.2% Adrian Bunk 380691 1.2% James Bottomley 367435 1.2% Linus Torvalds 325954 1.0% Ralf Baechle 319859 1.0% Paul Mackerras 279454 0.9% Sam Ravnborg 270118 0.8% David S. Miller 254574 0.8% Christoph Hellwig 238749 0.8% Mauro Carvalho Chehab 232793 0.7% Uwe Kleine-König 215560 0.7% Russell King 209362 0.7% Benjamin Herrenschmidt 195707 0.6% Jeff Garzik 190724 0.6% Paul Mundt 185781 0.6% David Howells 183872 0.6%

It should be repeated that these numbers are highly approximate. For example, while Andrew Morton was indeed a prolific code contributor in the 2.5.x and early 2.6 days, he didn't write quite that many patches; a lot of patches from others that went through him lost their authorship information on the way. That information is generally present in the changelog - somebody could try to make a new repository with proper credits given some time - but, for now, we'll have to make do with fuzzy numbers. The per-employer numbers are necessarily even fuzzier - to the point that they are most likely not worth showing here. Suffice to say that, in general form, they resemble the numbers we have been showing for the last few years.

For those who are curious about just the post-2.6.0 kernels, the numbers don't change that much. Since 2.6.0, there have been 264,706 changesets contributed by 7,725 developers adding 8.7 million lines of code.

One other exercise with this data seemed interesting: a determination of who have been the most consistent contributors over those nine years and some. After running a script to track which developers contributed to each major release, twelve developers were found who had contributed to all 41 of them. Additionally, a handful of developers have gotten code into almost every release. The most consistent developers are:

Most consistent developers 2.6.0-3.0
Developer	Releases	Missed releases
Linus Torvalds	41
David S. Miller	41
Greg Kroah-Hartman	41
Andrew Morton	41
Christoph Hellwig	41
Alan Stern	41
James Bottomley	41
Randy Dunlap	41
Russell King	41
Al Viro	41
Stephen Hemminger	41
Andi Kleen	41
Jens Axboe	40	v2.6.1
Jean Delvare	40	v2.6.4
Dave Jones	40	v2.6.35
Benjamin Herrenschmidt	40	v2.6.1
Jeff Garzik	40	v2.6.36
Ingo Molnar	39	v2.6.2 v2.6.5
Herbert Xu	39	v2.6.3 v2.6.5
Patrick McHardy	39	v2.6.2 v2.6.6
Dmitry Torokhov	38	v2.6.3 v2.6.4 v2.6.6
Rusty Russell	38	v2.6.1 v2.6.15 v2.6.39
Matthew Wilcox	38	v2.6.14 v2.6.36 v3.0
Dave Kleikamp	38	v2.6.26 v2.6.33 v2.6.37
Len Brown	38	v2.6.1 v2.6.17 v2.6.39
Oliver Neukum	38	v2.6.4 v2.6.14 v2.6.37
Wim Van Sebroeck	38	v2.6.4 v2.6.6 v3.0
Andrew Vasquez	38	v2.6.0 v2.6.1 v2.6.5
James Morris	38	v2.6.16 v2.6.37 v2.6.39
Neil Brown	37	v2.6.1 v2.6.2 v2.6.3 v2.6.6
Trond Myklebust	37	v2.6.1 v2.6.2 v2.6.8 v2.6.10
Paul Mackerras	37	v2.6.1 v2.6.3 v2.6.38 v2.6.39
Bjorn Helgaas	37	v2.6.3 v2.6.20 v2.6.39 v3.0
Tony Lindgren	37	v2.6.0 v2.6.1 v2.6.5 v2.6.20
Nicolas Pitre	37	v2.6.3 v2.6.4 v2.6.5 v2.6.23
Stephen Rothwell	37	v2.6.1 v2.6.2 v2.6.3 v2.6.7
David Howells	36	v2.6.1 v2.6.2 v2.6.3 v2.6.4 v2.6.6
Eric Sandeen	36	v2.6.1 v2.6.8 v2.6.11 v2.6.17 v3.0
Ralf Baechle	36	v2.6.1 v2.6.3 v2.6.4 v2.6.7 v2.6.38
Arjan van de Ven	36	v2.6.1 v2.6.3 v2.6.13 v2.6.14 v3.0
David Brownell	36	v2.6.33 v2.6.34 v2.6.37 v2.6.38 v3.0

Your editor, who only got changes into 32 releases during this time, knows what an accomplishment it is to consistently contribute to every release over such a long period of time.

But, then, creating the kernel and the development process we have over the course of the last 20 years is an impressive accomplishment. There are few development projects which have lasted this long, gone this far, and have been more vital than ever. It has been fun to watch. It seems likely that things will remain just as fun over the next 20 years - one could argue that we have just begun.

Comments (20 posted)

The structured logging challenge

The debate over the concept of "user-friendly disk names" was rekindled this week with the posting of a new version of Nao Nishijima's persistent device names patch. The disagreements over this particular feature remain; it is possible that the change will be merged regardless. At the core of this discussion, though, is a concept which goes beyond adding a user-specified name to specific devices; it's the bigger problem of getting structured data out of the kernel.

After all these years, the main mechanism by which the kernel passes information to user space remains the lowly printk() function. It is, needless to say, a useful and flexible way of getting messages out, but it imposes almost nothing on the structure of those messages. That leads to all kinds of output like (from drivers/net/de620.c):

    printk(KERN_WARNING "%s: Thanks, I feel much better now!\n", dev->name);

or the famous message from drivers/char/lp.c:

    printk(KERN_INFO "lp%d on fire\n", minor);

System administrators should not be faulted for wondering what they should do in response to messages like these.

There have been some changes to impose structure on printk() output, starting with the addition of a marker for the severity level of each message. It still is not hard to find printk() calls without severity levels, though; actually enforcing the use of these markers has proved hard to do. A bit more structure is added by dev_printk() (and variants like dev_err()), but the use of these functions is even less universal.

The lack of structure means that there is little consistency between messages; any two network drivers will almost certainly print different things to indicate the same situation. Kernel messages can also vary over time; messages emitted by printk() are also not normally seen as part of the kernel ABI, despite the fact that changing them can break scripts that try to extract useful information from the system logs. So it's not surprising that, about one year ago, Andrew Morton said:

Various people have tried to improve the situation in spots; the user-friendly disk names, by trying to attach a consistent name to devices, is one such attempt. The netoops patch from Google is another; it helps Google figure out why machines are crashing without requiring operators to actually dig through the logs. But these changes are far from an overall framework for structured data from the kernel.

There have been a few attempts to make such frameworks over the years; all have fallen far short of making it into the kernel. It's not hard to come up with plausible reasons for this failure. The amount of work required is huge, especially if one wants to add structure to the bulk of interesting communications from kernel space. Developers like printk(); they are less likely to be enamored of some other interface which requires more work to use, is less flexible (by design) in its output, and which may well have to sit alongside the existing printk() logging. Coming up with a structured format which meets everybody's needs - and which will not have to be supplemented with a "version 2" format in the coming years - presents its own challenges.

It must also be said that kernel developers, as a whole, see little value in standardized, structured kernel logging information. It will not help them to debug their kernels. The fact that a lot of users want this sort of feature is far from irrelevant to the development community, but experience has shown that a lack of developer interest can make it much harder to get changes merged - especially if those changes are wide-ranging and disruptive.

If this problem is ever to be solved, it would seem that two things need to be found: a mechanism which looks like it could work and a motivation for kernel developers to accept it. The motivation can probably found in a combination of (1) their paychecks as customers continue to push for this capability and (2) the prospect of a continuing stream of ad hoc patches adding structure to various corners without solving the real problem. But that leaves open the problem of finding a workable solution.

Your editor has a half-baked thought on this matter based on the realization that the kernel already has a nice mechanism for passing structured data to user space. On almost any contemporary system, the /dev directory is managed by the udev daemon; udev works by receiving highly-structured messages from the kernel describing the coming and going of devices, changes in their configuration, firmware load requests, and more. It is an established protocol which enables sophisticated user-space responses to kernel events. Udev and the associated "uevent" mechanism had some early growing pains, but this code is now stable, functional, and almost universally used. Perhaps it's time for this mechanism to take on some new duties.

Uevents work because the format is simultaneously structured and flexible; it can be extended when the need arises. The generation of events is almost entirely done automatically by the driver core; most driver authors need not do anything to cause them to happen and, indeed, may not even know that this mechanism is operating underneath the hood. Driver authors don't have to make their own events; they would have to go out of their way to prevent them from happening.

Logging of other types of events will probably require explicit support in the relevant kernel code; that is the part needing some extra thought. The creation of uevents by hand is a bit of a labor-intensive business; the relevant code tends to look like:

	retval = add_uevent_var(env, "ACTION=%s", action_string);
	if (retval)
		goto exit;
	retval = add_uevent_var(env, "DEVPATH=%s", devpath);
	if (retval)
		goto exit;
	retval = add_uevent_var(env, "SUBSYSTEM=%s", subsystem);
	if (retval)
		goto exit;

Clearly, any attempt to place this kind of code in every logging location is not going to get very far. What is needed is a useful set of helper functions. These functions, for maximum utility, would probably be tied fairly tightly to the underlying subsystems. Storage drivers could have functions to report block errors, device changes, and multipath connectivity changes. Network drivers would need to report events like carrier loss, excessive checksum errors, or duplicate MAC addresses. All kernel code could benefit from helpers to log allocation failures or failed assertions. In each case, the helper would standardize the format of the reported information while allowing the addition of information specific to the call site.

The addition of a new set of logging functions would necessarily require changes to drivers to use those functions. So it would take time to achieve anything close to comprehensive coverage, and 100% coverage would never happen. But, then, we still don't have 100% coverage for the KERN_* severity markers. If this interface proved useful, one could imagine that the code paths of interest to enterprise distribution customers would be covered in relatively short order.

But, then, there are probably several things fatally wrong with this idea; the structured logging problem will likely remain unsolved for some time yet. But the problem will not go away; if anything, the need to recognize and automatically respond to system events will only increase. Someday somebody will come up with a solution that works and that can be adopted with minimal pain; until then, printk() remains the only show in town.

Comments (2 posted)