LWN Weekly Edition
Front pageSecurity
Kernel development
Distributions
Development
Announcements
->One big page
This page
Previous weekFollowing week
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Kernel developmentBrief items Kernel release statusThe current development kernel is 3.0-rc7, released on July 11. "I think I said -rc6 might be the last -rc. I lied. Things have been pretty quiet, but there's enough new stuff here that I wanted to do another -rc, and we still have some issues with the RCU changes causing problems when RCU events happen before the scheduler has been fully initialized etc. So -rc7 is out there." See the full changelog for all the details.Stable updates: the 2.6.32.43 and 2.6.33.16 stable kernel updates were released on July 13; 2.6.39.3 came out on July 8. All three contain the usual list of important fixes - quite a few of them, in the 2.6.39.3 case.
Quotes of the week
The changelog forgot to tell us the user-visible effects of the
bug. That was really really bad of it. Bad changelog. No bone
for you.
-- Andrew Morton
Lawyers: they tend to seem like reasonable and smart people when
taken individually. And yet they try to argue whether you remember
the specific date of a newsgroup posting you wrote. Never mind that
the posting has the date clearly stated on it, and is a reply to
(and is itself replied to) by other newsgroup postings that also
have that date.
-- Linus
Torvalds
What will that kernel be called?When Linus announced the 3.0-rc1 kernel at the end of May, he noted that it was actually identifying itself as "3.0.0". There were enough scripts out there expecting that third digit to exist that he didn't want to break them from the outset. Still, he was hopeful that the final ".0" could go away:
We'll have the usual 6-7 weeks to wrestle it into submission, and
get scripts etc cleaned up, and the final release should be just
"3.0". The -stable team can use the third number for their
versioning.
Fast forward to 3.0-rc7, and the kernel is still identifying itself as 3.0.0. While hard information is relatively scarce, it seems that there are still some programs and scripts out there that will break without that final number. In a sense, the three-number versioning scheme has become a part of the kernel ABI over the years. One can argue that anything which breaks was written in a silly way to begin with, but the goal is still to avoid breaking user space. So, while no definitive answer has been given, it looks like we're going to have three-number mainline releases for some time. It would be most surprising to see the stable updates starting with anything other than 3.0.1, though. There is one related question, though: what about programs that cannot cope with anything other than "2.6.x" as the kernel version number? Evidently such programs exist. Saying that kernel releases must start with "2.6" forevermore stretches even the firmest commitment to binary compatibility; it's not something that is going to happen. In an attempt to make life easier for people stuck with such programs, Andi Kleen has implemented a new system personality that pretends the 3.0 change never happened. If a program is run under that personality, it will think that the kernel version is 2.6.40; 3.1 will look like 2.6.41, etc. Andi says "I know this is somewhat ugly, but I didn't find a better workaround, and compatibility to existing programs is important." As of this writing, though, the patch has not made it into the mainline.
Kernel development news Who wrote 3.0 - from two points of viewLinus Torvalds had hoped to release the 3.0 kernel after -rc6, but reality, as is its wont, intervened; thus, 3.0-rc7 was released on July 11. That probably is the last development release for 3.0, though. Tradition dictates that we take a look at the contributor statistics for this development cycle, which we will now do.This kernel release inaugurates the beginning of the 3.x series of kernels. As has been mentioned many times here, there is nothing particularly special about the 3.0 release; it has been, in many ways, a relatively boring development cycle. But it still provides a good opportunity to look back over a longer period of time. But, before doing that, we'll start with this cycle, which has, as of 3.0-rc7, seen 9,007 changesets contributed by 1,110 developers. The kernel grew 113,000 lines in this development cycle - a relatively modest figure by contemporary standards. The most active developers during this cycle were:
K. Y. Srinivasan topped the list of changeset contributors with a massive set of cleanups to the Microsoft HV driver in the staging tree; it's impressive to see how much cleanup less than 15,000 lines of code can require. David Miller made a lot of changes in the networking subsystem; some were warning fixes and such, while others were more substantial. Dan Williams contributed Intel's "isci" storage driver, merged in 3.0-rc6. Jonathan Cameron contributed a lot of work to rationalize the industrial I/O (iio) subsystem and prepare it for an eventual merge into the mainline. Takashi Iwai continues to do large amounts of work in the ALSA sound driver subsystem. The isci driver put Dan Williams at the top of the "lines changed" column. Larry Finger's contribution is largely negative (in line counts - not in value): he removed the rt2860sta and rt2870sta drivers from the staging tree now that the mainline driver can replace them. Dmitry Kravkov appears due to a firmware update; the bnx2x driver is one of the few which still has firmware in the mainline kernel tree. Vasanthakumar Thiagarajan also removed a lot of code, mostly through the process of eliminating duplication between Atheros wireless drivers. Mauro Carvalho Chehab removed the obsolete Micronas drx397xD driver. A total of 184 employers (that we were able to identify) participated in the 3.0 cycle; the most active among them were:
There are few surprises here. Microsoft at 4% of the total changes is unusual; one assumes that presence will not be permanent: even the HV drivers can only need so much cleaning up. The percentage of changes from hobbyists continues to drop; whether that's a bad thing (the kernel is becoming increasingly unapproachable to volunteer developers) or a good thing (it's impossible for anybody who can hack the kernel to remain unemployed) is still not clear.
A longer-term lookThe release of 3.0 provides as good an opportunity as any to look at the entire 2.6 series. Thanks to the BitKeeper history tree put together by Thomas Gleixner, it is possible to get detailed information back almost to the beginning of the 2.5 development cycle, which can be thought of as the set of -rc kernels leading up to 2.6.0. This information is far from complete, unfortunately. The 2.5.0 through 2.5.3 releases predate the BitKeeper transition, and thus appear as big patches from Linus. Even thereafter, a lot of early changes appear to have been contributed by the maintainer they passed through instead of the actual author; it took a while to establish the infrastructure to properly credit all work. Still, there is enough data there to work with. The history from the beginning of the 2.5 development series covers about 9.5 years of development. During this time, some 291,664 changesets were contributed by 8,078 developers; those changes added 10.5 million lines of code. Here are the most active developers over that extended period:
It should be repeated that these numbers are highly approximate. For example, while Andrew Morton was indeed a prolific code contributor in the 2.5.x and early 2.6 days, he didn't write quite that many patches; a lot of patches from others that went through him lost their authorship information on the way. That information is generally present in the changelog - somebody could try to make a new repository with proper credits given some time - but, for now, we'll have to make do with fuzzy numbers. The per-employer numbers are necessarily even fuzzier - to the point that they are most likely not worth showing here. Suffice to say that, in general form, they resemble the numbers we have been showing for the last few years. For those who are curious about just the post-2.6.0 kernels, the numbers don't change that much. Since 2.6.0, there have been 264,706 changesets contributed by 7,725 developers adding 8.7 million lines of code. One other exercise with this data seemed interesting: a determination of who have been the most consistent contributors over those nine years and some. After running a script to track which developers contributed to each major release, twelve developers were found who had contributed to all 41 of them. Additionally, a handful of developers have gotten code into almost every release. The most consistent developers are:
Your editor, who only got changes into 32 releases during this time, knows what an accomplishment it is to consistently contribute to every release over such a long period of time. But, then, creating the kernel and the development process we have over the course of the last 20 years is an impressive accomplishment. There are few development projects which have lasted this long, gone this far, and have been more vital than ever. It has been fun to watch. It seems likely that things will remain just as fun over the next 20 years - one could argue that we have just begun.
The structured logging challengeThe debate over the concept of "user-friendly disk names" was rekindled this week with the posting of a new version of Nao Nishijima's persistent device names patch. The disagreements over this particular feature remain; it is possible that the change will be merged regardless. At the core of this discussion, though, is a concept which goes beyond adding a user-specified name to specific devices; it's the bigger problem of getting structured data out of the kernel.After all these years, the main mechanism by which the kernel passes information to user space remains the lowly printk() function. It is, needless to say, a useful and flexible way of getting messages out, but it imposes almost nothing on the structure of those messages. That leads to all kinds of output like (from drivers/net/de620.c):
printk(KERN_WARNING "%s: Thanks, I feel much better now!\n", dev->name);
or the famous message from drivers/char/lp.c:
printk(KERN_INFO "lp%d on fire\n", minor);
System administrators should not be faulted for wondering what they should do in response to messages like these. There have been some changes to impose structure on printk() output, starting with the addition of a marker for the severity level of each message. It still is not hard to find printk() calls without severity levels, though; actually enforcing the use of these markers has proved hard to do. A bit more structure is added by dev_printk() (and variants like dev_err()), but the use of these functions is even less universal. The lack of structure means that there is little consistency between messages; any two network drivers will almost certainly print different things to indicate the same situation. Kernel messages can also vary over time; messages emitted by printk() are also not normally seen as part of the kernel ABI, despite the fact that changing them can break scripts that try to extract useful information from the system logs. So it's not surprising that, about one year ago, Andrew Morton said:
The kernel's whole approach to messaging is pretty haphazard and
lame and sad. There have been various proposals to improve the
usefulness and to rationally categorise things in way which are
more useful to operators, but nothing seems to ever get over the
line.
Various people have tried to improve the situation in spots; the user-friendly disk names, by trying to attach a consistent name to devices, is one such attempt. The netoops patch from Google is another; it helps Google figure out why machines are crashing without requiring operators to actually dig through the logs. But these changes are far from an overall framework for structured data from the kernel. There have been a few attempts to make such frameworks over the years; all have fallen far short of making it into the kernel. It's not hard to come up with plausible reasons for this failure. The amount of work required is huge, especially if one wants to add structure to the bulk of interesting communications from kernel space. Developers like printk(); they are less likely to be enamored of some other interface which requires more work to use, is less flexible (by design) in its output, and which may well have to sit alongside the existing printk() logging. Coming up with a structured format which meets everybody's needs - and which will not have to be supplemented with a "version 2" format in the coming years - presents its own challenges. It must also be said that kernel developers, as a whole, see little value in standardized, structured kernel logging information. It will not help them to debug their kernels. The fact that a lot of users want this sort of feature is far from irrelevant to the development community, but experience has shown that a lack of developer interest can make it much harder to get changes merged - especially if those changes are wide-ranging and disruptive. If this problem is ever to be solved, it would seem that two things need to be found: a mechanism which looks like it could work and a motivation for kernel developers to accept it. The motivation can probably found in a combination of (1) their paychecks as customers continue to push for this capability and (2) the prospect of a continuing stream of ad hoc patches adding structure to various corners without solving the real problem. But that leaves open the problem of finding a workable solution. Your editor has a half-baked thought on this matter based on the realization that the kernel already has a nice mechanism for passing structured data to user space. On almost any contemporary system, the /dev directory is managed by the udev daemon; udev works by receiving highly-structured messages from the kernel describing the coming and going of devices, changes in their configuration, firmware load requests, and more. It is an established protocol which enables sophisticated user-space responses to kernel events. Udev and the associated "uevent" mechanism had some early growing pains, but this code is now stable, functional, and almost universally used. Perhaps it's time for this mechanism to take on some new duties. Uevents work because the format is simultaneously structured and flexible; it can be extended when the need arises. The generation of events is almost entirely done automatically by the driver core; most driver authors need not do anything to cause them to happen and, indeed, may not even know that this mechanism is operating underneath the hood. Driver authors don't have to make their own events; they would have to go out of their way to prevent them from happening. Logging of other types of events will probably require explicit support in the relevant kernel code; that is the part needing some extra thought. The creation of uevents by hand is a bit of a labor-intensive business; the relevant code tends to look like:
retval = add_uevent_var(env, "ACTION=%s", action_string); if (retval) goto exit; retval = add_uevent_var(env, "DEVPATH=%s", devpath); if (retval) goto exit; retval = add_uevent_var(env, "SUBSYSTEM=%s", subsystem); if (retval) goto exit; Clearly, any attempt to place this kind of code in every logging location is not going to get very far. What is needed is a useful set of helper functions. These functions, for maximum utility, would probably be tied fairly tightly to the underlying subsystems. Storage drivers could have functions to report block errors, device changes, and multipath connectivity changes. Network drivers would need to report events like carrier loss, excessive checksum errors, or duplicate MAC addresses. All kernel code could benefit from helpers to log allocation failures or failed assertions. In each case, the helper would standardize the format of the reported information while allowing the addition of information specific to the call site. The addition of a new set of logging functions would necessarily require changes to drivers to use those functions. So it would take time to achieve anything close to comprehensive coverage, and 100% coverage would never happen. But, then, we still don't have 100% coverage for the KERN_* severity markers. If this interface proved useful, one could imagine that the code paths of interest to enterprise distribution customers would be covered in relatively short order. But, then, there are probably several things fatally wrong with this idea; the structured logging problem will likely remain unsolved for some time yet. But the problem will not go away; if anything, the need to recognize and automatically respond to system events will only increase. Someday somebody will come up with a solution that works and that can be adopted with minimal pain; until then, printk() remains the only show in town.
Patches and updates Kernel trees
Core kernel code
Development tools
Device drivers
Memory management
Architecture-specific
Security-related
Virtualization and containers
Benchmarks and bugs
Miscellaneous
Page editor: Jonathan Corbet |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||