Kernel development [LWN.net]

Release status

Kernel release status

The current 2.6 prepatch is 2.6.23-rc9, released on October 1. This -rc release was unexpected, but there were enough fixes added since -rc8 that Linus felt one more cycle was called for. The final 2.6.23 release should happen almost any time.

There have been about a dozen patches added to the mainline repository since -rc9.

The current -mm tree is 2.6.23-rc8-mm2. Recent changes to -mm include delayed allocation support for ext4 and a number of fixes. Andrew notes that -mm now has almost 32MB of patches relative to the mainline - an indication of what is to come once the 2.6.24 merge window opens.

Looking forward: it appears that the 2.6.24 cycle will begin with the i386/x86_64 merger. More information about 2.6.24 can be found in Andrew Morton's 2.6.24 merge plans document. It looks like 2.6.24 will include a bunch of memory management work, more anti-fragmentation patches, per-device write throttling, the LSM non-modules patch, file-based capabilities, more memory layout randomization, control groups (formerly containers), PID namespaces, kernel markers, and much more. Remember that this list covers only patches to be merged by Andrew; the bulk of the code going into 2.6.24 will get there directly from the subsystem maintainers.

Comments (none posted)

Kernel development news

Quotes of the week

We must not ignore people who tell us that "there is something wrong going on here", just because they are unable to analyze it themselves. Very often where we end up saying "we dont know what's going on here" it's likely _our_ fault. We also must not hide behind "please do these 10 easy steps and 2 kernel recompiles and 10 reboots, only takes half a day, and come back to us once you have the detailed debug data" requests.

-- Ingo Molnar

I was staring in astonishment at the pending sysfs patch pile last night. Forty syfs patches and twenty-odd patches against driver core and the kobject layer. That's a huge amount of churn for a core piece of kernel infrastructure which has been there for four or five years. Not a good sign. I mean, it's not as if, say, the CPU scheduler guys keep on rewriting all their junk.

oh, wait..

Andrew Morton

For me, given my threat model and how much my time is worth, life is too short for SELinux.

-- Ted Ts'o

Comments (8 posted)

Linux driver project gets a full-time leader

By Jake Edge
October 3, 2007

The Linux Driver Project (LDP) just got a big boost, courtesy of Novell and Greg Kroah-Hartman. The project was announced last January to much acclaim, but has languished since, buried under Kroah-Hartman's "day job" and Linux kernel development work. Now, he will be able to devote much more time to the project as his employer, Novell, has shifted his job responsibilities to work full-time on the LDP.

The original project announcement was released in conjunction with the Freedom HEC conference and was described by Kroah-Hartman as a "lame publicity stunt", because it just reiterated the standard Linux driver development model: with some hardware and some information, a driver for your device will be written. There was a new wrinkle, though; an arrangement worked out with the Linux Foundation to allow driver developers to sign non-disclosure agreements (NDAs) with hardware vendors to get access to documentation and other information about the device. NDAs for driver development are controversial in some quarters, but are often required by hardware vendors.

There are numerous benefits for Linux: the drivers will all be licensed under the GPL, will get merged into the mainline tree, and be available for all Linux users. Other free operating systems may be able to use the source to write drivers for their systems, as well. Kroah-Hartman notes that a surprising added benefit is for new kernel developers:

Another wonderful benefit that I never had imagined in the beginning is that we are now providing a way for developers who want to write something "real" to have a place to go. The biggest response I got from my first announcement was from developers wanting to help out. I had over 80 people sign up to help out as they wanted to be able to contribute to Linux, but did not previously know how to do so in a tangible manner. This project gives them a place where they can develop and maintain a real driver for the kernel community.

Now that he has time to devote to LDP, Kroah-Hartman has put together two mailing lists along with a wiki to track the project. There is a mailing list for each of the two main roles, developers and project managers. The role of a project manager will be to facilitate the development, making sure that the driver hacker has what they need to get the job done and keeping the company, for whom the driver is being written, informed of the status. In short, they will shepherd an individual driver in much the same way that Kroah-Hartman is coordinating the LDP as a whole.

In less than a week since the project restart, there are five driver projects up for grabs, including a "clean-up and get merged" project that would be suitable as a first driver for someone just starting out in Linux driver development. Project managers are lining up to take on the drivers as well. The numbers of volunteers have grown, but as Kroah-Hartman notes, publicity is something the project still needs:

We currently have over 200 people signed up to be a developer, so we doing quite well there. We also have over 25 people signed up to be a project manager, so I think we are good there too. What we do need the most help on right now is to find more companies that need our help. Spreading the word that this service is available and open to any company is the biggest importance I think at this time.

Already, there are drivers for many different kinds of devices in the pipeline:

[...] audio codec devices, USB timestamp devices, VOIP devices, video camera devices, lots of different types of data acquisition devices, as well as some custom bus interconnects and even some whole system-on-a-chip devices.

Kroah-Hartman plans to reconnect with various companies that contacted him since January, but fell by the wayside. As that happens and the word gets out about the project, there should be driver development projects suitable for a wide range of interests and various levels of kernel experience. By providing a self-contained project that is targeted at inclusion in the mainline, more developers will be exposed to that process, which should expand the ranks of kernel hackers.

Linux already supports more hardware than any operating system has before and the LDP will only extend that lead. There are huge benefits for Linux, the developers and project managers, the companies whose devices will be supported, as well as for distributors like Novell. There may be complaints about signing NDAs, but the drivers will be free, not obfuscated; once companies see how easy it is to get a high-quality driver into the kernel, they will certainly come back for more. This can only be a good thing for all free software systems, not just Linux.

Comments (6 posted)

What's in the realtime tree

By Jonathan Corbet
October 3, 2007

It has been almost exactly three years since Sven-Thorsten Dietrich posted a set of realtime improvements to the linux-kernel list. That particular body of code was upstaged by the realtime preemption work done by Ingo Molnar and others, but it deserves some credit for kicking off a development effort which continues to this day. After three years, many parts of the realtime preemption patch set have been merged into the mainline kernel, including dynamic tick support, a rewritten interrupt subsystem, mutexes, priority inheritance, high-resolution timers, and more. At this point, we are all running kernels which have benefited from the realtime preemption project.

The job of merging the realtime preemption work into the mainline is not complete, though. Indeed, a look at the 2.6.23-rc8-rt1 tree announcement shows that there are nearly 400 individual patches sitting there. This seems like a good opportunity to have a look at the realtime tree and see what remains to be merged.

The core of this patch set remains the realtime mutex code. When the kernel is configured for realtime operation, a bunch of (improved, but still scary) preprocessor magic causes normal spinlocks to be replaced by realtime mutexes, which have different properties. In particular, realtime mutexes are fully preemptible and have priority inheritance. With most kernel spinlocks replaced by these mutexes, there are few places in the kernel which are able to cause arbitrary latencies.

Merging realtime mutexes should, in theory, not be a problem; they are not actually used unless explicitly configured into the kernel, and it is assumed that most users will not configure things that way. Such a fundamental change to a core mutual exclusion mechanism will always raise eyebrows, though. So there have been no attempts to merge this code so far, and it is likely that most of the other parts of the realtime tree will find their way into the mainline first.

Code which could go in sooner is the threaded interrupt handlers patch. Putting interrupt handlers into threads allows them to be scheduled along with most other system activities and eliminates another potential source of latency. It also can improve the robustness of the system and make it easier to find bugs in interrupt code. So this patch could be merged and possibly even made the default - though there will always be a small number of interrupt handlers which must be run directly.

Also in the realtime tree is a patch which moves all software interrupt processing into dedicated threads. Then there is a patch which allows hardware interrupt handling threads to process any pending software interrupts before yielding the processor. This optimization avoids the need for a context switch to a separate software IRQ thread to get those interrupts delivered.

One of the sticking points with the realtime patches has been their interaction with the read-copy-update mechanism. The current code requires that preemption be disabled while references to RCU-protected data structures exist, but disabling preemption ruins the guarantees that the realtime code is trying to provide. The answer is a somewhat more complicated, preemptible RCU implementation. With luck, LWN will have an article on how preemptible RCU works in the near future.

Nick Piggin's lockless pagecache patches have found their way into the realtime tree. These patches make a number of low-level changes to the memory management and radix tree code so that some pagecache operations can be done without taking any locks. This code has been in circulation for some time without making it into the mainline, but it seems like a win in a number of scalability situations. Another patch (by Peter Zijlstra) gets rid of the locking in the kmap() code, improving latencies in systems using high memory.

The wisdom of allowing Java programs to mess with physical memory is not a topic which should need further discussion here. Another patch which has been out of the mainline for quite some time - and likely to remain that way - is Ted Ts'o's /dev/rmem facility. This device allows direct access to physical memory - a feature which is required on any system which wants to pass the realtime Java conformance tests. The wisdom of allowing Java programs to mess with physical memory is not a topic which should need further discussion here.

The realtime tree contains an extensive set of tools for tracking down parts of the kernel which cause excessive latencies. This code has, over the years, been put to good use in identifying and breaking up kernel code which hogs the processor unnecessarily. These patches would seem like a good match for the mainline, especially given recent discussions on the value of adding more instrumentation to the kernel. The first step in solving problems is being able to measure them.

For reasons which are unclear to your editor, the realtime tree contains the venerable realtime security module, which was definitively refused entry into the mainline a few years ago. The module is marked as being obsolete - but it is still there.

Quite a few other changes can be found in this tree. The SLUB allocator is not an option for realtime kernels. Instead, this tree uses a modified version of the slab allocator which replaces interrupt-based single-CPU locking with a set of specific per-CPU locks. The global files_lock has been removed in favor of tightly-locked per-CPU lists. To help with the creation of such lists, a new locked-list type has been added. The tasklet code has been reworked for better threading, but the tasklet elimination patch is not present. There's also quite a few architecture-specific patches adding various features (such as clockevents) needed by the realtime tree and fixing problems.

All told, there is a lot of code sitting in the realtime tree despite all of the mainline merging which has happened over the last couple of years or so. The stated plan is to merge most of this code in the not-too-distant future, but it is not clear when that will happen. In particular, some of the core realtime developers are likely to be severely distracted by the i386/x86_64 merger during the 2.6.24 cycle, so they may not manage to move much of the realtime code toward the mainline.

Comments (4 posted)

SMACK meets the One True Security Module

By Jonathan Corbet
October 2, 2007

The Simplified Mandatory Access Control Kernel is a security module designed to harden Linux systems through the addition of mandatory access control policies; it was covered here last August. Like SELinux, SMACK works by attaching labels to processes, files, and other system objects and implementing rules describing what kinds of access are allowed by specific combinations of labels. Unlike SELinux, though, SMACK was designed specifically for simplicity of administration.

The posting of version 3 of the SMACK patch inspired a certain amount of discussion. Andrew Morton's take led things off:

I don't know enough about security even to be dangerous. I went back and reviewed the August thread from your version 1 submission and the message I take away is that the code has been well-received and looks good when considered on its own merits, but selinux could probably be configured to do something sufficiently similar.

I'd have trouble declaring that "but" to be a reason to not merge smack. I'm more thinking "let's merge it and see if people use it".

Andrew concluded by suggesting that the SMACK patch should be merged for 2.6.24. There was some discussion about specific pieces of the patch (some developers are more impressed by CIPSO network labels than others), but there does not seem to be a whole lot of opposition to merging the SMACK security module. It is generally seen as being well-written code which makes sense from a security point of view.

There was one predictable exception, though: whenever a new security module is considered for merging the SELinux developers tend to oppose it. This time, James Morris voiced a more general complaint: SMACK would become the second in-kernel user of the Linux security module (LSM) framework. That, in turn, would make it almost impossible to remove LSM from the kernel. James claims:

If LSM remains, security will never be a first class citizen of the kernel. Application developers will see multiple security schemes, and either burn themselves trying to support them, or more likely, ignore them. Core kernel developers will continue to not have enough information to understand what the LSM hooks in their code are supposed to be doing, leading to long term maintenance issues and potential security problems.

The SELinux programmers, it seems, would rather that SELinux became the one security framework supported by Linux, with all development effort going into making SELinux better.

The problem, of course, is that, even after a few years with only SELinux in the kernel, there has not been a glut of application developers working to support it. The vision of each application shipping with an SELinux policy file has not come to be. Some progress has been made in the creation of higher-level tools for the management of SELinux policies, and the Fedora and Red Hat developers have come a long way toward making a general-purpose distribution work with a limited set of SELinux policies, but SELinux simply remains too complex for most people to deal with. SELinux may work out of the box on an RHEL system, but as soon as the system administrator runs into something which breaks, chances are that SELinux will be turned off.

The SELinux developers would presumably argue that the way to address these problems is to focus on a single security framework within the kernel. Rather than create competing, simplified frameworks, it would be better to implement approaches like AppArmor or SMACK within SELinux and, in the process, make SELinux better. Those developers argue that pluggable security modules, like pluggable schedulers, succeed only in splitting development effort and preventing the creation of a truly general solution:

Do you really want to encourage people to roll their own security module rather than working toward a common security architecture and a single balanced solution (which doesn't necessarily mean SELinux, mind you, but certainly could draw from parts of it)? As with pluggable schedulers, the LSM approach prevents cross pollination and forces users to make poor choices.

The answering argument is that there are many security environments and differing sets of user needs; there is no sign that a single security approach can ever work in all situations. And, even if such a unified approach were possible, getting security developers to agree on it would still be a major challenge. Ted Ts'o writes:

The real problem with security is that there are no "hard numbers", as Linus puts it. Instead, there are different sets of requirements in terms of what is a valid threat model --- which will very depending on the environment and how the servers are deployed, and what the capabilities are of the adversary trying to penetrate said system --- and how end users are willing to compromise between security and convenience.

The LSM architecture was a result of the very first kernel summit, where Linus stated that he had no wish to choose between the various security ideas in circulation and, instead, wanted the kernel to be able to support all of them. He has not changed his mind since; he still doesn't see any imminent consensus on security approaches, and still wants Linux to be flexible in this regard. So his pronouncement was:

So LSM stays in. No ifs, buts, maybes or anything else.

When I see the security people making sane arguments and agreeing on something, that will change. Quite frankly, I expect hell to freeze over before that happens, and pigs will be nesting in trees. But hey, I can hope.

So it looks like the way is clear for a merger of SMACK in 2.6.24. Once that happens, it would not be surprising to see another push made by the developers of security modules like AppArmor and TOMOYO Linux; in fact, the TOMOYO Linux patches have just been reposted. In the end, SELinux will, despite the apparent wishes of its developers, have to compete with other security approaches for attention from developers, distributors, and users. There will be no One True Security Module for Linux in the foreseeable future.

Comments (13 posted)

Patches and updates

Kernel trees