Kernel development [LWN.net]

Release status

Kernel release status

The current 2.6 prepatch remains 2.6.10-rc2; Linus has released no prepatches since November 14.

Patches continue to accumulate, slowly, in Linus's BitKeeper repository. These include the un-deprecation of MODULE_PARM() (it is generating too many warnings, and the fixes will not be merged before 2.6.10), a new major number (180) for the "ub" USB storage driver, some x86 single-stepping fixes, a large number of "sparse" annotations, the token-based memory management fix, a memory technology device (and JFFS2) update, a frame buffer device update, some user-mode Linux patches, some page allocator tuning, and a few architecture updates.

The latest patch from Andrew Morton is 2.6.10-rc2-mm4. Recent changes to -mm include a new "DMA32" memory zone on the x86_64 architecture, some big architecture updates, a quiet fix for a seemingly exploitable x86_64 buffer overflow, and lots of fixes.

The current 2.4 prepatch is 2.4.29-pre1, which was released by Marcelo on November 25. The bulk of this patch consists of driver and filesystem backports from 2.6.

Comments (none posted)

Kernel development news

Experiment: archive links in quoted messages

Perhaps the longest of longstanding LWN reader requests is the provision of archive links for messages we quote from the mailing lists. This has been a hard request to satisfy, for a couple of reasons. One is that we have learned, the hard way, that it is best to host a copy of the message ourselves. Web archives can go away or reorganize and break links; then we get a lot of mail about non-functioning links on LWN. The other, larger reason, though, has always been a simple matter of time. Producing the LWN Kernel Page requires reading through a few dozen lists, some of which have a lot of traffic. Reading those lists through a web archive - or even just finding links to quotes messages - would slow down the process and result in less content being written. That has not been a tradeoff we have been willing to make.

But we never forgot the request. Of course, it helps that certain readers regularly remind us... Starting this week, we have a tentative solution. The Gmane archive makes it easy to read through archived lists and create URLs to them. Thanks to a bit of script hacking, many of the quoted messaged linked to in this page now have an Archive-link header pointing to the Gmane version of the message, and to Gmane's thread view as well.

This feature should be considered experimental for now; whether it is retained depends on whether readers find it useful, and whether Gmane proves to be sufficiently reliable over time. We're curious to hear whether these links are worthwhile. With luck, an ancient item can now be scratched off the "to do" list.

Comments (5 posted)

Kernel headers and user space

The use of kernel headers in user space has long been discouraged. The kernel headers are not written with user space in mind, and they can change at any time. The proper way for user-space applications to interface with the kernel is by way of the C library, which provides its own structures and, when necessary, translates them into whatever the current kernel expects. This separation helps to keep user-space programs from breaking when the kernel changes.

Unfortunately, things do not always work that way, and some user-space programs still end up including kernel headers directly. These programs may simply be old, or they may need access to declarations which are not available in the C library include files - strange ioctl() codes, for example. So the kernel code still tries to make it possible for user space to include some header files. In these files, kernel-specific code is contained within #ifdef __KERNEL__ blocks and hidden from user space. This technique works, but it is brittle and adds extra cruft to the kernel code base. Intermixing internal kernel definitions with those needed by user space also makes it easy to break the user-space API.

The kernel developers have, for years, wanted to improve this situation. The latest attempt came in the form of this RFC from David Howells. This proposal would create some new directories in the kernel source tree: include/user and some architecture-specific variations (such as include/user-i386). When a portion of a kernel header file is found to be needed by user space, it would be placed into a separate file in one of those directories, and the new file would be included into the old one. At this point, the definitions needed by user space will have been separated out, but no visible changes will have been made; user space can still include the old file and get what it needs.

At some future point, when user space is deemed to have been fixed, all of the __KERNEL__ references could be removed from the old files. At that point, any application still including the internal header files would break.

One part of the idea which did not get very far was using standard C types (such as uint16_t and such) for the user-kernel interface. The problem with that idea is that the kernel cannot count on those types being consistently defined for all configurations, and cannot create its own definitions for the standard types. So the kernel/user interface must continue to be defined using kernel-specific types (__u16 and such).

Linus was not all that enthusiastic about the idea in general. To him, it looks like an exercise in rearranging things without specific goals and with the possibility of breaking things which work now:

We undeniably have existing users of kernel headers. That's just a fact. If we break them, it doesn't _matter_ how the kernel headers look, and then "existing practice" is about as good an organization as anything, and breaking things just to break things is not something I'm in the least interested in. "Beauty" comes secondary to "usefulness".

What he would like to see is more specific discussions which identify specific, problematic header files and what will be done to fix them. In the end, the header files might just get split up in the way described by Mr. Howells. It is more likely to happen as a long and slow process, however, and not as a massive, coordinated reorganization.

Comments (1 posted)

Merging swsusp2

Once upon a time, the 2.6 kernel had three software suspend implementations - two of which were in the mainline - but none which were seen to work all that well. Since then, the two mainline implementations have been merged, and the out-of-tree swsusp2 implementation has come a long way. Still, two implementations of a low-level core function seems like too many, so there is interest in bringing them together in the mainline. Swsusp2 developer Nigel Cunningham has made a new effort in that direction by posting a set of 51 patches which merge swsusp2 into the 2.6 kernel.

There is a great deal of code in these patches. Some of the more interesting pieces include:

A set of new exported symbols; these include fundamental things like sys_ioctl(), avenrun, and a number of low-level swap functions. These exports were poorly received; there is currently a great interest in reducing the number of symbols exported to modules. A patch which, instead, exports fundamental interfaces is bound to encounter some resistance.
The swsusp2 code wants these symbols exported because the entire suspend mechanism can be built as a module and loaded only when the system is to be suspended. This can be a nice feature; swsusp2 is a lot of code, and it is all excess baggage anytime the system is actually being used. The costs of making swsusp2 modular may prove too high for that feature to be accepted into the mainline, however.
A change to the workqueue API allowing the creator of a workqueue to specify whether that queue should be left running during the suspend process. Some workqueues perform tasks which are needed during the image writeout process, and thus cannot be stopped prematurely.
A replacement refrigerator, the code charged with putting all processes on hold so that the system is in a quiet state before suspending. The intrusive macros which characterized the swsusp2 refrigerator some time ago are gone. The new version relies upon a PF_SYNCTHREAD task flag to keep processes (temporarily) from being frozen while they are doing some work involved with writing data to disk.
A complicated I/O infrastructure for writing the suspend image to disk. The new I/O code is said to be much faster, and it supports features like image compression, writing to swap files or LVM devices, etc. Future plans call for more esoteric features, such as suspending across a network. These features may be nice, but it has been pointed out that the swsusp2 I/O code alone outweighs the entire mainline software suspend code by a significant amount.
Various types of "nice displays" when a suspend or resume is in progress. These displays include progress bars, splash screens, and more.

Nigel offers a number of reasons for merging swsusp2. It is claimed to be much faster as a result of the use of asynchronous I/O, readahead on resume, and (for systems with slow drives) image compression. It is far more configurable; users can select the sort of display they like, image compression and/or encryption, etc. Suspending to swap files, LVM devices, and more is supported. And so on. There is little disagreement that swsusp2 offers some nice features, but there is some concern over how Nigel is trying to proceed:

I'm thus seeking to simply merge the existing code, let Pavel and others get to the point where they're ready to say "Okay, we're satisfied that suspend2 does everything swsusp does and more and better." Then we can remove swsusp.

The kernel developers are increasingly resistant to wholesale merging of large blocks of code - especially when that code duplicates functionality already found in the kernel. They would rather see a series of incremental patches, each of which takes a small, useful step in the right direction. Nigel does not want to do that; swsusp2 is vastly different, internally, than the mainline suspend code, and evolving one into the other looks like a long, painful, and pointless job. He may have to do that work, however, before any of the swsusp2 code can be merged.

A bigger obstacle, however, may be the fact that, while swsusp2 was being developed, the mainline software suspend code was progressing too. Your editor is able to reliably suspend to memory and disk with a vanilla 2.6.9 kernel. SUSE enables software suspend and calls it a feature in its 9.2 release. Since the in-kernel suspend code seems to actually work, enthusiasm for replacing it with a larger, more complex version is not as high as it might otherwise have been. The ultimate fate of swsusp2 may yet be to contribute its best improvements to the mainline, but to never be merged in its entirety..

Comments (3 posted)

Patches and updates

Kernel trees