Kernel development [LWN.net]

Brief items

Kernel release status

The current 2.6 prepatch is 2.6.11-rc4, released by Linus on February 12. This prepatch, intended to be the last before 2.6.11 is released, is dominated by small fixes; it also contains some architecture updates, a serial ATA blacklist (for poorly-behaved drives), some extra checking for read() and write() calls (covered in last week's Kernel Page), a largish Radeon framebuffer update, and an IDE update. The long-format changelog has the details.

Linus's BitKeeper repository contains a handful of fixes, including a patch for a few memset() calls in the S/390 code which had the arguments reversed - and somehow seemed to work anyway.

The current -mm prepatch is 2.6.11-rc3-mm2. Recent changes to -mm include the addition of the realtime Linux security module, an NFS update, and various fixes.

The current 2.4 prepatch is 2.4.30-pre1, released by Marcelo on February 10. As would be expected for a kernel in deep maintenance mode, this prepatch contains a relatively small set of fixes and a couple of driver updates.

Comments (1 posted)

Kernel development news

Quote of the week

However, then when I start looking at n_tty_receive_room() and n_tty_receive_buf(), my stomach gets a bit queasy. I have this horrid feeling that I had something to do with the mess, but I'm going to lash out and blame somebody else, like tytso, for most of it....

I'd love for somebody to try to take a look at where n_tty goes wrong, but I think that for now I'll just make the fix be the cheezy "limit tty chunks to 2kB". It's worked for a decade, it can work for a bit longer ;)

-- Linus Torvalds finds a problem and "fixes" it

Comments (none posted)

The 2.4-hf tree launches

The 2.4 kernel is now in a deep maintenance mode; all of the exciting activity is happening in 2.6 instead. As a result, several months can pass between 2.4 releases. That delay should not normally be a problem, but it can be an issue for users who get bitten by a bug, or who need an important security fix. There are still quite a few systems running 2.4 kernels, after all, and not everybody wants to wait for months for a fix to a show-stopper bug.

It does not appear that the 2.4 process will speed up, however. Instead, Willy Tarreau, working with Marcelo, has created a new 2.4 "hotfix" tree; the first (announced) release is 2.4.29-hf2. This tree is created entirely by cherry-picking patches out of the 2.4 BitKeeper repository; as such, it contains only patches which will be part of the next official 2.4 kernel. The tree contains a few security fixes (none of which appear particularly urgent), one "critical fix" (for a panic in LVM), and various other bug fixes.

The latest -hf patches will always be available on Willy's site.

Comments (2 posted)

Recent Changes to /sbin/hotplug

February 16, 2005

This article was contributed by Greg Kroah-Hartman.

With the recent announcement of a replacement for the existing Linux Hotplug script project with a version written in C called hotplug-ng, attention has been renewed as to how the whole Linux hotplug process works. This article is an attempt to explain this process, describing the history of how we got here, and pointing out the directions in which things will probably be changing.

The /sbin/hotplug userspace interface for the kernel was created late in the 2.3 kernel development process (yes, way back then.) It was intended to be used to notify userspace that the kernel had discovered a new device. This was done so that userspace could then go and try to load a module for this new device, or do any other type of initialization and setup that might be needed. For a very good explanation of how userspace could determine what driver was needed for what device, and some examples of some very simple /sbin/hotplug implementations, see this paper from the 2001 Ottawa Linux Symposium.

With this humble beginning, the linux-hotplug project started, and over time, a nice collection of shell scripts were created by a number of developers, led primarily by David Brownell. These scripts are installed by almost all Linux distribution, and enable USB, SCSI, Firewire, PCI, and a number of other types of drivers to be loaded automatically when a device is inserted into the system. Thanks to these scripts, Linux accomplished a very good "it just works" feeling for a lot of users of the 2.4 and 2.6 kernels.

As time went on, more and more projects wanted to be notified by the kernel that something had happened so that it could try to do things automatically for the user. Things like:

start up and shut down networking interfaces automatically
mount storage devices and show an icon on the desktop

So different hooks were patched into the linux-hotplug scripts for these projects, and everyone was happy.

Things got complex

Then along came

udev

and the 2.6 kernel, and the frailty of the existing hotplug scripts were really felt. The 2.6 kernel changed the way hotplug events were created by the kernel. Instead of only emitting an event for a limited set of devices, everything that had a kobject registered in sysfs created a hotplug event. Due to the driver model conversion of all different busses in the kernel, now hotplug events were being created for many more things than the linux-hotplug scripts cared about. People realized that this was going to cause a big mess and a new type of /sbin/hotplug program was proposed and created.

If you look at the current version of /sbin/hotplug, it is now a very simple bash script:

DIR="/etc/hotplug.d"
for I in "${DIR}/$1/"*.hotplug "${DIR}/"default/*.hotplug ; do
        if [ -f $I ]; then
                test -x $I && $I $1 ;
        fi
done
exit 1

What this script now does is allow any program to be called for hotplug events from the kernel. Every time a hotplug event is created by the kernel, the script passes execution on to any program listed in the /etc/hotplug.d/ subdirectories. If a program wants to be notified of all hotplug events, they add themselves to the /etc/hotplug.d/default/ directory. If they only care about a single type of bus event, they place themselves in the proper /etc/hotplug.d/BUSNAME/ directory. The program name must end with .hotplug in order to make package managers life simpler.

So, for example, if you want to be notified of all USB hotplug events, put a symlink to your program in the /etc/hotplug.d/usb/ directory that ends in .hotplug . A typical /etc/hotplug.d tree one of my Gentoo-based systems looks like the following:

/etc/hotplug.d/
`-- default
    |-- 10-udev.hotplug -> ../../../sbin/udevsend
    |-- 20-hal.hotplug -> /usr/libexec/hal.hotplug
    `-- default.hotplug

This arrangement means that udevsend is called first for any hotplug event, followed by HAL, and then finally, the default linux-hotplug scripts.

The recent hotplug-ng announcement merely replaces the existing /sbin/hotplug bash script with a tiny executable program that does the exact same thing. This is useful for machines that have limited memory available, or generate a very high number of hotplug events.

Another noted goal of the hotplug-ng project was to replace the existing linux-hotplug bash scripts for loading modules for new devices with small executable programs. It shipped 3 examples of this, one for USB, PCI and SCSI devices. Soon after the announcement, a IEEE1394 program was submitted for inclusion in the package.

How a module is found

When the kernel finds a new device and registers it with sysfs, a hotplug event is generated that describes the new device in a bus specific manner through a number of different environment variables. For example, a USB device creates the following variables when it is found:

        PRODUCT=idVendor/idProduct/bcDevice
        TYPE=bDeviceClass/bDeviceSubClass/bDeviceProtocol
        INTERFACE=bInterfaceClass/bInterfaceSubClass/bInterfaceProtocol

Variable	Format	Description
`PRODUCT`	value/value/value	idVendor/idProduct/bcdDevice, from the USB device descriptor. Numbers are hexadecimal, without leading '0x' or zeros.
`TYPE`	value/value/value	bDeviceClass/bDeviceSubClass/bDeviceProtocol, from device descriptor. When 0// is seen, a variable of type INTERFACE is also provided. Numbers are decimal.
`INTERFACE`	value/value/value	bInterfaceClass/bInterfaceSubClass/bInterfaceProtocol, only for device class zero. Linux 2.6 gives each interface its own hotplug event, and /sys/$DEVPATH/bInterfaceNumber tells them apart. Earlier kernels only reported the first interface. Numbers are decimal.

The hotplug scripts then split those environment variables apart into individual numbers, and then search the /lib/modules/KERNEL_VERSION/module.*map files for the proper matching module for this device. The module.*map files are created by the depmod program in the module-init-tools package by picking out all of the MODULE_DEVICE_TABLE() information from the individual drivers. See the previously mentioned OLS article for more information about this process.

This scanning of the module.*map files by shell scripts has been determined by people to take a relatively long amount of time. The hotplug-ng project tries to solve this by bypassing these files completely, and relying on the fact that the modprobe program can use module aliases to determine what module to load. If you look at the output of the modinfo program on a module from a 2.6 kernel, you will notice a lot of alias entries:

$ modinfo tulip
filename:       /lib/modules/2.6.11-rc4/kernel/drivers/net/tulip/tulip.ko
author:         The Linux Kernel Team
description:    Digital 21*4* Tulip ethernet driver
license:        GPL
version:        1.1.13
parmtype:       tulip_debug:int
parmtype:       max_interrupt_work:int
parmtype:       rx_copybreak:int
parmtype:       csr0:int
parmtype:       options:array of int
parmtype:       full_duplex:array of int
vermagic:       2.6.11-rc4 SMP PENTIUM4 gcc-3.4
depends:        
alias:          pci:v00001011d00000009sv*sd*bc*sc*i*
alias:          pci:v00001011d00000019sv*sd*bc*sc*i*
alias:          pci:v000011ADd00000002sv*sd*bc*sc*i*
alias:          pci:v000010D9d00000512sv*sd*bc*sc*i*
alias:          pci:v000010D9d00000531sv*sd*bc*sc*i*
alias:          pci:v0000125Bd00001400sv*sd*bc*sc*i*
alias:          pci:v000011ADd0000C115sv*sd*bc*sc*i*
alias:          pci:v00001317d00000981sv*sd*bc*sc*i*
alias:          pci:v00001317d00000985sv*sd*bc*sc*i*
alias:          pci:v00001317d00001985sv*sd*bc*sc*i*
alias:          pci:v00001317d00009511sv*sd*bc*sc*i*
alias:          pci:v000013D1d0000AB02sv*sd*bc*sc*i*
alias:          pci:v000013D1d0000AB03sv*sd*bc*sc*i*
alias:          pci:v000013D1d0000AB08sv*sd*bc*sc*i*
alias:          pci:v0000104Ad00000981sv*sd*bc*sc*i*
alias:          pci:v0000104Ad00002774sv*sd*bc*sc*i*
alias:          pci:v00001259d0000A120sv*sd*bc*sc*i*
alias:          pci:v000011F6d00009881sv*sd*bc*sc*i*
alias:          pci:v00008086d00000039sv*sd*bc*sc*i*
alias:          pci:v00001282d00009100sv*sd*bc*sc*i*
alias:          pci:v00001282d00009102sv*sd*bc*sc*i*
alias:          pci:v00001113d00001216sv*sd*bc*sc*i*
alias:          pci:v00001113d00001217sv*sd*bc*sc*i*
alias:          pci:v00001113d00009511sv*sd*bc*sc*i*
alias:          pci:v00001186d00001541sv*sd*bc*sc*i*
alias:          pci:v00001186d00001561sv*sd*bc*sc*i*
alias:          pci:v00001186d00001591sv*sd*bc*sc*i*
alias:          pci:v000014F1d00001803sv*sd*bc*sc*i*
alias:          pci:v00001626d00008410sv*sd*bc*sc*i*
alias:          pci:v00001737d0000AB09sv*sd*bc*sc*i*
alias:          pci:v00001737d0000AB08sv*sd*bc*sc*i*
alias:          pci:v000017B3d0000AB08sv*sd*bc*sc*i*
alias:          pci:v000010B9d00005261sv*sd*bc*sc*i*
alias:          pci:v000010B9d00005263sv*sd*bc*sc*i*
alias:          pci:v000010B7d00009300sv*sd*bc*sc*i*
srcversion:     2B43BFCB982491A0D0794EC

Those module alias values are created directly from the MODULE_DEVICE_TABLE() values in the driver, and match the modules.*map files information. So, the hotplug-ng programs build up the module alias based on the environment variables passed to it, and then invokes the modprobe program directly. This greatly speeds up the whole module loading process. On this authors slow laptop, it went from 2 seconds to load a USB module for a newly seen device, to less than 1 with the hotplug-ng programs.

Disruption in the force

This was all well and good, until Roman Kagan made the very obvious observation that this whole process of creating environment variables, and then splitting them apart was incredibly stupid. Why not have the kernel itself just create the module alias string in the first place and add that to the hotplug call? That way the whole userspace process could be made incredibly simple. Sometimes the developers that are closest to the problem miss obvious issues like this as they forget to step back and view the whole picture properly. This revelation was received very well , and it will be added to the kernel after 2.6.11 is released, allowing the hotplug-ng programs to be made even smaller.

But what about `udev`?

One wrinkle on the whole hotplug process is the udev program. Originally, udev only wanted to pay attention to the hotplug events of devices that had a driver already loaded and wanted a node created in the /dev directory. But, in order to do this properly, it needed to listen to all events, sort them in the proper order, and then operate on them. This placement of everything in a sequential order by event generation, made Kay Sievers (one of the main udev developers) realize that he could just make udev operate as the main /sbin/hotplug process.

With the release of the 050 release of udev, if /sbin/udevsend is the kernel hotplug program (it can be changed by modifying the value of the /proc/sys/kernel/hotplug file), then it operates like the original /etc/hotplug.d multiplexer program as well as handling all of the udev device node generation. This ensures that the /etc/hotplug.d/ invocations happen in the proper order, and in sequence for the same device. Gentoo Linux already supports this mode of operation.

However, not every user wants to use udev. Because of that, the hotplug-ng project is continuing, even if it seems like they are competing against each other in implementing the same functionality. As the same developers are doing the work in both programs, all users of Linux benefit with a faster module loading process, and further advancements in hotplug functionality.

Comments (15 posted)

Patches and updates

Kernel trees