LWN.net Logo

Not logged in

Log in now

Create an account

Subscribe to LWN

Weekly Edition

Return to the Kernel page

Recent Features

Shumway lands in Firefox

LWN.net Weekly Edition for October 3, 2013

Integrity and embedded devices

NUMA scheduling progress

LWN.net Weekly Edition for September 26, 2013

Printable page
Weekly edition Kernel Security Distributions Contact Us Search
Archives Calendar Subscribe Write for LWN LWN.net FAQ Sponsors

Recent Changes to /sbin/hotplug

February 16, 2005

This article was contributed by Greg Kroah-Hartman.

With the recent announcement of a replacement for the existing Linux Hotplug script project with a version written in C called hotplug-ng, attention has been renewed as to how the whole Linux hotplug process works. This article is an attempt to explain this process, describing the history of how we got here, and pointing out the directions in which things will probably be changing.

The /sbin/hotplug userspace interface for the kernel was created late in the 2.3 kernel development process (yes, way back then.) It was intended to be used to notify userspace that the kernel had discovered a new device. This was done so that userspace could then go and try to load a module for this new device, or do any other type of initialization and setup that might be needed. For a very good explanation of how userspace could determine what driver was needed for what device, and some examples of some very simple /sbin/hotplug implementations, see this paper from the 2001 Ottawa Linux Symposium.

With this humble beginning, the linux-hotplug project started, and over time, a nice collection of shell scripts were created by a number of developers, led primarily by David Brownell. These scripts are installed by almost all Linux distribution, and enable USB, SCSI, Firewire, PCI, and a number of other types of drivers to be loaded automatically when a device is inserted into the system. Thanks to these scripts, Linux accomplished a very good "it just works" feeling for a lot of users of the 2.4 and 2.6 kernels.

As time went on, more and more projects wanted to be notified by the kernel that something had happened so that it could try to do things automatically for the user. Things like:

  • start up and shut down networking interfaces automatically
  • mount storage devices and show an icon on the desktop
So different hooks were patched into the linux-hotplug scripts for these projects, and everyone was happy.

Things got complex

Then along came udev and the 2.6 kernel, and the frailty of the existing hotplug scripts were really felt. The 2.6 kernel changed the way hotplug events were created by the kernel. Instead of only emitting an event for a limited set of devices, everything that had a kobject registered in sysfs created a hotplug event. Due to the driver model conversion of all different busses in the kernel, now hotplug events were being created for many more things than the linux-hotplug scripts cared about. People realized that this was going to cause a big mess and a new type of /sbin/hotplug program was proposed and created.

If you look at the current version of /sbin/hotplug, it is now a very simple bash script: 

DIR="/etc/hotplug.d"
for I in "${DIR}/$1/"*.hotplug "${DIR}/"default/*.hotplug ; do
        if [ -f $I ]; then
                test -x $I && $I $1 ;
        fi
done
exit 1

What this script now does is allow any program to be called for hotplug events from the kernel. Every time a hotplug event is created by the kernel, the script passes execution on to any program listed in the /etc/hotplug.d/ subdirectories. If a program wants to be notified of all hotplug events, they add themselves to the /etc/hotplug.d/default/ directory. If they only care about a single type of bus event, they place themselves in the proper /etc/hotplug.d/BUSNAME/ directory. The program name must end with .hotplug in order to make package managers life simpler.

So, for example, if you want to be notified of all USB hotplug events, put a symlink to your program in the /etc/hotplug.d/usb/ directory that ends in .hotplug . A typical /etc/hotplug.d tree one of my Gentoo-based systems looks like the following: 

/etc/hotplug.d/
`-- default
    |-- 10-udev.hotplug -> ../../../sbin/udevsend
    |-- 20-hal.hotplug -> /usr/libexec/hal.hotplug
    `-- default.hotplug
This arrangement means that udevsend is called first for any hotplug event, followed by HAL, and then finally, the default linux-hotplug scripts.

The recent hotplug-ng announcement merely replaces the existing /sbin/hotplug bash script with a tiny executable program that does the exact same thing. This is useful for machines that have limited memory available, or generate a very high number of hotplug events.

Another noted goal of the hotplug-ng project was to replace the existing linux-hotplug bash scripts for loading modules for new devices with small executable programs. It shipped 3 examples of this, one for USB, PCI and SCSI devices. Soon after the announcement, a IEEE1394 program was submitted for inclusion in the package.

How a module is found

When the kernel finds a new device and registers it with sysfs, a hotplug event is generated that describes the new device in a bus specific manner through a number of different environment variables. For example, a USB device creates the following variables when it is found: 
        PRODUCT=idVendor/idProduct/bcDevice
        TYPE=bDeviceClass/bDeviceSubClass/bDeviceProtocol
        INTERFACE=bInterfaceClass/bInterfaceSubClass/bInterfaceProtocol
Variable Format Description
PRODUCT value/value/value idVendor/idProduct/bcdDevice, from the USB device descriptor. Numbers are hexadecimal, without leading '0x' or zeros.
TYPE value/value/value bDeviceClass/bDeviceSubClass/bDeviceProtocol, from device descriptor. When 0/*/* is seen, a variable of type INTERFACE is also provided. Numbers are decimal.
INTERFACE value/value/value bInterfaceClass/bInterfaceSubClass/bInterfaceProtocol, only for device class zero. Linux 2.6 gives each interface its own hotplug event, and /sys/$DEVPATH/bInterfaceNumber tells them apart. Earlier kernels only reported the first interface. Numbers are decimal.

The hotplug scripts then split those environment variables apart into individual numbers, and then search the /lib/modules/KERNEL_VERSION/module.*map files for the proper matching module for this device. The module.*map files are created by the depmod program in the module-init-tools package by picking out all of the MODULE_DEVICE_TABLE() information from the individual drivers. See the previously mentioned OLS article for more information about this process.

This scanning of the module.*map files by shell scripts has been determined by people to take a relatively long amount of time. The hotplug-ng project tries to solve this by bypassing these files completely, and relying on the fact that the modprobe program can use module aliases to determine what module to load. If you look at the output of the modinfo program on a module from a 2.6 kernel, you will notice a lot of alias entries: 

$ modinfo tulip
filename:       /lib/modules/2.6.11-rc4/kernel/drivers/net/tulip/tulip.ko
author:         The Linux Kernel Team
description:    Digital 21*4* Tulip ethernet driver
license:        GPL
version:        1.1.13
parmtype:       tulip_debug:int
parmtype:       max_interrupt_work:int
parmtype:       rx_copybreak:int
parmtype:       csr0:int
parmtype:       options:array of int
parmtype:       full_duplex:array of int
vermagic:       2.6.11-rc4 SMP PENTIUM4 gcc-3.4
depends:        
alias:          pci:v00001011d00000009sv*sd*bc*sc*i*
alias:          pci:v00001011d00000019sv*sd*bc*sc*i*
alias:          pci:v000011ADd00000002sv*sd*bc*sc*i*
alias:          pci:v000010D9d00000512sv*sd*bc*sc*i*
alias:          pci:v000010D9d00000531sv*sd*bc*sc*i*
alias:          pci:v0000125Bd00001400sv*sd*bc*sc*i*
alias:          pci:v000011ADd0000C115sv*sd*bc*sc*i*
alias:          pci:v00001317d00000981sv*sd*bc*sc*i*
alias:          pci:v00001317d00000985sv*sd*bc*sc*i*
alias:          pci:v00001317d00001985sv*sd*bc*sc*i*
alias:          pci:v00001317d00009511sv*sd*bc*sc*i*
alias:          pci:v000013D1d0000AB02sv*sd*bc*sc*i*
alias:          pci:v000013D1d0000AB03sv*sd*bc*sc*i*
alias:          pci:v000013D1d0000AB08sv*sd*bc*sc*i*
alias:          pci:v0000104Ad00000981sv*sd*bc*sc*i*
alias:          pci:v0000104Ad00002774sv*sd*bc*sc*i*
alias:          pci:v00001259d0000A120sv*sd*bc*sc*i*
alias:          pci:v000011F6d00009881sv*sd*bc*sc*i*
alias:          pci:v00008086d00000039sv*sd*bc*sc*i*
alias:          pci:v00001282d00009100sv*sd*bc*sc*i*
alias:          pci:v00001282d00009102sv*sd*bc*sc*i*
alias:          pci:v00001113d00001216sv*sd*bc*sc*i*
alias:          pci:v00001113d00001217sv*sd*bc*sc*i*
alias:          pci:v00001113d00009511sv*sd*bc*sc*i*
alias:          pci:v00001186d00001541sv*sd*bc*sc*i*
alias:          pci:v00001186d00001561sv*sd*bc*sc*i*
alias:          pci:v00001186d00001591sv*sd*bc*sc*i*
alias:          pci:v000014F1d00001803sv*sd*bc*sc*i*
alias:          pci:v00001626d00008410sv*sd*bc*sc*i*
alias:          pci:v00001737d0000AB09sv*sd*bc*sc*i*
alias:          pci:v00001737d0000AB08sv*sd*bc*sc*i*
alias:          pci:v000017B3d0000AB08sv*sd*bc*sc*i*
alias:          pci:v000010B9d00005261sv*sd*bc*sc*i*
alias:          pci:v000010B9d00005263sv*sd*bc*sc*i*
alias:          pci:v000010B7d00009300sv*sd*bc*sc*i*
srcversion:     2B43BFCB982491A0D0794EC
Those module alias values are created directly from the MODULE_DEVICE_TABLE() values in the driver, and match the modules.*map files information. So, the hotplug-ng programs build up the module alias based on the environment variables passed to it, and then invokes the modprobe program directly. This greatly speeds up the whole module loading process. On this authors slow laptop, it went from 2 seconds to load a USB module for a newly seen device, to less than 1 with the hotplug-ng programs.

Disruption in the force

This was all well and good, until Roman Kagan made the very obvious observation that this whole process of creating environment variables, and then splitting them apart was incredibly stupid. Why not have the kernel itself just create the module alias string in the first place and add that to the hotplug call? That way the whole userspace process could be made incredibly simple. Sometimes the developers that are closest to the problem miss obvious issues like this as they forget to step back and view the whole picture properly. This revelation was received very well , and it will be added to the kernel after 2.6.11 is released, allowing the hotplug-ng programs to be made even smaller.

But what about udev?

One wrinkle on the whole hotplug process is the udev program. Originally, udev only wanted to pay attention to the hotplug events of devices that had a driver already loaded and wanted a node created in the /dev directory. But, in order to do this properly, it needed to listen to all events, sort them in the proper order, and then operate on them. This placement of everything in a sequential order by event generation, made Kay Sievers (one of the main udev developers) realize that he could just make udev operate as the main /sbin/hotplug process.

With the release of the 050 release of udev, if /sbin/udevsend is the kernel hotplug program (it can be changed by modifying the value of the /proc/sys/kernel/hotplug file), then it operates like the original /etc/hotplug.d multiplexer program as well as handling all of the udev device node generation. This ensures that the /etc/hotplug.d/ invocations happen in the proper order, and in sequence for the same device. Gentoo Linux already supports this mode of operation.

However, not every user wants to use udev. Because of that, the hotplug-ng project is continuing, even if it seems like they are competing against each other in implementing the same functionality. As the same developers are doing the work in both programs, all users of Linux benefit with a faster module loading process, and further advancements in hotplug functionality.

(Log in to post comments)

Recent Changes to /sbin/hotplug

Posted Feb 17, 2005 18:54 UTC (Thu) by jwb (guest, #15467) [Link]

I think the author should have mentioned, at least in a footnote, that he is the one who initially cooked up and promoted udev.

Recent Changes to /sbin/hotplug

Posted Feb 17, 2005 19:56 UTC (Thu) by gregkh (subscriber, #8) [Link]

Would that really matter? I was describing how udev fit into the hotplug picture, not trying to advocate for it one way or another.

Recent Changes to /sbin/hotplug

Posted Feb 17, 2005 20:28 UTC (Thu) by jwb (guest, #15467) [Link]

It hardly matters now because I added the footnote myself. But you have a history of taking a barely-usable project and presenting it as a fait accompli. I think your sentence "along came udev" is rather disingenuous in light of the fact that you wrote it as well as hotplug-ng.

If you were to write "I released a new project 6 days ago and now, in my own opinion, the entire world of kernel hotplug has been turned on its head" you'd get a rather different article. The reader deserves to know the relationship between the author and the subject matter.

An LWN article about hotplug-ng would normally highlight the relative youth of the project (not even a week) and the functionality of the previous hotplug system which you have chosen to remove; in other words, giving the reader the complete picture of how hotplug-ng will affect his use of Linux. That's why we pay for LWN, after all.

Recent Changes to /sbin/hotplug

Posted Feb 18, 2005 0:45 UTC (Fri) by dang (guest, #310) [Link]

Flame off, please.

And Greg, thanks for the nice overview of the genesis, point, and implimentation of this change.

Why still keeping both ?

Posted Feb 18, 2005 10:17 UTC (Fri) by xav (guest, #18536) [Link]

Now that they have more functionality in common, I don't understand why keeping both udevsend and hotplug. IMHO the better way would be to keep only udevsend, and just tell it not to create devices nodes in case of a system configured for static /dev.
But then, what do I know.

Recent Changes to /sbin/hotplug

Posted Feb 21, 2005 6:29 UTC (Mon) by mdz@debian.org (subscriber, #14112) [Link]

For what it's worth, the current Ubuntu development branch (Hoary) uses udevstart as the hotplug handler by default, and this approach has worked quite well so far.

Recent Changes to /sbin/hotplug

Posted Feb 21, 2005 18:48 UTC (Mon) by jschrod (subscriber, #1646) [Link]

Thanks for this very nice overview.

One comment, though: The article would have been improved, IMO, if you would have informed

  • how mature hotplug-ng is; i.e., how tested in many environments
  • which major distributions use it (e.g., I notice that SUSE does not use it (yet?))
  • is this change wholeheartly accepted by the hotplug developer community, or is there resistance to keep the old script
This would have helped to evaluate this new development. The nice thing on LWN articles is that they often provide help for this evaluation as well, not ``just'' the technical facts.

Cheers, Joachim

Recent Changes to /sbin/hotplug

Posted Mar 1, 2005 1:56 UTC (Tue) by gregkh (subscriber, #8) [Link]

Being the primary author of hotplug-ng, any opinion as to an "evaluation" of this project might be considered suspect :)

That being said, here's your answers:

> How mature hotplug-ng is
Not at all, it was only 1 week old at the time of writing this article.

> which major distros use it
None. Not even any minor ones that I know of. It's still under heavy development.

> Is it accepted by the hotplug developer community.
Not fully yet. There is a push by others to have udev do the work of hotplug-ng instead. See the linux-usb-devel mailing list for more details of this discussion if you are interested.

/sbin/udevstart : I miss something

Posted Feb 24, 2005 9:42 UTC (Thu) by danielos (subscriber, #6053) [Link]

With the release of the 050 release of udev, if /sbin/udevstart is the kernel hotplug program (it can be changed by modifying the value of the /proc/sys/kernel/hotplug file), then it operates like the original /etc/hotplug.d multiplexer program as well as handling all of the udev device node generation. 
/etc/hotplug.d/
`-- default
    |-- 10-udev.hotplug -> ../../../sbin/udevsend
    |-- 20-hal.hotplug -> /usr/libexec/hal.hotplug
    `-- default.hotplug
It is ok to have udevsend there?
On Debian could it be managed by debconf ?

/sbin/udevstart : I miss something

Posted Mar 1, 2005 1:57 UTC (Tue) by gregkh (subscriber, #8) [Link]

> Is it ok to have udevsend there?

Yes, that is how udev works. I catches the hotplug messages and acts on them to add and remove your /dev nodes.

/sbin/udevstart : I miss something

Posted Mar 2, 2005 12:39 UTC (Wed) by danielos (subscriber, #6053) [Link]

Well, I tried to use /sbin/udevstart as hotplug multiplexer, udev version 054. It does not create device node and do not load modules.

I just do not understand if udevstart do all work and then works as the hotplug multiplexer and then call udevsend that redo the same work, or if udevsend understand it and stop.

Recent Changes to /sbin/hotplug

Posted Feb 26, 2005 7:18 UTC (Sat) by k8to (subscriber, #15413) [Link]

If we are discussing maturing of tools, I am curious when udev will grow support for the normal semantics of the administrative tools chmod and chown.

Recent Changes to /sbin/hotplug

Posted Mar 1, 2005 1:58 UTC (Tue) by gregkh (subscriber, #8) [Link]

What do you mean by "normal semantics"?

udev supports user, group, and mode settings for /dev nodes today.

Or do you mean that if you run chmod or chown on a /dev node, udev will notice this and pick it up for next time? If so, I don't know of anyone who is working on this feature for udev, sorry.

Recent Changes to /sbin/hotplug

Posted Aug 17, 2006 17:52 UTC (Thu) by kpidamale (guest, #37113) [Link]

1 2 3 4 5 6 7 8 9

Recent Changes to /sbin/hotplug

Posted Apr 30, 2006 1:22 UTC (Sun) by maro (subscriber, #34315) [Link]

Albeit plenty late, I just want to thank you for the overview of the process. I'm still using hotplug-ng, even though it seems development has stopped :-(

Copyright © 2005, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds