Linux drivers in user space — a survey

Writing device drivers in user space, rather than as kernel modules, is a topic that comes up from time to time for a variety of reasons. The kernel's approach to user-space drivers varies considerably depending on the type of device involved. The recent posting of a patch set aimed at allowing LED drivers to be written as user-space programs seems like a suitable opportunity to have a look at the range of options currently available.

For it to be possible to write a device driver in user space it is necessary for the kernel to export the required interfaces. There are two different sorts of interfaces, that meet different needs, that the kernel can export; I will call them "upstream" and "downstream" interfaces.

When one reflects on the tree-like nature of the driver model, as described in an earlier article, it is clear that there is a chain, or path, of drivers from the root out to the leaves, each making use of services provided by the driver closer to the root (or "upstream") and providing services to the driver closer to the leaf (or "downstream"). An upstream interface allows a user-space program to directly access services provided by the kernel that normally are only accessed by other kernel drivers. A downstream interface allows a user-space program to instantiate a new device for some specific kernel driver, and then provide services to it that would normally be provided by some other kernel driver.

Upstream interfaces

An upstream interface is one that provides access to some hardware, possibly more directly than with the standard interfaces. In several cases this is provided not with a new interface but with a slight modification to an existing interface. Opening a block device with the O_DIRECT flag allows directly reading from and writing to that device without involving the page cache or the readahead and write-behind that it supports. Similarly, direct access to a serial port is obtained by opening a TTY device and disabling certain termios settings such as ECHO and ICANON. The documentation for cfmakeraw() identifies 16 such flags that are cleared.

Direct access to a network device can be achieved by creating a network socket using the AF_PACKET address family and specifying the SOCK_RAW communication type. This socket can then be bound to a particular interface or a particular Ethernet protocol type. A slightly less direct interface can be had by using SOCK_RAW with AF_INET. This still provides the routing and other functionality common to all IP protocols, but gives complete control over the payload of each IP packet.

Moving on to more purpose-built interfaces, the sg and bsg drivers (SCSI generic and block SCSI generic) both provide direct access to SCSI devices, or other devices such as SATA that use a compatible protocol. They allow SCSI command descriptor blocks (CDBs) to be sent to devices and to have results returned. The bsg interface is integrated with the block layer and supports a newer version of the sg interface that includes support for bidirectional commands. libsgutils is the recommended mechanism for making use of these interfaces, rather than working directly with /dev/sgN. Similarly, libusb provides a direct interface to USB devices, allowing arbitrary USB commands to be sent to any connected USB device.

I2C and SPI — 2-wire and 4-wire buses for communicating between integrated circuits on the same board — can be directly accessed via special-purpose character devices. For I2C, the i2c-tools package provides a scriptable interface. For SPI there do not appear to be any packaged solutions, though the armbedded.eu web site provides some code that would be worth trying for anyone who is interested.

All the interfaces listed so far are always available, to sufficiently privileged processes, if the kernel knows about the target device at all. Other interfaces require the kernel to be explicitly instructed to export a low-level interface. In the case of GPIOs (general-purpose I/O pins) and power regulators, this is as simple as adding some directives to the device-tree description of the hardware. The devices then appear in sysfs complete with attribute files allowing relevant settings to be changed and values to be read.

Finally, and requiring even more in-kernel support, is the UIO framework, which is intended for devices that are accessed through memory-mapped device registers, as is the norm for devices attached to PCI and similar buses. A simple in-kernel device driver can be written using the UIO framework that allows a user-space program to map that register bank into its own memory, and also to respond to interrupts from the device. This does not provide generic access to any PCI device, but does make it easy to get user-space access to a particular device of interest, so that the bulk of the driver can be developed, debugged, and maintained outside of the kernel.

This variety of different interface styles could be seen as a hodge-podge that is just crying out to be unified. On the other hand, different sorts of devices really are different and need different sorts of interfaces. Part of the role of an operating system like Linux is to hide as much of that difference as possible behind uniform abstractions. It should not be surprising that, if we want to bypass those abstractions and access the devices directly, we will be confronted by the variety that Linux generally tries to hide.

Downstream interfaces

Where upstream interfaces provide direct access to hardware, downstream interfaces allow a program to emulate some hardware and so provide access to other programs that expect to use a particular sort of interface. Rather than just providing a different sort of access to an already existing device, a downstream interface must make it possible to create a new device, configure it, and then provide whatever functionality is expected of that device type.

Probably the first driving force for these downstream interfaces was the introduction of networking and the consequent desire to allow a program on one computer to work with a device on another computer. With this came pseudo TTYs (PTYs), which are likely the oldest downstream interface in Unix. They allow a TTY to be created on which a user can log in and run programs that don't need to be aware that they are not attached to a physical terminal. The text entered can easily come from anywhere on the network, and the output generated can go back to the same place (or elsewhere).

The desire for network access to storage brought about such things as nbd, the network block device, and NFS, the network file system. Their design differs from that of PTYs in that they don't just provide an interface to user space that a network service could use but, instead, create the network connection themselves and define a protocol to carry the data and control over that connection. The most likely reason for this is that managing a storage service in a user-space program is prone to deadlocks. If the program ever needs to allocate memory, the kernel might choose to free up memory by writing out to a storage device, and if that device is managed by the program allocating memory it could easily deadlock. It is much safer to bypass user space and send directly to the network.

These network protocols can still serve as downstream interfaces in that they make it possible to instantiate a block device (with nbd) or a filesystem (with NFS) and provide services to it. This has been used to good effect with automounting programs such as amd (subsequently renamed to am-utils) that present as an NFS filesystem that contains only directories and symlinks (thus avoiding any deadlock issues) and transparently mounts filesystems when they are first accessed.

Though using NFS for this purpose is quite effective, it is not perfect; due to the limited possible interactions with the Linux virtual filesystem layer, filesystems must be mounted somewhere else and the NFS filesystem only contains a symbolic link to the real mount point. To address this shortcoming, Linux provides a dedicated downstream interface for creating filesystems, autofs, which supports the extra interactions required to automount filesystems directly onto directories.

Similarly there is a downstream interface for writing filesystems that is careful about how it interfaces with the page cache, and manages to avoid the writeback deadlocks described above: FUSE.

As part of FUSE there is CUSE, which allows character devices to be implemented in user space. There does not appear to be a corresponding "BUSE" for implementing block devices in user space, though some years ago there was a proposal for "ABUSE" which aimed to do just that. Block devices can be implemented in user space on a remote machine using nbd and presumably that is sufficient to meet most needs.

Networking plays a role in the next pair of examples too; the TUN and TAP drivers allow network devices to be emulated. TAP sends and receives Ethernet frames, so any networking protocol can be used with a TAP device. TUN works at the IP level, which is simpler and often sufficient providing there is no need to handle non-IP protocols such as ARP. These can most obviously be used for tunneling and creating virtual private networks (VPNs) but could also be used for user-space monitoring and filtering of network traffic.

Network devices, block devices and character devices (which include TTYs) cover all the device types that Unix supported before Linux came along. Linux has added a variety of new device types, some of which can be implemented in user space.

The input subsystem provides a standard interface for input device such as keyboards, mice, joysticks, touch pads, and similar devices. These are exposed to user space as character devices, so it might be possible to emulate them using CUSE, but it is more convenient if they are integrated with the rest of the input subsystem, and that is what uinput allows. If a program opens /dev/uinput and issues some ioctl() commands, a new input device is created. Events will be reported on that device when they are written to the file descriptor opened on /dev/uinput.

User-space LEDs — how and why

The latest addition to the collection of downstream interfaces is conceptually similar to uinput but it allows the emulation of LED devices rather than input devices. To support this functionality, it introduces a new device called /dev/uleds. Opening this device and writing the name of a new device (zero-padded to 64 bytes) will create an LED device with the given name.

There is no option to configure any other aspects of the LED, but there is not much that could be configured anyway. A LED can generally indicate the number of brightness levels that it can support; LEDs created with uleds always support 256 brightness levels. Whenever the brightness is changed, a byte can be read which reports the new level. An LED can also indicate that it knows how to blink so that, when needed, it can be given a single "blink" request rather than periodic "on" and "off" requests. A uleds device cannot be used to experiment with this functionality, but it could undoubtedly be added later using an ioctl() if a need was found.

The particular need that is driving the development of this interface by David Lechner is the desire to make two embedded systems compatible with one another. "I would like to make a userspace program that works the same on both devices." If that program accesses an LED device directly, the device must appear to be present on both systems; where it isn't physically present it can now be emulated, possibly using a widget on a graphic display.

At much the same time Marcel Holtmann had been working on a similar interface to allow the testing of LED triggers from the Bluetooth subsystem. Various subsystems can be connected to a LED, using a trigger, to signal the current state of that subsystem. Without an LED device, it is hard to test those triggers. With the ability to emulate a LED device, that impediment to development need no longer exist.

The 4th revision of the user-space LEDs patch set was posted in mid-September and appears to have addressed all the issues that reviewers found. We can expect the code to land in mainline for Linux 4.10. It seems unlikely that this will be the last device type that someone will want to emulate. Some devices, such as power regulators, seem so intimately related to hardware that it is hard to imagine an emulator ever being wanted. Others, like maybe a GPIO, might usefully be provided with a downstream interface for emulation. Whether there turns out to be a genuine need for that is something we will have to wait to see.