LCE: Systemd two years on

It has now been a little more than two years since the systemd project began, and in that time a lot has happened. So Lennart Poettering, one of the main developers of systemd, took the opportunity to look back at the progress to date, in a talk at LinuxCon Europe 2012 . Kay Sievers, the other principal systemd developer, was also in the audience, and offered some pieces of background information during the talk.

The first announcement of systemd was in April 2010. The first distribution based on systemd, Fedora 15, came out in May 2011. Since then, it has appeared as the default init system in a number of other distributions, including openSUSE, Mageia, Mandriva, Arch Linux, and others. Systemd is also included in Debian and Gentoo, but it is not the default init system. The most notable absence from the list is, of course, Ubuntu. Lennart expressed the hope that Ubuntu might still switch over, but sounds somewhat less confident of that these days than he has in the past.

Lennart reported that the systemd project is in good health, with 15 committers (including committers from a range of distributors) and 374 contributors to date, although the latter number is inflated by the incorporation of udev into systemd; if the udev contributors that have contributed to that project during its seven-year lifespan are excluded, then systemd has had about 150 contributors. There are about 30 contributors in any given month, and around 490 subscribers to the project mailing list.

One of the goals of systemd was to be able to boot a system without requiring shell scripts, and that goal was achieved about a year ago. The problem with shell scripts, of course, is that they make it difficult to implement the necessary parallelization and asynchronous logic that are needed for fast boot times. As a result of eliminating shell scripts from the boot phase, the number of processes created during system start-up has fallen from 1500 to about 150. On a system with an SSD, booting a full user space can be accomplished in under one second on a standard PC-architecture laptop. Users of Fedora are unlikely to achieve boot times that are quite that fast, because of dependencies like LVM, but boot times of 10 seconds should be achievable "out of the box" with Fedora. Thus, whereas once upon a time the kernel was fast and user space was slow to boot, these days the position has reversed. Lennart remarked that the ball is now back in the kernel's court in terms of improving boot speeds.

Lennart noted that systemd started out as an init system, and as such replaces the classical System V init (sysvinit) program. But along the way, the developers figured out that booting a system is not solely the job of PID 1 (the PID created for the first process on the system, which executes the init program). There are other processes as well, such as init scripts. So the developers redefined systemd a little more, to be not just an init system, but also a platform. Thus, systemd includes a couple of mini-services that are useful for booting. These services include, for example, reading ahead files that were used during the last boot so that booting proceeds faster this time around. Systemd also added a number of other components, such as udev and a system logger, and replacements for pieces like ConsoleKit. The goal of including these services and components within systemd is to create a more integrated boot system.

These changes are sometimes perceived as feature creep in what is supposed to be simply an init system, which prompts the question: is it bloated? Lennart's response is an emphatic "no". In the systemd platform, PID 1 is simply a unit control system. Other tasks are performed by other pieces that ship as part of the systemd package. He likened this to being a repackaging of what once were separate tools that interacted during the boot phase. That repackaging allows a more integrated approach to the boot process that also eliminates some code and feature duplication that occurred across the formerly separate components. By eliminating duplication, and providing a more integrated approach to the boot components, the resulting system is easier for system administrators to manage. The increasing scope of systemd has created more dependencies on other packages, but Lennart noted that almost all of these dependencies are optional, and only required if users want the optional features available in systemd.

At this point, an audience member asked about the status of systemd with Busybox for embedded systems. Lennart noted that he works mainly on desktop systems, but the project certainly does care about embedded systems.

Systemd integrates with some other operating system components. For example, Dracut can pass performance information to systemd, as well as other information, such as whether the root file system was consistency checked. Systemd integrates closely with udev—so closely that udev was eventually merged into systemd, a decision that was, as Lennart noted, widely discussed, and somewhat controversial. Systemd integrates with D-Bus activation, so that if a D-Bus name is requested, a service will be started. Systemd integrates with the Plymouth boot screen splash tool, so that the two systems coordinate their handling of the screen and passwords. It integrates with the Gummiboot boot loader for EFI, so that it is possible to get boot statistics such as measures of the amount of time spent in BIOS, bootloader, kernel initialization, in init ramdisk, and in user-space activation.

Like the Linux kernel, systemd targets the full spectrum of platforms, with features for mobile, embedded, desktop, and server systems, and Lennart remarked that it was interesting that these diverse platforms gained common benefits from systemd. For example, the control-group management features for systemd are of interest for resource management in both server and embedded systems.

An audience member noted that the systemd project has not focused on porting systemd to other operating systems, such as FreeBSD, and asked whether there was no FreeBSD developer who said they'd like to port systemd to FreeBSD? That has not happened so far, Lennart said. The problem is that systemd is tightly bound to various Linux-specific features, such as control groups, for which there are no equivalents on other operating systems. Kay noted that attempts to port udev to other systems went nowhere, and that would be a necessary first step to port systemd.

Systemd has made a number of components in the traditional init system obsolete. The most obvious of these are, of course, sysvinit and init scripts, which are replaced by small C programs that can be parallelized during boot. In addition, systemd made a number of other components obsolete, such as ConsoleKit, pm-utils, and acpid (the ACPI daemon). Systemd's socket-activation feature renders inetd obsolete. Lennart emphasized that while systemd renders all of these components obsolete, most of the components (with obvious exceptions such as sysvinit) can still be used in parallel with systemd, if desired.

At this point, an audience member opined that the classical UNIX init process was designed to be as small and robust as possible, but that this seemed to contrast with where systemd was going. Is systemd at risk of becoming a large single point of failure? Lennart pointed out that while systemd replaces many earlier tools, it performs those tasks in separate processes. However, the notable difference is that the various components operate in a much more integrated fashion.

Lennart then turned to a discussion of the Journal. The Journal is a recently announced systemd component, which is posited as the successor of the traditional syslog logging system. The syslog system is nearly 30 years old, renders log messages in plain text, and does not lend itself to seemingly simple tasks such as showing the last ten log messages produced by a particular service. There are many problems with syslog. For example, it does not index the log and it does not verify the source of log messages.

The systemd developers concluded that logging is an essential part of service management, and none of the current solutions was adequate for systemd's needs, so they came up with the Journal. The Journal has structured metadata about every log event, including eighteen fields with information such as user ID, group ID, and SELinux context. The log is indexed, and secured so that the logged data is trustworthy. The Journal is also rate limited, so that no individual service can clog the log. The Journal has forward secure sealing, so that, periodically (by default, every fifteen minutes) a cryptographic tag is added to the logged data, and the cryptographic key is discarded. The effect is that an attacker who breaks into the system can't alter history in the log (other than possibly the log entries created in the current interval). Work is ongoing to add network logging functionality to the Journal.

Unfortunately, Lennart ran out of time, so that he was unable to go over his thoughts about the future of systemd. However, after two years, it's clear that systemd is an established part of the Linux landscape, and there are increasing signs that it is moving toward becoming an essential part of the operating system.