Highlights from systemd v258: part one

The next release of systemd has been percolating for an unusually long time. Systemd releases are usually about six months apart, but v257 came out in December 2024, and v258 just now seems to be nearing the finish line; the third release candidate for v258 was published on August 20 (release notes). Now is a good time to dig in and take a look at some of the new features, enhancements, and removals coming soon to systemd. These include new workload-management features, a concept for multiple home-directory environments, and the final, once-and-for-all removal of support for control groups version 1.

Lennart Poettering has been doing his now-traditional parade of systemd feature highlights on Mastodon for v258 by posting about one feature daily (with some gaps for weekends, etc.); he started on May 21, and is now up to the 46th installment as of August 26. This installment will focus on some of the new features, with a follow-up article after the v258 release to cover others.

Gone or going

The systemd folks have been banging the drum to warn users that control groups v1 support was going away at least since the v253 release in February 2023. Support was disabled in v257, but it could be re-enabled by setting the SYSTEMD_CGROUP_ENABLE_LEGACY_FORCE=1 option on the kernel command line. Now v1 support is gone for good, and systemd only supports control groups v2. Poettering said that this has allowed the project "to remove a lot of code and simplify the logic of cgroup handling quite substantially".

The minimum kernel version for v258 has been bumped to 5.4, which was released in 2019. The minimum recommended kernel version is 5.7, which was released in 2020. (LWN subscribers can view the release history of all Git-era kernels via our kernel source database.) The support for System V service scripts is deprecated and will be removed in v259. The release notes suggest that developers update software to include a standard systemd unit file to ensure future compatibility. Additionally, systemd v258 sheds support for System V run levels and no longer provides initctl, runlevel, and telinit commands. (These were actually just symbolic links to systemctl, which would emulate the behavior of those legacy commands.)

The project has made OpenSSL the only supported cryptography backend for the systemd-resolved and systemd-importd services; support for GnuTLS and libgcrypt has been removed in v258 in an effort to use just one cryptography provider. Poettering polled the systemd-devel mailing list in April to ask if anyone was using these backends before they were removed, but there were no replies.

Workload management

Systemd uses the concept of slices to allow managing the resources of a group of processes; it uses unit-configuration files with the ".slice" suffix that specify resource limits to be applied using control groups. With v258, systemd is extending this to provide some basic workload management functionality by adding two new controls for .slice units: ConcurrencySoftMax= and ConcurrencyHardMax=.

These control knobs add limits to the maximum number of units, what most folks would call jobs, that may be active for a slice at the same time. The soft concurrency limit dictates how many jobs can be active at the same time; if there are more units to be run than slots available, then the others will be queued up to run after the active units finish. If the hard concurrency limit is set, then any additional jobs above that limit will simply fail. For example, if one sets ConcurrencyHardMax=5 and there are six units, then the sixth will fail.

This is a fairly coarse-grained knob for workload control; Poettering said that the project may eventually extend workload control to address CPU, I/O, memory limits, and so on, to give greater control over defining when workloads should run.

Home areas

Each regular user on a Linux system gets a single home directory, usually /home/user, where their configuration information and data are stored. This is typically sufficient, but in some cases it might be useful to be able to have more than one home directory. For example, one might want separate home environments for specific development projects or one home directory for work and another for personal projects.

In v258, systemd has introduced limited support for separate login environments called "home areas". Each home area is maintained in a subdirectory, under /home/user/Areas, of the user's regular home directory. For example, a person who does freelance work could have a separate home area for each work project, such as /home/user/Areas/proj1 that would have configuration files and data used for a specific project.

To log into that home area, they would add "%" and a "proj1" suffix to their username ("user%proj1") and use their regular password. That would set /home/user/Areas/proj1 as $HOME, and provide a cleaner separation between data and configuration files used for that project and their regular home environment. Poettering explained on Mastodon how he uses this feature:

My personal usecase for this goes something like this: I regularly build disk images for VMs of different distributions on my host, for building and testing systemd and other stuff. And I want access to my host user's home dir in each when booting them up, but not necessarily log directly into my host's home directory, but keep a separate area for each such image, so that the build trees in it can be distinct, do not leak context into each other, and can be created anew and flushed out easily. I also want full access to all my data from each such VM to simplify my work. And the area concept allows me to do all that.

All files still have the user's regular user ID (UID) and group ID (GID): there is no privilege separation between areas, and there are no plans to add it, so a user can still access files in their home directory or other home areas while logged in with "user%proj1".

The addition of the areas feature comes with a new "defaultArea" user record field, which (as the name suggests) can be used to specify a default home area. The standard /home/user is the default if no area is specified. This can be specified with the "homectl --default-area=" command.

Currently, the areas feature does not allow logging into a full graphical session using this feature because systemd does not start a per-area service manager for the user. However, the release notes indicate that this is planned for "one of the next releases". Poettering has said that the areas feature is only implemented for users managed with systemd-homed; it is unlikely that it will ever support "classic" Linux users defined in /etc/passwd. It is also not possible to use the "%area" syntax with SSH, at least at this time. The project has added an option (--area=) to the run0 command to allow a user to directly enter a specific home area when using run0 to elevate or acquire different user privileges.

Quotas for tmpfs

With systemd v258, users are automatically assigned a quota for /dev/shm and /tmp if /tmp is backed by a tmpfs. The default is 80% of the size assigned to the tmpfs. The goal for this feature is to prevent any user from effectively performing a denial-of-service attack by filling up /tmp or /dev/shm.

This goes hand-in-hand with another change in systemd v258: new user-record fields to specify quotas for /tmp and /dev/shm, as well as support for managing those fields with the homectl command. The fields are tmpLimit, tmpLimitScale, devShmLimit, and devShmLimitScale; they can be set or modified using the homectl command. For example, to set a user's /tmp quota to max out at 20% of the available tmpfs filesystem, one would use "homectl --tmp-limit=20%".

System reset

Systemd added systemd-repart to automatically manage disk partitions in v245, which was released in 2020. One of its features is a "factory reset" mode, invoked with the --factory-reset=yes option, that erases existing partitions to reset an installation to vendor defaults. With v258, systemd has added the systemd-factory-reset tool to provide a framework for a more complete reset that includes TPM keys and other bits.

In describing this tool on Mastodon, Poettering said that he believed "a fundamental aspect of OS security must be a secure way to return the OS into a well-defined, secure state" if the system had been compromised. It was not sufficient to reinstall an operating system, he said, because that would not take care of resetting the TPM key. If the old key is not invalidated, then all secrets associated with a compromised install would remain valid.

The current implementation only addresses resetting storage partitions and the TPM key, but the documentation notes that other services can be defined and plugged into the factory-reset target as well.

Miscellaneous changes

The "systemctl start" command now has a verbose (-v) option, which will display the service's logs as it is started until its startup is completed. This should make it somewhat easier to debug problems with service startup. Previously, if a service failed to start for some reason, it would be necessary to use a command like "journalctl -u" command to view the log. A feature request was created for a verbose option in 2015; Poettering said that it has been a much-requested feature since, but it was difficult to implement due to the number of sources that contribute to the log stream.

Systemd v258 has added support for generating hostnames with a dynamic part; one can request a hostname with a static string and a string that is assigned dynamically. This is done by including question mark characters ("?") in /etc/hostname. Each "?" is replaced with a nibble that is hashed from the system's machine-id.

This might be useful, for example, when managing a fleet of systems; it might be desirable to assign a static hostname part to designate (say) what the system is used for, or some other common detail for a subset of the fleet. For example, if the hostname is specified as "web-????????" in /etc/hostname for web servers then the hostname might be set to "web-98b0e71d". If there are multiple database servers they might be "db-????????", and so forth. There is not a preset limit on how many question marks can be used when setting the dynamic portion of the hostname, but Poettering notes that the kernel has a limit of 63 characters for a hostname.

It is possible to require certain conditions before a systemd unit is executed by specifying them in a systemd unit file. For example, one might want to ensure that a unit is only executed on the arm64 hardware architecture, which can be expressed with ConditionArchitecture=arm64. These are evaluated by systemd-analyze.

The ConditionKernelVersion= check was introduced in systemd v244, which allowed a test for the kernel version as returned by uname -r. For instance, one might use ConditionKernelVersion=6.12.*deb* to check whether a system is running a version of the 6.12 Linux kernel on a Debian system. With v258, ConditionKernelVersion= has been replaced with ConditionVersion=, which currently also supports a check for the version of glibc or systemd in use.

This release also adds ConditionKernelModuleLoaded=, which is used to check if a specified kernel module is already loaded. That logic can be used to skip or enable a service, which may be useful to speed up the boot process by suppressing unneeded modprobe activity when it is not needed.

Systemd supports the use of extension images, which are discoverable disk images (DDIs) that are overlay filesystems that can be applied to a system to extend directory hierarchies. System extensions (sysext) are used to extend /usr or /opt, while configuration extensions (confext) are used for /etc. This can be useful to add software to an immutable system, for example. This blog post from the Flatcar Container Linux project about system extension images is a good overview of how they might be used.

With systemd v258, it is now possible to automatically reload confext images associated with a service when that service is reloaded via "systemctl reload" or similar. See the systemd-sysext man page for more on the commands and unit files associated with system extensions.

Testing systemd

There is quite a bit to chew on in systemd v258. Those wanting to test systemd v258 immediately can find up-to-date packages via openSUSE's Open Build Service (OBS) for Arch Linux, Debian, Fedora, openSUSE, and Ubuntu. Developers who want to contribute to systemd—or just tinker with development versions of systemd—can follow the "hacking on systemd" documentation to use mkosi to build images with the systemd source. It takes 20-30 minutes to set up using mkosi, compile systemd, and build an image—though some of that is dependent on the speed of one's computer.

In the next installment we'll take a look at new features related to disk quotas, service-manager improvements, systemd-resolved changes, and more.