A report from the 2022 Image-Based Linux Summit

One of the motivations for the summit was the simple fact that much of the wider ecosystem has been thinking about the same set of problems. For example, our employer, Microsoft, has made use of a lot of the concepts covered by the summit in the recently announced ARM64-based Azure offload SoC, which is running a custom, security-hardened Linux distribution. While we were thinking, tinkering, and writing about new ways to improve the current state of the art, it became obvious to us that many vendors are working, more or less, in the same space, doing similar work with varying degrees of overlap. However, little to no collaboration was happening. The summit was meant to identify and agree on common concepts and come up with a set of initial specifications. Some of them already have reference implementations.

So we invited technical representatives from the engineering groups of various vendors and distributions that have been known to work on related topics. The summit was intentionally kept small, as it was meant to be a series of conversations and brainstorming sessions, with no fixed agenda or presentations — a BoF-style event. The 30 participants met in the Microsoft office in Berlin and discussed a range of topics from a list that the authors and participants had put together in advance. The topics covered were focused around the idea of shipping Linux via images and with enhanced security features.

Participants were affiliated with numerous companies or projects, including Canonical, Ubuntu Core, Debian, Gnome OS, Fedora CoreOS, Red Hat, Endless OS, Arch Linux, openSUSE, Flatcar, Microsoft, Amazon/AWS, Meta, System Transparency, systemd, image-builder/osbuild, mkosi, and rpm-ostree. As a result of this summit, the Linux Userspace API ("UAPI") Group was founded; it is a community for people with an interest in innovating how we build, deploy, and run modern Linux operating systems. It serves as a central gathering place for specs, documentation, and ideas. The associated uapi-group.org website contains the meeting minutes from the summit and links to current specifications and ideas.

Building Blocks for Images

Several concepts (and, of course, acronyms) that are supported by the systemd project were at the center of many discussions:

• A UKI (Unified Kernel Image) is a secure-boot-signed UEFI executable file that wraps a kernel image, an initrd image, a kernel command line, and more. A UKI can boot on EFI and integrates nicely with systemd-boot.
• Discoverable Disk Images (DDIs) are self-described filesystem images, heavily inspired by Canonical's Snaps, that have been enhanced to follow the Discoverable Partitions Specification (DPS). DDIs are wrapped in a GPT partition table that may contain root (or /usr/) filesystems, system extensions, system configurations, portable services, containers, and more, all of which are protected by dm-verity and combined into one image.
• Credentials pass secure bundles of data across components (hypervisor, firmware, system manager, container manager) and into system services.
• A system extension (sysext) is a DDI that can be overlaid on top of /usr/ (or /opt/) in a secure and atomic manner using read-only OverlayFS. A sysext can be used to extend a base filesystem to, for example, allow modular but pre-built initrds.
• A system configuration (syscfg) is a DDI that can be overlaid on top of /etc/ and provide a way to securely extend the configuration files of a system. This idea is still being developed, and will support the initrd, rootfs, system services, portable services, or nspawn images.

These concepts have all been supported by systemd and its collection of tools for some time, with the exception of syscfg, which is being developed now. They all support signed dm-verity for online and offline, kernel-enforced, integrity protection.

Usage of sysext is planned to be proposed for Fedora 39's initrd logic; it should close a major gap in the security story of Linux. So far, initrds have always been built locally and used in an unprotected manner, neither signed nor measured; thus they are lacking any verification and are at the mercy of any attacker who gains (online or offline) write access to the local disk. By switching to UKIs, we can provide a base, shared, vendor-built initrd that provides common components and a series of sysext DDIs that are added when needed, providing support for less commonly needed hardware or storage subsystems such as iSCSI or NFS. A specification for UKIs is available. For a deeper look, readers should refer to the aforementioned blog post.

Configuration, building, and deployment

We discussed how to handle local configuration at length. Various distributions, such as openSUSE, are pushing developers to use libeconf, which follows the same configuration scheme used by systemd. With this mechanism, the vendor's default configuration resides in /usr/, the ephemeral override is in /run/, and the persistent override is in /etc/. Others, like those based on OSTree, ship defaults in /usr/etc/ and then copy them over to /etc/ on instantiation. At Microsoft, we are working on a different approach in the form of syscfg, which is similar to sysext, but for configuration. The same principles are followed: configuration overlays are protected with dm-verity, signed, stackable, and applicable to the whole system or individual services/containers.

A large variety of options came up when discussing how to build images. Every single distribution has its own build system (as expected). The systemd project provides the systemd-repart tool, which understands DDIs and can run on each system (for initial partitioning or provisioning, or for factory reset), so there is some hope that it will be adopted for other uses too. The mkosi image-building tool is gaining some traction as it supports building layered sysexts and UKIs. But, by and large, every vendor is on its own with its own custom solution, and this situation is unlikely to change. One area where we hope to standardize is the production of a software bill of materials (SBOM), with some tentative agreement among the participants that the SPDX standard is the way to go. Some distributions, including SUSE and Flatcar, already go beyond this and provide full SLSA provenance.

After building an image, the next obvious step is getting it onto a system. Again, each vendor has its own methods here. Systemd recently introduced systemd-sysupdate as the run-time tool to pull down images from a configurable source. It is likely that the server side will remain unique to each vendor, but the hope is that, at least, the local client could be shared. Systemd-sysupdate can integrate nicely with systemd-boot and the rest of the ecosystem. Also discussed was resurrecting casync and integrating it with systemd-sysupdate to replace the use of curl seen in systemd-sysupdate now.

Updates and rollback

On the topic of upgrades, participants briefly discussed how to minimize the associated disruption. This is of great interest to Amazon and Microsoft, as image upgrades cause service interruptions. There are two competing approaches used in production. One is to use CRIU, which works best for single-process containers. The other is to use persistent memory and teach individual services how to make use of it, allowing restarting from fast memory or, even, keeping state intact across a kexec reboot. A standard user-space solution is desirable for the latter but is currently missing; some work is ongoing in that direction by Microsoft, but it is still the early days. A suggestion was made to implement an "exitrd" in systemd that would be similar to a kexec, allowing the system to skip the hardware/firmware stage of a reboot and saving some time when only the operating-system DDI is updated and the kernel doesn't change. In this case, the system could simply shut down user space to an "exitrd", swap the mount point of the DDI for the new one, and start user space again. This would allow skipping the kernel phases of shutdown and booting.

Almost all of the participants have implemented a form of operating-system rollback or factory-reset mechanism. This is highly desirable when distributing an image-based Linux system; one of the advantages is being able to roll back to a known-working state. There are important differences among the distributions though. Those using Btrfs (SUSE) rely on snapshots, while Ubuntu Core provides a more traditional recovery system that is stored in the EFI system partition.

But factory reset can mean two different things: either to restore the operating system to its original version, or to restore the entire system to the factory state, wiping all local data. Tools provided by systemd have long supported the latter, and will be enhanced to let the factory reset be triggered by a UEFI variable and/or a special UUID being set on the OS partition. A new target unit will be introduced as a synchronization point, so that any custom action that needs to happen on factory reset can be pulled in automatically. Image builders can then choose to provide a boot menu entry to let users trigger this functionality.

Other vendors, including Android, are using the well-known A/B pattern for rollbacks and updates. One of the next action items is to implement secure rollback protection in systemd components such as systemd-cryptsetup using TPM counters, as this is more scalable than relying on denylists that eventually become too big to be manageable. The response to the BootHole vulnerability alone, for example, used about one-third of the revocation space available on UEFI systems; see this page for details.

One of the areas where there was immediate agreement was boot assessment; when an image-based system uses an A/B scheme, a way to signal when a boot is "good" or "bad" is needed. The "boot-complete target" will fulfill this purpose. It is already available, and it sounded like more distributions will start using it; services doing any kind of local assessment will be able to use it as the synchronization point. Systemd will integrate this mechanism with a timer, so that if a good state is not reached within the configured time limit, a rollback and reboot will be triggered. Additionally, update success or failure can be reported when using advanced update protocols such as Omaha, for instance via the open-source Nebraska server.

Security

Enabling TPM-based security by default was also the subject of a lot of discussions. Right now, such security on generic Linux is pretty much opt-in. By switching to UKIs with embedded and signed PCR policies, we can make the TPM measurements predictable. This means that automatically enabling TPM-backed disk encryption by default becomes possible because there will no longer be a need to re-seal secrets every time a component changes. A document on signed TPM PCR policies will be available soon. It was agreed to set up a "registry" for PCRs, with the intention of helping developers and vendors avoid conflicts and overlaps.

There is a lot of interest in remote attestation in the context of confidential computing, and there are various solutions and competing standards. On a local node, some solutions rely on the IMA log, some on the TPM log, and some on completely custom registries. There is a lot of ongoing change at this stage and, while the work on image-based systems is largely a prerequisite for remote attestation, no clear action item or feature request came out of this discussion.

Conclusions and future work

The projects represented at the summit can be divided in three camps: image-based deployments, ostree-based deployments, and Btrfs-based deployments. So while it is natural that there was no complete overlap covering all participant projects, there was consensus that, on a significant number of topics, collaboration and standardization are indeed possible.

Finally, given that participants felt that the summit was productive and useful, it was agreed to meet again in the future, perhaps in co-location with another conference to facilitate travel. While there is no perfect agreement between all vendors on all topics, there are enough similarities that we came out of this with a series of action items and a plan. The UAPI Group, with all participants as members, will let us collect and collaborate on documentation and specifications. The initial specifications for DDIs, DPS, and a PCR registry are already in place and published on the UAPI website, with more to come.