Portable system services

In the refereed track of the 2016 Linux Plumbers Conference, Lennart Poettering presented a new type of service for systemd that he calls a "portable system service". It is a relatively new idea that he had not talked about publicly until the systemd.conf in late September. Portable system services borrow some ideas from various container managers and projects like Docker, but are targeting a more secure environment than most services (and containers) run in today.

There is no real agreement on what a "container" is, Poettering said, but most accept that they combine a way to bundle up resources and to isolate the programs in the bundle from the rest of the system. There is also typically a delivery mechanism for getting those bundles running in various locations. There may be wildly different implementations, but they generally share those traits.

Portable system services are meant to provide the same kind of resource bundling, but to run the programs in a way that is integrated with the rest of the system. Sandboxing would be used to limit the problems that a compromised service could cause.

If you look at the range of ways that a service can be run, he said, you can put it on an axis from integrated to isolated. The classic system services, such as the Apache web server or NGINX, are fully integrated with the rest of the system. They can see all of the other processes, for example. At the other end of the scale is virtual machines, like those implemented by KVM, which are completely isolated. In between, moving from more integrated to more isolated, are portable system services, Docker-style micro-services, and full operating system containers such as LXC.

Portable system services combine the traditional, integrated services with some ideas from containers, Poettering said. The idea is to consciously choose what gets shared and what doesn't. Traditional services share everything, the network, filesystems, process IDs, init system, devices, and logging, but some of those things will be walled off for portable system services.

This is the next step for system services, he said. The core idea behind systemd is service management; not everything is Docker-ized yet, but everything has a systemd service file. Administrators are already used to using systemd services, so portable services will just make them more powerful. In many cases, users end up creating super-privileged containers by dropping half of the security provided by the container managers and mostly just using the resource bundling aspect. He wants to go the other direction and take the existing services and add resource bundling.

Integration is good, not bad, Poettering said; having common logging and networking is often a good thing. Systemd currently recognizes two types of services: System V and native. A new "portable" service type will be added to support this new idea. It will be different from the other service types by having resource bundling and sandboxing.

To start with, unlike Docker, systemd does not want to be in the business of defining its own resource bundling format, so it will use a simple directory tree in a tarball, subvolume, or GUID Partition Table (GPT) image with, say, SquashFS inside. Services will run directly from that directory, which will be isolated using chroot().

The sandboxing he envisions is mostly concerned with taking away the visibility of and preventing access to various system resources. He went through a long list of systemd directives that could be used to effect the sandboxing. For example, PrivateDevices and PrivateNetwork are booleans that restrict access to any but the minimal devices in /dev (e.g. /dev/null, /dev/urandom) and to provide only a loopback interface for networking. PrivateTmp gives the service its own /tmp, which removes a major attack surface. There is a setting to have a private user database that only has three users (root and a service-specific user; all other user IDs are mapped to nobody). Some other settings protect various directories in the system by mounting them read-only for the service, there is a setting to disallow realtime scheduling priority, another to restrict kernel-module loading, and so on.

Many of those are already present in systemd and more will be added. The systemd project has been working to make sandboxing more useful for services, Poettering said. He would like to see a distribution such as Fedora turn these features on for the services that it ships. Another area systemd will be working on is per-service firewalls and accounting.

Unlike native or System V services, portable services will have to opt out of these sandboxing features if they don't support them. In fact, he said, if systemd were just starting out today, native services would have been opt-out for the sandboxing options, but it is too late for that now.

There are some hard problems that need to be solved to make all of this work. One is that Unix systems are not ready to handle dynamic user IDs. When a portable service gets started, an unprivileged user for the service gets created, but is not put into the user database (e.g. passwd file). If a file is created by this user and then the service dies, the file lingers with a user ID that is unknown to the system.

One way to handle that is to restrict services with a dynamic user ID from being able to write to any of the filesystems, so using that feature will require the ProtectSystem feature that mounts the system directories read-only. Those services will get a private /tmp and a directory under /run that they can use, but those will be tied to the lifecycle of the service. That way, the files with unknown (to the system) user IDs will go away when the service and user ID are gone.

Dynamic users are currently working in systemd, which is a big step forward, Poettering said. Right now, new users are installed when an RPM for a service is installed, which doesn't really scale. Dynamic users make that problem go away since the user ID is consumed only while the service is running.

Another problem that people encounter is that if they try to install a service in a chroot() environment, they will need to copy the user database into the chroot(). The idea behind the PrivateUsers setting is to make chroot() work right. That setting will restrict the service to only have three users, one is the dynamic user for the service and the other two are root and nobody. Most distributions agree on the user ID for root and nobody, so that will help make the portable services able to run on various distributions.

D-Bus is incompatible with a chroot() environment because there is a need to drop policy files into the host filesystem. For now, that is an unsolved problem, but the goal is to move D-Bus functionality into systemd itself. That is something the project should have done a long time ago, Poettering said. The systemd D-Bus server would then use a different policy mechanism that doesn't require access to the host filesystem.

He stressed that systemd is not building its own Docker-like container manager; it is, instead, providing building blocks to take a native service and turn it into a portable one. So systemd will only have a simple delivery mechanism that is meant to be used by developers for testing, not for production use. Things like orchestration and cluster deployment are out of scope for systemd, Poettering declared.

He showed a few examples of his vision of using the systemctl command to start, stop, and monitor portable services on the local host or a remote one, though it was not a demo. It was not entirely clear from the talk how far along things are for portable system services. The overall goal is to take existing systemd services, add resource bundling and sandboxing, and make "natural extensions to service management" to support the portable services idea, he said.

Various questions were asked at the end. For example, updating the services is something that is out of the scope for systemd. That can be handled with some other tool, but the low-level building blocks will be provided by systemd. Another question concerned configuration. Different configurations of a service would require building a different bundle with those changes, as the assumption is that configuration gets shipped in the bundle.

Having the security be opt-out is useful, but how will additions to the security restrictions be handled? Existing services could break under stricter rules. Poettering said that it was something he was aware of, but had not come up with a solution for yet. He wants to start with a powerful set of restrictions out of the box, but perhaps defining a target security level for a particular service could help deal with this backward incompatibility problem.

[ Thanks to LWN subscribers for supporting my travel to Santa Fe for LPC. ]