Development

Creating containers with systemd-nspawn

Typically Lennart Poettering gives his conference talks about various aspects of the systemd init replacement, and his presentation at LinuxCon Europe was in the same vein. But, instead of the core functionality of systemd, he spoke about a mostly unknown utility that ships with it: systemd-nspawn. The tool started as a debugging aid for systemd development, but has many more uses than just that, he said. In fact, systemd-nspawn is like the chroot command—but it is a "chroot on steroids" according to the title of his talk.

Poettering began by noting that most people think of systemd as an init system, which it is, but that's just where it started and it is more than that now. Systemd is a set of "components needed to build up an operating system on top of the Linux kernel", he said. As part of the development of systemd, the team looked at various kernel features to see if they were relevant to the project.

One of the features considered was containers. Containers on Linux usually means either using LXC or libvirt LXC, he said. Those two are, he stressed, totally separate projects despite the name similarity. Both are quite different from the well-known (and understood) chroot command (and underlying system call). There is no configuration required for chroot, unlike the other two. The systemd project needed a way to run inside of containers or virtual machines, but wanted a simple tool that was more like chroot than either LXC or libvirt LXC. Enter systemd-nspawn.

The idea was to write a tool that does much of what LXC and libvirt LXC do, but is easier to use. It is targeted at "building, testing, debugging, and profiling", not at deployment. systemd-nspawn uses the same kernel APIs that the other two tools use, but is not a competitor to them because it is not targeted at running in a production environment.

Like chroot, systemd-nspawn "just works" with "no configuration". The latter is not quite true, Poettering said, but the configuration has been deliberately kept simple. As an example, he showed the yum command needed to create a minimal Fedora 19 installation in a directory (similar commands for multiple distributions are available in the man page). That became the basis for his subsequent demos.

After setting up a distribution directory, one can boot a container with a simple command (as root):

    systemd-nspawn -bD dir

The -D specifies the root directory for the container and -b says to boot it using systemd inside the container. Omitting -b is similar to booting a kernel with the init=/bin/bash command-line parameter, which results in a root shell. While he called it "booting" a container, there is no actual kernel boot that occurs as all of the containers are running under the host kernel. Poettering then showed that starting the container goes through the normal startup sequence for the distribution by starting various services inside the container and so on. When complete, you get a login prompt.

Logging in as "root" with no password enters the container, which, unsurprisingly, looks like a Fedora 19 installation. It is a "full container", Poettering said; additional software can be installed inside it using yum, for example. He showed a ps command both inside and outside the container to show that the processes were running on the system (of course) but that they had different PIDs inside and outside.

The container will automatically get its network configuration and time from the host, but set its hostname based on the directory name (or -M name). It also bind mounts /etc/resolv.conf from the host so that name resolution works inside the container. As one might expect, when finished with the container you can poweroff to shut it down or use reboot to restart it.

Poettering then moved on to some tools that make it easier to work with the "nspawn containers" as well as some work that the team has done to make standard tools report things like container names. For example, cgls (which was systemd-cgls in earlier systemd releases) shows control groups and their processes in a tree-like structure similar to that of pstree. Also, systemd-cgtop shows control groups in at top-like display, sorting them based on which are using the most CPU time.

Another addition is the machinectl command that manages "machines" (either containers or virtual machines) for systemd. When nspawn creates a new container, it registers that machine with systemd over D-Bus. Those machines can then be monitored and managed using machinectl. For example:

    machinectl status mname

That will show status of the machine called mname. That name is also integrated with tools like ps so that one can specify machine as an output column to see which container a process is running in. The machine name registration is also done by libvirt LXC, so those containers are treated similarly; so far, though, LXC is not using the facility. One of the goals is to eventually allow systemctl (the systemd management program) to take a machine-name argument and have it operate on the instance inside the machine.

The integration with machine names means that systemd-nspawn does require a system that has been booted by systemd in order to function. Earlier versions of systemd shipped with an independent nspawn, but that has fallen by the wayside.

Centralizing the system log information for the nspawn containers, while still allowing getting that information on a per-container basis, is handled by integration with the Journal. Using the -j option to nspawn will link the container's journal with that of the host. The Journal is "a little like syslog except that it is indexed", Poettering said. With linked journals, the system logs for multiple containers can be monitored or queried from the host.

Another feature of nspawn is that it can isolate the container from the host network. As mentioned earlier, by default nspawn inherits the network of the host, but the --private-network argument will create a container without any network devices other than loopback. That is "ideal for build systems", Poettering said, which shouldn't need the network after the initial package retrieval.

Nspawn is quite useful in a number of scenarios and the systemd team has used it extensively to debug systemd itself, he said. Normally, an init system is difficult to debug, but when you can use gdb, strace, and similar tools from the host to the programs running in a container, it makes it much easier. It is a tool that more people, especially in the "DevOps" community, should be aware of, he said—his talk, and articles like this, will hopefully start getting that word out.

[I would like to thank the Linux Foundation for travel assistance to Edinburgh to attend LinuxCon Europe.]

Comments (3 posted)

Quotes of the week

— Two out of many Ambrose Bierce–like definitions written by Thomas Figg

Comments (1 posted)

Blender 2.69 released

Version 2.69 of the Blender 3D modeling/rendering/animation studio has been released. Additions in this version include the ability to track plane surfaces in the motion tracker, improvements to the rendering of hair, skies, and subsurface scattering, the import of FBX files, and numerous enhancements to the modeling tools.

Comments (2 posted)

Valgrind 3.9.0 released

Version 3.9.0 of the Valgrind debugging tool set is available. "3.9.0 is a feature release with many improvements and the usual collection of bug fixes. This release adds support for MIPS64/Linux, Intel AVX2 instructions and POWER8 instructions. DFP support has been added for S390. Initial support for hardware transactional memory has been added for Intel and POWER platforms. Support for Mac OS X 10.8 (Mountain Lion) has been improved. Accuracy of Memcheck on vectorized code has been improved."

Full Story (comments: 15)

DRI3 and Present available for X.org

Keith Packard has released version 1.0 of the DRI3000 DRI3 and Present protocol extensions for X.org, as well as libxshmfence, a supporting library that exposes an event API on top of Linux futexes. DRI3 is the new implementation of X's direct rendering infrastructure, which we most recently looked at in October.

Comments (none posted)

SystemTap 2.4 available

Version 2.4 of the SystemTap monitoring and diagnostic tool is available. This release includes smarter error reporting, virtual machine probing, and improved suggestions. "Better suggestions are given in many of the semantic errors in which alternatives are provided. Additionally, suggestions are now provided when plt and trace probes could not be resolved."

Full Story (comments: none)

Django 1.6 released

Version 1.6 of the Django web framework has been released. The new version simplifies the default templates for projects and applications, activates clickjacking protection measures, and supports persistent database connections. Transaction handling has also been rewritten.

Comments (none posted)

Development newsletters from the past week

• Caml Weekly News (November 5)
• What's cooking in git.git (November 1)
• Perl Weekly (November 4)
• PostgreSQL Weekly News (November 3)
• Ruby Weekly (October 31)
• Tor Weekly News (November 6)

Comments (none posted)

Seigo: on introducing new ideas to free software communities

Aaron Seigo offers some advice on how to approach a development community with a new idea. "There is a saying that goes, 'it is easier to ask forgiveness than permission'. However, when it when comes to introducing plans and ideas to a community, this is often very bad advice. It too easily puts people into a position where they feel a binding decision has already been made on their behalf, and when that happens people often stop listening effectively and start talking loudly. This is, needlessly to say, not good for communication."

Comments (98 posted)