Development

Systemd programming part 2: activation and language issues

This is the second half of a pair of articles looking at systemd as a programming language for the specification and management of system services. Part 1 was concerned with modularity issues and how services can be configured. This part continues with a look at the various ways to control the activation of services before getting into an overall look at issues with systemd's language. While systemd has been extensively discussed as an init system, there is value in regarding it from a language point of view as well.

Activation options

With systemd, simply installing a unit file in the appropriate directory does not mean it is automatically active. This is much the same model as SysVinit uses, but it is a contrast to udev and upstart, which treat files as active the moment they are installed. There are subtle differences between systemd and SysVinit, though, which gave me my first hint that just continuing to use the SysVinit scripts for nfs-utils wasn't going to work, even though systemd provides some degree of support for these scripts.

With SysVinit, scripts are installed in /etc/init.d and then linked to /etc/rcN.d, or possibly /etc/init.d/rcN.d, for some value of N. This linking is often performed by the insserv utility, which will examine the header of each script and choose appropriate names for the links, so that the scripts are all run in the correct order. If a script has a "Required-Start" header referring to some other script and that other script has not been enabled (i.e. not linked into some directory), then insserv will complain and identify the missing dependency. It is then a simple matter to rerun insserv listing the extra dependency.

As SysVinit has relatively few scripts with few dependencies, this form of dependency handling does not cause an issue. With systemd, where nfs-utils alone has 14 unit files, having to explicitly enable all of them could get cumbersome. So where insserv treats a dependency as "this must already be enabled", systemd normally treats a dependency as "start this whether it is explicitly enabled or not" (though systemd has a rich language for these dependencies which we will get to in due course).

When systemd reads a SysVinit script, though, it takes a slightly different and more conservative approach to interpreting the dependency headers. It correctly treats Required-Start as implying an ordering (using the After systemd directive), but does not encode the "that service must already be enabled" meaning as it has no equivalent concept. A more liberal approach would translate Required-Start to Requires, causing the named script to be run even if not enabled. That might often be correct, but could be seen as reading more in to the dependency than is intended.

This different default behavior can create different expectations. When insserv nfs reports:

    insserv: FATAL: service portmap has to be enabled to use service nfs

the system administrator will naturally run insserv portmap and move on. However, when "systemctl enable nfs" works, but a subsequent "systemctl start nfs" fails because rpcbind (which is the new name for portmap) isn't running, the administrator's response is likely to be less forgiving. For complete compatibility with SysVinit, systemd would need a dependency mechanism which makes it impossible to enable something unless some other requirement were already enabled, but that doesn't really fit into systemd's model.

With that little diversion out of the way, we should look at how units can be activated in systemd. Systemd is often described as using dependencies for unit activation and, while there is certainly truth in that, it is far from the full story. For the full (or, at least, fuller) story we will start with the "mdadm" package which provides services to assemble, monitor, and manage MD RAID arrays. One particular service is provided by running

     mdadm --monitor

as a daemon. This daemon will watch for any device failures or other interesting events and respond to them, possibly by sending email to the administrator, or possibly by finding a "spare" device on some other array and moving it across (thus allowing spare sharing between arrays). This daemon should be running whenever any md array is active, but otherwise it is completely unnecessary. This requirement is achieved by creating a systemd unit file to run mdadm --monitor and using the SYSTEMD_WANTS setting in a udev rules file.

Udev (which is a whole different story when it comes to language design) is notified when any device comes online and can perform numerous tests and take actions. One of those actions is to set an environment variable (SYSTEMD_WANTS) to the name of some unit. When systemd subsequently receives events from udev, it will receive the full environment with them. Systemd interprets SYSTEMD_WANTS by adding a Wants directive to the .device unit (possibly newly created) corresponding to the device in the event. So a udev rules file which detects a new MD array and sets SYSTEMD_WANTS=mdmonitor.service will cause mdadm --monitor to run at exactly the correct time.

Note that there is no need to explicitly enable this service. As udev rules are always enabled and the udev rule directly requests the systemd service, it just happens. No activation needed. This signaling from udev to systemd is an event much like the events talked about in the context of upstart. While systemd may not only use events for activation, it certainly does use them — and not only events from udev.

When an NFSv2 or NFSv3 filesystem is mounted, then, unless network file locking has been explicitly disabled, the rpc.statd process must be running to ensure proper lock recovery after a reboot. Rather than have this process always running, /sbin/mount.nfs (which is used to mount all NFS filesystems) will check that rpc.statd is running and, if not, will start it. When systemd is being used it is best to do this by running:

    systemctl start rpc-statd.service

which, again, is much like an event.

When mount.nfs checks to see if rpc.statd is running, it attempts to contact it via a network socket. It is well known that systemd supports socket-based activation, so it would be ideal to use that to activate rpc.statd. However, rpc.statd, like most ONC-RPC services, does not use a well-known port number, but instead chooses an arbitrary port and registers it with rpcbind. So systemd would need to do this too: bind an arbitrary port, register that port with rpcbind, and start rpc.statd when traffic arrives on that socket. This is certainly possible; SunOS used to ship with a version of inetd which did precisely this. Solaris still does. Whether it is worth adding this functionality to systemd for the two or maybe three services that would use it is hard to say.

The remainder of the daemons that make up the complete NFS service are not triggered by events and so must be explicitly activated by being tied to specific well-known activation point "targets", which are not unlike SysVinit run levels. Even there, the distinction between the systemd approach and the use of events in upstart is not as obvious as one might expect.

As we shall see, a dependency relationship is created between nfs-server.target and multi-user.target so that when multi-user.target is started, nfs-server.target is started too. As upstart jobs broadcast a "starting" signal when they are starting, and can register to, for example, "start on starting multi-user" the net effect is, to some degree at least, similar.

There is a key difference here, though, and it isn't really about events or dependencies, but about causality. In upstart, a job can declare "start on" to identify which event it should start on. So each job declares the events which cause it to run. Systemd, despite its rich dependency language, has no equivalent to "start on", an omission that appears to be deliberate. Instead, each event — the starting or stopping of a unit — declares which jobs (or units) need to be running. The dependency language is exactly reversed. With upstart, each job knows what causes it to start. With systemd, each job knows what it causes to start.

While systemd has no equivalent to "start on", it has something related that we must study to understand how the remaining nfs-utils daemons are started. This is represented by the "WantedBy" and "RequiredBy" directives, which are quite different from the "Requires" and "Wants" etc. dependency directives. "WantedBy" plays no role in determining when to start (or stop) any service. Instead, it is an instruction on how to enable a specific unit. The directive:

    WantedBy=some-unit.target

means "the best way to enable this unit is to tell some-unit.target that it Wants us." It is possible to tell any unit that it wants another unit, either by creating a drop-in file as described in part 1 to add a "Wants" directive, or by creating some special symbolic links that systemd interprets in a similar way to drop-ins. The easiest way, though, is to run:

    systemctl enable servicename

This command responds to the WantedBy directive in servicename.service by creating the special symlink so that some-unit.target thinks it wants servicename.

So in our collection of nfs-utils unit files, a few of them have WantedBy directives so they can be properly enabled. The rest get activated by "Wants" or "Requires" lines in those main files. Two of the unit files fit this pattern perfectly. nfs-server.target and nfs-client.target are WantedBy=multi-user.target or remote-fs.target, and then they Want or Require other units. The other two target unit files are a bit different, and to understand them we need to revisit the problem of configuration.

One of the purposes of the current configuration setup for nfs-utils in openSUSE is to optionally start the daemons which support using Kerberos to secure the NFS traffic. If you trust your local network, then Kerberos security is pointless and it is a waste to even run the daemons. However, if you want to use NFS over an untrusted network, then running rpc.gssd and rpc.svcgssd really is a must ("gss" here stands for "Generic Security Services"; while they were designed to be generic, the practical reality is that they only support Kerberos).

So we have the situation that nfs-server.target wants rpc-svcgssd.service, but only if network security is wanted, and this latter choice is configured by an environment variable. This is a requirement that systemd really cannot manage. It has a number of Condition directives to disable units in various cases, but none of them can be controlled using an environment variable. This suggests that either the sysadmin or the configuration tool (and possibly both) will need to use some other mechanism. The most obvious mechanism is systemctl and particularly:

	systemctl enable rpc-svcgssd

to enable a service if it is off by default, or:

	systemctl mask rpc-svcgssd

to mask (disable) a service that is otherwise on by default.

There is a complication though: in the pre-existing configuration that I was trying to mirror with systemd units, there are two services, rpc.gssd and rpc.svcgssd, that are both controlled by a single configuration item NFS_SECURITY_GSS. These need to be started in different circumstances: rpc.gssd is required if the NFS server is started or an NFS filesystem is mounted, while rpc.svcgssd is only required if the server is started. So we cannot simply have an nfs-secure.target which needs both of them and can be manually enabled. Systemd is powerful enough to make this set of requirements achievable, though it does seem to be a bit of a stretch.

The draft unit-file collection contains an nfs-secure.target unit which can be enabled or disabled with systemctl, but it doesn't actually start anything itself. Instead it is used to enable other units. The two related units (rpc-gssd.service and rpc-svcgssd.service) now gain the directive:

    Requisite=nfs-secure.target

This means that those services want nfs-secure, and if it isn't already running, they fail. This has exactly the desired effect. After "systemctl enable nfs-secure.target" the GSS daemons will be started when required; after "systemctl disable nfs-secure.target" they will not.

Having four different targets which can selectively be enabled (the fourth being a target similar to nfs-secure.target but which enables the blkmapd daemon to support part of the "pNFS" extension; not needed by most sites) might seem like it is providing too much choice. Here again, systemd comes to the rescue with an easy mechanism for distribution packagers to take some of the things I made optional and make them always enabled. A distribution is encouraged to provide one or more "preset" files listing systemd units that should be automatically enabled whenever they are installed. So if a distribution was focused on high levels of security, it could include:

	enable nfs-secure.target

in the preset file. This would ensure that, if nfs-utils were ever installed, the security option would be activated by default. This feature encourages upstream unit file developers to be generous in the number of units that require enabling, being confident that while it provides flexibility to whose who need it, it need not impose a cost on those who don't.

In summary, systemd provides a pleasing range of events and dependencies (many of which we have not covered here) which can be used to start units. It is unfortunate, however, that enabling or disabling of specific units is not at all responsive to the environment files that systemd is quite capable of reading. The choice to not automatically activate any installed unit file is probably a good one, although it is an odd contrast to udev, which is included in the same source package.

Language issues

Having a general interest in programming language design, I couldn't help looking beyond the immediate question of "can I say what I need to say" to the more subjective questions of elegance, uniformity, and familiarity. Is it easy to write good code and hard to write bad code for systemd?

One issue that stuck me as odd, though there is some room for justification, is the existence of section headings such as [Unit] or [Service] or [Install]. These don't really carry any information, as a directive allowed in one section is not allowed in any other, so we always know what a directive means without reference to its section. A justification could be that these help ensure well-structured unit files and thus, help us catch errors more easily.

If that is the case, then it is a little surprising that the concern for error detection doesn't lead to unit files with syntax errors immediately failing so they will be easily noticed. The reasoning here is probably that an imperfectly functioning system is better than one that doesn't boot at all. That is hard to argue against, though, as a programmer, I still prefer errors to fail very loudly — I make too many of them.

More significant than that is the syntax for conditionals. The directive:

	ConditionPathExists=/etc/krb5.keytab

will cause activation of the unit to only be attempted if the given file exists. You can easily test for two different files, both of which must exist, with:

	ConditionPathExists=/etc/krb5.keytab
	ConditionPathExists=/etc/exports

or even for a disjunction or 'or' condition:

	ConditionPathExists=|/etc/mdadm.conf
	ConditionPathExists=|/etc/mdadm/mdadm.conf

If you want to get more complicated, you can negate conditions (with a leading !) and have the conjunction of a number of tests together with the disjunction of some other tests. For example:

	A and B and (C or D or E)

To achieve this, the A and B conditions are unadorned, while C, D and E each have a '|' prefix. However, you cannot have multiple disjunctions like

	(A or B) and (C or D)

Now, I'm not sure I would ever want a conjunction of multiple disjunctions, but it seems worth asking why the traditional infix notation was not used. From reading around, it is clear that the systemd developers want to keep the syntax "simple" so that the unit files can be read by any library that understands the "ini" file format. While this is a laudable goal, it isn't clear that the need for some unspecified program to read the unit files should outweigh the need for humans to read and understand them.

It also doesn't help that while the above rules cover most conditions, they don't cover them all. As already noted, "Requisite" is largely just a condition which might be spelled "ConditionUnitStarted", but isn't; it also doesn't allow the '|' or '!' modifiers.

The final inelegance comes from the ad hoc collection of directives which guide how state changes in one unit should affect state changes in another unit. Firstly, there are 13 directives that identify other units and carry various implications about the relationship — sometimes overlapping, sometimes completely orthogonal:

Requires, RequiresOverridable, Requisite, RequisiteOverridable, Wants, BindsTo, PartOf, Conflicts, Before, After, OnFailure, PropagateReloadsTo, ReloadPropagateFrom

Of these, Before and After are inverses and do not overlap with any others, while Requires, Requisite, and Wants specify dependencies with details that differ on at least two dimensions (active vs passive and hard vs soft). Several, such as PartOf, PropagateReloadsTo, ReloadPropagateFrom, and OnFailure, are not dependencies, but instead specify how an event in one unit should cause an action in another.

Together with these, there are some flags which fine-tune how a unit responds in various circumstances. These include:

StopWhenUnneeded, RefuseManualStart, RefuseManualStop

While there is clearly a lot of expressive power here, there is also a lot of ad-hoc detail which makes systemd harder to learn. I get a strong feeling that there is some underlying model that is trying to get out. If the model were fully understood, the language could be tuned to expose it, and the programmer would more easily select exactly the functionality that is required.

Some small part of the model is that the relationship between two units can be:

• ordered: Before or After or neither;
• active, where one unit will start another, or passive, where it won't;
• dependent, which can take one of three values: no dependence, must have started, or must still be running; and
• overridable, which indicates whether an external request has extra force, or no force at all.

Given that systemd already uses special characters to modify meaning in some directives, it is not hard to come up with a set of characters which could allow all these details, and possibly more, to be specified with a single "DependsOn" directive.

These only really cover the starting of units, and even there, they aren't complete. The model must also include the stopping of units as well as "restart" and "reload" events. Creating a simple model which covers all these aspects without leading to an overly verbose language would be a real boon. Using special characters for all of the different details that would turn up may well cause us to run out of punctuation, but maybe there is some other way to describe the wide range of connections more elegantly.

A pragmatic assessment

While it did feel like a challenge to pull together all the ideas needed to craft a clean and coherent set of unit files, the good news is that (except for one foolish cut-and-paste error) the collection of unit files I created did exactly what I expected on the first attempt. For a program of only 168 lines this might not seem like a big achievement, but, as noted, the fact that only 168 lines were needed is a big part of the value.

These 14 units files are certainly not the end of the story for nfs-utils. I'm still learning some of the finer details and will doubtlessly refine these unit files a few times before they go live. Maybe the most valuable reflection is that I'm far more confident that this program will do the right thing than I ever could be of the shell scripts used for SysVinit. Speaking as a programmer: a language that allows me to say what I want succinctly and gives me confidence that it will work is a good thing. The various aesthetic issues are minor compared to that.

Comments (20 posted)

Quotes of the week

— Gervase Markham

— Chris Siebenmann

Comments (4 posted)

Docker 0.8 released

Version 0.8 of the Docker container-creation system has been announced. This release brings some changes to the development process: "First, this is the first Docker release where features take the backseat to quality: dozens and dozens of bugfixes, performance boosts, stability improvements, code cleanups, extra documentation and improved code coverage – that’s the primary feature in Docker 0.8." The project will also be doing time-based monthly releases going forward. There are still some new features, including a Btrfs storage driver and Mac OS support; see the changelog for details.

Comments (15 posted)

GDB 7.7 released

Version 7.7 of the GDB debugger is out. It features improved Python scripting support, a number of new commands, support for a few new targets, and more.

Full Story (comments: 1)

Glibc 2.19

Version 2.19 of the GNU C library has been released. It includes a lot of bug fixes, support for a number of new locales, better SystemTap support, and more; see this article for more information.

The EGLIBC 2.19 release is also in the works, with an important note:

It appears that changes in glibc development have made this fork unnecessary going forward.

Comments (6 posted)

Mozilla announces "Firefox Accounts"

The Mozilla Blog has an announcement about "Firefox Accounts," a new service being rolled out by the browser vendor. The post describes this venture as "as a safe and easy way for you to create an account that enables you to sign in and take your Firefox with you anywhere. With Firefox Accounts, we can better integrate services into your Web experience". The announcement does not shed light on what services will be involved, other than the fact that it will incorporate the existing Firefox Sync. The service is testable on Mozilla's Aurora pre-release builds of Firefox.

Comments (16 posted)

Apache SpamAssassin 3.4.0 available

The SpamAssassin 3.4.0 release is out. "This is a major release. It introduces over two years of bug fixes and features since the release of SpamAssassin 3.3.2 on June 16, 2011." Changes include use of the Redis backend for Bayesian data storage, native IPv6 support, and, of course, lots of rule changes.

Full Story (comments: 3)

Development newsletters from the past week

• What's cooking in git.git (February 5)
• What's cooking in git.git (February 7)
• LLVM Weekly (February 10)
• OCaml Weekly News (February 11)
• Perl Weekly (February 10)
• PostgreSQL Weekly News (February 9)
• Python Weekly (February 6)
• Ruby Weekly (February 6)
• This Week in Rust (February 9 )
• Tor Weekly News (February 12)

Comments (none posted)

Jones: The EFI System Partition and the Default Boot Behavior

On his blog, Peter Jones writes about the boot process for UEFI, looking at the requirements for EFI System Partitions (ESPs), how the BootOrder variable is used, falling back to removable media, and more. It may be more than you wanted to know about UEFI booting. "There’s nothing truly special about an ESP. It isn’t an ESP because of the GPT GUID and label, nor because of the file system type. Those are how the firmware identifies a partition, and the file system it contains, as candidates to treat as the ESP, when it really needs to find one. The only factor in determining if a partition is the ESP is this: is the firmware attempting to use it as the ESP? At the same time, the requirements for the ESP give us latitude; we know that we can use UEFI’s APIs to find correctly constructed FAT file systems, but there’s no need for those to be the ESP. In fact, even when we create multiple partitions with the ESP’s GUID and label, there’s no requirement that the firmware looks at more than one of them if it needs to find the ESP, and there’s no guarantee as to which one it will pick, either."

Comments (14 posted)

Mozilla To Sell Ads In Firefox Web Browser (AdvertisingAge)

AdvertisingAge is reporting that Mozilla will be selling ads in Firefox. In particular, the "New Tab" page that normally has nine of the most frequently visited sites shown will, for new users, show ads and "pre-packaged content" in the new feature called "Directory Tiles". The Mozilla blog gives a bit more detail: "Some of these tile placements will be from the Mozilla ecosystem, some will be popular websites in a given geographic location, and some will be sponsored content from hand-picked partners to help support Mozilla’s pursuit of our mission. The sponsored tiles will be clearly labeled as such, while still leading to content we think users will enjoy. We are excited about Directory Tiles because it has inherent value to our users, it aligns with our vision of a better Internet through trust and transparency, and it helps Mozilla become more diversified and sustainable as a project."

Comments (101 posted)