Systemd programming part 1: modularity and configuration

Systemd's positive and negative features have been discussed at length; one of the first positives I personally noticed was seen from my perspective as an upstream package maintainer. As the maintainer of mdadm and still being involved in the maintenance of nfs-utils, one of my frustrations was the lack of control over, or even much visibility into, the way these packages were integrated into the "init" system on each distribution. Systemd has the potential to give back some of that control while still giving flexibility to distributors and administrators; this article (and the one that follows) will look at systemd's programming features to show how that works.

Once upon a time, all the major distributions were using SysVinit and each had their own hand-crafted scripts to handle the startup of "my" packages. While I could possibly have copied all of those scripts into the upstream package and tried to unify them, it would have been problematic trying to do this for all distributions, and it is unlikely that many distributions would have actually used what was offered. We did have scripts for Debian in the nfs-utils package for a while, but it turned out that this didn't really help the Debian developer at the time and they were ultimately removed when they proved to be more of a hindrance.

Not all packagers will care about having this visibility or control. However, with both mdadm and nfs-utils there are issues with recovery after failures which are not entirely straightforward to deal with. It is very possible for a distribution maintainer to put together an init script which works perfectly well in most cases, but can fail strangely in rare cases. I first came across this problem with nfs-utils, which has subtle ordering requirements for starting daemons on the server if a client held a lock while the server rebooted. Most (possibly all) distributions got this wrong. The best I could do to fix it was to get a fix into the distribution I was using at the time and update the README file. Whether that actually helped any other distribution is something I'll never know.

It was against this backdrop that I first considered systemd. The contrast to SysVinit was and is substantial. While the configuration files for SysVinit are arbitrary shell scripts with plenty of room for individual expressiveness and per-distribution policy, the configuration for systemd is much more constrained. Systemd allows you to say the things you need to say but provides very little flexibility in how you say them; systemd also makes it impossible to say things irrelevant to the task at hand, such as color coding status messages. This means that there is much less room for per-distribution differences and, thus, much less motivation for distribution packager to deviate from a configuration provided by upstream.

So when it came time to replace the SysVinit scripts that openSUSE uses for nfs-utils with some systemd unit files I decided to see if I could do it "properly." My initial baseline definition for "properly" was that the unit files should be suitable for inclusion in the upstream nfs-utils package. This in turn means they must be suitable for every distribution to use, and must be of sufficient quality to pass review by my peers without undue embarrassment.

This effort, together with the work I had already done to convert mdadm to use systemd unit files, caused me to look at systemd from the perspective of programming and programming language design. Systemd is a powerful and flexible tool that can be programmed through a special-purpose language to unite the various tools in the packages that I help maintain, in order to provide holistic functionality.

Modularity

In the early days of Unix, there was a script called /etc/rc which started all the standard daemons, and maybe it would run /etc/rc.local to start a few non-standard ones. When you only have 32KB of RAM, you probably don't want to run so many daemons that the lack of modularity provided by these files becomes a problem. As the capacity of hardware increased so too did the need for modularity, with modern SysVinit allowing a separate script (or possibly scripts in the plural) for each distinct package.

Systemd takes this one step further by allowing, and in fact requiring, a separate unit file for each daemon or for each distinct task. For packages that just provide a single daemon this is probably ideal. For nfs-utils, this requirement borders on being clumsy.

The draft unit-file collection I recently posted for review has 14 distinct unit files with a total of 168 lines (including blanks and occasional comments). They replace two SysVinit scripts totaling 801 lines, so the economy of expression cannot be doubted. However, it does mean that I cannot simply open "the unit files" in an editor window and look over them to remind myself how it works or look for particular issues. For mdadm, which has 4 systemd unit files this is a minor inconvenience. For nfs-utils it really feels like a barrier.

These 14 files include eight which actually run daemon processes, two which mount special virtual filesystems which the tools use for communicating with the kernel, and four which are "target" units. "Targets" are sometimes described as "synchronization points" in the systemd documentation. A target might represent a particular level of service such as network-online.target or multi-user.target. For nfs-utils we have, for example, nfs-server.target. This target starts all the various services required for, or useful to, NFS service. The individual daemons don't necessarily depend on each other, but the service as a whole depends on the collection and so gets a separate target.

These target units are well placed to provide a clean module structure. There is sometimes a need for unit files belonging to one package to reference unit files belonging to another package, such as the reliance on rpcbind.target in several nfs services. Restricting such references to target units would allow a clean separation between the API (.target) and the implementation (.service etc). Unfortunately the "systemctl" command handles an abbreviated unit name but it assumes a ".service" suffix rather than a ".target" suffix. This tends to discourage the use of targets and blurs the line between API and implementation.

This ability to collect units together into a virtual unit while allowing the individual details of the units to be managed separately is a distinct "plus" for systemd from the modularity perspective. The insistence that units each have their own file, together with systemctl's unfortunate default behaviour, are small "minuses".

Configuration

As a programmer I try to choose sensible defaults but also to allow users of my code to customize or configure some settings to meet their particular needs. As a programmer using the systemd language I have two sorts of users to keep in mind: distribution maintainers who will package my project for particular distributions, and system administrators who will use it to create a useful computer system.

I want both of these groups to be able to make any configuration changes they need, but at the same time I want to retain some degree of control. If I find a bug, for example some subtle ordering issue for different daemons, then I want to be able to fix that in the upstream package and have some confidence that the changes will flow through to all users. In this I seem at variance with the systemd developers, or at least with an opinion expressed two and a half years ago in systemd for Administrators, part IX.

That document asserts that "systemd unit files do not include code", and that "they are very easy to modify: just copy them from /lib/systemd/system to /etc/systemd/system and edit them there". The first of these points is largely a philosophical one, but one which the attentive reader will already see that I disagree with. Any time I am writing specific instructions to bring about a specific effect, I am writing code — whether it is written in C, Python, or systemd unit-file language.

The second is a more practical concern. If system administrators were to take this advice and replace one of my carefully crafted unit files, then the bug-fix I release may not have a chance to work for them — an outcome I would rather avoid.

So I don't want system administrators to feel the need to edit my unit files, but equally I don't want distribution packagers to edit them either. Partly this is a pride issue — I want upstream to be a perfect fit for everyone. Partly this is a quest for uniformity — packagers should feel free to send any patches they require upstream so that everyone benefits. And partly it is a support issue. If I get a bug report, I want to be able to ask the reporter to fetch and install the current upstream version and be fairly confident that it won't break any distribution-specific features.

So my goal, as a programmer, is to ensure my users can configure what they need without having to change my code. Once again systemd gets a mixed score-card.

Systemd allows for so-called "drop-ins" to extend any unit file. When looking for the file to load a particular unit, systemd will search a standard list of directories for a file with the right name. Early in this list are directories for local overrides such as /etc/systemd/system mentioned above. Late in the list is the location for the primary file to use — /lib/systemd/system in the quote above, though often /usr/lib/systemd/system on modern installations.

After finding the unit file, systemd will search again, this time for a directory with the same name as the file except with ".d" appended. If that is found, any files in the directory with names ending ".conf" are read and can extend the unit description. These ".conf" files are referred to as "drop-ins."

The ordering of directives in unit files is largely irrelevant so any directive can be replaced or, if relevant, extended by a drop-in file. So for example an "ExecStartPre" directive could be given to add an extra program to be run before the service starts. This facility allows both the packager and sysadmin to impose a variety of configuration changes without having to edit my precious unit files. Things like WorkingDirectory, User and Group, CPUSchedulingPriority, and more can easily be set if that is desired to meet some local need. However, there is one common area of configuration that isn't so amenable to change through drop-in files: the command-line arguments for the process being run.

When I examine the configuration that openSUSE SysVinit scripts allow for various daemons in nfs-utils, the one that stands out is the adding of extra command-line options. This may involve setting the number of threads that the NFS server might use, allowing mountd to run multi-threaded or provide alternate management of group IDs, or setting an explicit port number for statd to use, among others.

It is certainly true that a drop-in file can contain text like:

    ExecStart=
    ExecStart=/some/program --my --list=of arguments

where the assignment of an empty string removes any previous assignment (ExecStart is defined as a list, so that a "one-shot" service can run a list of processes) and the second assignment gives the desired complete list of command line arguments. However this approach can only replace the full set of arguments, it cannot extend them.

This gets back to my desire for control. If an upstream fix adds or changes a command-line argument, then any installation which uses a drop-in to replace the arguments to that daemon will miss out on my change.

There is also a question of management here. Many distributions don't expect system administrators to edit files directly but, instead, provide a tool like YaST (on openSUSE) or debconf (on Debian) which manages the configuration through a GUI interface or a series of prompts. For such a tool, it is much easier to manipulate a file with a well-defined and simple structure — such as the /etc/sysconfig files in openSUSE and Fedora or the similar /etc/defaults files in Debian — which are essentially a list of variable assignments. It is quite straightforward for user-interface tools to manipulate this file, and then for the SysVinit scripts to interpret the values and choose the required command line arguments.

While systemd can certainly read these same configuration files (with the EnvironmentFile directive) and can expand environment variables in command line arguments to programs, it lacks any sophistication. Non-optional parameters are easily handled, so:

    /usr/sbin/rpc.nfsd -n $USE_KERNEL_NFSD_NUMBER

would work as expected, but optional parameters cannot be managed so easily. While the Bourne shell (which interprets SysVinit scripts) and Upstart would both support:

    /usr/sbin/rpc.mountd ${MOUNTD_PORT:+-p $MOUNTD_PORT}

to include the "-p" only if $MOUNTD_PORT were non-empty, systemd cannot do this. It could be argued that this syntax has little to recommend it and that systemd might be better off without it. However, there is a clear need for this sort of functionality, at least unless or until various configuration management tools learn to create complete command lines directly.

There are at least two possible responses that are worth considering to address this need. The first is to modify all the daemons to accept any command line configuration also from environment variables. For example mountd could be changed to examine the MOUNTD_PORT environment variable if no -p option were given, and use that value instead.

This could be seen as a rather intrusive change. However, since I am looking from the perspective of an upstream developer, writing code in each daemon is not really harder or easier than writing code in systemd unit files. It is also very much in the style of systemd, which seems to encourage authors of daemons to design them to work optimally with systemd, recommending use of libsystemd-daemon, for example, when appropriate. This can maximize the effective communication between systemd and the services it runs, and remove any "middle-men" which don't really add any long-term value.

So this seems the best solution for the longer term, but as it requires some degree of agreement among developers and co-ordination between distributions, it doesn't seem like the best short-term solution.

The other option is to create a "middle-man" exactly as suggested. That is, have a script which can read a distribution-specific configuration file, interpret the contents, and construct the complete command lines for a collection of daemons. These command lines would be written to a temporary file which systemd could read before running the daemons.

So we could have an "nfs-config.service" unit like:

    [Unit]
    Description=Preprocess NFS configuration
    [Service]
    type=oneshot
    RemainAfterExit=yes
    ExecStart=/usr/lib/systemd/scripts/nfs-utils_env.sh

which is tasked with creating /run/sysconfig/nfs-utils. The "nfs-utils_env.sh" script would be distribution-specific to allow the packager to create a perfect match for the configuration options that the local config tool supports.

Then each nfs-utils unit file would contain something like:

    [Unit]
    Wants=nfs-config.service
    After=nfs-config.service
    [Service]
    EnvironmentFile=-/run/sysconfig/nfs-utils
    ExecStart=/usr/sbin/rpc.mountd $MOUNTD_ARGS

so that the "nfs-config" service would be sure to have run first and created the required EnvironmentFile, which would contain, among other things, the value of MOUNTD_ARGS.

Assessing this support that systemd provides (or fails to provide) for configuration objectively is hard. On the one hand, it seems to require me to create this "nfs-config" service which seems a bit like a hack. On the other it encourages me to press forward and change my daemon processes to examine the environment directly which is probably a better solution than any sort of service that systemd could provide to convert environment variables directly.

So, generally, for configuration, systemd gets a thumbs-up, but I'm reserving final judgment for now.

Thus far, we have seen a pattern with modularity and service configuration, where it provides good functionality but also a few significant frustrations. This pattern will continue in the second part when we look at how unit files may be "activated" to become running services, and how the set of unit directives as a whole fare when viewed as a programming language.

Index entries for this article
GuestArticles	Brown, Neil

---

to post comments