LWN: Comments on "Systemd programming part 1: modularity and configuration"

Systemd programming part 1: modularity and configuration

g1pi — Sun, 16 Mar 2014 19:59:04 +0000

On my linux system (rel. 3.2) an unprivileged user cannot read the environment of other users' processes, either via "ps axel" or directly.

Don't know if that depends on some sysctl variable, but I always get an -EACCESS (permission denied) whenever I try "cat /proc/X/environ" (for X not a process of mine).

Systemd programming part 1: modularity and configuration

khim — Sun, 16 Mar 2014 12:11:20 +0000

Imagine a dumb sysadm reports a problem and before you can help him, you have to explain how to dump the environment of a running process, and then try to guess why the environment differs from the one he (or you) expected.

This is already needed (things like LANG or LC_ALL can effect daemons in funny way). Thankfully with systemd it's easier to trace since it cleans up environment by default (while sysvinit scripts don't do that and distribution-specific helpers do that in not-entirely-consistent-way).

Systemd programming part 1: modularity and configuration

khim — Sun, 16 Mar 2014 12:06:37 +0000

Currently a glance at the output of "ps -elf" tells everybody (even non-privileged users) how the daemon is configured at runtime.

And with environment variables you can get the same info with “ps axel” (again, even non-privileged users can do that). Is it such a big difference?

Systemd programming part 1: modularity and configuration

g1pi — Sun, 16 Mar 2014 10:10:29 +0000

"The first is to modify all the daemons to accept any command line configuration also from environment variables."

In my opinion, this is going to cause lots of headaches to sysadms and users of all distributions. Currently a glance at the output of "ps -elf" tells everybody (even non-privileged users) how the daemon is configured at runtime.

If configuration migrates into environment, only privileged users can examine it. Imagine a dumb sysadm reports a problem and before you can help him, you have to explain how to dump the environment of a running process, and then try to guess why the environment differs from the one he (or you) expected.

Systemd programming part 1: modularity and configuration

sethml — Fri, 21 Feb 2014 06:15:11 +0000

I honestly cannot tell if you're being serious or sarcastic. I vote for sarcastic.

Systemd programming part 1: modularity and configuration

farnz — Wed, 19 Feb 2014 11:02:12 +0000

It's useful in conjunction with your configuration management tool (cfengine, puppet, chef, ansible, whatever you use). Your CM tool tells you what you're pushing, systemd-delta tells you how that differs on a given system from what the distro shipped. In this use case (the one I've used it on), you run it on your test system whenever you update the underlying distro, to check that your overrides are still as you'd expect.

Alternatively, you can plumb it into your monitoring tools - check that the output of systemd-delta matches the expected delta, given your CM tool configuration, and complain if it changes.

It's more of a low level piece of plumbing than a complete tool - it gives you just enough to hook into your higher level processes so that you can easily check that you're doing what you expect.

Systemd programming part 1: modularity and configuration

nix — Sat, 15 Feb 2014 19:30:33 +0000

It's probably most interesting for developers' development machines. These are often systems in which the developer *is* the sysadmin, and which accumulate all sorts of hacks over time. With systemd-delta, you can easily see what those hacks are (in one domain) so as to more easily drop them when no longer necessary.

Systemd programming part 1: modularity and configuration

neilbrown — Thu, 13 Feb 2014 21:34:42 +0000

I can't seem to get excited about systemd-delta. It seems to be a tool for sysadmins, yet it doesn't seem to scale to multiple systems in a sensible way. To me, this is a contradiction.
Sysadmins who administer a single machine are a tiny minority I suspect.

Please do not reintroduce shell scripts when we don't need them

Darkmere — Thu, 13 Feb 2014 17:13:18 +0000

Yep, looking over the socket activations, I found some of my traditional bastards were there and enabled. PCMCIA, iscsi-related things and so on.

Off with their heads! Though I should look a bit at what
Listen: @ISCSIADM_ABSTRACT_NAMESPACE (Stream)
means.

Please do not reintroduce shell scripts when we don't need them

mathstuf — Thu, 13 Feb 2014 14:17:03 +0000

If you run "systemctl disable sshd", you get a warning that sshd.socket may still cause its activation and a command you can use to disable it as well. I don't know which version this was implemented in, but in pretty sure it was after 204.

Systemd programming part 1: modularity and configuration

farnz — Thu, 13 Feb 2014 14:08:10 +0000

When it comes to sysadmin overrides, it's worth mentioning systemd-delta; this is a command provided by systemd to let system administrators look at their changes in a diff format.

It tells you what you've done, and where you did it, so that you can undo unnecessary changes later.

Please do not reintroduce shell scripts when we don't need them

Darkmere — Thu, 13 Feb 2014 13:02:30 +0000

I have to read up more on what socket-activated things I have on my system, going to be interesting I think.

I'm a tad worried that I might have some things socket-activated where I thought they were disabled.

Please do not reintroduce shell scripts when we don't need them

kisko — Thu, 13 Feb 2014 11:12:19 +0000

In this case sshd would be an ambigious selection because there's also sshd.socket unit present.

Systemd programming part 1: modularity and configuration

mchapman — Tue, 11 Feb 2014 13:15:04 +0000

> When does systemd split the line into arguments? If it does it after expanding environment variables, putting -p into the variable would work.

The docs [1] say:

> Basic environment variable substitution is supported. Use "${FOO}" as part of a word, or as a word of its own on the command line, in which case it will be replaced by the value of the environment variable including all whitespace it contains, resulting in a single argument. Use "$FOO" as a separate word on the command line, in which case it will be replaced by the value of the environment variable split up at whitespace, resulting in zero or more arguments. [...]

So yes, it would be possible to put -p into the variable itself -- just use the no-brace form.

This does seem a bit of a round-about way of doing things though. If a daemon can't read a config file of its own, and if it can't work with the environment variables given to it, it might make sense just to use a wrapper script to convert the environment variables to appropriate command-line options fo0r the daemon. The wrapper script would then be used in ExecStart. Seems a lot neater than having a separate nfs-config.service to generate the command-line options in just the right form.

[1] http://www.freedesktop.org/software/systemd/man/systemd.s...

Systemd programming part 1: modularity and configuration

robbe — Tue, 11 Feb 2014 12:57:12 +0000

> /usr/sbin/rpc.mountd ${MOUNTD_PORT:+-p $MOUNTD_PORT}
A few comments to this problem
"sh -c ..." should work and give you the desired shell syntax. Seems clumsy, though.

When does systemd split the line into arguments? If it does it after expanding environment variables, putting -p into the variable would work.

At least for debconf, I don't see the big problem. There's always a .postinst file (usually written in bourne shell), which generates a configuration file from the contents of the debconf database. So if your original version would do something like (error handling elided):

db_get nfs-utils/mountd_port
echo "MOUNTD_PORT=$RET" >> $configfile
db_get nfs-utils/mountd_descriptors
echo "MOUNTD_DESCRIPTORS=$RET" >> $configfile

An alternative would be

db_get nfs-utils/mountd_port
port="$RET"
db_get nfs-utils/mountd_descriptors
descr="$RET"
printf "ExecStart=\nExecStart=/usr/sbin/mountd -p $port -d $descr" >> $dropinfile

Systemd programming part 1: modularity and configuration

lmb — Mon, 10 Feb 2014 13:43:38 +0000

I had a similar experience with one of my smallish pet projects - a shared storage poison pill based fencing mechanism. It has rather strong requirements of where in the cluster stack's start/stop sequence it must be run. And whether admins should be allowed to start/stop it manually (the answer being no, too much potential for shooting themselves in the foot and then complaining about data corruption.)

While simple in theory, this was not possible to achieve easily with LSB compliant init scripts (the dependency model just isn't powerful enough), and one ended up patching init scripts for corosync or pacemaker to put it in the proper place.

With systemd, this was a breeze: https://github.com/l-mb/sbd/blob/master/src/sbd.service The dependency model is just that much more powerful, and it allows the maintainer to insert hard requirements in other services. This greatly improves modularity.

As for the options on the commandline, I decided to handle this in a wrapper instead of the daemon itself. Mostly, I guess, because I was a bit lazy and this made it a little bit easier to remain compatible with previous code and ways of integrating sbd.

If I ever transition to being compatible just with systemd (or an init system at least as powerful), I'd probably remain in the foreground, handle a proper config file, etc.

Please do not reintroduce shell scripts when we don't need them

Darkmere — Mon, 10 Feb 2014 10:43:22 +0000

It used to be that systemctl would throw a big "huh? Whatchatalkingabout?" When you didn't specify .service, which made "systemctl restart sshd" annoying and hard to use. So they made .service be the default.

Maybe it should balk if there's an ambigious selection (".service and .target? I'm not making a choice!" )

Systemd programming part 1: modularity and configuration

neilbrown — Fri, 07 Feb 2014 22:45:11 +0000

Looking forward to reviewing those patches.

Make sure they include plugins for all knows configuration UIs to work with this new config file.

Given that

- configurators can already read/write environment files
- systemd can already load them into the environment, as can the shell
- daemons can get them out of the environment using "getenv()", which is already in the standard library

there seems little point in creating new config files for any configuration that can be provided by a simple environment file.

What is why I won't be providing patches to read config files.

Systemd programming part 1: modularity and configuration

bfields — Fri, 07 Feb 2014 21:54:53 +0000

"The daemon could read a configuration file (say /etc/nfsd.conf) directly..."

Yeah, I think that would be better in the long run.

Code at git://linux-nfs.org/nfs-utils, patches to linux-nfs@vger.kernel.org....

Systemd programming part 1: modularity and configuration

iq-0 — Fri, 07 Feb 2014 16:12:00 +0000

Which can be seen as a testament that systemd is capable enough to not need all sorts of workarounds for this complex case (the configuration file customization discussion aside, not that it doesn't matter but that's more a bikeshed discussion).

Please do not reintroduce shell scripts when we don't need them

Siosm — Fri, 07 Feb 2014 16:07:39 +0000

From the article:
> Unfortunately the "systemctl" command handles an abbreviated unit name but it assumes a ".service" suffix rather than a ".target" suffix. This tends to discourage the use of targets and blurs the line between API and implementation.

Maybe this issue should also be reported upstream?

Please do not reintroduce shell scripts when we don't need them

Siosm — Fri, 07 Feb 2014 15:59:41 +0000

> The config services can look up a host's name and ip address in a flat-file describing the entire cluster and generate all of the configuration files necessary to customize the host.

As far as I understand this looks like a one time process, not really a case for a systemd unit and more adapted to "configuration management" tools usage.

Moreover, you could use systemd unit specifiers (http://www.freedesktop.org/software/systemd/man/systemd.u...) to tell your service to start using a particular configuration depending on the hostname.

Also, why is the maintainer discussing this issue here and not on the systemd mailing list where he could get official recommendations/suggestions from the developers themselves?

Please do not reintroduce shell scripts when we don't need them

Shewmaker — Fri, 07 Feb 2014 15:24:30 +0000

I've found that config services can be extremely useful in cluster settings where I don't want to have hundreds or thousands of slightly different config files to manage. The config services can look up a host's name and ip address in a flat-file describing the entire cluster and generate all of the configuration files necessary to customize the host.

That way you get one system image instead of many, and can treat a cluster like a single system instead of thousands.

I'm not saying that everything should be optimized for the cluster use case, but it would be nice if distributions kept that more in mind and didn't actively make it more difficult with their new configuration tools (e.g. RedHat's NetworkManager has been unfriendly towards automation, although I've heard it has a command line now).

Systemd programming part 1: modularity and configuration

Alphix — Fri, 07 Feb 2014 04:30:48 +0000

I've looked at the init.d scripts in Debian (and I've toyed with converting a few of them to systemd unit files as a learning experience). I think it's worth mentioning that the nfs scripts are actually some of the most complex init.d scripts that I've seen so far (multiple daemons which depend on each other, kernel modules, virtual file systems, parameters, etc).

Just something to keep in mind when reading this article.

Systemd programming part 1: modularity and configuration

vonbrand — Thu, 06 Feb 2014 18:16:35 +0000

Way overkill if the typical configuration is a single file, or at most three, with a dozen lines or so each.

Systemd programming part 1: modularity and configuration

dashesy — Thu, 06 Feb 2014 15:32:42 +0000

Any modular programming language deserves (and benefits from) a nice IDE. If there are 14 files that one needs to read in order to get the overall picture, it needs a project workspace associating them together. Syntax highlighting, code assist, and go-to-definition also become handy to investigate (and perhaps fix) a foreign code base of more than few files.

Please do not reintroduce shell scripts when we don't need them

josh — Thu, 06 Feb 2014 10:09:35 +0000

Agreed entirely; if the override mechanism feels deficient in some way, extend it in systemd, not by creating a magic custom mechanism.

Please do not reintroduce shell scripts when we don't need them

Siosm — Thu, 06 Feb 2014 09:23:04 +0000

I don't really like the "nfs-config.service" approach as this brings unnecessary complexity, introduce another specific file for configuration and spawn a mostly useless shell.

Using ".d" directory or copy-pasting the unit file feels like the sensible way to go, using systemd-delta to track changes.

The author is concerned that administrators won't see the changes if the units are copied/tweaked. This should be fixed elsewhere, for example in packages update scripts where a warning could say:
"<unit> file has changed since the previous version. There are differences between /etc/systemd/system/<unit> and the /usr/lib/systemd/system/<unit> file included in this packages. Please review them using systemd-delta".

This is a new way to manage services and administrators must learn how to use systemd.

Systemd programming part 1: modularity and configuration

raven667 — Thu, 06 Feb 2014 06:49:39 +0000

Sometimes I need to shut my mouth and read the article rather than just the Unread Comments interface as then I would have seen that Neil had this well under control in the main article and wouldn't have been redundant.

Now that I've read the article I understand Neil's point but it is my opinion that giving the system admin good tools to audit changes which diverge from the default is sufficient, although sure there could be an even better system which doesn't even require the audit and integration, I'd rather keep the system simple, his example of having units run scripts to generate environment files which alter the behavior of other units is too abstract and complex (hacky) for me.

The straightforwardness of systemd-delta appeals to me, in the same way that other tools which show you diffs between the packaged and modified version of a config and allow you to merge the changes when you update.

Systemd programming part 1: modularity and configuration

raven667 — Thu, 06 Feb 2014 06:00:41 +0000

That does seem very "transitional". One of the benefits that you get with the config file arrangement for systemd is a standard supported way to override configs without conflicting with the packaged default files. You could create a new service file in /etc/systemd which changes the default command to include whatever arguments you want, and then audit any such changes with systemd-delta to show what you've hand-edited and exactly where it differs from the default. That removes a lot of the need for /etc/default or /etc/sysconfig files in the first place, which was the inherent conflict when editing /etc/init.d scripts between your edits and the packaged versions.

Systemd programming part 1: modularity and configuration

zenaan — Thu, 06 Feb 2014 04:40:21 +0000

Many daemons provide just such an option.

Also:

> While systemd can certainly read these same configuration files (with the EnvironmentFile directive) and can expand environment variables in command line arguments to programs, it lacks any sophistication. Non-optional parameters are easily handled, so:
>⋅
> /usr/sbin/rpc.nfsd -n $USE_KERNEL_NFSD_NUMBER
>⋅
> would work as expected, but optional parameters cannot be managed so easily. While the Bourne shell (which interprets SysVinit scripts) and Upstart would both support:
>⋅
> /usr/sbin/rpc.mountd ${MOUNTD_PORT:+-p $MOUNTD_PORT}

So change MOUNTD_PORT to MOUNTD_PORT_ARG:

/usr/sbin/rpc.mountd $MOUNTD_PORT_ARG ...

Where eg, /etc/default/my-service.conf contains eg:

MOUNTD_PORT_ARG="-p 12345"

I know it's clunky, but it is trivial to implement, and the author Neil _is_ talking transition here ("longer term" vs "short-term")...

Systemd programming part 1: modularity and configuration

zuki — Wed, 05 Feb 2014 23:54:22 +0000

The daemon could read a configuration file (say /etc/nfsd.conf) directly...