Systemd programming, 30 months later

Posted Sep 27, 2016 15:30 UTC (Tue) by matthias (subscriber, #94967)
In reply to: Systemd programming, 30 months later by bandrami
Parent article: Systemd programming, 30 months later

> ... but some problems scream out for an imperative solution, and this article seems to highlight one.

Is this really true? I guess, you are talking about the generator section.

The problem is that systemd cannot determine all dependencies of NFS, because it cannot read the proprietary configuration files. The solution is to have a very small program that parses the configuration file and provides the dependencies to systemd in a declarative syntax.

I am quite sure that a solution that would meet the demands of an imperative system like sysvinit would be far worse. Note that this solution covers most use cases and the administrator does not need to configure any ordering with this solution. I do not want to see a sysvinit script that manages these dependencies and works without manual configuration (like specifying an order of init scripts) by the admin.

True, the helper program is written in C, which is imperative, but this choice is because there is already a C-parser for the exports-file. Otherwise, any functional language would just do as well. There is really no need for something imperative.

I had a similar problem with managing mount dependencies for a very special use case. I just added a few dependencies myself. This was much easier than rewriting init scripts that do not match precisely my needs. How do you specify in the usual sysvinit scripts that a service needs to to run after filesystem A is mounted and before filesystem B (both of the same type)? You basically have to throw away the distribution scripts and write some new script that handles your case.

Systemd programming, 30 months later

Posted Sep 27, 2016 16:44 UTC (Tue) by bandrami (guest, #94229) [Link]

> I am quite sure that a solution that would meet the demands of an imperative system like sysvinit would be far worse.

Why? I've got something like that in most of my old init scripts:

somed -f
anotherd -f
athirdd -f

Dependencies are very simple if all you're doing is explicitly writing out a sequence of commands (sure, there are some edge cases of "make sure anotherd hasn't just forked but is actually doing something useful", though for most daemons systemd doesn't (yet) help with that either, and that's a stack selection problem anyways since you shouldn't be using a daemon that behaves badly to begin with).

> I am quite sure that a solution that would meet the demands of an imperative system like sysvinit would be far worse.

That's why I said "site-local". As a sysadmin, I don't need an init system that meets every or even most use cases; I need an init system that meets my use cases. I still haven't migrated systemd to any of our production systems (though like I said I'm warming to it) and I feel like something would have to not work, or systemd offer me something I need and don't have, to finally make that happen (we ship turnkey 2Us to clients all over the world, so this really has to be fire and forget for me). I'm not a distribution maintainer and I totally agree that for a distribution maintainer the systemd way is much, much better; the problem is that as a working sysadmin the best that gets me is where I am right now, with a lot of disruption and uncertainty in the process.

Systemd programming, 30 months later

Posted Sep 27, 2016 17:42 UTC (Tue) by dps (guest, #5725) [Link] (13 responses)

The case cited in *much simpler* if you use system V init instead of systemd. Just list the filesysyems in an order which works in /etc/fstab and system V init script will mount them sequenially in that order. I think the systemd solution described is not only more complex but also a lot more fragile... I can see ways of what has been described failing to cope with creatively used symlinks.

I don't use systemd on stripped down firewall systems because it requires both dbus and kernel features than I am not likely to actively remove becuase the firewall machine has no business using that feature.

Systemd programming, 30 months later

Posted Sep 27, 2016 17:54 UTC (Tue) by matthias (subscriber, #94967) [Link] (12 responses)

We talked about a service B that needs to start after filesystem A, but before filesystem C. In the article this service is the NFS server, in my case this was a different service. How can I tell sysvinit that it should mount A, then start service B and then mount C?

In all distros that I know, the solution would be to throw away the init scripts doing the mounts and create your own scripts. The problem is that all mounts are done by the same script, which would have to do some work before B is started and some work after. The solution is to create a separate init script for each filesystem. Then ordering of these scripts is possible. Creating a file that says B requires A and has to start before C is way easier.

In the article it is not that easy, as for a general solution, the exports file needs to be parsed to determine the dependencies. No ordering of filesystems in fstab will help with that. NFS server has to start in between.

Systemd programming, 30 months later

Posted Sep 27, 2016 18:01 UTC (Tue) by bandrami (guest, #94229) [Link] (11 responses)

> How can I tell sysvinit that it should mount A, then start service B and then mount C?

Umm...

#!/bin/sh
...
mount /A
Bctl start
mount /C
mount -a
...

I mean, seriously: people act like this is some kind of arcane dark art for reasons that still elude me. Your run control script is the series of commands, in order, your computer needs to execute to start up. Yes, that means a sysadmin needs to write a few lines of shell on a server. That's kind of what we're paid for.

Systemd programming, 30 months later

Posted Sep 27, 2016 18:17 UTC (Tue) by matthias (subscriber, #94967) [Link] (5 responses)

This is definitely no solution for a distro. And also myself, I do not want to do this, even if I can. I prefer declarative languages. You can also get paid writing code in these.

In modern multitasking systems it is not that easy. You need some synchronization, i.e., mount /C should only fire when service B is ready. The sysvinit way of doing this is adding a bunch of sleep commands and praying that there never will be a delay bigger than the configured sleep time. OK, usually sleep 1 or 2 should be enough. But this is neither nice nor fast.

Systemd programming, 30 months later

Posted Sep 27, 2016 18:39 UTC (Tue) by bandrami (guest, #94229) [Link] (4 responses)

> The sysvinit way of doing this is adding a bunch of sleep commands

No, the much more common SysV way of doing it is busy-looping some kind of polling command which, incidentally, is what systemd does for about 90% of services (this may change in the future as they learn to talk back to systemd). The only thing you ever really want to sleep for is hardware that may or may not be present and takes a while to power up (which thankfully has never applied to any server I manage).

> This is definitely no solution for a distro.

And distro init scripts are not and have never been a solution for me as a professional sysadmin; the best ones are the Slackware style that are nearly brain-dead and do enough to get the system at a point where you can configure it how it is supposed to be. They're far too general and try to be all things for all people. For my laptop? Fine; I don't really care.

> And also myself, I do not want to do this, even if I can. I prefer declarative languages.

I find them way too opaque for high-pressure debugging. I want to know the actual commands the server is executing, in the order they are executed, and I want cat and grep to be enough to view them (since I may not even have ncurses when I need it)

Systemd programming, 30 months later

Posted Sep 28, 2016 21:32 UTC (Wed) by smcv (subscriber, #53363) [Link] (3 responses)

> busy-looping some kind of polling command which, incidentally, is what systemd does for about 90% of services

I don't think any of the supported service types need to busy-wait? systemd's structure is basically an event loop, letting the kernel do all the waiting (like typical network servers).

Type=notify services can wait on the notify socket, using sd-event (basically a wrapper around epoll).

Type=dbus services can subscribe to NameOwnerChanged signals, then wait on the D-Bus socket with sd-event.

Type=forking services (those that double-fork, do their real work in the grandchild, and exit from the original process when the grandchild is ready, like a typical sysvinit-managed daemon) can wait for the original process to exit (by giving sd-event a signalfd that waits for SIGCHLD). Type=oneshot services (those that do some work in a blocking way and then exit, without leaving long-running subprocesses behind) are similar.

Type=simple and Type=idle have no concept of "ready" (they start the service then just continue as though it was already ready) so there's nothing to wait for.

Systemd programming, 30 months later

Posted Sep 29, 2016 0:33 UTC (Thu) by bandrami (guest, #94229) [Link] (2 responses)

Right, and type-"notify" and "dbus" are not very many services in September 2016, are they?

In the future, there will be more of them. As it is, Systemd is either polling or simply blithely ignoring most services, just like SysV did.

Systemd programming, 30 months later

Posted Sep 29, 2016 0:59 UTC (Thu) by neilbrown (subscriber, #359) [Link]

> Right, and type-"notify" and "dbus" are not very many services in September 2016, are they?

$ date
Thu Sep 29 10:55:16 AEST 2016
$ grep -h Type= /usr/lib/systemd/system/*.service | sort | uniq -c| sort -n
      9 Type=idle
     17 Type=simple
     18 Type=notify
     21 Type=dbus
     23 Type=forking
     92 Type=oneshot

Systemd programming, 30 months later

Posted Sep 29, 2016 6:36 UTC (Thu) by pbonzini (subscriber, #60935) [Link]

Types other than dbus or notify don't poll, either.

Systemd programming, 30 months later

Posted Sep 28, 2016 21:18 UTC (Wed) by rahvin (guest, #16953) [Link]

Your forgot the 20 lines of sleep commands to make sure the daemon has time to load, the 50 lines of code to verify the daemon didn't crash and the mount is up before the second mount command runs. Then the 50 lines of code to handle restarting and shutting down these in the reverse order, etc.

As he said, you replace the sysv init script with your own custom script. And then you reinvent the wheel every single time. Some of us would prefer not to do that. I like that this upstream service file takes care of all but the freakiest edge case. 99/100 I won't even need to touch these system files now.

Systemd programming, 30 months later

Posted Sep 29, 2016 16:03 UTC (Thu) by drag (guest, #31333) [Link] (3 responses)

> I mean, seriously: people act like this is some kind of arcane dark art for reasons that still elude me.

It's always been trivial if you can have make a huge amount of assumptions about your environment, have a static environment, are dealing with perfectly behaving software, and are the only person that will ever need to use or maintain these systems.

When everything is simple and predictable then the simple is trivial.

The more reality diverges from the 'simple case' the more of a 'arcane art' it becomes.

Lets look at your example:

mount /A
Bctl start
mount /C
mount -a

Ok, mount /A has been mounted for a long time. Needs to do fsck check. How are the other services in your system going to deal with having that file system, btctl start, mount /C and mount -a not running for another 15 minutes or so after start up? Nothing started before it will ever be in any danger of timing out. You are also completely fine with having a inaccessible system if it runs, say, before sshd does?

Brtl start to mount /C... Are you absolutely sure there is no race condition there?

If 'mount /C' depends on a service started by 'Bctl start' and 'Bctl' does the daemon double fork then there is no way this script actually takes into account the fact that 'mount /C' is going to be ran LONG before 'Bctl' ever completes it's initialization in a fast system. May work just fine in a VM or something, but it's very likely to fail randomly. so on and so forth. Unless the 'Bctl' script contains a lot of logic itself so that it does not exit before B is started... which in that case you are just hiding the bulk of your example behind a secret-perfect command.

Basically your example pretty much going to fail in even trivial use cases.

When I do something I try to take the time to do it right the first time so I never have to deal with it again. This is definitely not something that I would find remotely acceptable except in simple one-off cases.

I've also written plenty of Init scripts and because of that I really would like to avoid doing such things.

Systemd programming, 30 months later

Posted Sep 29, 2016 17:30 UTC (Thu) by nybble41 (subscriber, #55106) [Link] (2 responses)

> If 'mount /C' depends on a service started by 'Bctl start' and 'Bctl' does the daemon double fork then there is no way this script actually takes into account the fact that 'mount /C' is going to be ran LONG before 'Bctl' ever completes it's initialization in a fast system.

If 'Bctl' is written incorrectly, that is. Not that there aren't plenty of examples of such incorrect behavior, but the right approach would be first to complete the initialization, *then* to fork. The initial process shouldn't exit until the daemon is ready to deal with clients. (Note that systemd also requires this behavior from Type=forking services to support proper dependency handling. The service will be considered ready for use as soon as the initial process exits.)

I do completely agree with you about the other limitations of hand-written, system-specific init scripts. In practice systemd unit files are superior in every significant way, though they do require a certain amount of retraining.

Systemd programming, 30 months later

Posted Sep 29, 2016 21:43 UTC (Thu) by drag (guest, #31333) [Link] (1 responses)

> If 'Bctl' is written incorrectly, that is.

Exactly. Effectively he moved the difficulty of dealing with the sequence of these commands out of the example script and into the 'brctl' and 'mount' commands. There is no escaping the complex logic required to do these things correctly.

Systemd programming, 30 months later

Posted Oct 7, 2016 18:46 UTC (Fri) by lsl (subscriber, #86508) [Link]

That logic needs to be in the B service, one way or another. Only the service itself can know when it considers itself ready for operation, so it is going to signal that readiness by exiting from the parent, starting to accept connections from a socket, signaling an eventfd or something entirely else. Whatever the mechanism, the underlying logic needs to be there.

The nice thing about socket activation (as implemented by systemd) is that it can piggyback on a mechanism we're probably using anyway as part of normal operation and for which the kernel already does lots of the ordering orchestration. It might not fit all services but when it does it's a very nice thing to have.