The Grumpy Editor's guide to surviving the systemd debate [LWN.net]

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 23:36 UTC (Fri) by neilbrown (subscriber, #359)
In reply to: The Grumpy Editor's guide to surviving the systemd debate by anselm
Parent article: The Grumpy Editor's guide to surviving the systemd debate

> How much more complex can you get?

There are different sorts of complexity, or at least different locations for it.

You have have a very simple engine with very complex configuration, and you can have a very complex engine with very simple configuration.

sysvinit leans towards the first, systemd leans towards the second.

My preferences is generally towards having a simple engine (which is probably why I like coding in C). It is certainly appropriate to add functionality to a simple engine, but it is best if it can be kept simple (well-defined, uniform, orthogonal)

systemd is excellent in terms of functionally. I'm less convinced that it has maintained internal simplicity.

to post comments

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 1:24 UTC (Sat) by anselm (subscriber, #2796) [Link] (7 responses)

My preferences is generally towards having a simple engine

“Simple” is generally a good idea, but there is such a thing as “too simple”. The sysvinit “engine” is really too simple for what it needs to do, which is why it is not particularly good at what it is doing, so people need to tack on important functionality by way of shell scripts. Note: If your engine uses a hundred shell scripts which are all substantially the same, this might tell you that perhaps some functionality could be usefully added to your engine such that your configuration does not need to be quite that complex. Also for the longest time even the complex configuration of sysvinit needed considerable ad-hoc human expertise to manage; in the history of sysvinit over the last 30 years or so, the automatic management of dependencies between services hasn't been around all that long, and that is one area where a little more complexity in the engine would really have been quite helpful.

Also, if one considers not just sysvinit but the whole “basic plumbing” that systemd replaces (including such beauties as the distribution-specific boot scripts or inetd/xinetd), then any claim of simplicity goes right out the window. The individual components on their own may be pretty simple but their totality isn't (and distribution-specific design choices figure into it, too, so if you have figured out how distribution X does its thing, that may or may not apply to distribution Y), and in spite of all the complexity they don't really form a coherent system that works well together. Systemd is a lot more advanced in that respect.

Which is not to say that systemd is the epitome of internal simplicity – but the systemd developers do care about things like code reuse, and facts like that all the configuration files use the same format (and presumably share the same parser) do suggest that systemd isn't the most complex piece of software in existence, either. It is probably a lot simpler internally than, say, the Linux kernel, or for that matter the sum total of sysvinit and friends.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 1:44 UTC (Sat) by dlang (guest, #313) [Link] (6 responses)

actually, what you dismiss as a whole pile of shell scripts doing almost the same thing is actually frequently a small amount of boilerplate that is calling the helper functions (as scripts or binaries) that provide exactly the type of extensions that you are saying need to be added

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 5:42 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

Except that helper functions are not really helpful at all. They don't do namespacing, SELinux or even bad old chrooting. They barely provide PID file tracking, and even that without daemon readiness protocol or anything like this.

And if you do try to add them, then scripts become even more incomprehensible. Witness, the Gentoo's BIND initscript as an example: http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/n...

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 19:04 UTC (Sat) by anselm (subscriber, #2796) [Link] (4 responses)

Possibly. On my Debian system, many of the shell scripts in question still are more than 100 lines long (sometimes considerably so), still contain mostly boilerplate, and still don't do various things one would reasonably wish to be done when a service is started. There are ways of reducing the amount of repetition but they do not appear to have caught on in most mainstream Linux distributions. I for one am looking forward to these scripts being replaced by systemd unit files that for the most part will fit on one screen.

Then there is the fact that these shell scripts are impossible to change and adapt to local preferences without running the risk of having to do it over and over again whenever the distribution issues an updated version, or having to prevent the distribution from installing and activating its version of the script in addition to the one the local administrator provided under a different name. Systemd solves this in a much more elegant manner by cleanly separating the distribution-provided default from any local changes.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 16, 2014 21:39 UTC (Sun) by rgmoore (✭ supporter ✭, #75) [Link] (3 responses)

There's also the general point that writing configuration files as programs in a Turing-complete programming language is a gross violation of separation of policy from mechanism. I would probably lump it in with things that count as design anti-patterns.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 17, 2014 1:46 UTC (Mon) by neilbrown (subscriber, #359) [Link] (2 responses)

> There's also the general point that writing configuration files as programs in a Turing-complete programming language is a gross violation of separation of policy from mechanism.

Is it?
What if your policy is algorithmic in nature, or is more complex than the designers of the config file imagined?

TCL, lua, javascript are all languages that effectively allow very sophisticated tools to be configured in very sophisticated ways.
Are they all gross violations? (I almost added PHP to the list, but that might hurt my argument....)

If your configuration language supports substitution or conditionals - or maybe even just sequencing - then you probably want to start looking at a real language, because iteration probably isn't far away.

The line between "policy" and "implementation" is very real, but it is in your mind, not in the computer. And that is where it should be enforced.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 0:07 UTC (Tue) by nix (subscriber, #2304) [Link] (1 responses)

TCL, lua, javascript are all languages that effectively allow very sophisticated tools to be configured in very sophisticated ways.
Are they all gross violations?

Further, at least two of these (Lua and Tcl) explicitly started as ways to write, well, Turing-complete configuration files: configury and program in the same language. Javascript didn't, but then it grew JSON, which is, well, exactly the same thing.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 2:40 UTC (Tue) by zlynx (guest, #2285) [Link]

Sort of. But serious users of JSON never stick it into a string and eval(). That way lies nightmares and security holes.

In best practice JSON is non-executable.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 9:07 UTC (Sat) by fandingo (guest, #67019) [Link] (6 responses)

Why do you speak in metaphors and veiled examples in your several replies? What's the metaphor: game engines, search engines, steam engines, locomotives?

Your last comment further up talks about two distinct problems that you recently experienced. Why did you omit the details? To be honest, this sort of thing is used by systemd detractors all the time. The (unintentional) subterfuge masks either their lack of knowledge or fraudulent purpose. I'm not saying that you're arguing in bad faith, but if you're going to detract from something, it's only reasonable to provide enough details so we know what you're talking about.

Continuing about this comment, I assume that your metaphor is about mechanical engines. Are you really suggesting that more primitive engines are better? Should we all desire steam engines despite the fact that would make air travel impossible? Sure it's more straightforward to understand "dump the water in there and shove the coal into the fire," but to continue the metaphor the drawbacks are preposterous. Again I hope I'm getting the metaphor right, but it seems like the piston-driven airplane and jet engines are perfect corollaries to sysV and systemd. The engineering level of a piston engine is quite a bit lower and has far lower tolerances. The jet engine, on the other hand, is both far, far more powerful and efficient, and it is also substantially more reliable and requires practically no maintenance.

But anyways, init systems are not engines. It's pretty useless to talk about technical things in metaphors. We can shoehorn in whatever exemplary or derogatory metaphor we want without actually talking about anything.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 12:02 UTC (Sat) by viro (subscriber, #7872) [Link] (1 responses)

"The jet engine, on the other hand, is both far, far more powerful and efficient, and it is also substantially more reliable and requires practically no maintenance."

Off-topic, but... Jet engines getting "practically no maintenance" are the stuff of which the nightmares are made... Ever seen the effects of fatigue crack in the wrong place that has grown past the critical length due to inadequate inspections? Gets _really_ unpleasant when e.g. the shaft fails, separating compressor from the turbine and leaving the latter with no load...

There's a whole lot of failure modes in those puppies and airlines trying to save money by cutting down on maintenance ended up with serious trouble after a while. Including a lot of dead bodies...

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 12:56 UTC (Sat) by pizza (subscriber, #46) [Link]

> Off-topic, but... Jet engines getting "practically no maintenance" are the stuff of which the nightmares are made...

There are far fewer failure modes in Jet engines -- due to many fewer moving parts and much much much much lower vibration, thanks to the lack of reciprocating mass. Consequently, they require considerably less ongoing maintenance than piston engines.

(But only a fool or airline execs would consider them maintenance-free)

OT, yes...

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 17, 2014 0:57 UTC (Mon) by neilbrown (subscriber, #359) [Link] (1 responses)

> Why do you speak in metaphors

I didn't think I was. I thought "engine" had a fairly well defined meaning in the software world - it is the bit on the inside that does all the work. It might be a game engine or search engine and you suggest, or a physics engine or an inference engine or a database engine or... whatever is required.

You can often think of a software system as having an engine and an interface.

No, I don't think a more primitive engine is better. I think having core functionality that meets the need is very important and systemd has a lot of that - which is good. I want a sophisticated engine, but I don't want a complicated engine.

I think 'git' provides a valuable contrast. It is in some ways a new and different thing, much like systemd. It uses some well established ideas and combines and presents them in a way that adds a lot of value. And there are people who seem to hate git too - though not as much as systemd it seems.

A key part of the design of git is that core functionality (the so-called plumbing) is directly available if you want to use it. On top of this more friendly interfaces are provided. Originally this was just shell scripts combining the core parts, though I think that over time more and more is being converted to C. This is sophisticated but not too complicated. Core functionality is well defined and directly accessible. This depends on having a rich language for combining the components: shell.

systemd has a lot of valuable core functionality too. But it is not always exposed in a way that provides direct access. The configuration language is extremely simplistic. Quite a few of the directives provided are composites of core functionality - so they set an agenda ("set combinations apply") rather than provide raw functionality.

To be more specific, there are mechanisms for interactions between different units. In upstart (if I remember correctly), such interactions involve "signals" and I think there is full low-level control of signals.
systemd designers deemed that signals were not sufficient (Which is probably correct) and provide something else. What exactly? It seems to be a combination of signals and requirements.

"Requisite" is a requirement with no signal
"Wants" is a signal with no requirement
"Requires" is a combination of the two.

There is more subtlety to the interactions - there is a "reload" signal and maybe a "restart" signal, though that might have requirement implications. There are inverse requirements (conflicts).

But the configuration language doesn't provide access to the core features - I'm not even entirely sure what all the core features are precisely. It just provides a number of directives that pre-combine these features in ways that seem to be useful.

So I do want a sophisticated engine - I want socket activation and process group monitoring and timeouts and auto-mounts and device triggers etc etc. I think all that is great stuff.
But I want direct access to all of it. I want to be in control (yes - I'll admit to being a control freak). I want to be able to access the different functions in ways that the designers never envisaged.

Things become "complicated" when you don't have direct access and so need to work around limitations and do things in no-obvious ways just to get access to the bits you want.

(I'll reply about the examples separately).

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 17, 2014 18:58 UTC (Mon) by cebewee (guest, #94775) [Link]

A nice property of a restricted language is that it is usually much more accessible to static analysis (and turing-complete languages aren't). This might be one of the reasons why the systemd developers lean on this side of exposing functionality.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 17, 2014 1:33 UTC (Mon) by neilbrown (subscriber, #359) [Link] (1 responses)

> Why do you speak in ... veiled examples in your several replies?

Sometimes too much detail can distract people from thinking about the point you are trying to make. So including it at the wrong time might not help. But I'm happy to share.

These are two issues that I recently hit. They are *not* examples which try to say "systemd is horrible and doomed can never work", they are meant as examples that say "systemd does have some real complexity, does have steep regions on the learning curve, isn't entirely transparent". It could be that they are problems for me only because I haven't learned enough yet - which just points to the learning-curve issue.

Firstly, I really like that using systemd makes it a lot easier for upstream packages to provide the start-up config file: providing init scripts in upstream was always problematic.
But if a distro has a pre-existing practice of using a particular name for a service, and upstream chooses a different name, then there is an issue to be resolved. I would like to support both names, at least during a transition period. That means we need an effective alias, so "foo-daemon" and "food" can be used interchangeably.

systemd *has* alias functionality using syslinks. If you create "food.service" as a link to "foo-daemon.service" then you can "start" and "stop" and "reload" and request "status" using either name. Which is great.
But you cannot "enable" food.service. Is this by design or is it a bug? From what I know of systemd I cannot really decide and the documentation only seems to mention aliases in asides - I cannot find a section "how aliases work".

So I created 'food.service' as a separate service with (hopefully) enough "Requires" and "PartOf" directives pointing at "foo-daemon.service" that systemd will make sure they are either both running or neither are.

Secondly, I had some fun with dracut (not that dracut is much fun ... I find it particularly difficult even remembering what it is called!! Whose idea was it to name key system software after towns in New England? And will they please stop it!?!)

Dracut uses systemd (which seems appropriate) and has a bunch of shell scripts which are run by a queue manager (I can't decide if there is something ironic about that.... can systemd not manage the queue? should it?).

Anyway in a particular circumstance an md/raid array will *not* be assembled when all the devices have appeared, and this is correct. There needs to be a timeout to give up waiting for more devices - then the array is assembled. If that array contains an LVM volume and the LVM volume contains the root filesystem, then dracut needs to:
1/ wait the timeout
2/ assemble the md array
3/ activate the LVM volume

all of which it does correctly. During this time, systemd is waiting for the root device to appear.
How long does it wait? According to "systemd.unit", "JobTimeoutSec" defaults to zero - except for devices units. It doesn't tell me what the default is for device units :-( It turns out to be 90 seconds - DefaultTimeoutStartSec in /etc/systemd/system.conf. But the timeout in step '1' above in 120 seconds. How can this ever work?
I'm not entirely sure, but it does is most cases. In the above case, systemd times out between steps 2 and 3. So if the md array contained the root device directly it would work. If not, it doesn't because systemd doesn't wait the extra second it takes to activate the volume.

If I increase DefaultTimeoutStartSec to 120 .. it still doesn't work. 130? No. 300? Yes. Now it works.

I'm sure there is a good explanation for this. It might even make sense. But "simple" "transparent" "gentle learning curve" aren't the words I'm focussing on just now.

Maybe I need to enable some tracing somewhere.... old habits of adding "echo" or "printk" as just going to have to change...

The Grumpy Editor's guide to surviving the systemd debate

Posted Mar 13, 2015 0:37 UTC (Fri) by neilbrown (subscriber, #359) [Link]

For the record, the problem with dracut was that it has a systemd 'generator' which only creates the important files the first time it is run: rootfs-generator.sh

When you "systemctl daemon-reload", systemd will remove everything from /run/systemd/generator and then re-run all the generators. As rootfs-generator.sh only created files in that directory the first time it is run, a subsequent 'systemctl daemon-reload' will revert the effect of that generator.

The key effect of the generator is to disable systemd timeouts on the device which holds the root filesystem - as dracut has its own timeout mechanism. So a 'daemon-reload' will re-instate the default timeout.

When dracut does timeout waiting for the md array to assemble, and forces the assembly of all degraded md array, it also runs "daemon-reload" (as part of giving up on resume-from-disk). This will cause systemd to give up on the root device immediately, typically a second or so before it is actually ready.

Does that make sense? Anyway, the primary problem was not a problem with systemd ... except...

The real goal of dracut here is to generally disable the systemd timeouts which wait for devices. Lots of systemd is configurable at runtime using systemctl, but this thing isn't. It can only be configured in /etc/systemd/system.conf before systemd starts. 'daemon-reload' doesn't reload system.conf.

So dracut has this complex (and buggy) code to over-ride timeouts that systemd doesn't let it configure in a sensible fashion. So maybe systemd is partly to blame, after all...