The Grumpy Editor's guide to surviving the systemd debate [LWN.net]

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 13, 2014 4:09 UTC (Thu) by neilbrown (subscriber, #359)
Parent article: The Grumpy Editor's guide to surviving the systemd debate

> There is plenty of experience by now to show that Linux systems still work when systemd is installed

To me, this seems to miss the point. One could equally say "There is plenty of experience by now to show that Intel-based systems still work when MS-Windows is installed".

It isn't about whether it works, it is about what it is like to work with.
So some important issues are complexity, transparency, familiarity, learning curves, etc.

> But even if systemd turns out to be a wrong turn in the end, the current level of anxiety and heat is not called for.

This, on the other hand, is spot on. All the heat makes is very hard to address the other issues coherently.

to post comments

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 13, 2014 8:56 UTC (Thu) by rvfh (guest, #31018) [Link] (2 responses)

> It isn't about whether it works, it is about what it is like to work with.

OK, but most user don't work with the init system, right? It's just a few developers that are affected, and far less people than are participating in discussions, right?

> So some important issues are complexity, transparency, familiarity, learning curves, etc.

OK, but as was said before, the learning curve for the old init or even upstart, when starting from scratch (not your case I suppose) is less steep with systemd... At least that's my personal experience after recently writing the service file for my [very simple] daemon for sysvinit, upstart and systemd.

I would go as far as to say that your voice would probably be one of the most useful, but maybe it gets buried in the noise...

Note: I used to maintain a Slackware-based distribution for a private company about 10 years ago.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 22:39 UTC (Fri) by neilbrown (subscriber, #359) [Link]

> OK, but most user don't work with the init system, right? It's just a few developers that are affected, and far less people than are participating in discussions, right?

I wonder if that is true. I don't have any data - do you?

It isn't just developers that might need to work with an init system, it is also sysadmins, and many hobbyists serve as their own sysadmin.

You've probably heard the quote:
> Simple things should be simple, complex things should be possible.

With systemd, lots of simple things really are simple. It has very focussed domain-specific "knowledge" and does many of the things that you often want very easily. This makes the initial learning curve very gentle as you suggest.

But simple things aren't really all that interesting - of course those are easy. The complex things provide a much better test for what a system is like to work with. That is where the learning curve gets interesting.

With init.d scripts, most people will have been exposed to shell programming already so there is an opportunity for skills transfer. shell is undoubtedly a horrible language, but it is a powerful and flexible language and you can usually achieve what you need to. It is easy to add tracing (echo, set -x), easy to add special cases. Possible to address ad-hoc problems with ad-hoc solutions.

systemd is much better in terms of core functionality and over-all structure. But is it not a complete language. It seems to deliberately avoid being a programming language in the normal sense (witness the odd syntax for conditionals). So if you want to do "normal" things, it is very easy. But as you deviate from "normal" the learning curve gets very steep. When I do, I start to feel like I am fighting against it instead of working with it.

I've had two non-normal issues with systemd recently. They *should* be easy - but I didn't find them so. In one case I think I cobbled together a solution, but I don't really like it. Maybe the fact that the "obvious" approach didn't work is just a bug. For some reason, I just don't feel like trying to find out if it is.
In the other case systemd was behaving very strangely w.r.t. timeouts. I found a work-around but I still don't understand why it wasn't simple. Maybe I will if/when I try harder.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 21, 2014 17:56 UTC (Fri) by Wol (subscriber, #4433) [Link]

> > It isn't about whether it works, it is about what it is like to work with.

> OK, but most user don't work with the init system, right? It's just a few developers that are affected, and far less people than are participating in discussions, right?

Except you're making a major mistake that far too many IT workers make. You're assuming that systems have a sys-admin.

How many systems are like mine? I have a home network - two gentoo PCs, a Windows 7 laptop, a bunch of Android phones ... yep, I may be a sys-admin, but I'm a sys-admin merely because I'm a user and I'm the only person to admin my systems.

I want an init system that is simple to use, and SysVInit most certainly is NOT. Currently (while I may not be able to blame it on the init system, I can certainly blame it on the lack of time to trouble-shoot) I have a system with broken raid and nfs. I don't want to add init weirdos to the rest of my boot-time cussing ...

Cheers,
Wol

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 13, 2014 9:35 UTC (Thu) by anselm (subscriber, #2796) [Link] (21 responses)

So some important issues are complexity, transparency, familiarity, learning curves, etc.

In that department systemd runs rings around the traditional setup (possibly not as far as “familiarity” is concerned but then again it hasn't been around that long). Remember that the traditional setup consists of components from diverse sources that have been cobbled together with lots of duct-tape but no overarching design and little interaction. Between init, the init scripts, /etc/fstab, inetd/xinetd, cron, and friends, no two configuration file formats are the same, and the components don't talk to one another except on the most basic and accidental level. (Even within sysvinit the components don't really talk to one another; the PID 1 process doesn't know or care whether service daemons are running or not, so you need more components and more duct-tape to retro-fit service management.) As far as complexity is concerned, the bulk of the configuration files in a sysvinit-based system are shell scripts. How much more complex can you get? Especially if 90% of every init script is basically boiler plate (but possibly with subtle and undocumented tweaks) but that boiler plate is different from one distribution to the next.

It turns out that systemd is really rather easier to work with than the traditional setup simply because it is much more consistent with itself and has been designed with maintainability in mind. For example, with systemd there is a clean separation between default configurations that the distribution provides and local changes. This makes package upgrades a lot easier since you don't need to go in and manually reconcile your own changes to a configuration file with the changes to the same file that the distro made in the meantime.

All the heat makes is very hard to address the other issues coherently.

It would also help if many of the systemd detractors would stop to learn at least a little bit about the ideas behind systemd (and their implementation). Much of the systemd “criticism” is really not based on actual systemd but on some straw-man version postulated by detractors. Many people who were initially opposed to systemd found out more about it and then rather liked what they saw.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 13, 2014 18:10 UTC (Thu) by skorgu (subscriber, #39558) [Link] (3 responses)

Systemd in and of itself, sure. It is a bit of a lightning rod/poster child for a definite change in how systems are built though and teasing those two roles apart isn't always easy. I was trying to debug an early-boot process in CentOS 7 this week which supports both a complex set of shell scripts in the form of /etc/init.d/network (now started through systemd) and NetworkManager.

I decided NetworkManager was the Future, made my peace with the capitalization choices, read some docs and tried to use it. No dice, as far as I could tell the networkmanager service started but NetworkManager-wait-online.service didn't which caused kdump to determine there was no network which prevented the multi-user.target from being reached.

Ok. Let's look at the wait-online.service.

ExecStart=/usr/bin/nm-online -q --timeout=30

Great! I run nm-online and it hangs for 30 seconds before failing. What criteria is it using to check that the network is up? --help is useless, "nmcli networking connectivity" says "full" and I have no idea where to look next. strace and looking at the code upstream led me to the conclusion that it's doing something via dbus but to understand what exactly would require me to understand a good chunk of the NetworkManager/dbus codebase.

Fine. Let's try the init script way. Hm, doesn't work, looking at the logs (journalctl is good, why isn't -l the default though?) shows me that udev was renaming eth0 -> em1 at the same time as the network was trying to bring up em1, classic race condition. I wonder if there's a delay I can add?

grep -i delay /etc/sysconfig/network-scripts/*

Bam, add LINKDELAY=10 to the ifcfg-em1 file and move on with my life.

Is the problem here systemd? No, it's clear that this specific case was networkmanager's fault and systemd was faithfully executing what it was told to. However the /model/ of reducing complexity by tighter coupling, dbus and C instead of files and shell scripts, monolithic entities instead of loose components made this much harder to debug.

There's no reason to foam at the mouth and cancel my LWN subscription, we're all adults here and should be able to say "hey this isn't great" without going off the rails. I have no doubt that in a few years these bugs will have tripped up enough people and been fixed, better tools will exist or simply be more widely known and it'll be fine, probably better than the old world.

However we *are* all adults here and shouldn't pretend that systemd is just PID 1 and not also a central and highly visible component of a fundamental change in how linux systems are built and operated or that there aren't tradeoffs in discoverability and debugability being made.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 13, 2014 19:33 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

A better solution would have been to add a em1 dependency to the NetworkManager target. It can be done by creating a socket unit depending on the em1 device and adding it to the dependency list.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 18:59 UTC (Fri) by skorgu (subscriber, #39558) [Link] (1 responses)

I didn't think of that, thanks for the suggestion!

Though I'm not sure it would have actually helped the NetworkManager case since the network-online target /never/ came up, even when the system manifestly had connectivity (and even nmcli agreed).

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 19:03 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

I don't think that socket targets actually depend on the network-online target, they are activated directly by udev events. So it should have helped.

Also, can you file a bug for this? It's clearly an issue that should be fixed.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 6:07 UTC (Fri) by ploxiln (subscriber, #58395) [Link]

If only systemd was just for running services. If only!

I was very excited when I read Lennart's first blog post about systemd, years ago, it was just about the socket-activation model. That's pretty cool.

But then you thrown in excessive use of cgroups, journald, logind (and the discontinuing of consolekit), udev in the same repo! dbus! tempfiles! cron! block devices, filesystems! all this stuff really doesn't need to be coupled to init, it really really doesn't. Setups in arch linux or gentoo or slackware or plain debian were super simple in all of these areas. Yeah it sucked that services weren't properly managed, processes not properly tracked and restarted, but that was really it.

Now, if you want to use arch linux or fedora or more and more other distros, you have to do a shitload of work to avoid systemd's huge web of stuff.

Systemd used to have a config option to not use some cgroup controllers on the cgroups it created, like the memory controller. I used that option. It went away. Now I have to compile a custom kernel with those controllers disabled.

Systemd failed to shutdown (shutdown!) multiple times for me over the past couple years (I've been using arch linux, they switched quite a while ago).

Worst of all, for me, the output of "systemctl --all" is ridiculous, like 300 lines, of which maybe 20 are interesting. All sorts of stuff is enabled by default, there are services without actual unit files in the system, loads of virtual filesystems I haven't used and probably won't use anytime soon are mounted, and seriously cluttering up /proc/mounts, it's more than twice as long and twice as wide as it was before. I mask loads of stuff when I set up a new system. I have to unmount some stuff in a late-running rc.local because some stuff isn't even a virtual "unit" or whatever.

This is the ubuntu/windows model that I used (other) linux distros to get away from. Loads of crap by default, then I have to clean it up, if I want to end up with a system that I understand and fully control. And now all this systemd stuff comes together, I can't trim or simplify pieces.

I've used initscripts, daemontools, systemd, upstart, both personally and professionally. I can deal with it. But I don't like it being suggested that I'm backwards and closed-minded for not being happy about it.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 23:36 UTC (Fri) by neilbrown (subscriber, #359) [Link] (15 responses)

> How much more complex can you get?

There are different sorts of complexity, or at least different locations for it.

You have have a very simple engine with very complex configuration, and you can have a very complex engine with very simple configuration.

sysvinit leans towards the first, systemd leans towards the second.

My preferences is generally towards having a simple engine (which is probably why I like coding in C). It is certainly appropriate to add functionality to a simple engine, but it is best if it can be kept simple (well-defined, uniform, orthogonal)

systemd is excellent in terms of functionally. I'm less convinced that it has maintained internal simplicity.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 1:24 UTC (Sat) by anselm (subscriber, #2796) [Link] (7 responses)

My preferences is generally towards having a simple engine

“Simple” is generally a good idea, but there is such a thing as “too simple”. The sysvinit “engine” is really too simple for what it needs to do, which is why it is not particularly good at what it is doing, so people need to tack on important functionality by way of shell scripts. Note: If your engine uses a hundred shell scripts which are all substantially the same, this might tell you that perhaps some functionality could be usefully added to your engine such that your configuration does not need to be quite that complex. Also for the longest time even the complex configuration of sysvinit needed considerable ad-hoc human expertise to manage; in the history of sysvinit over the last 30 years or so, the automatic management of dependencies between services hasn't been around all that long, and that is one area where a little more complexity in the engine would really have been quite helpful.

Also, if one considers not just sysvinit but the whole “basic plumbing” that systemd replaces (including such beauties as the distribution-specific boot scripts or inetd/xinetd), then any claim of simplicity goes right out the window. The individual components on their own may be pretty simple but their totality isn't (and distribution-specific design choices figure into it, too, so if you have figured out how distribution X does its thing, that may or may not apply to distribution Y), and in spite of all the complexity they don't really form a coherent system that works well together. Systemd is a lot more advanced in that respect.

Which is not to say that systemd is the epitome of internal simplicity – but the systemd developers do care about things like code reuse, and facts like that all the configuration files use the same format (and presumably share the same parser) do suggest that systemd isn't the most complex piece of software in existence, either. It is probably a lot simpler internally than, say, the Linux kernel, or for that matter the sum total of sysvinit and friends.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 1:44 UTC (Sat) by dlang (guest, #313) [Link] (6 responses)

actually, what you dismiss as a whole pile of shell scripts doing almost the same thing is actually frequently a small amount of boilerplate that is calling the helper functions (as scripts or binaries) that provide exactly the type of extensions that you are saying need to be added

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 5:42 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

Except that helper functions are not really helpful at all. They don't do namespacing, SELinux or even bad old chrooting. They barely provide PID file tracking, and even that without daemon readiness protocol or anything like this.

And if you do try to add them, then scripts become even more incomprehensible. Witness, the Gentoo's BIND initscript as an example: http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/n...

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 19:04 UTC (Sat) by anselm (subscriber, #2796) [Link] (4 responses)

Possibly. On my Debian system, many of the shell scripts in question still are more than 100 lines long (sometimes considerably so), still contain mostly boilerplate, and still don't do various things one would reasonably wish to be done when a service is started. There are ways of reducing the amount of repetition but they do not appear to have caught on in most mainstream Linux distributions. I for one am looking forward to these scripts being replaced by systemd unit files that for the most part will fit on one screen.

Then there is the fact that these shell scripts are impossible to change and adapt to local preferences without running the risk of having to do it over and over again whenever the distribution issues an updated version, or having to prevent the distribution from installing and activating its version of the script in addition to the one the local administrator provided under a different name. Systemd solves this in a much more elegant manner by cleanly separating the distribution-provided default from any local changes.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 16, 2014 21:39 UTC (Sun) by rgmoore (✭ supporter ✭, #75) [Link] (3 responses)

There's also the general point that writing configuration files as programs in a Turing-complete programming language is a gross violation of separation of policy from mechanism. I would probably lump it in with things that count as design anti-patterns.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 17, 2014 1:46 UTC (Mon) by neilbrown (subscriber, #359) [Link] (2 responses)

> There's also the general point that writing configuration files as programs in a Turing-complete programming language is a gross violation of separation of policy from mechanism.

Is it?
What if your policy is algorithmic in nature, or is more complex than the designers of the config file imagined?

TCL, lua, javascript are all languages that effectively allow very sophisticated tools to be configured in very sophisticated ways.
Are they all gross violations? (I almost added PHP to the list, but that might hurt my argument....)

If your configuration language supports substitution or conditionals - or maybe even just sequencing - then you probably want to start looking at a real language, because iteration probably isn't far away.

The line between "policy" and "implementation" is very real, but it is in your mind, not in the computer. And that is where it should be enforced.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 0:07 UTC (Tue) by nix (subscriber, #2304) [Link] (1 responses)

TCL, lua, javascript are all languages that effectively allow very sophisticated tools to be configured in very sophisticated ways.
Are they all gross violations?

Further, at least two of these (Lua and Tcl) explicitly started as ways to write, well, Turing-complete configuration files: configury and program in the same language. Javascript didn't, but then it grew JSON, which is, well, exactly the same thing.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 2:40 UTC (Tue) by zlynx (guest, #2285) [Link]

Sort of. But serious users of JSON never stick it into a string and eval(). That way lies nightmares and security holes.

In best practice JSON is non-executable.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 9:07 UTC (Sat) by fandingo (guest, #67019) [Link] (6 responses)

Why do you speak in metaphors and veiled examples in your several replies? What's the metaphor: game engines, search engines, steam engines, locomotives?

Your last comment further up talks about two distinct problems that you recently experienced. Why did you omit the details? To be honest, this sort of thing is used by systemd detractors all the time. The (unintentional) subterfuge masks either their lack of knowledge or fraudulent purpose. I'm not saying that you're arguing in bad faith, but if you're going to detract from something, it's only reasonable to provide enough details so we know what you're talking about.

Continuing about this comment, I assume that your metaphor is about mechanical engines. Are you really suggesting that more primitive engines are better? Should we all desire steam engines despite the fact that would make air travel impossible? Sure it's more straightforward to understand "dump the water in there and shove the coal into the fire," but to continue the metaphor the drawbacks are preposterous. Again I hope I'm getting the metaphor right, but it seems like the piston-driven airplane and jet engines are perfect corollaries to sysV and systemd. The engineering level of a piston engine is quite a bit lower and has far lower tolerances. The jet engine, on the other hand, is both far, far more powerful and efficient, and it is also substantially more reliable and requires practically no maintenance.

But anyways, init systems are not engines. It's pretty useless to talk about technical things in metaphors. We can shoehorn in whatever exemplary or derogatory metaphor we want without actually talking about anything.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 12:02 UTC (Sat) by viro (subscriber, #7872) [Link] (1 responses)

"The jet engine, on the other hand, is both far, far more powerful and efficient, and it is also substantially more reliable and requires practically no maintenance."

Off-topic, but... Jet engines getting "practically no maintenance" are the stuff of which the nightmares are made... Ever seen the effects of fatigue crack in the wrong place that has grown past the critical length due to inadequate inspections? Gets _really_ unpleasant when e.g. the shaft fails, separating compressor from the turbine and leaving the latter with no load...

There's a whole lot of failure modes in those puppies and airlines trying to save money by cutting down on maintenance ended up with serious trouble after a while. Including a lot of dead bodies...

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 15, 2014 12:56 UTC (Sat) by pizza (subscriber, #46) [Link]

> Off-topic, but... Jet engines getting "practically no maintenance" are the stuff of which the nightmares are made...

There are far fewer failure modes in Jet engines -- due to many fewer moving parts and much much much much lower vibration, thanks to the lack of reciprocating mass. Consequently, they require considerably less ongoing maintenance than piston engines.

(But only a fool or airline execs would consider them maintenance-free)

OT, yes...

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 17, 2014 0:57 UTC (Mon) by neilbrown (subscriber, #359) [Link] (1 responses)

> Why do you speak in metaphors

I didn't think I was. I thought "engine" had a fairly well defined meaning in the software world - it is the bit on the inside that does all the work. It might be a game engine or search engine and you suggest, or a physics engine or an inference engine or a database engine or... whatever is required.

You can often think of a software system as having an engine and an interface.

No, I don't think a more primitive engine is better. I think having core functionality that meets the need is very important and systemd has a lot of that - which is good. I want a sophisticated engine, but I don't want a complicated engine.

I think 'git' provides a valuable contrast. It is in some ways a new and different thing, much like systemd. It uses some well established ideas and combines and presents them in a way that adds a lot of value. And there are people who seem to hate git too - though not as much as systemd it seems.

A key part of the design of git is that core functionality (the so-called plumbing) is directly available if you want to use it. On top of this more friendly interfaces are provided. Originally this was just shell scripts combining the core parts, though I think that over time more and more is being converted to C. This is sophisticated but not too complicated. Core functionality is well defined and directly accessible. This depends on having a rich language for combining the components: shell.

systemd has a lot of valuable core functionality too. But it is not always exposed in a way that provides direct access. The configuration language is extremely simplistic. Quite a few of the directives provided are composites of core functionality - so they set an agenda ("set combinations apply") rather than provide raw functionality.

To be more specific, there are mechanisms for interactions between different units. In upstart (if I remember correctly), such interactions involve "signals" and I think there is full low-level control of signals.
systemd designers deemed that signals were not sufficient (Which is probably correct) and provide something else. What exactly? It seems to be a combination of signals and requirements.

"Requisite" is a requirement with no signal
"Wants" is a signal with no requirement
"Requires" is a combination of the two.

There is more subtlety to the interactions - there is a "reload" signal and maybe a "restart" signal, though that might have requirement implications. There are inverse requirements (conflicts).

But the configuration language doesn't provide access to the core features - I'm not even entirely sure what all the core features are precisely. It just provides a number of directives that pre-combine these features in ways that seem to be useful.

So I do want a sophisticated engine - I want socket activation and process group monitoring and timeouts and auto-mounts and device triggers etc etc. I think all that is great stuff.
But I want direct access to all of it. I want to be in control (yes - I'll admit to being a control freak). I want to be able to access the different functions in ways that the designers never envisaged.

Things become "complicated" when you don't have direct access and so need to work around limitations and do things in no-obvious ways just to get access to the bits you want.

(I'll reply about the examples separately).

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 17, 2014 18:58 UTC (Mon) by cebewee (guest, #94775) [Link]

A nice property of a restricted language is that it is usually much more accessible to static analysis (and turing-complete languages aren't). This might be one of the reasons why the systemd developers lean on this side of exposing functionality.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 17, 2014 1:33 UTC (Mon) by neilbrown (subscriber, #359) [Link] (1 responses)

> Why do you speak in ... veiled examples in your several replies?

Sometimes too much detail can distract people from thinking about the point you are trying to make. So including it at the wrong time might not help. But I'm happy to share.

These are two issues that I recently hit. They are *not* examples which try to say "systemd is horrible and doomed can never work", they are meant as examples that say "systemd does have some real complexity, does have steep regions on the learning curve, isn't entirely transparent". It could be that they are problems for me only because I haven't learned enough yet - which just points to the learning-curve issue.

Firstly, I really like that using systemd makes it a lot easier for upstream packages to provide the start-up config file: providing init scripts in upstream was always problematic.
But if a distro has a pre-existing practice of using a particular name for a service, and upstream chooses a different name, then there is an issue to be resolved. I would like to support both names, at least during a transition period. That means we need an effective alias, so "foo-daemon" and "food" can be used interchangeably.

systemd *has* alias functionality using syslinks. If you create "food.service" as a link to "foo-daemon.service" then you can "start" and "stop" and "reload" and request "status" using either name. Which is great.
But you cannot "enable" food.service. Is this by design or is it a bug? From what I know of systemd I cannot really decide and the documentation only seems to mention aliases in asides - I cannot find a section "how aliases work".

So I created 'food.service' as a separate service with (hopefully) enough "Requires" and "PartOf" directives pointing at "foo-daemon.service" that systemd will make sure they are either both running or neither are.

Secondly, I had some fun with dracut (not that dracut is much fun ... I find it particularly difficult even remembering what it is called!! Whose idea was it to name key system software after towns in New England? And will they please stop it!?!)

Dracut uses systemd (which seems appropriate) and has a bunch of shell scripts which are run by a queue manager (I can't decide if there is something ironic about that.... can systemd not manage the queue? should it?).

Anyway in a particular circumstance an md/raid array will *not* be assembled when all the devices have appeared, and this is correct. There needs to be a timeout to give up waiting for more devices - then the array is assembled. If that array contains an LVM volume and the LVM volume contains the root filesystem, then dracut needs to:
1/ wait the timeout
2/ assemble the md array
3/ activate the LVM volume

all of which it does correctly. During this time, systemd is waiting for the root device to appear.
How long does it wait? According to "systemd.unit", "JobTimeoutSec" defaults to zero - except for devices units. It doesn't tell me what the default is for device units :-( It turns out to be 90 seconds - DefaultTimeoutStartSec in /etc/systemd/system.conf. But the timeout in step '1' above in 120 seconds. How can this ever work?
I'm not entirely sure, but it does is most cases. In the above case, systemd times out between steps 2 and 3. So if the md array contained the root device directly it would work. If not, it doesn't because systemd doesn't wait the extra second it takes to activate the volume.

If I increase DefaultTimeoutStartSec to 120 .. it still doesn't work. 130? No. 300? Yes. Now it works.

I'm sure there is a good explanation for this. It might even make sense. But "simple" "transparent" "gentle learning curve" aren't the words I'm focussing on just now.

Maybe I need to enable some tracing somewhere.... old habits of adding "echo" or "printk" as just going to have to change...

The Grumpy Editor's guide to surviving the systemd debate

Posted Mar 13, 2015 0:37 UTC (Fri) by neilbrown (subscriber, #359) [Link]

For the record, the problem with dracut was that it has a systemd 'generator' which only creates the important files the first time it is run: rootfs-generator.sh

When you "systemctl daemon-reload", systemd will remove everything from /run/systemd/generator and then re-run all the generators. As rootfs-generator.sh only created files in that directory the first time it is run, a subsequent 'systemctl daemon-reload' will revert the effect of that generator.

The key effect of the generator is to disable systemd timeouts on the device which holds the root filesystem - as dracut has its own timeout mechanism. So a 'daemon-reload' will re-instate the default timeout.

When dracut does timeout waiting for the md array to assemble, and forces the assembly of all degraded md array, it also runs "daemon-reload" (as part of giving up on resume-from-disk). This will cause systemd to give up on the root device immediately, typically a second or so before it is actually ready.

Does that make sense? Anyway, the primary problem was not a problem with systemd ... except...

The real goal of dracut here is to generally disable the systemd timeouts which wait for devices. Lots of systemd is configurable at runtime using systemctl, but this thing isn't. It can only be configured in /etc/systemd/system.conf before systemd starts. 'daemon-reload' doesn't reload system.conf.

So dracut has this complex (and buggy) code to over-ride timeouts that systemd doesn't let it configure in a sensible fashion. So maybe systemd is partly to blame, after all...