LWN: Comments on "Debian reconsiders init-system diversity"

Debian reconsiders init-system diversity

Cyberax — Fri, 29 Nov 2019 06:12:38 +0000

I remember that it used Unix process groups to track the double-forked daemons (cgroups-lite, lol). But it did not work reliably if the daemon itself tried to use process groups. This was not uncommon, and usually utilized for the same purpose - to track its own children.

Debian reconsiders init-system diversity

flussence — Fri, 29 Nov 2019 04:04:20 +0000

That kind of got overshadowed by the systemd thing going on at the time. It was called BIND10.

Debian reconsiders init-system diversity

flussence — Fri, 29 Nov 2019 03:55:39 +0000

djb's daemontools had a workaround for that. I've never used it though, as I've never encountered anything that misbehaved in that way. The only problem I've had was with nfsd, and that's not a real daemon anyway.

Debian reconsiders init-system diversity

jrigg — Thu, 28 Nov 2019 19:11:02 +0000

Out of of more than 170 comments in this thread I couldn't find any discussing the subject of Debian policy towards alternative init systems. I may have missed some, but that's disappointing.

Debian reconsiders init-system diversity

jezuch — Thu, 28 Nov 2019 11:13:53 +0000

The same reason people cling to sysvinit, I guess? But we haven't yet had the obligatory "you'll pry bind from my cold dead hands" flamewar yet?

Debian reconsiders init-system diversity

smurf — Thu, 28 Nov 2019 08:09:51 +0000

I think most (if not all) of them had a foreground mode before systemd, for logging to the stderr / debugging etc..

Happily the once-common failure mode of

* daemonize, which involves …
* redirecting stdout+stderr to /dev/null
* the daemon or one of its libraries runs into a problem
* part/all of the problem/crash report is written to, you guessed it, standard error

is a thing of the past. Don't ask me how often I found that kind of thing with strace.

Debian reconsiders init-system diversity

Cyberax — Thu, 28 Nov 2019 07:18:46 +0000

Yep. Pretty much.

runit can't handle classic double-forking daemons. Fortunately, because of systemd most daemons these days have a foreground mode.

Debian reconsiders init-system diversity

flussence — Thu, 28 Nov 2019 02:39:28 +0000

I guess all my systems just vanished in a puff of logic then.

Debian reconsiders init-system diversity

farnz — Wed, 27 Nov 2019 17:33:48 +0000

That's not what that document says, though - it says that SysV init supports two different ways to manage initscripts, and they are only documenting one of the two.

Debian reconsiders init-system diversity

jwilk — Wed, 27 Nov 2019 17:09:03 +0000

Debian has supported multiple (sysvinit-compatible) init systems since at least 1999:

https://sources.debian.org/src/debian-policy/3.1.1.1/poli...

Debian reconsiders init-system diversity

anselm — Wed, 27 Nov 2019 17:01:09 +0000

BIND 9 actually borders on the usable. In addition, while there are loads of purported “BIND replacements” they often implement only that part of the functionality of a DNS server that their authors like or agree with. If one desires a DNS server which actually does everything that the RFCs say a DNS server should do, the number of feasible substitutes for BIND is suddenly not all that large anymore.

Which is not to say that there aren't BIND replacements that are useful in many contexts (I'm a happy user of PowerDNS and dnsmasq, for example) – it's just that the “BIND replacements” tend to come with their own sets of issues. Hence, caveat emptor.

Debian reconsiders init-system diversity

Cyberax — Tue, 26 Nov 2019 21:14:01 +0000

Runit/daemontools can't work with arbitrary daemons, though. They require software to be written specifically for them.

Debian reconsiders init-system diversity

flussence — Tue, 26 Nov 2019 20:42:20 +0000

>And we dropped from ~10k lines of init scripting to <500 lines of helper setup scripts. It was a *huge* jump in maintainability and I'll never go back.

If that's your metric for success, you should have used runit (which predates systemd by a decade).
Here's how much code it takes to boot my desktop and server, with parallel startup and service dependency ordering:

~ $ find /etc/service/ -follow -name run -exec wc -l '{}' + 2>/dev/null | tail -1
174 total
~ $ ssh $server !!
327 total

Debian reconsiders init-system diversity

mathstuf — Tue, 26 Nov 2019 14:09:16 +0000

How much of that is because:

- libesystemd is *not* ABI/API compatible?
- dpkg/apt-get is mis-behaving because it can't replace implementations of libraries it links to?

How are either of these problems libsystemd's fault rather than limitations of the supposed replacement and dpkg/apt-get bugs?

Debian reconsiders init-system diversity

farnz — Tue, 26 Nov 2019 13:55:23 +0000

Yes. After all, we're going from one monopoly (SysV style init), to another (systemd style init), with a small phase in the middle where distros used multiple new inits (upstart, openrc, systemd) in SysV style compatibility mode instead of their native modes.

It's not like SysV style init was something with multiple compatible implementations for most of its life, after all - why is the SysV monopolization good, while the newer styles are bad?

systemd is an init-system? i wish.

pizza — Tue, 26 Nov 2019 13:01:59 +0000

> I've worked on embedded projects where we ended up installing all sorts of crap because nobody noticed it was being installed alongside something we actually needed.

Embedded developers too lazy (or to be blunt, incompetent) to handle tasks that basic cannot be trusted to exercise the considerable diligence to write classic-style init scripts of the quality/reliability needed for embedded work.

> Also, did you look at the other comments indicating how difficult it is to replace that shared library? Not-systemd needs to replace the library with one that talks to not-systemd so that applications can use it. But presumably, all of systemd's components also use it as a utility library. So they won't run on not-systemd unless not-systemd also implements all of the utilities.

So... you're saying that the libsytemd authors, by following well-established best practices with respect to code reuse, should have instead re-implemented helper/utility functions into each and every component, and required all third-party users to do the same?

(and for the record, systemd-as-pid1 is emphatically *not* a requirement for the use of libsystemd functions)

systemd is an init-system? i wish.

immibis — Tue, 26 Nov 2019 12:29:47 +0000

> Except for logging (which has special considerations) "so tightly integrated" is actually nothing more than "resides in the same source tree and relies on a shared library."

This is actually a problem, because it makes it harder to not install one of them than it should be. I've worked on embedded projects where we ended up installing all sorts of crap because nobody noticed it was being installed alongside something we actually needed. Most likely with systemd we would've ended up building and installing everything, then deleting the unit files for the bits we didn't want, and leaving the binaries where they are.

When you want a package you:
* Download the package
* Build the package
* Install the package

and you shouldn't have to
* Review the package didn't automatically install a bunch of other packages that you now have to remove
* Delve into the package's build system to figure out how to disable those packages because it's not well documented
* Give up figuring out how to make it not build those packages, and just delete them after installing

When it's a distribution, this effect is amplified because if the package maintainer doesn't do it, then none of the distribution's users can do it even if they want to. Basically, when you make something the default, you ensure that 98% of people are stuck with it. "Defaults are arguably the most important design decisions you'll ever make as a software developer": https://blog.codinghorror.com/the-power-of-defaults/

This is great for increasing your market share. You can truthfully say "Well 98% of our users use systemd-resolved" without considering that many of those users don't even know they're running systemd-resolved because the build script just assumed they would and installed it anyway. And then you can say "Since 98% of our users use systemd-resolved anyway, we won't run the unit tests with it disabled." or "Since 98% of our users use systemd-resolved anyway, we'll refactor the build system with the side effect of making it mandatory in the next version, and those who complain are just whiny boomers."

Also, did you look at the other comments indicating how difficult it is to replace that shared library? Not-systemd needs to replace the library with one that talks to not-systemd so that applications can use it. But presumably, all of systemd's components also use it as a utility library. So they won't run on not-systemd unless not-systemd also implements all of the utilities.

Debian reconsiders init-system diversity

immibis — Tue, 26 Nov 2019 12:11:51 +0000

The difficulty of replacing libsystemd, mentioned in other comments, suggests that it is not modular enough.

Debian reconsiders init-system diversity

immibis — Tue, 26 Nov 2019 12:09:35 +0000

I guess s/he is referring to the core part written in C, not the collection of scripts.

Debian reconsiders init-system diversity

immibis — Tue, 26 Nov 2019 12:08:28 +0000

You're both right.

Debian reconsiders init-system diversity

immibis — Tue, 26 Nov 2019 12:06:31 +0000

Those are indeed all very nice features. Are they worth the risk of monopolization?

Debian reconsiders init-system diversity

immibis — Tue, 26 Nov 2019 12:02:40 +0000

Code quality problems at lower layers are more serious. The shell scripts in /etc/init.d can, at least in principle, all be thrown away and rewritten. And if one crashes, it only affects that service. By contrast, the core of the system (any system) is relatively unchangeable and bugs have a greater impact.

FUSE drivers can afford to be far less stable than in-kernel filesystem drivers - now imagine importing that instability into the kernel without cleaning it up.

Debian reconsiders init-system diversity

flussence — Sun, 24 Nov 2019 20:59:33 +0000

BIND is very... 1980s software. It's ugly, hard to use right, prone to falling over in a slight breeze, and has better replacements, yet everyone clings to it for some reason.

Debian reconsiders init-system diversity

Wol — Fri, 22 Nov 2019 10:59:24 +0000

> "GNOME developers, plus people who accept the default configuration of whatever desktop Linux distribution they installed".

Which is why my make.conf has "-gtk -gnome" in it ... (gentoo make options, for those who don't know). In other words, "disable them if you can".

Cheers,
Wol

Debian reconsiders init-system diversity

Wol — Fri, 22 Nov 2019 10:55:41 +0000

Or the horror that was bind, which was pointed out in the early days of systemd ...

It required the network service to be running when it shut down, else it would get stuck in an infinite loop. If your system happened to shut down networking before bind ... if it was in a remote co-lo you were SOL.

Oh and I think this took *years*, *if ever* to get fixed in the official repo.

Cheers,
Wol

Debian reconsiders init-system diversity

mathstuf — Thu, 21 Nov 2019 13:07:58 +0000

> The underlying problem is that something is not shutting down cleanly, and the system has been explicitly configured to wait indefinitely for it.

I agree. It sounds like some dependency specification problem. I see it as akin to "parallel build is busted, but the serial build works". Well, this means your rules aren't listing their dependencies properly. The right fix is to find that missing link and fix it, not say "well, then just use `-j1` and be happy". Granted, systemd is at times tight lipped about some of these things, but it takes some analysis of the logs of services and the systemd service event logs to figure out.

Debian reconsiders init-system diversity

mathstuf — Thu, 21 Nov 2019 13:04:46 +0000

> I don't have much more than "it's broken (sometimes)", which is about the most useless bug report possible....a complete waste of time for everyone concerned.

At some point, this is still useful to file. Maybe the developers have seen it before and know where to point you for the fix and/or discussion of a fix. Sitting in your corner with your puzzle you won't show anyone complaining that it's "impossible" doesn't do much to convince anyone that it's actually an "impossible" problem. There's no shame in saying "well crap, I'm out of ideas, anyone else have a thought about this problem?".

Debian reconsiders init-system diversity

pizza — Wed, 20 Nov 2019 00:38:19 +0000

> Getting the procedures right is really hard, but they have to be repeated in every script that does the same thing. The net result is that there are a lot of really bad init scripts out there. They are either copied and pasted, with all the drawbacks that entails, or they are rewritten from scratch, with the inevitable risk of reintroducing bugs.

Exhibits 1-$VeryBigNumber: improper handling of variables, arguments, or data that contains spaces, leading to all sorts of joy.

Debian reconsiders init-system diversity

rgmoore — Tue, 19 Nov 2019 23:40:28 +0000

There was a giant ecosystem of random shell scripts and utilities and conventions and tools that parsed comments to discover ordering and a mess of problems that were solved and resolved by every package or every distribution independently (creating users, creating temporary directories, and so forth).

To put this slightly differently, the biggest problem with SysV init is that it fails to separate policy from mechanism. An init script is inherently about both what is supposed to happen and how to do it. Since the "how" is hard and involves lots of details to get it right, it tends to overwhelm the "what". This leads to two problems:

Init scripts are very difficult for an inexperienced person to understand. The little bit of policy is buried in a great mess of mechanism, and it's generally written as part of the procedure rather than separated out where it's easy to understand. For example, instead of saying "this process should be run as a daemon", it includes a procedure for running it that way. This is completely opaque to someone who understands the goal but doesn't understand the mechanism.
Getting the procedures right is really hard, but they have to be repeated in every script that does the same thing. The net result is that there are a lot of really bad init scripts out there. They are either copied and pasted, with all the drawbacks that entails, or they are rewritten from scratch, with the inevitable risk of reintroducing bugs.

systemd deals with both these problems by separating policy (unit files) from mechanism (systemd itself). Unit files are much easier to understand than init scripts, solving problem 1. And mechanisms only need to be written and debugged once in systemd for every program it's starting to benefit, solving problem 2.

Debian reconsiders init-system diversity

jccleaver — Tue, 19 Nov 2019 16:36:33 +0000

It always surprises me that systemd came out of RedHat developers rather than Debian (or Slackware, or Gentoo, or some other distro ecosystem).

On RedHat/Fedora/RHEL/CentOS/Scientific Linux all of those other mechanisms were deprecated LONG ago, and you were supposed to use the /sbin/service and /sbin/chkconfig commands for a) controlling current state, and b) controlling boot and S/K## ordering. Initscripts were supposed to be immutable (replaced on update) and all config happened in /etc/sysconfig/<scriptname> or /etc/chkconfig.d/<scriptname>, which were treated as config files and remained on update. I'm not sure when these were introduced but it's got to have been at least 20 years now... Certainly there by the Cartman RedHat Linux release.

All these other mechanisms for doing something, other than as a 3am hack, strike me as incredibly dirty. Can't believe other ecosystems hadn't fixed it by now.

Debian reconsiders init-system diversity

tao — Tue, 19 Nov 2019 08:50:57 +0000

The compromise ended up being that any script that requires bash explicitly has to use /bin/bash as their shebang.
There are still a lot of scripts requiring bash, but at least any script using /bin/sh should run equally well (except faster) with /bin/sh pointing to /bin/dash.

Debian reconsiders init-system diversity

rahvin — Mon, 18 Nov 2019 22:56:11 +0000

I prefer these graphs from the Debian Popularity Contest:

Debian - SysV vs. systemd popularity as total number installed

Debian - SysV vs. systemd popularity as a Percentage of Installs

SysV use in Debian at this point is a rounding error.

Debian reconsiders init-system diversity

rahvin — Mon, 18 Nov 2019 22:35:13 +0000

Old? AC has only been out a few years and was the most recent standard until AX (wifi6) came out a few months ago. I'd hardly call that old. I know people still running routers that top out at G.

Debian reconsiders init-system diversity

jezuch — Mon, 18 Nov 2019 17:09:19 +0000

> A system cobbled together is malleable (you can always cobble something different.)

It's also brittle. Undocumented. Not discoverable. It fossilizes quickly and a cargo cult develops around it ;)

Debian reconsiders init-system diversity

jezuch — Mon, 18 Nov 2019 17:05:35 +0000

> You can replace Linux with FreeBSD, or Bash with Dash, for the most part.

Well, it took Debian literally years to make it possible to use dash as the default /bin/sh instead of bash. There was a huge campaign to rid all the scripts of "bashisms" - in fact I don't even know if it has been completed :)

Oh, and the kfreebsd architecture in Debian... Is not faring well, I think.

systemd is an init-system? i wish.

smurf — Mon, 18 Nov 2019 13:54:21 +0000

So who exactly forces you to use any of these "non-init" subsystems?

Feel free to ignore each and every one of them if you want to make your life more difficult than necessary. In the meantime I happen to be ecstatic about a cron replacement that finally doesn't fill my machine with 4718 hung jobs (by design, not because every cron job script author remembered how to correctly use "shlock"), a log system that finally doesn't lose data and can be queried without going insane, an ntp client that hasn't been designed for a machine that does nothing but NTP, and a "monopoly" that frankly doesn't exist.

You want a distribution without systemd? fine. Use one that didn't switch to systemd, or create your own. Then at least you can't blame somebody else for your broken system.

Or you might stop to ask whether there might be actual real-world usecases that require systemd to do logging or cron or networking or DNS – usecases that didn't work with the previous heap of independent services. It's not as if they sat down and decided to usurp /etc/network/interfaces (or whatever) from the blackness of their ugly little hearts, fer crissake!

NB: You might want to enlighten us WRT exactly which usecase you have in mind that's so incompatible with systemd that it requires special "catering".

systemd is an init-system? i wish.

pizza — Mon, 18 Nov 2019 12:32:36 +0000

> And they did just that; resolver, cron, logging, ntpd, etc etc are standalone daemons, not part of PID1.

whoops, I mistyped this; 'cron' functionality is part of the core. And that's a good thing. (Yay, no more long-runing cron tasks stepping on each other's toes, no more buggy dependency/prereq checking, no more inconsistent logging, etc..)

Debian reconsiders init-system diversity

pizza — Mon, 18 Nov 2019 12:10:27 +0000

> It often goes into a seemingly endless wait-loop - I say "seemingly" because I don't have hours or days to wait for the machine to finish shutdown and reboot, and frequently have to just physically power-cycle the machine just so I can reboot it sometime today rather than tomorrow. or next year.

Every time I've seen systemd stuck waiting on something, I've seen it get reported to the console, and start/complete times are logged. I've not personally seen any shutdown task without a hard upper time limit (and that limit is configurable globally) but I suppose one could exist elsewhere, perhaps as part of some distro/daemon/application-provided unit.

I might add that this is probably not a "systemd bug". The underlying problem is that something is not shutting down cleanly, and the system has been explicitly configured to wait indefinitely for it. In other words, it's most likely due to how your distribution (or software supplier, or admin) integrated the troublesome component.

Have you at least reported this bug? Clearly this is not desired behaviour.

Meanwhile, with respect to your ip routing bug. You didn't say how networking was configured; networkmanager? systemd-networkd? distro (which one?) ifup/down scripts? script that manually invokes ifconfig/ip tools? dhcp or static configuration? Are these routes statically/explcitily configured, or automagic with respect to the networking infrastructure? Details matter, and supplying that (here or as part of a bug report) doesn't require setting up VMs or whatnot.

(FWIW, this sounds a lot like what happened when I wrote an incorrect networkmanager hook script to selectively enable a VPN -- It worked, except when it didn't, ending up with multiple VPN instances stepping on each other's toes with ping times that would grow to 4-5s before the whole thing came crashing down and restarted itself..)

systemd is an init-system? i wish.

pizza — Mon, 18 Nov 2019 11:50:51 +0000

> None of these "systemd-borgd" built-ins are useful for machines that **don't** closely resemble the authors' personal desktop machines. They're not capable of being used for anything that doesn't match the devs' personal needs. and, worse, the devs aren't at all interested in catering to anything they don't personally need.

You do realize you're commiting the same "I don't care about those use cases, therefore they're not valid for anyone" hubris that you're accusing the systemd cabal of doing? (Meanwhile, the systemd developers have demonstrated time and again that they care about use cases beyond their personal needs...)

> And if they think they can do a better DNS resolver, a better cron, a better log service, a better ntpd, a better anything-that-isn't-actually-init then they can write a standalone daemon to prove it.

And they did just that; resolver, cron, logging, ntpd, etc etc are standalone daemons, not part of PID1.

> None of this stuff belongs in init. And none of it should be so tightly integrated with init.

Except for logging (which has special considerations) "so tightly integrated" is actually nothing more than "resides in the same source tree and relies on a shared library."

> 1. aren't privileged with special access to init's internals, hard-coded dependencies, or deliberate sabotage of alternatives within systemd
> 2. are easy to disable - without breakage of other stuff running on the system
> 3. are reasonably easy to replace with other alternative/competing implementations.

Except for journald, all three are already the case -- at least with respect to systemd and its authors.

Now distributions downstream may make, as a matter of policy, choices that prevent (2) and (3). Similarly, application/framework developers may decide, as a matter of policy, to rely on systemd-provided functionality.

Personally, I'm having a hard time seeing the decisions of third parties to use systemd features is somehow the fault of systemd's authors.

Debian reconsiders init-system diversity

zdzichu — Mon, 18 Nov 2019 08:40:31 +0000

Well, if you don't report bugs, they won't get fixed. This is the basic building block of opensource communicties. Both Debian and systemd itself have quite extensive pages on how to use various debugging facilities to figure out what's going on. It should be enough to prepare worthwhile bugreport.

But if it still not enough, I'd expect maintainer to provide targeted debugging instructions. If you feel there's no point in submitting bug reports, I'd strongly suggest to change your provider. There are many distributions to choose from.