On technological change

Posted Oct 10, 2014 9:18 UTC (Fri) by Seegras (guest, #20463)
Parent article: On the sickness of our community

I've probably run Unix and Linux longer than most of you.

In the beginning, I hadn't those new-fangled SysV initscripts. Everything went into my rc-script and all was good. But then came SysV, with all these scripts and even worse, that not-really-unix-but-Sun-way of calling scripts that call scripts that source other scripts. Of course there's a reason for it, and that is the package system. You want every package to be able to provide its own startup script. And you don't want every one of those to copy general methods, hence the calling and sourcing of other scripts. In the end I learned to cope with it. The system was clearly better suited for running systems that sport a package-management than the original BSD init.

Then there came some other init systems, either used complementary, such a supervisor, or on different systems, such as upstart on Ubuntu or launchd on MacOS X. Also clearly, they addressed shortcomings of the SysV init, like the unparalleled (sic!) slowness of the startup, or the fact that it was unclear to the system whether a process was really running and so on. I even tried initng and runit. It was clear, SysV needed to be replaced, at least on workstations and especially portable computers of all kind (for servers, it only became clear later (to me), when they also started to get hot-plug hardware and such).

Then came systemd. I mostly ignored it, wasn't really interested in it (just another upstart ;)), until Debian started looking into it. Well, they decided on it. And the point is, I trust Debian very much to make the right decision. For me, they've proven it time and again. From the package management to the DFSG, they just usually do the right thing.

Now I've got three of my four systems running with systemd, and they've each posed its own set of challenges. One needed a serial console. The other had initially no cgroups in the kernel (yes, systemd does not take that lightly, at least all mounts of encrypted filesystems will fail). And another still starts all kinds of garbage at startup, which I want on my system but no usually started. But I've now written my first .service file, and I'm starting to comprehend how systemd works. It has its kinks, but I think it _can_ be better than most of the other systems; it has room to the top, where most of the others already hit the ceiling.

Anyway. The point I actually want to make is that when some die-hard old unix nut can cope with changes from BSD init to systemd, I can only surmise that people grown up with upstart now crying against systemd haven't somehow learned that the world (including the unix or the open source world) constantly changes, and we have to adapt to it. And if you really don't like what your software is doing or where it's going, either get involved (and that means coding and participation, not ranting on forums and mailinglists), or there's boots.

My boots are made for walking.

I've walked away from Windows (3.11, good riddance, and to all the other crap Microsoft made later on), and I've walked away from closed source Unix, DLD, and Slackware, and SuSE and RPM-based distros, and sendmail, and qmail, and NCSA-httpd, and fvwm, and Gnome and hundreds of applications thet were useful at one point in time. I don't hate them (well, except maybe these abominations from Microsoft), and I don't need to go on spewing vitriol against them. Usually they just don't concern me. If you really don't like the "non-unix" way of systemd (why do you even accept SysV then?), why don't you try a different linux distro? Or *BSD?

to post comments

On technological change

Posted Oct 16, 2014 13:58 UTC (Thu) by ksandstr (guest, #60862) [Link] (6 responses)

You missed the part where sysvinit came out with dependency-ordered concurrent booting at about the time when upstart was marketed as superior for that reason. Indeed all systems installed from Debian stable as of today boot in exactly that manner, and it is quite zippy indeed.

The "sysvinit is teh sl0wz" argument hasn't held water since the time when it started being made.

On technological change

Posted Oct 16, 2014 16:09 UTC (Thu) by anselm (subscriber, #2796) [Link] (5 responses)

System-V init can indeed be made quite fast but that doesn't detract from the fact that it sucks rocks through a thin straw in a variety of other respects, many of which are solved by systemd.

systemd's unique benefits (if any)

Posted Oct 16, 2014 17:01 UTC (Thu) by ksandstr (guest, #60862) [Link] (4 responses)

Strangely enough I have not heard these arguments from systemd advocates at all in the past three-ish years. I've heard only the one about sysvinit being slow to boot due to supposed lack of concurrency, usually combined with a journalist's misconceptions about SMP and entirely ignoring block device scheduling.

The article you link doesn't seem to highlight any other issue besides perceived slowness which is addressed therein by read-ahead optimizations and "doing significant damage to X", and delays experienced while loading kernel modules through a script. Could you explain which of these problems is soluble in systemd's model in such a way that cannot be achieved with a dependency-ordered sysvinit script, please? I'd also appreciate a pointer to what direction the experimentation with regard to X ended up taking, as it's been six years since that article was researched and written.

systemd's unique benefits (if any)

Posted Oct 16, 2014 17:27 UTC (Thu) by anselm (subscriber, #2796) [Link] (3 responses)

Could you explain which of these problems is soluble in systemd's model in such a way that cannot be achieved with a dependency-ordered sysvinit script, please?

The important issues with sysvinit that systemd addresses don't really have a lot to do with speed; that systemd allows many systems to boot faster than before is a side benefit.

For example, there is that thing where sysvinit is unable to keep track of whether a service actually keeps running once it has been launched. The init script may return without an error but you still can't be sure that the service was in fact started successfully. Then there is that thing where systemd manages to figure out many if not most dependencies between services by itself, where sysvinit relies completely on error-prone explicit dependencies that a distribution maintainer needs to put in by hand. And so on. These are the issues that systemd is really about, not the speed of the boot process.

systemd's unique benefits (if any)

Posted Oct 16, 2014 18:44 UTC (Thu) by ksandstr (guest, #60862) [Link] (2 responses)

[Thank you for the thoughtful reply. I'll endeavour to respond in kind.]

>The init script may return without an error but you still can't be sure that the service was in fact started successfully.

Not from any indication of the script itself, no. However a quick `pidof` and a check of the PID file dropped by start-stop-daemon (or the service process itself) should reveal whether the process is up or not, which is all that systemd's "hey I'm up" library call does. (As implementation goes, meta-data to provide for such a function could easily be carried in init.d scripts, just as dependency information is.)

That's to say, the guarantee of service up-ness provided by a process' continued presence is a slim one given failure cases besides the ones that cause a process to disappear outright. For example in a network server that spawns child processes to handle client connections there may be a systemic failure that prevents client interactions altogether but not startup of the parent. Failures in this spectrum will not be thoroughly revealed by any mechanism short of per-application querying to the full breadth of its functionality.

So I'm not counting this as unique to systemd's architecture, nor as a (r/)evolutionarily useful feature.

>Then there is that thing where systemd manages to figure out many if not most dependencies between services by itself, where sysvinit relies completely on error-prone explicit dependencies that a distribution maintainer needs to put in by hand.

This only exchanges one category of failure (insufficient maintainer diligence) for another (software-induced lossage). I'd tend to prefer the former since the correctness of the latter should ideally be proven for all possible sets of dependencies whereas the former need only be correct for one initscript at a time.

Furthermore I'm suspicious that systemd's mechanism won't cover all possible dependencies between initscripts, leading to a situation where there's a (potentially crusty, brittle, and/or legacy-saddled) magic algorithm, a set of hand-crafted dependencies, and a policy for combining the two into something that the startup function can actually execute. If nothing else this conflicts with the ideal of parsimony in design; especially since the dependency list could be generated ahead of time (e.g. by the package maintainer) with a tool that's equivalent to systemd's magic, and then stored in place of existing metadata.

Compared to the lossage that systemd's growth of accessory dependencies and their expansion is causing, it's still my opinion that Debian's switch to systemd and packages depending on it is a grand mistake.

systemd's unique benefits (if any)

Posted Oct 16, 2014 19:15 UTC (Thu) by pizza (subscriber, #46) [Link]

> Not from any indication of the script itself, no. However a quick `pidof` and a check of the PID file dropped by start-stop-daemon (or the service process itself) should reveal whether the process is up or not, which is all that systemd's "hey I'm up" library call does. (As implementation goes, meta-data to provide for such a function could easily be carried in init.d scripts, just as dependency information is.)

History has shown that "could easily" doesn't quite work out so well, and pidfiles are not something that can be relied on.

And systemd's "hey, I'm up" library call uses linux's cgroups to definitively state whether or not a process and anything it's forked are still running. It's a completely reliable mechanism that doesn't require knowledge of process names and a gajillion scripts and daemons and whatnot to get things perfectly correct each and every time. (Hate to state the obvious, but they don't)

> This only exchanges one category of failure (insufficient maintainer diligence) for another (software-induced lossage). I'd tend to prefer the former since the correctness of the latter should ideally be proven for all possible sets of dependencies whereas the former need only be correct for one initscript at a time.

Wait, you say you prefer the former, because... the latter is superior?

Experience has also shown that the former tends to crop up far more often than the latter, unless you're counting "insufficient maintainer diligence" to be a "software-induced lossage" too.

Incidentally, the latter is precisely what systemd and its unit files provide: provably correct dependencies, performed in one place rather than needing to be re-implemented in shell by every single script.

> Furthermore I'm suspicious that systemd's mechanism won't cover all possible dependencies between initscripts, leading to a situation where there's a (potentially crusty, brittle, and/or legacy-saddled) magic algorithm, a set of hand-crafted dependencies, and a policy for combining the two into something that the startup function can actually execute.

Any problem that could throw off systemd would also throw off any other system that uses static dependency trees, yes, including sysvinit. Barring that, if there's some real-world thing that systemd's dependency mechanisms can't handle, I don't think I'm going out on a limb by saying that the systemd folks would love to hear about it. (When you consider that you can generate unit files on the fly to create dynamic dependency trees, systemd is *much* more capable of dealing with those crusty, magic, phase-of-the-moon sequencing tasks)

> If nothing else this conflicts with the ideal of parsimony in design; especially since the dependency list could be generated ahead of time (e.g. by the package maintainer) with a tool that's equivalent to systemd's magic, and then stored in place of existing metadata.

That's such a good idea that someone already did it -- it's called a systemd unit file. Seriously, you're arguing for a design that systemd already implements, and quite well at that.

systemd's unique benefits (if any)

Posted Oct 16, 2014 20:16 UTC (Thu) by anselm (subscriber, #2796) [Link]

However a quick `pidof` and a check of the PID file dropped by start-stop-daemon (or the service process itself) should reveal whether the process is up or not, which is all that systemd's "hey I'm up" library call does.

pidof and PID files can have nasty race conditions, and in particular you can't really trust a PID file from start-stop-daemon (this is documented as a last-resort type of approach for services which don't write their own PID files, and even if they do it's a haphazard thing because you can't rely on the PID file to go away when the service does etc. etc.)

On the other hand, the mechanisms systemd uses to find out that a service process has terminated are a lot more robust. Note that the usual method to start a service in the background with sysvinit is for the service process to be launched from an init script, and to double-fork so that it is adopted by PID 1, at which point PID 1 has no idea what the process is or where it came from. With systemd, PID 1 can start a service process directly within its own cgroup, which affords a lot more control. (It also makes the service processes that much more easier to implement since you don't need to go through the double-fork-etc. song and dance that sysvinit requires.)

For example in a network server that spawns child processes to handle client connections there may be a systemic failure that prevents client interactions altogether but not startup of the parent.

Which is too bad in your sysvinit setup – the init script returns success, and then the next init script is started, which if it depends on the other service will also fail etc. With systemd, there is a chance that systemd's activation methods can stand in for a service that doesn't come up (e.g., by queuing up connection requests from other services), at least to a point where the system is in a state that allows debugging.

This only exchanges one category of failure (insufficient maintainer diligence) for another (software-induced lossage). I'd tend to prefer the former since the correctness of the latter should ideally be proven for all possible sets of dependencies whereas the former need only be correct for one initscript at a time.

Whatever floats your boat. I personally would rather depend on a general piece of software that has been implemented once and thoroughly debugged than on distribution maintainers who need to be relied upon to do the Right Thing over and over again. The way sysvinit requires maintainers to come up with yet another (usually subtly different) init script for every new service has nothing to do with “parsimony of design“; I'm all in favour of 10-line declarative unit files for systemd over 200-line procedural init scripts for sysvinit.

This becomes especially true once we need to change one line in the configuration of how a service is started from the distribution default – with systemd this means a two-line file under /etc/systemd for a change that will automatically remain in force across package updates, while with sysvinit this means changing the actual init script and somehow making sure that an update of the underlying package doesn't blow the change to kingdom come.

Compared to the lossage that systemd's growth of accessory dependencies and their expansion is causing, it's still my opinion that Debian's switch to systemd and packages depending on it is a grand mistake.

You're perfectly entitled to your opinion. We'll see where we stand five years from now.