Init system support in Debian
The "systemd question" has roiled Debian multiple times over the years, but things had mostly been quiet on that front of late. The Devuan distribution is a Debian derivative that has removed systemd; many of the vocal anti-systemd Debian developers have switched, which helps reduce the friction on the Debian mailing lists. But that seems to have led to support for init system alternatives (and System V init in particular) to bitrot in Debian. There are signs that a bit of reconciliation between Debian and Devuan will help fix that problem.
The Devuan split was acrimonious, much like the systemd "debate" that preceded it. Many bits were expended in describing the new distribution as a waste of time (or worse), while the loudest Devuan proponents declared that systemd would cause the end of Debian and Linux as a whole. Over time, that acrimony has mostly been reduced to random potshots (on both sides); there is clearly no love lost between the pro and anti sides (whether those apply to systemd, Devuan, or both). Some recent developments have shown that perhaps a bit of thawing in relations is underway—that can only be a good thing for both sides and the community as a whole.
Holger Levsen alerted the debian-devel mailing list that the Debian "Buster" (i.e. Debian 10) release was in danger of shipping with only partial support for running without systemd. The problem is that two packages needed for running with System V init (sysvinit-core and systemd-shim) are not really being maintained. The shim is completely unmaintained and sysvinit-core has been languishing even though it has two maintainers listed.
While Thorsten Glaser at first downplayed
the problems, he changed
his tune somewhat after Ben Hutchings pointed
out that the sysvinit package has "many open bugs with patches that have not
been applied or answered
". The problem goes even deeper than that,
according
to Andreas Henriksson:
And, as Petter Reinholdtsen explained,
the sysvinit "team" is really just him at this point. He is "lacking
both the required spare time and interest to do [a] good
job, but still try to fix the gravest problems while waiting for someone
with time and interest to [adopt] the packages
". He too is
concerned that the packages will be removed before long. So Jonathan
Dowland suggested
looking elsewhere for help:
That set off a fairly predictable round of Devuan bashing, along with concerns that inviting Devuan developers to work in Debian would bring on a return of the mailing list battles of old. But it also resulted in a reply from Enzo Nicosia (also known as "KatolaZ"), who is one of the Devuan "Caretakers" team:
In the last four years there has been hatred from both camps (Debian vs Devuan), and there is no doubt that most of that could/should have been avoided on both parts. Grepping through email archives and [resurrecting] posts from 3 or 4 years ago won't help to move on, though.
I am not interested in chit-chat or flames, because those don't get packages released. The only reason I am here is that sysvinit is effectively getting kicked off Debian, and I think I can help avoiding that.
Several Debian developers were ready to let bygones be bygones and welcomed any effort toward keeping sysvinit alive in Debian. Ian Jackson invited Nicosia to a new debian-init-diversity mailing list. That mailing list has been expressly set up to avoid the hostility on (some) Debian mailing lists, Jackson said. It would appear to be the place where non-systemd init systems will be discussed and developed moving forward.
Back in debian-devel, there were other sysvinit proponents (such as Benda Xu) who did not see any real problems with the sysvinit package. But Reinholdtsen was quick to point out that attitude as helping to push sysvinit out of Debian. Beyond that, as Martin Pitt noted, the problems are far more widespread than simply being confined to the sysvinit package:
SysV init leaves all the really hard problems to these, as it cannot really do much by itself. That's a fact that people that keep yelling "but SysV init was so easy!" keep finessing..
So "how many RC bugs does sysvinit have" is a completely useless metric IMHO.
Bernd Zeimetz brought
up a related issue: "the typical package maintainer won't test
initscripts
". Many of them won't even have access to a system that
runs sysvinit, he said, so they can't test them well. In another part of
the long thread, though, Philipp Kern asked:
"Could someone reiterate about what the current state of init
diversity
is supposed to be?
" Russ Allbery had
some thoughts on that:
It is quite easy to write a sysvinit init script for most daemons that will basically work. I don't think the maintainer is obligated to do much more than that (for instance, I don't think you need to try to duplicate systemd hardening configuration or other features that are quite challenging to do under sysvinit without more tool support, although some of that may be coming in start-stop-daemon).
He suggested that maintainers could test the init scripts by moving the
systemd unit file aside and have systemd use the init script. "For
nearly all daemons
that don't involve tight system integration, this will be more than
adequate.
" In another
message, he explained that providing an init script is, at least partly,
a gesture of goodwill within the Debian community:
Overall, the dialog was relatively positive and the outcome may well lead to better maintenance of sysvinit for both Debian and Devuan moving forward. Given a little more time (and water under the bridge), Devuan's users could provide the testbed for the init scripts, which will obviously help Devuan, but will also be a boon for any Debian users of sysvinit. There is still likely to be something of chasm between the two distributions, but any rapprochement should be welcome news to most. In truth, what separates the two is pretty trivial in the grand scheme of things—the flames and acrimony notwithstanding.
Posted Oct 31, 2018 21:58 UTC (Wed)
by fartman (guest, #128226)
[Link] (84 responses)
Stop pushing sysvinit! Instead invest in communities that are trying to bring benefits systemd has in a way that does not require daemons to couple with them. There are alternatives to sysvinit that are well maintained, have an author upstream who works on bug fixes, and responds to community issues. I have been using the said project for a fair amount of time now and everything works. Everytime the sysvinit vs systemd debate is brought up, I die a little inside.
sysvinit is not comparable to systemd, init scripts are not comparable to the systemd-suite as a whole. You need proper supervision, things like runit and s6 provide it.
I use s6[1] therefore I would request the Debian/Devuan people to come forth, and more prominently, evaluate it against sysvinit, ditch sysvinit, and invest efforts in something considerably modern to today's standards. The s6 suite has a lot of the things you have in systemd (except the complex transactional dependency engine, which systemd and SMF are the only known implementations of), be it startup notifications, file descriptor holding, and so on. It can be used with an RC, such as *OpenRC* (which might be considered a choice for inclusion along with s6), but it comes with an RC of its own. The author works hard on making things work, but needs some help, it's been a one man show, but I think they're going in the right direction and not beating the old dead horse over and over. I won't make this too long and request you to look into it
Also, as far as systemd-shim is concerned, please, there is a decoupled version of logind called elogind already available and integrated in non-systemd distros (namely Void Linux and Gentoo Linux). As far as timesyncd, timedated, and hostnamed are concerned, they can already be built from systemd sources and made to work with minimal patching (and now consider this, s6 comes with a *compat binary for daemons that use the systemd notification protocol*, so you don't even need to patch out that part, win!).
This is a massive change, but you cannot cling on to sysvinit for the rest of the future. I would like to see some improvement, in that, things like s6, nosh, runit be considered, and even shipped by default. They're easily replaceable hence if anyone wants to go back to sysvinit, they can!
Summarising, I wish to see:
* Debian and Devuan to come together and let go sysvinit. Instead, choose a modern alternative like s6, runit, or nosh (In my opinion, preferably s6, as it will be the least painful of all three options, and does things better than the other two, supports more things when the entire suite is used together (again, nothing depending on each other at runtime, but just sharing a common library), but that is something debatable and I'd be happy if it is debated).
* Make the selected one the default, and ship with it (considering there are enough reasons I listed above that would help its adoption as the default).
* systemd-shim begone and replaced with elogind+timedated+hostnamed from upstream, each of which are maintained properly.
* Integrating sd_notify daemons with the s6 suite has a less painful path even when the said daemon uses its notification protocol (as s6 comes with compatibility code for it [3]). It has provision to perform file descriptor stashing for daemons without putting them in PID1, like systemd does.[2]
* The s6 suite also has "socket activation"[4] related tools (a marketing term for the UCSPI toolset, well known among daemontools users), and is a complete alternative solution to much of the problems systemd tries to solve, but in an arguably better way.
* There's a small s6-sudod[6] daemon to allow you to run things in a pristine context like systemd-run would (since PID1 forks off the service), moreover, a small accessrules library for access control over its socket, an idempotent database those rules are compiled to and loaded, an alternative to policykit.
* You can easily configure s6 to run as a user session manager, an alternative to systemd --user. elogind already supports the user bus, with some glue, it should be possible to support proper user session (or with enough will :) ).
* Small codebase, less things to do, less bugs, less maintainence.
* Sandboxing is modifying the execution context of the service before executing into it, something easily done with execline, it's accompanying tools, and if need be, Linux specific tools like unshare/nsenter from util-linux/busybox. This is how it happens in systemd as well, a child process is forked off from PID1 which takes care of setting up the execution context of the service before executing into it. Moving into cgroups and applying resource limits is just writing a few values to /sys/fs/cgroup and related files.
* s6 bundles are concise to write and maintain, come with a small shell of their own to minimise boilerplate, avoid forking, save resources[5] and this burden can be shifted to a team that maintains them for all packages (because asking people to learn how to write s6 bundles will unnecessarily over power all other benefits and lead to it being ruled out, I'm afraid).
Note that I am not enforcing any of these, I am in favor on discussion happening on what to choose, but please, PLEASE, don't make the mistake of choosing sysvinit again (or more precisely, initscripts, s6 works just fine with sysvinit - the init). There are better alternatives, they need some polish and integration and they will end up working as well as systemd does, giving you most - if not all features (and flexible enough to implement the missing yourself).
Here are some links:
Posted Oct 31, 2018 22:02 UTC (Wed)
by fartman (guest, #128226)
[Link]
Posted Oct 31, 2018 22:08 UTC (Wed)
by rweikusat2 (subscriber, #117920)
[Link] (39 responses)
[*] I'm one of the people Mr Bradley Kuhn considers to be "morally reprehensible" in that I work as developer based on a "standard" software industry employment contract --- everything I write which could conceivably be useful for anything automatically becomes "proprietary IP" of my employer. Hence, this is (unfortunatety) all "secret code".
Posted Nov 3, 2018 14:03 UTC (Sat)
by drag (guest, #31333)
[Link] (38 responses)
People have been solving these problems for decades now. Over and over again.
And eventually somebody came along, solved them, wrote it all in C, created a formal project and got all of the major distributions to adopt it. They call it Systemd.
Posted Nov 4, 2018 19:56 UTC (Sun)
by rweikusat2 (subscriber, #117920)
[Link] (37 responses)
systemd is one of half a gazillion replacement inits written in $compiled_language which roughly all provide about the same feature set (and make the same errors), there's nothing special about it beyond "it's the RedHat replacement init!". It's arguably an improvement compared to the mess RedHat, SuSE, Debian etc used to ship in /etc/init.d. So is upstart, runit, s6, monit, $younameit etc, they just never became "mainstream" because RedHat didn't adopt any of them.
But the "mess in /etc/init.d" RH, SuSE, etc used to ship wasn't created by some force of nature, it just represents about 20 years of "organic shell script growth" based on lack of sensible support tools for process management and Bourne shell ad hocery. A large part of this codeheap isn't really good for anything and could just be deleted (I still fondly remember a comment in one SuSE init script complaining that it was necessary to work around bugs in another SuSE init script --- why not just fix them?). Another large part can be condensed into a set of support tools written in C which can be reused in other init scripts (or simliar situations).
That's not much different from what systemd and other members of the zoo offer, it's just more flexible in that it retains a fully capable high-level, standardized programming language as backbone engine instead of some non-extensible, declaratory language intepreted by a dedicated interpreter which has some random set of "process management support code" aggregated in one (or some) fairly large program(s).
Posted Nov 4, 2018 22:00 UTC (Sun)
by Cyberax (✭ supporter ✭, #52523)
[Link] (20 responses)
Everything else simply follows. If you can isolate daemons then you don't need PID files. You can place limits on their resource use and reliably redirect logs.
If you are aware of any other pre-systemd init system that does the same, I'm all ears.
Posted Nov 5, 2018 20:07 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (7 responses)
"PID files" are a grotty, traditional UNIX hack to prevent multiple instances of a program from running (a very dubious practice to begin with) and to enable some measure of process control by sending signals to the process whose pid is recorded in a pid file (which may or may not be the process actually supposed to get these signals). A well-behaved program won't 'escape' (or at least can be asked to avoid doing so) from its parent. Hence, the parent can reliably send signals to it and can provide an interface enabling other software to do so. It can also provide "only one running instance" feature reliably and can set resource limits for processes which either aren't privlileged or at least don't actively sabotage management.
[*] So far, I haven't encountered the case of a program which actively resisted management. Hence, I had no reason to create a tool for supporting this.
Posted Nov 5, 2018 20:18 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (6 responses)
Socket activation and fully asynchronous design are also new.
> For the usual case that this is not the case, normal parent-child relationship and ensuring that managed processes don't background themselves are entirely sufficient. The lxc-project has been using them for containers since before systemd existed (Wikipedia puts its beginnings at 2010. In 2010, I was using lxc as base for virtualized IPsec VPN servers for mobile clients).
> Hence, the parent can reliably send signals to it and can provide an interface enabling other software to do so. It can also provide "only one running instance" feature reliably and can set resource limits for processes which either aren't privlileged or at least don't actively sabotage management.
The earliest example of such process is DJB's daemontools. It follows the same ideology as systemd, it has declarative service description and it uses traditional UNIX process groups to track processes. It doesn't have dependencies or socket activation, though. It's also not asynchronous so a badly-written script can lock it up.
Upstart attempted to do better but its design honestly sucks. It also uses hacks like ptrace() to detect double-forking daemons.
Posted Nov 6, 2018 20:42 UTC (Tue)
by rweikusat2 (subscriber, #117920)
[Link] (3 responses)
Socket activation with the intent to reduce system boot time and provide better overall system performance saw the light of the day with the 4.3BSD inetd.
Fully asynchronous, that is, event-driven servers have also existed for a long time. Dan Kegel's original C10K paper mentioned the concept. The (older) racoon IKEv1 daemon also works in this way. This is again a design choice: Manage more than one process with the same process management process and accomodate buggy or intentionally hostile "other software". One could argue that buggy other software should rather be fixed and that one shouldn't be using intentionally hostile one.
The sysvinit program actually supports (some very limited form) of process management: It's possible to define per-runlevel commands which will be started automatically whenever a runlevel is entered and restarted if they ever terminate. Traditionally, this is unused except for gettys (to support user logins on all kinds of text terminals, virtual, physical, serial console etc). But the initscripts/ runlevel management system is completely independent of that: All init knows about this is that it's supposed to execute the command /etc/init.d/rc with the runlevel number as argument (both configurable). This means it's possible to use the sysinit program (or any other kind of "init init" which restricts itself to this role) together with any other process management program/ tool. Eg, the one I'm using is just a command which is executed during service start. Basically, it will execute another command with certain arguments in a subprocess. Should this other command terminate, it will be respawned. Further, the command creates an AF_UNIX stream socket for external process control. This can be used to query the status of the managed command, to initiate a stop or restart and to send abitrary signals to it.
If the process serving as init ever terminates, the kernel will panic. But ordinarily, death of a parent process doesn't affect its children. They'll be inherited by the init process which will eventually collect their exit status to prevent them from turning into zombies. Otherwise, backgrounding a program by forking and terminating the parent process wouldn't be possible.
Posted Nov 6, 2018 21:51 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
I'm not aware of any other init system that does this, even to this day. There were scattered bits and pieces here and there, but not a coherent design.
Posted Nov 7, 2018 17:04 UTC (Wed)
by rweikusat2 (subscriber, #117920)
[Link] (1 responses)
But this again starts to drift from a technically interesting topic (technically interesting to me at least) towards marketing statements ("This shoesine is brilliant!") ... :-)
Posted Nov 7, 2018 17:47 UTC (Wed)
by pizza (subscriber, #46)
[Link]
That's like saying that Linux is just a reimplementation of UNIX.
(Along with the implication that there nothing new, different, or otherwise improved between the two)
Posted Nov 12, 2018 2:44 UTC (Mon)
by fest3er (guest, #60379)
[Link] (1 responses)
I'll put forth that sysvinit *is* the parent that manages its children. It simply has never been used that way. Instead, the old method of using RC scripts (did this come fom Berzerkely?) was used (or continued). The very useful part of sysvinit (most of what's in inittab) was relegated to the status of a curious-looking red-headed stepchild useful only for running getty.
If sysvinit was extended to, say, 128 run levels to handle dependencies, actually used as intended, and possibly extended to facilitate enabling/disabling/activating/deactivating daemons without needing to edit inittab, sysvinit would do everything a lot of people need without a lot of complexity.
Posted Nov 12, 2018 9:39 UTC (Mon)
by anselm (subscriber, #2796)
[Link]
Sysvinit can actually deal with more runlevels than the customary six. It's just that people seldom bother to define more of them.
That still wouldn't give you the consistency and convenience that systemd provides. Remember that much of the advantage that systemd confers, from the POV of system management, is in configuration, e.g., the obvious division between system-provided and locally-provided configuration settings which makes it easier to install a new upstream version of a software package without stomping on local customisations. You could obviously come up with a management layer for sysvinit that generates the real /etc/inittab file from a set of snippets in various directories, but the simple fact is that that hasn't yet been done, and now that systemd exists (and can be pared way down if required) the need for it is much less than it may have been earlier.
Posted Nov 5, 2018 21:28 UTC (Mon)
by luto (guest, #39314)
[Link] (11 responses)
Posted Nov 5, 2018 23:02 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (8 responses)
Posted Nov 5, 2018 23:12 UTC (Mon)
by luto (guest, #39314)
[Link] (7 responses)
Posted Nov 5, 2018 23:17 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (6 responses)
Saving kernel size is a valid concern but I'm not sure how much you're going to save if you disable all cgroups controllers.
Posted Nov 5, 2018 23:24 UTC (Mon)
by luto (guest, #39314)
[Link] (5 responses)
Posted Nov 5, 2018 23:28 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (4 responses)
Posted Nov 12, 2018 17:22 UTC (Mon)
by nix (subscriber, #2304)
[Link] (3 responses)
Posted Nov 12, 2018 19:54 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
Posted Nov 13, 2018 0:18 UTC (Tue)
by nix (subscriber, #2304)
[Link] (1 responses)
That's... useful. As in, I think I'll have a use for it in my own work in the next week. :) I wish I'd realised v2 worked this way years ago!
Posted Nov 13, 2018 0:21 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Nov 5, 2018 23:19 UTC (Mon)
by anselm (subscriber, #2796)
[Link]
The subreapers would give you the opportunity to track processes that exit, but you'd still need to use cgroups to do resource control.
Posted Mar 13, 2019 16:30 UTC (Wed)
by nopsled (guest, #129072)
[Link]
It would be nice if Linux had the PROC_REAP_KILL stuff. I think one could improve the FreeBSD interface a tad bit when incorporating it, but the concept looks sound.
systemd ofcourse has one supervisor per context (system or user), but this will be especially useful for supervisors that have one monitor per service (s6, runit style), and that these can be nested supervisors (so an option to not cross subreaper boundaries where each supervisor is a subreaper, or a way to kill inherited children only, and a way to kill direct children only, and OR the flags to do both). You could get creative, but if done right, this could solve a lot of woes with process management.
Posted Nov 5, 2018 13:17 UTC (Mon)
by anselm (subscriber, #2796)
[Link] (15 responses)
It makes more sense to think of systemd as “basic plumbing” rather than “init”. Systemd does include an init process but it tries to address many low-level OS issues that the other replacement inits don't. One advantage of this is that systemd offers a unified approach to configuration that actually extends the same features of, e.g., subprocess creation to wherever subprocesses are created in systemd. With sysvinit or, for that matter, the other replacement inits, this sort of thing is still handled in an ad-hoc manner in a variety of tools that have been cobbled together over decades with no cohesive design.
In theory it would of course be perfectly possible to come up with a set of reusable “support tools written in C” that init systems could avail themselves of, and make these popular across distributions, but nobody except Lennart Poettering and Kay Sievers actually condescended to doing that in practice. Now, from a practical POV, that ship has sailed, since the adoption of systemd by virtually all mainstream distributions has placed the bar for basic-plumbing replacement a lot higher than it used to be when sysvinit+cron+at+inetd+… was the “state of the art”. Anything put forward now must not just provide a significant advantage over sysvinit+…, but over systemd, and with the amount of clever engineering that has gone into systemd over the last decade or so, that is a considerably more difficult task.
Also note that nobody forced Red Hat, SUSE, Debian, … into adopting systemd. Out of the existing init system replacements at the time it just happened to be the one with the most enticing value proposition. (It's also worth reiterating, since people here have been calling systemd “Red Hat's init”, that Lennart Poettering and Kay Sievers started systemd on their own time; nobody from Red Hat went to them and said “write an init replacement for Red Hat Linux”.)
Systemd lets you call out to shell scripts if you need to. But in general, adding the shell to the mix only makes things more complicated for those many if not most cases which are really quite simple and straightforward. The declarative configurations of systemd are easier to read, understand, reason about, standardise upstream, maintain, and teach, than shell scripts and you can still write and use old-fashioned sysvinit-style init scripts with systemd if you absolutely must.
Posted Nov 5, 2018 20:09 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (14 responses)
The set of "open source developers" on this planet is a proper subset of the set of developers.
Posted Nov 5, 2018 21:23 UTC (Mon)
by anselm (subscriber, #2796)
[Link] (13 responses)
You conveniently neglected to quote an important part of my sentence, namely
As you say, init replacements are a dime a dozen, but none of them really achieved a lot of traction (with the possible exception of Canonical's Upstart). It is to Lennart's and Kay's credit that they managed to come up with something so convincing that virtually all major Linux distributions have rallied around it, and the resulting network effects are a major win for the platform.
(In the non-open-source area, practically every provider of proprietary Unix had come up with its own replacement for sysvinit even before systemd got started – Linux used to be the last holdout of sysvinit – but that is neither here nor there.)
Posted Nov 5, 2018 22:12 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (12 responses)
[...]
I've ignored that on purpose because while I'm somewhat interested in discussing the technology, I have zero interest to engage in a discussion of the surrounding politics and other social mechanics as that's bound to degrade into an entirely pointless flamewar of a very familiar shape quickly.
Posted Nov 5, 2018 23:17 UTC (Mon)
by anselm (subscriber, #2796)
[Link] (11 responses)
For the record, I agree that there have been enough flame wars about systemd. But it is important to get one's basic facts right, including that (a) systemd was not started as a Red Hat project, and certainly by now includes active developers from many parts of the community, and (b) other distributions (whose developers, on the whole, aren't gullible incompetents either) adopted systemd because they felt it was technically the Right Thing to do, not because they were somehow (how exactly?) coerced to do so by Red Hat. Remember that Debian in particular did not take the decision to default to systemd at all lightly.
The fact that most major Linux distributions did adopt systemd as their default init system differentiates systemd from all of the other init system replacements (except Upstart), and as such it is reasonable to ask what the other init system replacements “did wrong” if they're as clearly superior to systemd as their proponents claim but yet, unlike systemd, didn't manage to convince multiple mainstream Linux distributions of their merits. It's easy to blame “politics and other social mechanics” but we should not altogether discount the fact that systemd, too, includes a good dash of clever engineering.
In any case, no matter what people think about systemd, it's not going away anytime soon. It is certainly not going away unless somebody comes up with something that the major distributions consider obviously better than systemd, and by now that is a pretty high bar to clear (especially since being “only” an init system replacement is no longer enough; to compete successfully with systemd there needs to be some proposition for the overall basic-plumbing aspect of things). It's still not entirely inconceivable, but somebody will eventually have to start writing code, rather than LWN comments about how terrible systemd is.
Posted Nov 6, 2018 8:41 UTC (Tue)
by ras (subscriber, #33059)
[Link] (8 responses)
As a Debian Developer watching on when this happened, what they "did wrong" had nothing to do with the merits of the init system. ConsoleKit and a few other things were bit rotting badly at the time, and logind, dbus and a few other things got moved into the same source ball as systemd. I remember wondering consolekit and logind did that was so important - it turned out the most important thing is gnome needed something to give the local user (ie the one running the X server) permission to access local resources (like the sound card, and the keyboard), and gnome decided to use what systemd provided. (As I recall gnome didn't have a whole pile of alternatives to choose from.)
Gnome was Debian's default Window manager, so it was game over.
I use lxde on my laptop, and administer a whole pile of servers. Systemd works, but what has to happen is so trivial I don't have a clue what that 10MB of code in systemd could possibly be doing. I'm pretty sure I would of been happier with something much simpler. In fact I needed to boot a Debian chroot on a system without systemd once, and as this article points out the sysv scripts are indeed bit rotting so I wrote a shell script to interpret systemd's .service files for a headless server. It was under 300 lines. It makes the 1/2 a million lines in systemd a bit of a puzzle.
Posted Nov 6, 2018 10:37 UTC (Tue)
by lkundrak (subscriber, #43452)
[Link] (6 responses)
It's more like 1 MB: If you also count other tools from the systemd repository then their file name usually gives a good clue about what the tool does. See: systemd-resolved -- a resolver. systemd-timedated -- manages time and date. systemd-hostnamed -- manages host name. Et cetera. Also, both the init daemon and the supplementary tools are thoroughly documented. If you're really curious I wholeheartedly recommend trying out the man utility. It could turn out useful in your "administering a pile of servers" endeavour. In an unlikely case the documentation won't be sufficient, you should know that the source is available for free and is perhaps one of more reasonable C programs that you'll find running on a typical Linux system.
Hope this helps.
Posted Nov 6, 2018 22:36 UTC (Tue)
by ras (subscriber, #33059)
[Link] (5 responses)
I was referring to the installed size reported by Debian. If you are going to use size, you should at least include the additional 1.5MB in libsystemd-shared-232.so. If you want just count binary size, you should probably be include everything in /lib/systemd, which is over 5MB. That misses a bit of it installed elsewhere of course. The single figure is maybe less accurate, but is a damned sight easier to come by and I doubt it effects the error margin overly.
But if you insist on using systemd + libsystemd-shared's binary size, it's 2.5MB versus 34K for a sysV init I have lying around.
> Hope this helps.
Not really. You''re talking to someone who has an accepted pull request for the systemd documentation. You are, right there is a *lot* of documentation, and it's very good. But don't underestimate the value of only something that does a similar job that only needs a few pages of documentation.
Posted Nov 6, 2018 22:39 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link] (4 responses)
Posted Nov 6, 2018 23:18 UTC (Tue)
by ras (subscriber, #33059)
[Link] (3 responses)
True, but these are usually there anyway, and on the systemd side of the ledger I didn't count libc or systemd's other 20 shared dependencies.
The Debian installed size is so much easier. Looking on an old machine, the installed size of sysV init and ancillary stuff is about 900KB. Adding inetd and cron increases it by 0.4MB. 1/10th the size does seem about right.
This stuff is not that hard. I wrote an "house keeping" system for a container as a dash script. It booted, started and stopped services, provided timers, did scheduled backups, rotated log files, maintained a status web page for monitoring and provided a framework for building containers using it. It was 3K lines and would do a fine job of managing my laptop.
Not all init system's can be simple. An init system for clusters like kubernetes is complex for reasons. But for a single machine with a couple of tasks, it's damned hard to see what added value systemd's complexity brings.
Posted Nov 6, 2018 23:41 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
Posted Nov 6, 2018 23:58 UTC (Tue)
by ras (subscriber, #33059)
[Link] (1 responses)
How did you pull that off? On Debian stretch systemd + libsystemd-shared-232.so alone are 3.3MB.
Hmmm. Looking at the debian archives the systemd v44 .orig.tar.gz ball dates from 6 years ago, and is 860K. systemd v239 .orig.tar.gz is 6.8MB. Have you noticed a 8 fold increase in functionality?
Posted Nov 11, 2018 11:24 UTC (Sun)
by mpr22 (subscriber, #60784)
[Link]
Most trivially, v239 includes udev (whose tarball at the time of the merger in v183 was about 70% the size of the systemd tarball) and the udev HWDB (which was merged later, and is itself pretty huge).
Posted Nov 11, 2018 10:13 UTC (Sun)
by ms_43 (subscriber, #99293)
[Link]
There was an opportunity to contribute the necessary maintenance and bug fixes to ConsoleKit to keep it working, but instead the anti-systemd camp decided to troll the Debian technical committee decision process. I'm very happy that this turned out not to be a successful strategy.
[1] https://github.com/systemd/systemd/compare/v29...v30
Posted Nov 6, 2018 17:44 UTC (Tue)
by rweikusat2 (subscriber, #117920)
[Link] (1 responses)
Posted Nov 7, 2018 1:13 UTC (Wed)
by anselm (subscriber, #2796)
[Link]
That would be to the point if the claim was that systemd is good because it is popular. A more likely proposition, however, is the converse: that systemd is popular because – at least in the eyes of the developers in most mainstream Linux distributions who decide what init system to default to – it is good.
Posted Oct 31, 2018 22:18 UTC (Wed)
by jccleaver (guest, #127418)
[Link] (27 responses)
s6, and any other service manager that doesn't require being executed as PID 1 for its operation, such as DJB's daemontools, or even xinetd/inetd, can function as a completely distinct add-on to any sysv-style init system.
When I say "I want sysvinit", I don't mean "ban all other forms of service management"; I mean "I don't want systemd running the boot sequence (before services have begun), and I sure as hell don't want it as PID 1".
I've run daemontools for 15 years on RH systems, fired off from /etc/init.d/svscan with a simple /sbin/chkconfig svscan on + /sbin/service svscan start. Works great.
I can't speak for the Debian side either, and I'm aware that their init system as a whole (prior to the whole systemd mess everywhere) wasn't quite as clean as the Red Hat initscripts environment, but saying "NO TO SYSVINIT" misses the point about this completely.
All I want is a reliable PID1; a reliable, *deterministic* boot sequence; and a simple environment from which to launch other things in userland.
If you need a service manager, either because your daemon doesn't know how to behave as a proper *nix daemon, or because you need more comprehensive restart and logging capabilities, then install a service manager of your choosing (be it xinetd, s6, daemontools, or -yes- even systemd-not-as-PID1) and run that daemon through that. Otherwise, a traditional init script serves its purpose well and gets out of the way of a system administrator quickly and easily.
Posted Nov 1, 2018 2:06 UTC (Thu)
by milesrout (subscriber, #126894)
[Link] (25 responses)
'I want sysvinit' does not mean 'I don't want systemd'. That's a false dichotomy. There are lots of options that aren't systemd and aren't sysvinit. Comparing systemd to sysvinit is pretty unfair to sysvinit. I can understand why someone would choose systemd if their only other option was sysvinit. I can't understand why someone would choose systemd if their other options include OpenRC.
Posted Nov 1, 2018 4:18 UTC (Thu)
by mchehab (subscriber, #41156)
[Link] (15 responses)
That pretty much depends on the use case.
For my most common use case it is just the opposite: systemd is a very poor replacement for SysV.
I have here several machines here that I use just for testing - most of the time Kernel patches.
When they start, I want two things to happen real quick (no more than a couple of seconds):
- enable network and sshd.
I also need a real quick reboot that won't wait for daemons to shut down, doing just sync and unmount - and, if unmount fails, reboot anyway - as the Kernel might have OOPSed.
Also, I don't have any usage for journald, as I'm using rsyslogd anyway, in order to export logs to some other machine.
Never found a way to do that with systemd.
Starting with logs, I never found a way to disable journald on modern systemd-based distros.
Also, during boot, sometimes systemd spends several minutes to start network on a system - and debugging the culpit that it is slowing down the network to start earlier is really hard. Last time I had to do that, I lost *several hours* and ended by uninstalling lots of stuff from the machine and by hacking several systemd service files.
Finally, systemd only enables shell at console after initializing everything else (changing this behavior is possible, but not trivial).
With sysV, implementing such use case is trivial: just remove links from rc3.d or rename them, in order to change the startup priority.
Posted Nov 1, 2018 4:42 UTC (Thu)
by fartman (guest, #128226)
[Link] (13 responses)
> When they start, I want two things to happen real quick (no more than a couple of seconds):
Define a target, make it pull these things, and make systemd boot to it on the kernel command line with systemd.unit=myspecialtarget.target. man systemd.target, systemd.unit, systemd (for debug shell).
> I also need a real quick reboot that won't wait for daemons to shut down, doing just sync and unmount - and, if unmount fails, reboot anyway - as the Kernel might have OOPSed.
systemctl reboot --force, https://www.freedesktop.org/software/systemd/man/systemct...
> Also, I don't have any usage for journald, as I'm using rsyslogd anyway, in order to export logs to some other machine.
Although I myself just configure it to log locally and pass things on to rsyslog.
> Starting with logs, I never found a way to disable journald on modern systemd-based distros.
You can't, you loose kernel logs, you lose stdout/stderr, you lose audit in the journal, etc. PID1 is tightly coupled with journald.
Better idea for it to maintain a tight buffer in volatile storage and pass things on to rsyslog (this way systemctl status works fine too).
> Also, during boot, sometimes systemd spends several minutes to start network on a system - and debugging the culpit that it is slowing down the network to start earlier is really hard. Last time I had to do that, I lost *several hours* and ended by uninstalling lots of stuff from the machine and by hacking several systemd service files.
Sounds like the respective *-wait-online service is not enabled and not activating network-online.target, and/or you're not depending on that from your unit file. Could be both, could be just one of those. Also, network filesystems automatically gain a dep (or use _netdev as an option in fstab).
> Finally, systemd only enables shell at console after initializing everything else (changing this behavior is possible, but not trivial).
https://www.freedesktop.org/software/systemd/man/kernel-c...
Posted Nov 1, 2018 11:57 UTC (Thu)
by mchehab (subscriber, #41156)
[Link] (12 responses)
I actually did the opposite: I disabled several services at the multi-user target, and had to patch some *.service files in order to reduce timeout and do some cleanups at the dependency chain.
Anyway, adjusting the target is the easiest part. The main problem is how to get rid of the long chain of dependencies that the network service has. See, for example, the time required to initialize network manager on an Intel i7-5557U CPU with M.2 SSD disks (on a machine whose I used lots of tricks to optimize it):
NetworkManager.service +1.241s
With systemd, the minimal time I was able to have network working is around 45-60 seconds. When using a mechanical HD, it takes *a lot* longer time.
The beauty of sysV is that it doesn't have a dependency chain: the scripts just follow the order defined by the admin. Optimizing the chain order is a way simpler.
Thanks for the tip with journald. I may try to play with some options there. If you see the above log, journald slowed network start by ~3.5 seconds.
> Sounds like the respective *-wait-online service is not enabled
I had to play with those. My main goal here is not to wait: it is just the opposite. Ideally, I would really love to have the network started even before mounting the disks, in order to be able to get the logs at the remote machine as soon as possible - and on some machines even use a network console.
Posted Nov 1, 2018 13:42 UTC (Thu)
by pizza (subscriber, #46)
[Link] (11 responses)
Does that include the BIOS/GRUB delays?
At home I have four systems with spinning rust that don't even take anywhere near long to get all the way from to the graphical login prompt. And yes, networking (wifi!) is up at that point. (Meanwhile, two servers each take longer than that to get through the POST...)
Posted Nov 1, 2018 17:09 UTC (Thu)
by flussence (guest, #85566)
[Link] (9 responses)
I just timed power-on to network up with a stopwatch. 13 seconds in the BIOS, 19 more seconds to desktop, 10 seconds more until `ip mon` came to life, so about 42 seconds total. This is on a 9 year old netbook with runit.
Posted Nov 1, 2018 17:45 UTC (Thu)
by mchehab (subscriber, #41156)
[Link] (8 responses)
BIOS/GRUB time a little better... 11 seconds on BIOS, 17 seconds for the first Kernel messages to appear.
However, network comes a lot late here:
1:23 for ping to answer; 1:35 for ssh to log into it.
This machine is a 3 years old NUC (Bios identifies it as NUC5i7RYB) with a 5th gen i7core. It was not supposed to take that 1,5 minutes for ssh do succeed on it. When booting the same machine with Debian and SysV (right now, it is booting Ubuntu/systemd), it takes about 4 times less for ssh to start working.
Posted Nov 1, 2018 18:14 UTC (Thu)
by pizza (subscriber, #46)
[Link]
Be careful, you're not exactly comparing apples to apples here. At the very least, both systems have to be configured to start up the same services before you can make any direct comparison.
Posted Nov 1, 2018 18:47 UTC (Thu)
by seyman (subscriber, #1172)
[Link] (6 responses)
That's certainly a long time. Can you run "systemd-analyse blame"? It might reveal something.
Posted Nov 1, 2018 19:55 UTC (Thu)
by mchehab (subscriber, #41156)
[Link] (5 responses)
26.999s lvm2-monitor.service
The top two things there are not related to network, so shouldn't prevent network to start earlier (it might prevent ssh, though).
The third item (the DNS resolver) took 10 seconds to start, and the only thing to blame would be systemd itself (as it is starting systemd-resolved):
resolvconf.service +10.902s
Anyway, while I'm enjoying this discussion, as it helps me to try optimize booting, we're getting a little OOT here: the point is that systemd seems to be just randomly adding delays to network start for no good reason, and there's no direct way to control it, making network start as early as possible.
In other words, with systemd, the system admin cannot tell it what are the priority services that should be started first.
Posted Nov 2, 2018 0:25 UTC (Fri)
by jccleaver (guest, #127418)
[Link] (1 responses)
In a nutshell, systemd is a developer's boot stamping on the face of a sysadmin/syseng... begging the question with their design choices on so many paradigm shifts that the administrator gives up trying to fight it, and eventually gives up trying to grok it at all.
Posted Nov 2, 2018 19:24 UTC (Fri)
by naptastic (guest, #60139)
[Link]
You've hit the nail on the head. However much of systemd is NIH syndrome, or enforcing vendor lock-in, or adding complexity so people will pay for support, or just obliviousness to reality, systemd is just... well, I remove it as fast as I can wherever I can, and every time I do, life gets better. My systems boot faster, shut down faster, have fewer random problems, and are easier to troubleshoot.
Posted Nov 2, 2018 0:45 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
I wonder if you have an unrelated issue in your systems somewhere.
Posted Nov 2, 2018 1:32 UTC (Fri)
by zlynx (guest, #2285)
[Link]
There is definitely something wrong. If it's reading from an M.2 drive benchmark that thing, because it's looking broken.
I believe my last NAS with dual Bobcat cores at 1.6 GHz booted faster than that.
I think Raspberry Pi's reading from slow sdcards are quicker.
Posted Nov 2, 2018 19:11 UTC (Fri)
by mchehab (subscriber, #41156)
[Link]
Yeah, there was indeed an issue: I build kernels with KASAN enabled. That, together with all stuff systemd tries to load and then unload at the initrd disk, made it really slow (it was spending ~30 seconds at initrd). I did a clean custom.target with the minimum stuff, with improved it a lot:
$ ls /etc/systemd/system/custom.target.wants/
Also removed an open-iscsi package (with alone saved almost 15 seconds).
After lots of cleanups, I was able to reduce the time for network to start to "just" 1 minute, with KASAN enabled.
Without KASAN, using a vanilla Ubuntu Kernel, it is now booting really quick:
Startup finished in 777ms (kernel) + 2.416s (initrd) + 8.385s (userspace) = 11.579s
Discovering it costed me an entire day of work. Doing the same with sysV would be trivial.
Posted Nov 1, 2018 17:26 UTC (Thu)
by mchehab (subscriber, #41156)
[Link]
Nope. Just after systemd starts:
$ systemd-analize
Startup finished in 5.095s (kernel) + 24.631s (initrd) + 54.179s (userspace) = 1min 23.906s
$ systemd-analyze critical-chain multi-user.target
multi-user.target @53.796s
> At home I have four systems with spinning rust that don't even take anywhere near long to get all the way from to the graphical login prompt. And yes, networking (wifi!) is up at that point. (Meanwhile, two servers each take longer than that to get through the POST...)
Systemd services setup seem to be written to prioritize starting the GUI. Here, GUI comes quick, but all the rest takes a lot more time. Usually, the *last* thing systemd starts are the text consoles (just after network).
On some of my machines, it may take up to 10 minutes to have them started - that was actually happening on that specific machine and forced me to actually remove packages from it (like lxc/lxd/snapcraft/libvirt/...).
Posted Nov 2, 2018 3:18 UTC (Fri)
by tytso (subscriber, #9993)
[Link]
I normally use it as as part of a VM image, but if you need to run it on a real x86 hardware that shouldn't be all that difficult. In any case, it satisfies your requirement that it should come up in seconds. So it can definitely be done:
https://github.com/tytso/xfstests-bld/blob/master/kvm-xfs...
Documentation:
https://github.com/tytso/xfstests-bld/blob/master/Documen...
Posted Nov 1, 2018 14:48 UTC (Thu)
by rweikusat2 (subscriber, #117920)
[Link] (8 responses)
This statement makes no sense. I'm tolerably familiar with the sysvinit code because I'm maintaining a fork of it. It contains some rudimentary process management which is virtually unused and presumably mainly exists (or came to exist) for compatibility with the really SysV init. Apart from that, it waits for run-level change requests and runs a (configurable) script in response to these. It also handles the exit status of processes it inherited because their parent processes exited while the children were still running. And that's it. That's a basic component which (very flexibly) enables implementing a variety of process manageing schemes on top of it, including the 'legacy' Linux distribution process not-management-system with all its horror workarounds for lack of sensible additional tools, pid files, pgrep, start-stop-daemon etc.
Considering this, your statement is somewhat like "I understand why someone would fit a combine harvester to a wheelbarrow after the wheel broke. After all, it has four them, meaning, greater reliabilty through redundant hardware!".
Posted Nov 1, 2018 14:56 UTC (Thu)
by rweikusat2 (subscriber, #117920)
[Link] (7 responses)
Posted Nov 2, 2018 18:56 UTC (Fri)
by jccleaver (guest, #127418)
[Link] (6 responses)
I'm not sure about the "certainly" there. Arguably? Yes. Certainly? No.
I can count the number of unexpected boot hangs traced back to a problem with the rc.sysinit or related bootup scripts in non-systemd systems going back to 2000 on maybe one hand, with the only regular issue being sendmail hanging the boot on a lack of DNS resolution of the local hostname. Virtually every other hang has been a result of hardware, or daemon bugs.
systemd is too clever for its own good in trying to parallellize boot, meaning the "init system" is much more likely to fail in unexpected and hard to trace ways. upstart, of course, is capable of "parallellizing" things too... but, crucially, in RHEL 6 this wasn't actually used. The vast majority of the boot logic was in /etc/rc.sysinit, not /etc/init/*. If systemd did the same a significant number of problems would be avoided. (Even if other philosophical problems remained.)
systemd proposes, and then implements, a paradigm shift among things that used to be discrete, including "the init system", as opposed to PID 1. I think there's room for reductionist debate at all levels.
Posted Nov 2, 2018 19:45 UTC (Fri)
by mpr22 (subscriber, #60784)
[Link] (5 responses)
AFAIK systemd doesn't do anything particularly "clever" when it comes to parallelizing system startup, but rather just says that by default, any pair of units which do not state themselves to require serialization with respect to each other shall be assumed parallelizable.
Posted Nov 2, 2018 20:28 UTC (Fri)
by xtifr (guest, #143)
[Link] (4 responses)
And yet, when Debian switched over to systemd as a default, it still reduced my boot time by an order of magnitude!
(Note that I do not really have any strong opinions about init systems one way or the other. But despite switching to systemd as part of Debian's automatic updates, and, more recently, because of a re-install of my system partitions from scratch, the only difference I've observed is a faster, cleaner boot. Heck, even my log files are still working the way I'm used to. Which rather surprised me, since "I hate binary log files" is one of the few complaints I've heard about systemd that actually had me at all worried.)
Posted Nov 2, 2018 20:56 UTC (Fri)
by lkundrak (subscriber, #43452)
[Link] (3 responses)
I have no idea how could I have lived without journalctl --since '5 minutes ago'.
Posted Nov 2, 2018 21:50 UTC (Fri)
by jccleaver (guest, #127418)
[Link] (2 responses)
tail -200 /var/log/messages ?
(and maybe adjust your number if you guessed your log velocity wrong)
Posted Nov 2, 2018 23:39 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Nov 5, 2018 11:57 UTC (Mon)
by jond (subscriber, #37669)
[Link]
Posted Nov 1, 2018 7:23 UTC (Thu)
by gfernandes (subscriber, #119910)
[Link]
Really?
Honestly?
Come on my friend. That's a bit like eating chalk pretending it's cheese.
Posted Nov 1, 2018 2:04 UTC (Thu)
by milesrout (subscriber, #126894)
[Link] (14 responses)
Maybe a few years ago the options were sysvinit vs systemd, and the decision to go with systemd was correct. But today, there is a wide variety of options, and the people that continue to act like the only alternative to sysvinit is systemd (which is bad, they say, so we should use sysvinit instead) or vice versa are being very misleading IMO.
Posted Nov 1, 2018 2:19 UTC (Thu)
by anselm (subscriber, #2796)
[Link] (3 responses)
There may be a “wide variety of options” today but package maintainers can't be expected to know and support them all. Many of them restrict themselves to systemd and sysvinit, which is what policy asks them to do.
So therefore, if you want to promote a new (non-default) init system in Debian you should be prepared to contribute support for that to most of the relevant packages. It is unlikely that you will get all of their maintainers to come up with the required configuration files from scratch, although they should in theory be willing to accept patches.
Posted Nov 1, 2018 4:48 UTC (Thu)
by zlynx (guest, #2285)
[Link] (2 responses)
To win them over to some other system write an automatic conversion tool that takes a source (like systemd unit files) and produces whatever configuration the other system needs.
Posted Nov 1, 2018 17:10 UTC (Thu)
by jccleaver (guest, #127418)
[Link] (1 responses)
Posted Nov 5, 2018 13:21 UTC (Mon)
by mjthayer (guest, #39183)
[Link]
This sub-thread struck me as a very constructive part of the discussion. Debian policy around unit files to keep them easily convertible to other init systems could be helpful.
Posted Nov 1, 2018 7:27 UTC (Thu)
by gfernandes (subscriber, #119910)
[Link] (7 responses)
But it must certainly is the *best*, most comprehensive, and well thought out solution there is.
Posted Nov 1, 2018 7:28 UTC (Thu)
by gfernandes (subscriber, #119910)
[Link]
Posted Nov 1, 2018 21:24 UTC (Thu)
by rweikusat2 (subscriber, #117920)
[Link] (5 responses)
Granted, that's not specific to systemd, that's a defect it shares with a lot of other "Jonathans very own init replacement"-programs.
Posted Nov 2, 2018 1:13 UTC (Fri)
by zlynx (guest, #2285)
[Link] (2 responses)
Wrapped in a retry loop.
You really cannot do anything about an app that crashes after it reports ready. It's not like you can force it to keep running.
I'm not actually sure what systemd would do there. It would have been ready, so it would have started dependent units. But then it would have noticed the service crash which makes it not-ready again. I don't think it actually stops the dependents. They just have to survive on their own I guess.
Posted Nov 2, 2018 19:51 UTC (Fri)
by rweikusat2 (subscriber, #117920)
[Link] (1 responses)
Posted Nov 2, 2018 20:07 UTC (Fri)
by zlynx (guest, #2285)
[Link]
The only problem is if the server startup time (giant Java apps) is so long that the clients time out before the server wakes up. Then you're back to needing readiness notifications.
Smart clients would retry connections on connection failure. Or they'd immediately exit, but their activation unit would restart them with some retry delay.
Posted Nov 2, 2018 19:32 UTC (Fri)
by mpr22 (subscriber, #60784)
[Link] (1 responses)
Posted Nov 2, 2018 19:56 UTC (Fri)
by rweikusat2 (subscriber, #117920)
[Link]
Posted Nov 3, 2018 4:32 UTC (Sat)
by ThinkRob (guest, #64513)
[Link] (1 responses)
At some point the size of a userbase and the secondary effects becomes a significant factor. Maybe even the most significant.
systemd almost certainly has the largest install base of any of the Linux init options at the moment, by a large margin. That alone makes it attractive: there's more support for it, documentation/help is easier to find, and it gets a lot more testing. Something else could take its place, but it would have to be a LOT better to overcome that inertia. (That doesn't mean it can't happen; that's how systemd won in the first place!)
So IMHO systemd is "great" because it's the most widely used tool that does the job.
Posted Nov 3, 2018 13:49 UTC (Sat)
by drag (guest, #31333)
[Link]
Sysvinit does not address most of the issues that low-level-but-above-kernel OS design requires and in the past this meant a lot of ad-hoc development work being forced on distribution maintainers, application developers, and sysadmins who really are not that interested in designing operating systems.
This ended up with thousands of people facing the same basic set of issues trying to solve them independently to significantly different degrees of success. How many tens of thousands of different variations of 'start/stop/status httpd' scripts does the world really need? How many hundreds daemon helper programs that only 10 people is going to make the world a better place?
The ultimate purpose of operating systems is to make applications easier to write and easier to consume. It is telling that people that are doing the actual important work of making services/applications easier to use on Debian don't use sysvinit except though some sense of obligation.
And, sure, nobody questions that there are not alternatives to doing what systemd does. Systemd isn't magic, it's just software. And like any other program it can all be done in a billion different variations with a billion different designs any of which can be perfectly viable.
It's not the design details that matters so much. It's the amount of work people put into making it work. And if nobody is able to put the same amount of work into any systemd alternatives and offer it up to wide-spread consumption then there really isn't any benefit to doing it that way for most people and most distributions.
Posted Nov 1, 2018 19:24 UTC (Thu)
by mads (subscriber, #55377)
[Link] (1 responses)
I just want to mention that systemd works great in Gentoo, it's easy to download a stage3 from gentoo.org that has a systemd profile already enabled, and it's easy to switch between an openrc based profile and a systemd based profile on an already built system. Can gladly recommend it, I've been using it for years on all of my Gentoo systems.
Posted Nov 8, 2018 13:36 UTC (Thu)
by Wol (subscriber, #4433)
[Link]
I tried to install gentoo/systemd on my laptop. I gave up - it's now running my second-choice distro, SUSE.
I am left with the *very* strong impression that there are several anti-systemd developers at the core of gentoo. And it shows in that there are a fair few references *to* systemd documentation, but *no pointers*, and the documentation is (apparently deliberately) quite hard to find.
So yes, I think gentoo is, sadly, one of the anti-systemd distros, even if it is quite easy (allegedly, I couldn't) to get a systemd version running.
Cheers,
Posted Nov 2, 2018 19:36 UTC (Fri)
by joey (guest, #328)
[Link] (1 responses)
Who are those many developers who switched? The only one I know of is Roger Leigh.
Otherwise, none of the names on https://devuan.org/os/team/ have been in the Debian keyring. Perhaps some of the anonymous/pseudonymous people are DDs (the original fork announcement claimed they had two other anonymous DDs), but this seems a thin thread for LWN to hang that statement on.
Posted Nov 5, 2018 11:58 UTC (Mon)
by jond (subscriber, #37669)
[Link]
Roger Leigh is the only one I know of, and I do miss him, almost as much as I miss Joey Hess.
Posted Nov 3, 2018 19:04 UTC (Sat)
by dps (guest, #5725)
[Link] (8 responses)
There are problems, which I have not solved, with systemd like pointless start jobs on disks which are already mounted which take 90s to fail. Just not mounting thar device early is not an option because that device contains /usr.
Things like NetorkMamager are a *huge* step backwards if you need simple and predictable networkong - doing it with nmcli is very hard and old style network configuration scripts work much better. Nothing in a dmz should even think of trusting dhcp,
It is possible to eliminate systemd, dbus and udev but this requires nontrivial effort. Removing udev, which obviously implements anti-features on some systems, breaks most supported init systems without changes which are hard to do because the system won't boot.
IMHO those that want simple, low dependency and bloat free init should be able to get it more simply. Static /dev might be out of fashion but I actively don't want automatic device creation so udev does not solve the problem.
P.S. I remember the days is SLS 1.03 (kernel version 0.99pl11), when udev and dbus did not exist and everybody used static /dev. If think that the dhcp rfc had not been written at that time.
Posted Nov 3, 2018 19:15 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
Posted Nov 3, 2018 19:38 UTC (Sat)
by zdzichu (subscriber, #17118)
[Link] (1 responses)
Posted Nov 5, 2018 13:56 UTC (Mon)
by nix (subscriber, #2304)
[Link]
Posted Nov 3, 2018 19:38 UTC (Sat)
by mpr22 (subscriber, #60784)
[Link] (4 responses)
Perhaps some of those that want that should team up and do the work to provide tools for arranging it, so that the rest of those who want it can get it.
Posted Nov 4, 2018 15:45 UTC (Sun)
by jospoortvliet (guest, #33164)
[Link] (3 responses)
Sometimes I wonder if the complainers are just jealous of Lennart who actually does manage to build BETTER solutions. After all his technology gets adopted widely and that wouldn't happen if it wasn't superior right? Isn't open source a meritocracy?
Posted Nov 5, 2018 15:56 UTC (Mon)
by dps (guest, #5725)
[Link] (1 responses)
Firewalls are not configured like regular boxes and what is desirable is different. A lot things which are better for regular systems are worse for firewalls and bastion hosts, which actively don't want most features. Some features are security features and obviously a good idea, and others may be required to do whatever the box is supposed to do. Anything else is anti-feature and should be resisted.
Those that have used Linux 0.90pl11 know how to do things without the modern features that depend on what is an anti-features on a paranoid host. System V init could handle this problem. Those things where the simple solution does not work out are almost all features that are useless on paranoid hosts in noisy data centres.
Posted Nov 5, 2018 16:50 UTC (Mon)
by anselm (subscriber, #2796)
[Link]
I don't think that if you're paranoid you would want to use anything on a firewall or bastion host that is as needlessly complicated as sysvinit.
Posted Nov 8, 2018 13:48 UTC (Thu)
by Wol (subscriber, #4433)
[Link]
The problem is those people who want (or need, let's face it, a lot of us DON'T need a big complicated init) a simple init usually aren't the people with the skill to understand the problem.
Those of us who need a complicated init will design an init to suit our needs, and those of us who want a simple init will have to put up with it because we don't have the skills to do it ourselves.
It's like distros - your average distro is a jack-of-all-trades and any one user probably would describe 90% of it as unwanted crap. Problem is, my unwanted crap is your can't live without application, and vice versa.
So you need an expert in the complicated system to strip out the 90%, to make a dedicated single purpose distro or init system.
Cheers,
Posted Nov 8, 2018 9:43 UTC (Thu)
by oldtomas (guest, #72579)
[Link] (1 responses)
Skimming the comment section is a sad experience and still reveals how deep that conflict is. The tendency to denigrate the "other side" seems still... irresistible.
Luckily, the tone within Debian has (mostly) reached a cooperative, almost friendly level.
To all of you (on any of both sides): re-read the Russ Allbery quotes up there in the article. This is the example to follow (it has been from the hottest phase in the systemd "war", btw).
Posted Nov 10, 2018 9:42 UTC (Sat)
by HelloWorld (guest, #56129)
[Link]
Posted Nov 12, 2018 20:13 UTC (Mon)
by dkg (subscriber, #55359)
[Link]
I am not doing this out of antipathy toward sysvinit or its users; i'm doing it because i can't responsibly maintain them. even if i ran systems that had sysvinit as pid 1 that i could use for testing, systemd offers so many better ways to ship tuned, constrained unit files that for moderately complex services (ones that need customized permissions or OS constraints, for example, or that use socket activation), i can't reliably assert that:
shipping a sysvinit file responsibly means i need to help people use it, and test and analyze the ways that it fails. I have limited cycles, and i don't want to maintain a lot more code, especially if there is some other codepath that is already established and i can relatively simply express what i want common services to do in a declarative manner. And i definitely don't want local admins who have the package installed using sysvinit having radically different active system semantics. for example, it would be pretty bad if a daemon runs as root under sysvinit, but as user foo under systemd.
Consequently, i have opened bugs asking for help with sysvinit maintenance (e.g. #833854 and #855653), but have gotten very little concrete help in making them work reliably.
If i can't offer something that is reliable, i think i'm doing debian a disservice in shipping it. Certainly, i'd never complain if an administrator wants to write their own initscripts (for sysvinit) or servicedirs (for runit) or any other system integration. And i don't want to accidentally clobber a locally-customized sysvinit script either! But if anyone wants to step up to offer to co-maintain that part of the package so that it has comparable support to the systemd unit files, i'd welcome it.
but i'm going to focus my Debian package maintenance time on things that seem more useful and efficient and widely applicable to the Debian ecosystem. Today, that's systemd.
NO TO SYSVINIT - or initscripts, rather!
What's more, this new combination of s6+s6-rc/OpenRC can be used in Debian/kFreeBSD (for all the 10 users there) and other Debian ports as they're portable to most Unices.
[1]: http://skarnet.org/software/s6/
[2]: http://skarnet.org/software/s6/s6-fdholder-daemon.html
[3]: https://skarnet.org/software/misc/sdnotify-wrapper.c
[4]: http://skarnet.org/software/s6/socket-activation.html
[5]: http://skarnet.org/software/execline/
[6]: http://skarnet.org/software/s6/s6-sudo.html
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
Nope. systemd's flash of brilliance was the use of cgroups to isolate daemons, which allowed it to actually reliably track them. For the first time ever. No other init system did this before.
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
I don't remember other init systems doing this, though. It's brilliantly simple like many other inventions.
Uhm, systemd was released in 2010. At best this is an example of convergent evolution.
The problem here is that SysV (and its ilk) doesn't really have a parent that can be used to manage its children. For this to work you need a specialized persistent process that stays up at all time, because if it dies all of its children will also be killed.
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
If sysvinit was extended to, say, 128 run levels to handle dependencies, actually used as intended, and possibly extended to facilitate enabling/disabling/activating/deactivating daemons without needing to edit inittab, sysvinit would do everything a lot of people need without a lot of complexity.
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
systemd is one of half a gazillion replacement inits written in $compiled_language which roughly all provide about the same feature set (and make the same errors), there's nothing special about it beyond "it's the RedHat replacement init!".
That's not much different from what systemd and other members of the zoo offer, it's just more flexible in that it retains a fully capable high-level, standardized programming language as backbone engine instead of some non-extensible, declaratory language
NO TO SYSVINIT - or initscripts, rather!
> could avail themselves of, [...] but nobody except Lennart Poettering and Kay Sievers actually condescended to doing that in
> practice.
NO TO SYSVINIT - or initscripts, rather!
to come up with a set of reusable “support tools written in C” that init systems could avail themselves of, and make these popular across distributions, but nobody except Lennart Poettering and Kay Sievers actually condescended to doing that
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
> I don't have a clue what that 10MB of code in systemd could possibly be doingNO TO SYSVINIT - or initscripts, rather!
$ size /usr/lib/systemd/systemd
text data bss dec hex filename
1238373 259368 416 1498157 16dc2d /usr/lib/systemd/systemd
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
But then you need bash/dash and a ton of other utilities (sed, awk, ...) for the initscripts. You can mostly get away with busybox these days, which is around 2MB.
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
[2] https://packages.debian.org/sid/dbus
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
- open a shell at the serial console;
NO TO SYSVINIT - or initscripts, rather!
> - enable network and sshd.
> - open a shell at the serial console;
Though you always have sysrq as the good old hammer.
Also, kill -RTMIN+15 1, https://www.freedesktop.org/software/systemd/man/systemd....
> Never found a way to do that with systemd.
They have https://www.freedesktop.org/software/systemd/man/systemd-... and https://www.freedesktop.org/software/systemd/man/systemd-...
That's what upstream says, you really can, but then need to set DefaultStandardOutput=/DefaultStandardError= in /etc/systemd/system.conf to whatever socket/fifo/etc you might want to pick those up from, or syslog. Also, in this case you end up losing native journal messages (unless you write something that forwards them to syslog).
You want systemd.debug_shell (started as early as possible, root shell).
NO TO SYSVINIT - or initscripts, rather!
└─dbus.service @47.065s
└─basic.target @46.210s
└─sockets.target @45.940s
└─lxd.socket @42.127s +3.542s
└─sysinit.target @39.893s
└─systemd-update-utmp.service @37.305s +2.312s
└─systemd-tmpfiles-setup.service @30.462s +1.908s
└─systemd-journal-flush.service @23.994s +3.461s
└─systemd-remount-fs.service @2.192s +9.791s
└─systemd-fsck-root.service @584542y 2w 2d 20h 1min 37.360s +821ms
└─systemd-fsckd.socket @5.262s
└─system.slice
└─-.slice
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
25.765s dev-mapper-localhost\x2d\x2dvg\x2dubuntu.device
10.902s resolvconf.service
9.526s keyboard-setup.service
9.497s systemd-modules-load.service
8.397s kmod-static-nodes.service
8.387s systemd-remount-fs.service
8.067s binfmt-support.service
7.825s sys-kernel-debug.mount
...
└─systemd-journald.socket
└─system.slice
└─-.slice
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSTEMD
NO TO SYSVINIT - or initscripts, rather!
You can whack it with a hammer and delete the dependencies of the network service. But your startup times look really unhealthy anyway, 10 seconds for resolvd is unbelievable.
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
anacron.service dbus.service networking.service rsyslog.service systemd-resolved.service thermald.service atd.service inetd.service NetworkManager.service ssh.service systemd-update-utmp-runlevel.service ttyUSB0.service cron.service irqbalance.service ondemand.service systemd-logind.service systemd-user-sessions.service
custom.target reached after 8.383s in userspace
NO TO SYSVINIT - or initscripts, rather!
multi-user.target reached after 53.796s in userspace.
└─mpd.service @52.285s +1.509s
└─network.target @52.258s
└─systemd-resolved.service @52.060s +197ms
└─tmp.mount @39.178s +6.959s
└─dev-localhost\x2dvg-tmp.device @37.992s
NO TO SYSVINIT - or initscripts, rather!
https://github.com/tytso/xfstests-bld/blob/master/Documen...
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
> "I hate binary log files" is one of the few complaints I've heardNO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
I've seen enough code like this (the tail-pipeline) but this just illustrates the 'skills' (or lack thereof) of the person who wrote this code.
However, it doesn't really matter in which way something which cannot possibly work is implemented. And "readiness notification" is such a something as it communicates information about a state at a time of check but there's nothing which stops this state from changing in the time between a somewhat later time of use. Check-and-use would need to be an atomic operation and this operation would need to be performed every time use is supposed to happen. The very fact that check is done by some system daemon and use by "something else" prevents this.
The underlying problem here (if any) is really a bug (or some bugs) in application desiring to do the use which cannot cope with the service they're trying to use being unavailable for some time. And the "readiness notification" is just a doomed attempt to work around this bug (or these bugs).
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
A program which, having become "ready", subsequently becomes "unready" without actually exiting or getting a fatal signal is a problem whether you're using a service supervisor or not.
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
Init system support in Debian
Init system support in Debian
Wol
Init system support in Debian
many of the vocal anti-systemd Debian developers have switched
Init system support in Debian
Init system support in Debian
Init system support in Debian
Add a device dependency to these jobs.
Init system support in Debian
Init system support in Debian
Init system support in Debian
Init system support in Debian
Init system support in Debian
Init system support in Debian
Init system support in Debian
Wol
Thanks!
Thanks!
"Both sides"? Is that you, Donald?
As a debian package maintainer, i'll speak up and say that i am one of the people guilty of starting to drop sysvinit-style initscripts from my packages. This makes me a bit sad because i really would like to support non-Linux kernels in Debian better, and systemd is Linux-only, of course.
Init system support in Debian