NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
Posted Nov 3, 2018 14:03 UTC (Sat) by drag (guest, #31333)In reply to: NO TO SYSVINIT - or initscripts, rather! by rweikusat2
Parent article: Init system support in Debian
People have been solving these problems for decades now. Over and over again.
And eventually somebody came along, solved them, wrote it all in C, created a formal project and got all of the major distributions to adopt it. They call it Systemd.
Posted Nov 4, 2018 19:56 UTC (Sun)
by rweikusat2 (subscriber, #117920)
[Link] (37 responses)
systemd is one of half a gazillion replacement inits written in $compiled_language which roughly all provide about the same feature set (and make the same errors), there's nothing special about it beyond "it's the RedHat replacement init!". It's arguably an improvement compared to the mess RedHat, SuSE, Debian etc used to ship in /etc/init.d. So is upstart, runit, s6, monit, $younameit etc, they just never became "mainstream" because RedHat didn't adopt any of them.
But the "mess in /etc/init.d" RH, SuSE, etc used to ship wasn't created by some force of nature, it just represents about 20 years of "organic shell script growth" based on lack of sensible support tools for process management and Bourne shell ad hocery. A large part of this codeheap isn't really good for anything and could just be deleted (I still fondly remember a comment in one SuSE init script complaining that it was necessary to work around bugs in another SuSE init script --- why not just fix them?). Another large part can be condensed into a set of support tools written in C which can be reused in other init scripts (or simliar situations).
That's not much different from what systemd and other members of the zoo offer, it's just more flexible in that it retains a fully capable high-level, standardized programming language as backbone engine instead of some non-extensible, declaratory language intepreted by a dedicated interpreter which has some random set of "process management support code" aggregated in one (or some) fairly large program(s).
Posted Nov 4, 2018 22:00 UTC (Sun)
by Cyberax (✭ supporter ✭, #52523)
[Link] (20 responses)
Everything else simply follows. If you can isolate daemons then you don't need PID files. You can place limits on their resource use and reliably redirect logs.
If you are aware of any other pre-systemd init system that does the same, I'm all ears.
Posted Nov 5, 2018 20:07 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (7 responses)
"PID files" are a grotty, traditional UNIX hack to prevent multiple instances of a program from running (a very dubious practice to begin with) and to enable some measure of process control by sending signals to the process whose pid is recorded in a pid file (which may or may not be the process actually supposed to get these signals). A well-behaved program won't 'escape' (or at least can be asked to avoid doing so) from its parent. Hence, the parent can reliably send signals to it and can provide an interface enabling other software to do so. It can also provide "only one running instance" feature reliably and can set resource limits for processes which either aren't privlileged or at least don't actively sabotage management.
[*] So far, I haven't encountered the case of a program which actively resisted management. Hence, I had no reason to create a tool for supporting this.
Posted Nov 5, 2018 20:18 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (6 responses)
Socket activation and fully asynchronous design are also new.
> For the usual case that this is not the case, normal parent-child relationship and ensuring that managed processes don't background themselves are entirely sufficient. The lxc-project has been using them for containers since before systemd existed (Wikipedia puts its beginnings at 2010. In 2010, I was using lxc as base for virtualized IPsec VPN servers for mobile clients).
> Hence, the parent can reliably send signals to it and can provide an interface enabling other software to do so. It can also provide "only one running instance" feature reliably and can set resource limits for processes which either aren't privlileged or at least don't actively sabotage management.
The earliest example of such process is DJB's daemontools. It follows the same ideology as systemd, it has declarative service description and it uses traditional UNIX process groups to track processes. It doesn't have dependencies or socket activation, though. It's also not asynchronous so a badly-written script can lock it up.
Upstart attempted to do better but its design honestly sucks. It also uses hacks like ptrace() to detect double-forking daemons.
Posted Nov 6, 2018 20:42 UTC (Tue)
by rweikusat2 (subscriber, #117920)
[Link] (3 responses)
Socket activation with the intent to reduce system boot time and provide better overall system performance saw the light of the day with the 4.3BSD inetd.
Fully asynchronous, that is, event-driven servers have also existed for a long time. Dan Kegel's original C10K paper mentioned the concept. The (older) racoon IKEv1 daemon also works in this way. This is again a design choice: Manage more than one process with the same process management process and accomodate buggy or intentionally hostile "other software". One could argue that buggy other software should rather be fixed and that one shouldn't be using intentionally hostile one.
The sysvinit program actually supports (some very limited form) of process management: It's possible to define per-runlevel commands which will be started automatically whenever a runlevel is entered and restarted if they ever terminate. Traditionally, this is unused except for gettys (to support user logins on all kinds of text terminals, virtual, physical, serial console etc). But the initscripts/ runlevel management system is completely independent of that: All init knows about this is that it's supposed to execute the command /etc/init.d/rc with the runlevel number as argument (both configurable). This means it's possible to use the sysinit program (or any other kind of "init init" which restricts itself to this role) together with any other process management program/ tool. Eg, the one I'm using is just a command which is executed during service start. Basically, it will execute another command with certain arguments in a subprocess. Should this other command terminate, it will be respawned. Further, the command creates an AF_UNIX stream socket for external process control. This can be used to query the status of the managed command, to initiate a stop or restart and to send abitrary signals to it.
If the process serving as init ever terminates, the kernel will panic. But ordinarily, death of a parent process doesn't affect its children. They'll be inherited by the init process which will eventually collect their exit status to prevent them from turning into zombies. Otherwise, backgrounding a program by forking and terminating the parent process wouldn't be possible.
Posted Nov 6, 2018 21:51 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
I'm not aware of any other init system that does this, even to this day. There were scattered bits and pieces here and there, but not a coherent design.
Posted Nov 7, 2018 17:04 UTC (Wed)
by rweikusat2 (subscriber, #117920)
[Link] (1 responses)
But this again starts to drift from a technically interesting topic (technically interesting to me at least) towards marketing statements ("This shoesine is brilliant!") ... :-)
Posted Nov 7, 2018 17:47 UTC (Wed)
by pizza (subscriber, #46)
[Link]
That's like saying that Linux is just a reimplementation of UNIX.
(Along with the implication that there nothing new, different, or otherwise improved between the two)
Posted Nov 12, 2018 2:44 UTC (Mon)
by fest3er (guest, #60379)
[Link] (1 responses)
I'll put forth that sysvinit *is* the parent that manages its children. It simply has never been used that way. Instead, the old method of using RC scripts (did this come fom Berzerkely?) was used (or continued). The very useful part of sysvinit (most of what's in inittab) was relegated to the status of a curious-looking red-headed stepchild useful only for running getty.
If sysvinit was extended to, say, 128 run levels to handle dependencies, actually used as intended, and possibly extended to facilitate enabling/disabling/activating/deactivating daemons without needing to edit inittab, sysvinit would do everything a lot of people need without a lot of complexity.
Posted Nov 12, 2018 9:39 UTC (Mon)
by anselm (subscriber, #2796)
[Link]
Sysvinit can actually deal with more runlevels than the customary six. It's just that people seldom bother to define more of them.
That still wouldn't give you the consistency and convenience that systemd provides. Remember that much of the advantage that systemd confers, from the POV of system management, is in configuration, e.g., the obvious division between system-provided and locally-provided configuration settings which makes it easier to install a new upstream version of a software package without stomping on local customisations. You could obviously come up with a management layer for sysvinit that generates the real /etc/inittab file from a set of snippets in various directories, but the simple fact is that that hasn't yet been done, and now that systemd exists (and can be pared way down if required) the need for it is much less than it may have been earlier.
Posted Nov 5, 2018 21:28 UTC (Mon)
by luto (guest, #39314)
[Link] (11 responses)
Posted Nov 5, 2018 23:02 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (8 responses)
Posted Nov 5, 2018 23:12 UTC (Mon)
by luto (guest, #39314)
[Link] (7 responses)
Posted Nov 5, 2018 23:17 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (6 responses)
Saving kernel size is a valid concern but I'm not sure how much you're going to save if you disable all cgroups controllers.
Posted Nov 5, 2018 23:24 UTC (Mon)
by luto (guest, #39314)
[Link] (5 responses)
Posted Nov 5, 2018 23:28 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (4 responses)
Posted Nov 12, 2018 17:22 UTC (Mon)
by nix (subscriber, #2304)
[Link] (3 responses)
Posted Nov 12, 2018 19:54 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
Posted Nov 13, 2018 0:18 UTC (Tue)
by nix (subscriber, #2304)
[Link] (1 responses)
That's... useful. As in, I think I'll have a use for it in my own work in the next week. :) I wish I'd realised v2 worked this way years ago!
Posted Nov 13, 2018 0:21 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Nov 5, 2018 23:19 UTC (Mon)
by anselm (subscriber, #2796)
[Link]
The subreapers would give you the opportunity to track processes that exit, but you'd still need to use cgroups to do resource control.
Posted Mar 13, 2019 16:30 UTC (Wed)
by nopsled (guest, #129072)
[Link]
It would be nice if Linux had the PROC_REAP_KILL stuff. I think one could improve the FreeBSD interface a tad bit when incorporating it, but the concept looks sound.
systemd ofcourse has one supervisor per context (system or user), but this will be especially useful for supervisors that have one monitor per service (s6, runit style), and that these can be nested supervisors (so an option to not cross subreaper boundaries where each supervisor is a subreaper, or a way to kill inherited children only, and a way to kill direct children only, and OR the flags to do both). You could get creative, but if done right, this could solve a lot of woes with process management.
Posted Nov 5, 2018 13:17 UTC (Mon)
by anselm (subscriber, #2796)
[Link] (15 responses)
It makes more sense to think of systemd as “basic plumbing” rather than “init”. Systemd does include an init process but it tries to address many low-level OS issues that the other replacement inits don't. One advantage of this is that systemd offers a unified approach to configuration that actually extends the same features of, e.g., subprocess creation to wherever subprocesses are created in systemd. With sysvinit or, for that matter, the other replacement inits, this sort of thing is still handled in an ad-hoc manner in a variety of tools that have been cobbled together over decades with no cohesive design.
In theory it would of course be perfectly possible to come up with a set of reusable “support tools written in C” that init systems could avail themselves of, and make these popular across distributions, but nobody except Lennart Poettering and Kay Sievers actually condescended to doing that in practice. Now, from a practical POV, that ship has sailed, since the adoption of systemd by virtually all mainstream distributions has placed the bar for basic-plumbing replacement a lot higher than it used to be when sysvinit+cron+at+inetd+… was the “state of the art”. Anything put forward now must not just provide a significant advantage over sysvinit+…, but over systemd, and with the amount of clever engineering that has gone into systemd over the last decade or so, that is a considerably more difficult task.
Also note that nobody forced Red Hat, SUSE, Debian, … into adopting systemd. Out of the existing init system replacements at the time it just happened to be the one with the most enticing value proposition. (It's also worth reiterating, since people here have been calling systemd “Red Hat's init”, that Lennart Poettering and Kay Sievers started systemd on their own time; nobody from Red Hat went to them and said “write an init replacement for Red Hat Linux”.)
Systemd lets you call out to shell scripts if you need to. But in general, adding the shell to the mix only makes things more complicated for those many if not most cases which are really quite simple and straightforward. The declarative configurations of systemd are easier to read, understand, reason about, standardise upstream, maintain, and teach, than shell scripts and you can still write and use old-fashioned sysvinit-style init scripts with systemd if you absolutely must.
Posted Nov 5, 2018 20:09 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (14 responses)
The set of "open source developers" on this planet is a proper subset of the set of developers.
Posted Nov 5, 2018 21:23 UTC (Mon)
by anselm (subscriber, #2796)
[Link] (13 responses)
You conveniently neglected to quote an important part of my sentence, namely
As you say, init replacements are a dime a dozen, but none of them really achieved a lot of traction (with the possible exception of Canonical's Upstart). It is to Lennart's and Kay's credit that they managed to come up with something so convincing that virtually all major Linux distributions have rallied around it, and the resulting network effects are a major win for the platform.
(In the non-open-source area, practically every provider of proprietary Unix had come up with its own replacement for sysvinit even before systemd got started – Linux used to be the last holdout of sysvinit – but that is neither here nor there.)
Posted Nov 5, 2018 22:12 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (12 responses)
[...]
I've ignored that on purpose because while I'm somewhat interested in discussing the technology, I have zero interest to engage in a discussion of the surrounding politics and other social mechanics as that's bound to degrade into an entirely pointless flamewar of a very familiar shape quickly.
Posted Nov 5, 2018 23:17 UTC (Mon)
by anselm (subscriber, #2796)
[Link] (11 responses)
For the record, I agree that there have been enough flame wars about systemd. But it is important to get one's basic facts right, including that (a) systemd was not started as a Red Hat project, and certainly by now includes active developers from many parts of the community, and (b) other distributions (whose developers, on the whole, aren't gullible incompetents either) adopted systemd because they felt it was technically the Right Thing to do, not because they were somehow (how exactly?) coerced to do so by Red Hat. Remember that Debian in particular did not take the decision to default to systemd at all lightly.
The fact that most major Linux distributions did adopt systemd as their default init system differentiates systemd from all of the other init system replacements (except Upstart), and as such it is reasonable to ask what the other init system replacements “did wrong” if they're as clearly superior to systemd as their proponents claim but yet, unlike systemd, didn't manage to convince multiple mainstream Linux distributions of their merits. It's easy to blame “politics and other social mechanics” but we should not altogether discount the fact that systemd, too, includes a good dash of clever engineering.
In any case, no matter what people think about systemd, it's not going away anytime soon. It is certainly not going away unless somebody comes up with something that the major distributions consider obviously better than systemd, and by now that is a pretty high bar to clear (especially since being “only” an init system replacement is no longer enough; to compete successfully with systemd there needs to be some proposition for the overall basic-plumbing aspect of things). It's still not entirely inconceivable, but somebody will eventually have to start writing code, rather than LWN comments about how terrible systemd is.
Posted Nov 6, 2018 8:41 UTC (Tue)
by ras (subscriber, #33059)
[Link] (8 responses)
As a Debian Developer watching on when this happened, what they "did wrong" had nothing to do with the merits of the init system. ConsoleKit and a few other things were bit rotting badly at the time, and logind, dbus and a few other things got moved into the same source ball as systemd. I remember wondering consolekit and logind did that was so important - it turned out the most important thing is gnome needed something to give the local user (ie the one running the X server) permission to access local resources (like the sound card, and the keyboard), and gnome decided to use what systemd provided. (As I recall gnome didn't have a whole pile of alternatives to choose from.)
Gnome was Debian's default Window manager, so it was game over.
I use lxde on my laptop, and administer a whole pile of servers. Systemd works, but what has to happen is so trivial I don't have a clue what that 10MB of code in systemd could possibly be doing. I'm pretty sure I would of been happier with something much simpler. In fact I needed to boot a Debian chroot on a system without systemd once, and as this article points out the sysv scripts are indeed bit rotting so I wrote a shell script to interpret systemd's .service files for a headless server. It was under 300 lines. It makes the 1/2 a million lines in systemd a bit of a puzzle.
Posted Nov 6, 2018 10:37 UTC (Tue)
by lkundrak (subscriber, #43452)
[Link] (6 responses)
It's more like 1 MB: If you also count other tools from the systemd repository then their file name usually gives a good clue about what the tool does. See: systemd-resolved -- a resolver. systemd-timedated -- manages time and date. systemd-hostnamed -- manages host name. Et cetera. Also, both the init daemon and the supplementary tools are thoroughly documented. If you're really curious I wholeheartedly recommend trying out the man utility. It could turn out useful in your "administering a pile of servers" endeavour. In an unlikely case the documentation won't be sufficient, you should know that the source is available for free and is perhaps one of more reasonable C programs that you'll find running on a typical Linux system.
Hope this helps.
Posted Nov 6, 2018 22:36 UTC (Tue)
by ras (subscriber, #33059)
[Link] (5 responses)
I was referring to the installed size reported by Debian. If you are going to use size, you should at least include the additional 1.5MB in libsystemd-shared-232.so. If you want just count binary size, you should probably be include everything in /lib/systemd, which is over 5MB. That misses a bit of it installed elsewhere of course. The single figure is maybe less accurate, but is a damned sight easier to come by and I doubt it effects the error margin overly.
But if you insist on using systemd + libsystemd-shared's binary size, it's 2.5MB versus 34K for a sysV init I have lying around.
> Hope this helps.
Not really. You''re talking to someone who has an accepted pull request for the systemd documentation. You are, right there is a *lot* of documentation, and it's very good. But don't underestimate the value of only something that does a similar job that only needs a few pages of documentation.
Posted Nov 6, 2018 22:39 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link] (4 responses)
Posted Nov 6, 2018 23:18 UTC (Tue)
by ras (subscriber, #33059)
[Link] (3 responses)
True, but these are usually there anyway, and on the systemd side of the ledger I didn't count libc or systemd's other 20 shared dependencies.
The Debian installed size is so much easier. Looking on an old machine, the installed size of sysV init and ancillary stuff is about 900KB. Adding inetd and cron increases it by 0.4MB. 1/10th the size does seem about right.
This stuff is not that hard. I wrote an "house keeping" system for a container as a dash script. It booted, started and stopped services, provided timers, did scheduled backups, rotated log files, maintained a status web page for monitoring and provided a framework for building containers using it. It was 3K lines and would do a fine job of managing my laptop.
Not all init system's can be simple. An init system for clusters like kubernetes is complex for reasons. But for a single machine with a couple of tasks, it's damned hard to see what added value systemd's complexity brings.
Posted Nov 6, 2018 23:41 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
Posted Nov 6, 2018 23:58 UTC (Tue)
by ras (subscriber, #33059)
[Link] (1 responses)
How did you pull that off? On Debian stretch systemd + libsystemd-shared-232.so alone are 3.3MB.
Hmmm. Looking at the debian archives the systemd v44 .orig.tar.gz ball dates from 6 years ago, and is 860K. systemd v239 .orig.tar.gz is 6.8MB. Have you noticed a 8 fold increase in functionality?
Posted Nov 11, 2018 11:24 UTC (Sun)
by mpr22 (subscriber, #60784)
[Link]
Most trivially, v239 includes udev (whose tarball at the time of the merger in v183 was about 70% the size of the systemd tarball) and the udev HWDB (which was merged later, and is itself pretty huge).
Posted Nov 11, 2018 10:13 UTC (Sun)
by ms_43 (subscriber, #99293)
[Link]
There was an opportunity to contribute the necessary maintenance and bug fixes to ConsoleKit to keep it working, but instead the anti-systemd camp decided to troll the Debian technical committee decision process. I'm very happy that this turned out not to be a successful strategy.
[1] https://github.com/systemd/systemd/compare/v29...v30
Posted Nov 6, 2018 17:44 UTC (Tue)
by rweikusat2 (subscriber, #117920)
[Link] (1 responses)
Posted Nov 7, 2018 1:13 UTC (Wed)
by anselm (subscriber, #2796)
[Link]
That would be to the point if the claim was that systemd is good because it is popular. A more likely proposition, however, is the converse: that systemd is popular because – at least in the eyes of the developers in most mainstream Linux distributions who decide what init system to default to – it is good.
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
Nope. systemd's flash of brilliance was the use of cgroups to isolate daemons, which allowed it to actually reliably track them. For the first time ever. No other init system did this before.
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
I don't remember other init systems doing this, though. It's brilliantly simple like many other inventions.
Uhm, systemd was released in 2010. At best this is an example of convergent evolution.
The problem here is that SysV (and its ilk) doesn't really have a parent that can be used to manage its children. For this to work you need a specialized persistent process that stays up at all time, because if it dies all of its children will also be killed.
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
If sysvinit was extended to, say, 128 run levels to handle dependencies, actually used as intended, and possibly extended to facilitate enabling/disabling/activating/deactivating daemons without needing to edit inittab, sysvinit would do everything a lot of people need without a lot of complexity.
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
systemd is one of half a gazillion replacement inits written in $compiled_language which roughly all provide about the same feature set (and make the same errors), there's nothing special about it beyond "it's the RedHat replacement init!".
That's not much different from what systemd and other members of the zoo offer, it's just more flexible in that it retains a fully capable high-level, standardized programming language as backbone engine instead of some non-extensible, declaratory language
NO TO SYSVINIT - or initscripts, rather!
> could avail themselves of, [...] but nobody except Lennart Poettering and Kay Sievers actually condescended to doing that in
> practice.
NO TO SYSVINIT - or initscripts, rather!
to come up with a set of reusable “support tools written in C” that init systems could avail themselves of, and make these popular across distributions, but nobody except Lennart Poettering and Kay Sievers actually condescended to doing that
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
> I don't have a clue what that 10MB of code in systemd could possibly be doingNO TO SYSVINIT - or initscripts, rather!
$ size /usr/lib/systemd/systemd
text data bss dec hex filename
1238373 259368 416 1498157 16dc2d /usr/lib/systemd/systemd
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
But then you need bash/dash and a ton of other utilities (sed, awk, ...) for the initscripts. You can mostly get away with busybox these days, which is around 2MB.
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
[2] https://packages.debian.org/sid/dbus
NO TO SYSVINIT - or initscripts, rather!
NO TO SYSVINIT - or initscripts, rather!
