LWN: Comments on "Debian TC vote on init system coupling"

Debian TC vote on init system coupling

anselm — Sun, 09 Mar 2014 13:25:24 +0000

This is particularly useful because in the systemd setup the configuration files provided by the distribution are cleanly separated from the local settings made by the administrator.

With something like System-V init, everyone unloads their init files in /etc/init.d, and it is difficult for an administrator to (a) see whether any of these scripts have indeed been changed to better suit local preferences, and (b) maintain such changes across updates from the distributor. Systemd makes this much more obvious because anything under /etc is by definition a local change, and updates to the distributor-provided configuration in /usr do not run the risk of either obliterating the local changes or else requiring a tedious manual merge.

Debian TC vote on init system coupling

rahulsundaram — Sun, 09 Mar 2014 05:13:59 +0000

Agreed on everything except the last part. There is no particular reason defaults shipped by the distribution needs to be part of /etc at all and systemd is hardly the only program shipping system configuration defaults in /usr. Dozens and dozens of programs do that. In systemd, you can override only what you want to:

Ex:

--
/etc/systemd/system/httpd.service.d/restart.conf

[Service]
Restart=always
RestartSec=30

man systemd.unit more details

Debian TC vote on init system coupling

dlang — Sun, 09 Mar 2014 04:19:29 +0000

Service startup is an administrator level activity, so configurations for service startups should be under /etc, including ones modified by the local administrator and those that are system defaults.

Debian TC vote on init system coupling

anselm — Sat, 08 Mar 2014 23:42:10 +0000

Here's your system administration convention:

/etc = stuff the local administrator may change
/usr = stuff the local administrator shouldn't change

The distribution provides basic systemd units below /usr, and local changes to those systemd units, or all-new systemd units, go into /etc. Systemd figures out how these go together, and provides tools that make it easy to find the effective configuration that is actually being used. It is actually quite nifty.

Exactly how does systemd »throw away« anything? If anything, System V init is ignoring the convention by putting loads of stuff into /etc that is really quite difficult to change without producing a maintenance nightmare.

Debian TC vote on init system coupling

raven667 — Sat, 08 Mar 2014 18:29:24 +0000

Just because a decision is stylistic or subjective doesn't mean it's arbitrary, it means that there are pros and cons of each approach and depending on how you weigh the assumptions you may logically arrive at different answers from other people.

Certainly in this case there are pros and cons of each approach, which I tried to highlight and provide some guesses of possible rationale for each, none of which was "just because".

Debian TC vote on init system coupling

rahulsundaram — Sat, 08 Mar 2014 18:21:45 +0000

You are making vague assertions 'just because'.

Debian TC vote on init system coupling

dlang — Sat, 08 Mar 2014 18:15:07 +0000

and systemd continues to throw away system administration conventions 'just because'

Debian TC vote on init system coupling

rahulsundaram — Sat, 08 Mar 2014 17:05:35 +0000

The distinction between /etc/ which is user configuration and /usr which is what the distribution provides isn't merely a stylistic choice. If more programs adopt this convention, it makes it much more easier to understand what customizations have been done or snapshotting etc. There are other ways of handling it including storing /etc in a revision control system and package management specific tweaks but I would prefer if such ideas were baked in as common conventions.

Debian TC vote on init system coupling

raven667 — Sat, 08 Mar 2014 15:53:13 +0000

I don't think this person is disagreeing with you about splitting the config files, the only concern, which is a subjective, stylistic one, is that the default files aren't in /etc, so a new administrator is going to look in /etc/systemd for configs and see some empty directories and have to search around more to find where the defaults are stored, if the defaults were also in /etc/systemd, in a subdirectory or whatever then they'd be slightly more discoverable. The downside is that it is more inviting to edit the default files rather than create override files which is going against the grain of how the files are organized, which is probably why it isn't done this way, to lower slightly the chance of a misconfig by someone editing the wrong file.

Debian TC vote on init system coupling

zdzichu — Sat, 08 Mar 2014 08:52:34 +0000

I personally believe the split is very useful, clearly separating customization done by administrator from distro defaults.
And systemctl make it dead simple to see where from configuration is gathered.

1) unit location is in the second line of status output:
$ systemctl status gdm
gdm.service - GNOME Display Manager
Loaded: loaded (/usr/lib/systemd/system/gdm.service; enabled)
…

2) If administrator does customistation by drop-in snippets, all are listed in next line:
$ systemctl status privoxy
privoxy.service - Privoxy Web Proxy With Advanced Filtering Capabilities
Loaded: loaded (/usr/lib/systemd/system/privoxy.service; enabled)
Drop-In: /etc/systemd/system/privoxy.service.d
└─after-tor.conf

3) administrator can review his own changes in /etc/systemd/system. To see final unit, ”systemctl cat” will merge all the changes and present the unit from systemd's point of view.

It is actually quite easy and clear to see what's going on.

Debian TC vote on init system coupling

elvis_ — Sat, 08 Mar 2014 06:25:12 +0000

It is an artificial distinction, unit files are configuration wherever they go. It just makes it harder for users to figure out what is going on when they are hidden away from /etc

I would much prefer unitfile.default and unitfile.localchanges in the same directory.

Debian TC vote on init system coupling

dlang — Mon, 03 Mar 2014 06:38:42 +0000

Linus talked about this sort of thing a lot in the early git days. He sees it as a high priority that he can be replaced by people just pointing at some other master git tree. Along similar lines, he has said that he expects that someday there will be a replacement for Linux that's created by someone making a stripped down, more efficient replacement.

Linus also rants about the need for new features to default to "off", that no new feature is worth hurting existing users over, and that the users (i.e. admins) need to have the options available to them to not do something that the developers think is neat and important.

Systemd development seems just the opposite, they know better than the users what's important, don't use systemd in any way other than the one they intend you to, etc.

Debian TC vote on init system coupling

rahulsundaram — Sun, 02 Mar 2014 14:26:05 +0000

*user* configuration should go into /etc and it does.

Debian TC vote on init system coupling

elvis_ — Sun, 02 Mar 2014 09:26:08 +0000

My one wish is that all the config files go in /etc, not scattered all over the system. It drives me insane.

Debian TC vote on init system coupling

k8to — Sat, 01 Mar 2014 06:48:40 +0000

This was in the era where Red Hat 5 or 6 (whichever shipped with glibc) segfaulted in common binaries on fresh install with no hardware errors.

Really everything was pretty fubar around that transition point, even without upgrades, so I wouldn't put too much weight on that scenario from 1997 or so.

Debian TC vote on init system coupling

khim — Fri, 28 Feb 2014 23:33:05 +0000

Indeed. Cgroups are designed to be unescapable and non-intrusive means of managing process groups thus they are the only logical choice for the init system, but if you don't really care about making usable system and only want to create safe system there are many possibilities. Besides ptrace there are seccomp (and no, not this new fancy seccomp-bpf but plain old seccomp: it's enough, surpisingly enough). You can use it to create completely safe init system. Why this is bad idea left as an excercise for a reader.

Debian TC vote on init system coupling

mgb — Fri, 28 Feb 2014 23:23:58 +0000

"Bound sockets should exclusively come from the init system."

And all P2P S/W should be trash-canned.

Not.

P.S. Does systemd support AF_APPLETALK?

Debian TC vote on init system coupling

mjg59 — Fri, 28 Feb 2014 22:35:23 +0000

ptrace a process that's running in another cgroup, ask it to exec another copy of you?

Debian TC vote on init system coupling

fandingo — Fri, 28 Feb 2014 22:24:35 +0000

> But now that we've accepted that using cgroups is not actually reliable

I'm still looking for a concrete example of a service process escaping cgroups when using the systemd single-writer that lives in PID 1. The only thing I can come up with is a full kernel exploit, and that breaks everything. Even khim's worst-case scenario of root with unconfined_t, can't escape other processes from service cgroups. The manager will not allow it, and there's no way to replace the manager.

Debian TC vote on init system coupling

javispedro — Fri, 28 Feb 2014 21:57:30 +0000

Yes. But now that we've accepted that using cgroups is not actually reliable, we can go back to discussing why the other approaches have been readily rejected. Even ptrace, as ugly and hacky as it is, is as reliable (as long as the processes that are being monitored run under a lower privilege level).

Debian TC vote on init system coupling

fandingo — Fri, 28 Feb 2014 21:06:23 +0000

I agree. I'd really like to see blanket prohibition on services calling bind(skt). Bound sockets should exclusively come from the init system.

That being said, while the current processes are "root" mostly in name. SELinux (and other LSMs) for the most part have neutered these processes.

The real security threat comes from kernel exploits, and unfortunately, neither changing service users nor using a LSM sandbox helps prevent kernel exploits that much.

Debian TC vote on init system coupling

Cyberax — Fri, 28 Feb 2014 20:06:29 +0000

It's not a security issue, really.

A root process can simply move itself to another cgroup hierarchy, since it has access to cgroups.

And a single writer model won't really protect against it because a malicious root process can simply ptrace or replace the cgroups manager with a modified version that allows it to do anything.

Additional confinement is needed to fix this problem, in any case. Be it namespaces, SELinux, AppArmor or something else.

Debian TC vote on init system coupling

khim — Fri, 28 Feb 2014 20:00:38 +0000

Once you are unrestricted root the game is finished. End of story. You can do whatever you want with a system (using DMA in GPU works just fine for that). There are plenty of ways to restrict root and, surprise, surprise, most of them make it impossible to escape from cgroup as a nice added bonius. But it's not systemd's place to solve that problem. You can not on one hand complain about systemd “feature creep” and on the other hand complain that it does not work as intrusion detection and/or prevention system.

Debian TC vote on init system coupling

vonbrand — Fri, 28 Feb 2014 19:27:37 +0000

You can't exclude malicious code, a vulnerability might allow the attacker to execute arbitrary code.

Debian TC vote on init system coupling

vonbrand — Fri, 28 Feb 2014 19:15:43 +0000

Fix them?

At the very least, this reduces the attack surface to a handful of daemons...

Debian TC vote on init system coupling

mathstuf — Fri, 28 Feb 2014 18:57:12 +0000

Maybe this is the security flaw being mentioned that Cyberax is questing for? Do you have more details?

Debian TC vote on init system coupling

fandingo — Fri, 28 Feb 2014 18:50:44 +0000

Yes, you can do that with socket activation, but there are two problems. First, not everything supports socket activation. Httpd is an example that keeps a main process running as root.

Second, even some services that do, still run as root. openssh-server supports socket activation, but the main process is still root, and every session has a root process used for permission separation.

Debian TC vote on init system coupling

vonbrand — Fri, 28 Feb 2014 18:36:27 +0000

You configure the port in the unit file, no privileges in the daemon required at all.

Debian TC vote on init system coupling

fandingo — Fri, 28 Feb 2014 16:59:00 +0000

Most daemons are privileged for a portion of their lifetime, even if it's only to bind to a low numbered port.

Debian TC vote on init system coupling

fandingo — Fri, 28 Feb 2014 16:57:26 +0000

Just so we're clear, I'm talking about the new cgroup implementation (single writer) with systemd as the cgroup manager (PID 1).

Short of a kernel exploit, there is no way to move out of cgroups or to a non-allowed cgroup. The cgroup manager won't allow it, and there's no way even a malicious program could kill the cgroup manager and register as a new manager with the kernel because the manager is PID 1, making it unkillable.

Yes, cgroupfs is insecure because it allows arbitrary writers, and you absolutely need something like SELinux to keep the privileged portions of daemons from causing a jailbreak. Fortunately, this mode of operation will disappear soon (at least for those using systemd).

Debian TC vote on init system coupling

peter-b — Fri, 28 Feb 2014 16:42:26 +0000

> Any daemon running as root can trivially escape a cgroup. You need something stronger in order to actually ensure "reliability".

Why are you running daemons as root?

Debian TC vote on init system coupling

khim — Fri, 28 Feb 2014 11:43:33 +0000

Any daemon running as root can trivially escape a cgroup.

Really? Let's exclude explicitly written malicious code (if you run malicious code under root in any UNIX system you are hosed anyway). How does that happen?

You need something stronger in order to actually ensure "reliability".

You need something stronger to run potentially malicious code, sure. But that's totally different task: given kernel's track record you can not even run malicious code from unknown origin under regular user (you need some level of sandboxing like NaCl or seccomp) thus of course systemd does not try to solve that problem.

Debian TC vote on init system coupling

javispedro — Fri, 28 Feb 2014 10:58:19 +0000

Because systemd is much, much more than just daemon management (which is part of the criticism, btw). The parts of systemd related to process management have been reimplemented to death in e.g. openrc (support for both cgroups and jails), launchd ("process groups"), smf, etc.

Debian TC vote on init system coupling

javispedro — Fri, 28 Feb 2014 10:54:48 +0000

Any daemon running as root can trivially escape a cgroup. You need something stronger in order to actually ensure "reliability".

Debian TC vote on init system coupling

anselm — Thu, 27 Feb 2014 14:21:59 +0000

I think this is a conspiracy theory. Linus is trying to make Linux the best operating system it can be. Linus does not "fight for the ability of the user to choose to move away from Linux". Linus does not work on BSD, Solaris, OS X, or Windows, simply so Linux users can have alternatives to Linux. These exist because other people actively prefer them to Linux, to the point where it is worthwhile for them to go to the trouble of keeping them around. If Microsoft suddenly decided to base future versions of Windows on the Linux kernel, Linus would for sure not try to convince them otherwise just to keep Linux users' options open.

And exactly how does Lennart, by doing for systemd exactly what Linus does for Linux, namely trying to make it the best init system for Linux it can be, take away your option to stick with System V init, inetd, cron, syslogd, etc. for as long as you like, if you actively prefer them? How are distribution makers coerced to jump onto the systemd bandwagon against their better judgment? How are developers of background services forcibly brainwashed into believing that systemd's features may make their software simpler, more secure, and more efficient? Isn't systemd just as much of an open-source program as the Linux kernel, such that people can decide at some point that Lennart has gone off his rocker and they don't trust him any more, and take further development into their own hands?

It's not the fault of Lennart and his colleagues (other than by doing good work on systemd) if systemd is so convincing that most mainstream Linux distributions have decided they want to make it their default init system. If anything it is SysV init etc.'s fault for sucking so much in comparison that many people (and especially people in charge of distributions, who are the ones who matter most in this game) believe systemd is the better long-term option. Lennart does not have magical powers to make everybody bend to his will; if systemd is becoming the new standard it is on its technical merits and not through conspiracy. The Debian TC vote only emphasises this.

Debian TC vote on init system coupling

nye — Thu, 27 Feb 2014 11:29:37 +0000

>So no, you are misunderstanding me. Does the above quote make things more clear?

No, it makes things even less clear, since the T option there is even looser than the N option you are complaining is too loose.

That T option specifically encourages loose coupling where feasible, which I would have expected to upset you even more than allowing maintainers to choose as they see fit.

Debian TC vote on init system coupling

AngryChris — Thu, 27 Feb 2014 10:18:16 +0000

I am referring to this, nye:
Debian's technical committee starts another init system vote [LWN]

== dependencies rider version T (Tight coupling) ==

   Software may require a specific init system to be pid 1.

   However, where feasible, software should interoperate with
   all init systems; maintainers are encouraged to accept
   technically sound patches to enable interoperation, even if it
   results in degraded operation while running under the init system
   the patch enables interoperation with.

== dependencies rider version L (Loose coupling) ==

   Software outside of an init system's implementation may not require
   a specific init system to be pid 1, although degraded operation is
   tolerable.

   Maintainers are encouraged to accept technically sound patches
   to enable improved interoperation with various init systems.

So no, you are misunderstanding me. Does the above quote make things more clear?

Debian TC vote on init system coupling

AdamW — Thu, 27 Feb 2014 07:42:32 +0000

Fedora upgradability has little to do with this rather vague concept of 'plumbing' and a lot to do with the fact that upgradability is really hard, and we ship new Fedora releases with giant piles of new things in them every six months, and we never really developed a culture of offering particularly guaranteed upgradability anyway.

Debian has a much slower release cycle and, as I understand it, spends rather a lot of the stabilization portion of its cycle on dealing with niggling issues in upgrade paths.

Fedora's stabilization period is about two weeks if we're lucky. We (I work Fedora QA) get to spend about three days running some basic upgrade tests, if we're lucky.

We could certainly improve Fedora's upgradability quite a lot if we spent much longer on our releases, but that's not really what Fedora's for.

Even with all those caveats, though, Fedora upgradability has improved substantially since systemd arrived.

Debian TC vote on init system coupling

dlang — Thu, 27 Feb 2014 05:32:54 +0000

Linus fights for the ability of the user to choose, including the ability for the user to choose to move away from Linux, and the ability for kernel devs to decide that Linus has gone off his rocker and they don't trust him any more.

Lennart fights for the one true way, and anyone who don't like it needs to shut up and get with the program.

this basic attitude is very different.

Debian TC vote on init system coupling

viro — Thu, 27 Feb 2014 04:51:14 +0000

*shrug* If they need it, they can write it. For sysvinit it took me about a weekend (IIRC - that was 16 years ago) to write and debug and most of that consisted of writing the marshalling code. That they already have, apparently, so it ought to be a few hours worth of work, from start to finish, including building and testing. Not my problem, seeing that I'm not using systemd for a lot of reasons. And no, for me the lack or presence of such reexec functionality isn't anywhere near those.

Don't read into that comment anything other than a surprise at the decision to use an on-disk file for passing the state. All software is full of WTFs of that kind and the life is too short to fix ones in the stuff one doesn't use. I have more than enough fun with the kernel and with the userland I do use, TYVM. If I run across something of that kind in e.g. OpenBSD kernel, or djbdns, I'll probably comment on it and move on. Same as with systemd. Has nothing to do with my (lack of) liking of their authors, before anyone goes into that - if I run across something fishy in glibc or openssh, I *will* do more than commenting, no matter how little I am fond of Ulrich and Theo resp.

Same reason why I won't contribute to GNOME or emacs - I don't use either, they are not within 0.01% of programs that are pleasant to read, I'm not interested in developing them and it's not even what I'm paid for. Bugs in e.g. nvi are something I will deal with (and had done so in the past), something similar in aforementioned emacs... apt-get remove emacs\* solves them all nicely, as far as I'm concerned.