Thanks [LWN.net]

Thanks

Posted Nov 19, 2014 15:39 UTC (Wed) by tjc (guest, #137)
In reply to: Thanks by gb
Parent article: Today's Debian technical committee resignation: Ian Jackson

> anyway now we all must go and use systemd in our work.

Not necessarily; there are still alternatives for those who don't care for systemd: Slackware, Gentoo, Crux -- perhaps others that I don't know about. And one can always switch to FreeBSD, PC-BSD, NetBSD, etc. without too much trouble. systemd is dominant, but not exclusive.

to post comments

Thanks

Posted Nov 19, 2014 16:19 UTC (Wed) by moltonel (subscriber, #45207) [Link] (1 responses)

Dare I mention that those distributions have plenty of other things going for them than just not being tied to systemd ? :)

Gentoo user here. Go have a look, the small compile time pain is largely worth the gain (no, this is not about performance). And you can use whichever init system you prefer.

Thanks

Posted Nov 20, 2014 11:01 UTC (Thu) by deepfire (guest, #26138) [Link]

Gentoo is passe, NixOS is the declaratively-defined new hotness nowadays : -)

A noticeable chunk of the haskell community is behind NixOS -- to the point that next-gen haskell.org infrastructure is being built on it..

Thanks

Posted Nov 19, 2014 16:33 UTC (Wed) by seyman (subscriber, #1172) [Link] (14 responses)

> there are still alternatives for those who don't care for systemd

There's also the possibility of forking an existing init system (or creating one from scratch) and making it better than systemd. It's a lot of work but it can be done.

Thanks

Posted Nov 21, 2014 15:29 UTC (Fri) by tjc (guest, #137) [Link] (13 responses)

I guess if it comes to a fork, Upstart would be the most viable candidate. Canonical has a copyright on the name, so that would have to be changed, but the code is under the GPLv2. That might be better than starting over yet again, unless, of course, someone is just itching to write an init system. I'm all in favor of writing code for the sake of writing code; I have no delusions of world dominion. :)

Thanks

Posted Nov 21, 2014 15:59 UTC (Fri) by anselm (subscriber, #2796) [Link] (12 responses)

That might be better than starting over yet again, unless, of course, someone is just itching to write an init system.

The problem with Upstart is that it has severe shortcomings that – at least according to its original lead developer – would require a major redesign to overcome. So it seems that you would in effect have to be “itching to write an init system” in order to make Upstart a viable long-term proposition.

This would be especially ludicrous given that systemd is basically such a redesign already, and has both a fairly vibrant developer community and widespread buy-in from mainstream distributions, so is likely to stick around unless something radically better comes along, which at this point is unlikely. Given this, it would probably make much more sense to fork systemd rather than Upstart.

Thanks

Posted Nov 21, 2014 18:48 UTC (Fri) by tjc (guest, #137) [Link] (10 responses)

> The problem with Upstart is that it has severe shortcomings that – at least according to its original lead developer – would require a major redesign to overcome.

What are the shortcomings? I've been using Upstart for several years without incident, but I don't require much of an init system (other than to "init").

Thanks

Posted Nov 21, 2014 19:31 UTC (Fri) by mjg59 (subscriber, #23239) [Link] (6 responses)

https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/4... is an example.

Thanks

Posted Nov 22, 2014 16:10 UTC (Sat) by smurf (subscriber, #17840) [Link] (5 responses)

To summarize: While an event-based init system seemed like a good idea at the time, upstart quickly ran into a whole lot of corner cases which are basically unsolvable without adding job state as separate concepts. Upstart has tried working around this by adding even more events (like "starting to start" and "finished starting") but they don't help in general.

Given this fact, a dependency-based system which uses events just to change the units' state simplifies the whole system and affords easier debugging, as it's statically introspectable – meaning you don't need an event history to figure out what's wrong, just the current job states and the dependency graph.

Thanks

Posted Nov 22, 2014 21:14 UTC (Sat) by mgb (guest, #3226) [Link] (4 responses)

Forgy invented http://en.wikipedia.org/wiki/Rete_algorithm in the 70's and it was widely adopted in the early 80's.

Sad to see the wheel painfully being reinvented again.

Thanks

Posted Nov 22, 2014 21:46 UTC (Sat) by smurf (subscriber, #17840) [Link] (1 responses)

Dependency resolution in an init system doesn't need anything remotely this elaborate. We don't even need alternates (what for?).

Thanks

Posted Nov 22, 2014 22:08 UTC (Sat) by mgb (guest, #3226) [Link]

> Dependency resolution in an init system doesn't need anything remotely this elaborate.

Rete is not elaborate. It is a clean elegant modular algorithm.

> We don't even need alternates (what for?).

Alternate networks. Sometimes alternate data stores. Alternate entropy sources - you want sshd ASAP but maybe not until you have some entropy. "If I can't get any entropy bring up sshd only on the private LAN and if that doesn't work fall back to telnetd on the private LAN."

As the "new" init systems try to invade serious servers they will keep on running into problems and adding more keywords and reliving the 70's until they eventually relearn the lessons of the early 80's.

Thanks

Posted Nov 22, 2014 23:33 UTC (Sat) by mjg59 (subscriber, #23239) [Link] (1 responses)

How would you apply this to an event-based init system?

Thanks

Posted Nov 23, 2014 0:15 UTC (Sun) by mgb (guest, #3226) [Link]

> How would you apply this to an event-based init system?

Your own http://lwn.net/Articles/622742/ answers that.

If you want functionality beyond inittab, an event-based init system is even worse than an ad-hoc dependency-based init system that has to keep adding keywords like RequiresMountsFor.

Thanks

Posted Nov 21, 2014 20:14 UTC (Fri) by Felix (subscriber, #36445) [Link] (2 responses)

jspaleta also compiled an impressive list https://lwn.net/Articles/582585/

Thanks

Posted Nov 21, 2014 21:02 UTC (Fri) by jspaleta (subscriber, #50639) [Link] (1 responses)

One request...

If someone does fork upstart and fix all the deep deep design problems. Please rename it as a new project. What you end up building will be a very different animal.

I have a couple of humble name suggestions:
upyours, upset, upchuck

-jef

Thanks

Posted Nov 22, 2014 15:23 UTC (Sat) by zuki (subscriber, #41808) [Link]

Funny... but not quite funny enough. Please give it a rest.

Thanks

Posted Nov 21, 2014 22:31 UTC (Fri) by seyman (subscriber, #1172) [Link]

> Given this, it would probably make much more sense to fork systemd rather than Upstart.

I don't see how this will win over the people who are complaining (assuming anything can, at this point). OpenRC always seemed the better choice, imho.

Thanks

Posted Nov 19, 2014 17:42 UTC (Wed) by gb (subscriber, #58328) [Link] (1 responses)

> Not necessarily; there are still alternatives for those who don't care for systemd: Slackware, Gentoo, Crux -- perhaps others that I don't know about. And one can always switch to FreeBSD, PC-BSD, NetBSD, etc. without too much trouble. systemd is dominant, but not exclusive.

I am tied to Linux not just as a hobby, but also at work. Debian at home, Red Hat at work. Every day I develop software for Linux on Linux, deploy software, packaging software, parsing logs, etc. This all soon be changed by systemd, wayland. New Linux. I hope it wouldn't like Gnome 3.

Thanks

Posted Nov 22, 2014 10:44 UTC (Sat) by Wol (subscriber, #4433) [Link]

Well, the systemd guys seem to view backwards compatibility as important. If you want to use systemd pid1 just to fire off your old SysVInit scripts and not use their new config files, that's a design goal. If it doesn't work, file a bug report.

And Wayland is being developed by the X guys. In effect it's X13 (seeing as the moniker X12 is already taken). So again, any failure in compatibility is a good candidate for a bug report.

Cheers,
Wol

Thanks

Posted Nov 19, 2014 18:08 UTC (Wed) by jonnor (guest, #76768) [Link] (31 responses)

I also think there is a pretty good chance that Debian will work ok without systemd for the forseeable future for a lot of usecases, now that the discussions are (hopefully) over and people can spend their energy on solving technical problems.

Thanks

Posted Nov 19, 2014 23:12 UTC (Wed) by andreashappe (subscriber, #4810) [Link] (30 responses)

I believe it will work until socket/service-activated services will get used more (if ever) -- otherwise it should be possible to create those init-scripts from systemd's configuration.

Thanks

Posted Nov 20, 2014 1:42 UTC (Thu) by JdGordy (subscriber, #70103) [Link] (29 responses)

At which time someone will make another shim so they work or everyone will wonder why the heck the community had such an ugly argument over this.

Thanks

Posted Nov 22, 2014 15:31 UTC (Sat) by zuki (subscriber, #41808) [Link] (28 responses)

> At which time someone will make another shim
One could argue that inetd/xinetd already do this, although not with the systemd protocol.

More recent systemd-activate from the systemd tree implements most of socket activation protocol including Accept=yes/no and multiple sockets (not all socket types, e.g. netlink is missing but that could be fixed). It is not tied to systemd as PID 1, so it could be used a shim easily.

Thanks

Posted Nov 22, 2014 20:19 UTC (Sat) by rodgerd (guest, #58896) [Link] (27 responses)

It's a reasonable argument.

(It would also be interesting to know how many "traditional Unix is the ONLY WAY" actually run ssh or their web servers out of inetd.)

Thanks

Posted Nov 23, 2014 1:29 UTC (Sun) by zuki (subscriber, #41808) [Link] (26 responses)

I think socket-activated sshd makes a lot of sense... Not the extreme of running a separate instance for each connection (Accept=yes in systemd parlance), but a single instance with delayed activation. For many notebooks, workstations, and dedicated servers sshd is used never or rarely, so not starting it unconditionally is a nice optimization. For this to really work nicely, sshd should learn to exit-on-idle. Then you get the best of both worlds.

Thanks

Posted Nov 23, 2014 3:05 UTC (Sun) by mgb (guest, #3226) [Link] (20 responses)

You save a few bytes of swap space and increase the likelihood of having to drive to the data center at zero dark thirty.

Not for me, thanks.

Thanks

Posted Nov 23, 2014 7:31 UTC (Sun) by smurf (subscriber, #17840) [Link] (19 responses)

How does socket-activating sshd increase any likelihood of problems?

I'd assume that this would decrease it, since this way you do not risk starting it too early (/var/log not yet mounted, or whatever else might cause sshd to fail to start), nor too late (i.e. not at all, if something earlier in the boot sequence fails)

Thanks

Posted Nov 23, 2014 7:42 UTC (Sun) by mchapman (subscriber, #66589) [Link] (18 responses)

> How does socket-activating sshd increase any likelihood of problems?

If the system is close to being out of memory, then I would expect a socket-activated SSH to be less likely to get you to a shell than a non-socket-activated SSH.

If your only remote access to the system is with SSH, and remotely rebooting it is infeasible or impractical, then this could be a concern.

Thanks

Posted Nov 23, 2014 9:01 UTC (Sun) by mgb (guest, #3226) [Link] (12 responses)

Also for handling upgrade problems.

Once your sshd process exists you don't want it to disappear unexpectedly.

Thanks

Posted Nov 23, 2014 9:29 UTC (Sun) by mchapman (subscriber, #66589) [Link]

No, I don't think socket-activation changes anything there. When you upgrade SSH, the package's post-installation scripts should cause it to be restarted if it's running as a non-socket-activated service.

Thanks

Posted Nov 23, 2014 16:37 UTC (Sun) by flussence (guest, #85566) [Link] (8 responses)

Reliably upgrading sshd doesn't have much in common with socket activation. You can still do the normal "restart daemon, log in on new sshd to verify it works, log out of old sshd when it's safe" routine, though that of course renders this micro-optimization pointless.

On the other hand, a system using cgroups to indiscriminately purge entire process ancestries will create exactly the problem you're describing... by design!

Thanks

Posted Nov 23, 2014 16:57 UTC (Sun) by mgb (guest, #3226) [Link] (5 responses)

> On the other hand, a system using cgroups to indiscriminately purge entire process ancestries will create exactly the problem you're describing... by design!

Exactly.

A hypothetical modular portable dependency-based init could have been a useful tool but systemd is the opposite of a tool - it is a restrictive imposition. Systemd's short-sighted design is extraordinarily counter-productive and this becomes glaringly obvious as systemd attempts to move from its desktop niche to the critical server and embedded spaces.

Thanks

Posted Nov 23, 2014 17:53 UTC (Sun) by zuki (subscriber, #41808) [Link] (4 responses)

>> Reliably upgrading sshd doesn't have much in common with socket
>> activation. You can still do the normal "restart daemon, log in on new
>> sshd to verify it works, log out of old sshd when it's safe" routine,
>> though that of course renders this micro-optimization pointless.
Not if exit-on-idle is implemented. If it is, you can do a test login after upgrade, and still have the process go away after while.

I agree that for a server this is pointless, but let's say that you are running containers or lightweight VMs, in multiple instances. Then avoiding starting the process can be a nice optimization.

>> On the other hand, a system using cgroups to indiscriminately purge
>> entire process ancestries will create exactly the problem you're
>> describing... by design!
> Exactly.
I don't grok. When an SSH session is started, the ssh instance and user process forked off it are moved to a separate scope unit, which of course also means a different subtree in the cgroup hierarchy. So individual sessions are completely independent of the ssh service. As for the sshd service itself, yes, all processes in it are stopped during restart. That pretty much describes the basic functionality of systemd achieved with cgroups. If you think something is wrong with "purging" all processes of a service when it is stopped, please explain.

[snip the rant]

Thanks

Posted Nov 23, 2014 18:06 UTC (Sun) by mgb (guest, #3226) [Link] (3 responses)

> If you think something is wrong with "purging" all processes of a service when it is stopped, please explain.

You should try pre-systemd Debian Stable which doesn't kill existing sshd connections during upgrades. It's great.

Thanks

Posted Nov 23, 2014 18:15 UTC (Sun) by zuki (subscriber, #41808) [Link] (2 responses)

>> If you think something is wrong with "purging" all processes of a
>> service when it is stopped, please explain.
> You should try pre-systemd Debian Stable which doesn't kill existing sshd
> connections during upgrades. It's great.

Once again: existing sshd connections *are* *not* *part* of sshd.service.
After they are established they are independent and are not touched when sshd.service is stopped or restarted.

(It is possible that there's a bug in your Debian package or setup or whatever... I can only say that it works for me and apparently for most people, and of course is *designed* to work this way. If it doesn't work for you, please provide the details and we'll work on a fix. Probably best to do this on the distribution bugtracker rather than here though.)

Thanks

Posted Nov 23, 2014 18:43 UTC (Sun) by mpr22 (subscriber, #60784) [Link] (1 responses)

Asking mgb for details about specific failures of systemd-based systems is a waste of electrons; they have explicitly stated on this site a refusal to use systemd or help debug systemd.

Thanks

Posted Nov 25, 2014 16:31 UTC (Tue) by nix (subscriber, #2304) [Link]

It's also clear from this subthread that mgb doesn't bother to actually read posts to which it responds: rather, it scans them for things that can be attacked, and ignores everything else, including inconvenient facts that might contradict its rants.

Thanks

Posted Nov 23, 2014 23:28 UTC (Sun) by mchapman (subscriber, #66589) [Link] (1 responses)

> On the other hand, a system using cgroups to indiscriminately purge entire process ancestries will create exactly the problem you're describing... by design!

Except it doesn't. There are two reasons for this.

First, pam_systemd will move the SSH child process into a different cgroup.

Second, even if it didn't do this, the sshd.service contains KillMode=process, which means only the main process is killed upon stop or restart. Other processes in the sshd.service cgroup are unaffected.

Thanks

Posted Nov 24, 2014 20:16 UTC (Mon) by smurf (subscriber, #17840) [Link]

This did happen, once upon a time. It was a bug. It was fixed rather quickly, for obvious reasons.

Too bad (from the point of view of the anti-systemd crowd, that is).

Thanks

Posted Nov 24, 2014 20:18 UTC (Mon) by smurf (subscriber, #17840) [Link] (1 responses)

It won't. Socket-activated processes don't get killed because they're idle. systemd ignores the socket while the process is running, so how could it know?

Thanks

Posted Nov 24, 2014 20:48 UTC (Mon) by mgb (guest, #3226) [Link]

> Socket-activated processes don't get killed because they're idle.

The subthread to which you are responding is concerned with zuki's suggestion http://lwn.net/Articles/622790/ that sshd should exit on idle.

Thanks

Posted Nov 24, 2014 13:02 UTC (Mon) by james (guest, #1325) [Link] (1 responses)

I'd expect a socket-activated sshd that is activated from init to be more likely to work (perhaps after a minute or two), simply because init won't be killed by the OOM killer.

Thanks

Posted Nov 25, 2014 16:32 UTC (Tue) by nix (subscriber, #2304) [Link]

sshd also takes precautions against being OOM-killed -- but for most other daemons your point is valid.

Thanks

Posted Nov 24, 2014 18:33 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

It's quite likely for OOM killer to kill your sshd, especially if it's inactive. Sure, you can play with OOM killer settings to make sure that your sshd (which is usually dropbear ssh) is not killed.

But of course, it's easier to do with systemd and socket-activation is useful because it will restart your daemon in case of failures...

Thanks

Posted Nov 24, 2014 19:27 UTC (Mon) by rodgerd (guest, #58896) [Link] (1 responses)

Of course, systemd makes it trivial to mark OOM settings on a per-service basis, too.

Thanks

Posted Nov 24, 2014 20:00 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

I'm actually thinking that combining socket activation and adjusting OOM killer score to make it _more_ likely to kill sshd is an even better strategy. This way, sshd will be killed first and will free up RAM that is likely wasted.

There's a slight possibility of a livelock when sshd is killed immediately after it's restarted by systemd. Perhaps the "OOMAdjustDelay" setting can be added to systemd?

Thanks

Posted Nov 23, 2014 9:32 UTC (Sun) by rodgerd (guest, #58896) [Link] (4 responses)

For some cases. Of course, OpenSSH isn't very traditional Unix. It's one project that replaces telnet, rlogin, rsh, rcp, and FTP. Why have a big monolithic blob? Shouldn't it be run as a set of loosely-coupled projects with seperate repositories?

The proper Unix way would be to have all those traditional services spawn out of inetd and use tools like tcpwrappers for access control; if you really need security you should be using something like stunnel rather than building it all into the basic tool. All that code living together is obviously a much bigger attack service.

And have you seen the attitude of the maintainer? He rejects portability patches and keeps saying rude things about other platforms!

All of this is against traditional Unix principles and will be a disaster for Unix, mark my words.

Thanks

Posted Nov 23, 2014 18:35 UTC (Sun) by dlang (guest, #313) [Link] (3 responses)

actually, the fact that ssh is telnet + ftp + vpn is an ongoing problem for security people who would like to allow some of this capability without allowing it all.

Thanks

Posted Nov 24, 2014 20:22 UTC (Mon) by smurf (subscriber, #17840) [Link] (2 responses)

It is, like, _so_ difficult to turn the unwanted features off in sshd_config, no?

Thanks

Posted Nov 25, 2014 2:22 UTC (Tue) by dlang (guest, #313) [Link] (1 responses)

yes, it's extremely hard to turn features off for some users while allowing it for others.

Thanks

Posted Nov 25, 2014 16:33 UTC (Tue) by nix (subscriber, #2304) [Link]

The keyword you're looking for is 'Match'. You can PermitTunnel on a group-by-group, IP-by-IP, user-by-user, or even local-port-by-port (?!) basis.