Poettering: Revisiting how we put together Linux systems

Posted Sep 2, 2014 19:18 UTC (Tue) by jackb (guest, #41909)
In reply to: Poettering: Revisiting how we put together Linux systems by Wol
Parent article: Poettering: Revisiting how we put together Linux systems

Except, as a gentoo user, I want to use systemd! Not because I use Gnome (I personally can't stand it), but because I'd like to have multiple stations on a single pc. Comes by default with systemd apparently, but OpenRC well I don't know - I got the impression it couldn't.

Likewise.

Except in my case, I really just want the feature of automatically restarting crashed daemons, and automatically funnelling all their console output into the journal.

I've made a lot of progress converting a lot of my (virtualized guest) machines to systemd, but it's been incredibly difficult and I can't convert the host machines due to bugs with systemd's dmcrypt/mdraid/lvm setup. (apparently they only test on Fedora or something)

I've had to deal with problems like how one release of systemd-networkd worked perfectly, but the next release it consistently failed to set an ip address as a dhcp client. No errors, no warning, no indication of what the problem might have been at all. After one update I had ~30 guest machines that couldn't get networking parameters that I had to manually log into and set up static IP addressing to get working again.

If it wasn't for the two features I listed at the beginning of this comment, I wouldn't even bother with systemd at all, and even given those improvements it's barely worth it to switch.

Poettering: Revisiting how we put together Linux systems

Posted Sep 2, 2014 20:20 UTC (Tue) by NightMonkey (subscriber, #23051) [Link] (22 responses)

"Except in my case, I really just want the feature of automatically restarting crashed daemons, and automatically funneling all their console output into the journal."

Those are behaviors that shouldn't be in the init system, if you like UNIX philosophical models of "do one thing and one thing well." These complicate an already complex job, better done by task-specific narrowly-scoped tools. Monit, Puppet, Chef, watchdog, and many other programs can do that simply defined task and do it well. And fix those crashing daemons! Crashing should never become accepted, routine program behavior! :)

For the console output, can't syslong-ng (or rsyslog, or similar) do that?

Poettering: Revisiting how we put together Linux systems

Posted Sep 2, 2014 20:51 UTC (Tue) by dlang (guest, #313) [Link] (6 responses)

no, syslog-ng and rsyslog do not capture stdout and stderr from programs that start on the system (they assume that if an application is generating output, it's because the programmer wants the user to see it instead if it going into some log somewhere)

There's nothing preventing you from running the output of any program into logger (or equivalent) to have that data sent to syslog-ng or rsyslog ( 2|logger -t appname.err |logger -t appname.stdout or something similar)

Poettering: Revisiting how we put together Linux systems

Posted Sep 4, 2014 15:20 UTC (Thu) by xslogic (guest, #97478) [Link] (5 responses)

Surely most daemons (...although I don't know about the systemd varieties...) close off stdin/stdout/stderr when they daemonise?

Poettering: Revisiting how we put together Linux systems

Posted Sep 4, 2014 15:51 UTC (Thu) by raven667 (subscriber, #5198) [Link] (4 responses)

"most" things you try to run as a daemon do the whole daemonization song-and-dance but not "all" things you want to run as daemons do so, systemd handles all cases correctly, not just the ones where a well-behaved daemon does all the right things.

Poettering: Revisiting how we put together Linux systems

Posted Sep 4, 2014 22:10 UTC (Thu) by Darkmere (subscriber, #53695) [Link] (3 responses)

Systemd actually encourages writing of "Daemons" that don't fork/close, but simply linger until done, which is wonderful as it copies the way "proper" daemons work with Runit and similar systems.

Poettering: Revisiting how we put together Linux systems

Posted Sep 4, 2014 22:36 UTC (Thu) by dlang (guest, #313) [Link] (2 responses)

so a"proper systemd" daemon acts differntly that a "proper Linux" daemon for any other init system?

why do so few people see the problems with this sort of thing?

Poettering: Revisiting how we put together Linux systems

Posted Sep 4, 2014 23:25 UTC (Thu) by anselm (subscriber, #2796) [Link]

The non-forking daemon approach as recommended for systemd is what basically every init system other than System-V init prefers (check out Upstart or, e.g., the s6 system mentioned here earlier). It allows the init system to notice when the daemon quits because it will receive a SIGCHLD, and the init system can then take appropriate steps like restart the daemon in question. In addition, it makes it reasonably easy to stop the daemon if that is necessary, because the init process always knows the daemon's PID (systemd uses cgroups to make this work even if the daemon forks other processes).

The »double-forking« strategy is needed with System-V init so that daemon processes will be adopted by init (PID 1). The problem with this is that in this case the init process does receive the notification if the daemon exits but has no idea what to do with it. The init process also has no idea which daemons are running on the system in the first place and where to find them, which is why many »proper Linux daemons« need to write their PID to a file just so the init script has a fighting chance of being able to perform a »stop« – but this is completely non-standardised, requires custom handling in every daemon's init script, and has a certain (if low) probability of being wrong.

In general it is a good idea to push this sort of thing down into the infrastructure rather than to force every daemon author to write it out themselves. That way we can be reasonably sure that it actually works consistently across different services and is well-debugged and well-documented. That this hasn't happened earlier is only too bad but that is not a reason not to start doing it now. People who would like to run their code on System-V init are free to include the required song and dance as an option, but few if any systems other than Linux actually use System-V init these days – and chances are that the simple style of daemon that suits systemd better is also more appropriate for the various init replacements that most other Unix-like systems have adopted in the meantime.

Poettering: Revisiting how we put together Linux systems

Posted Sep 5, 2014 8:34 UTC (Fri) by cortana (subscriber, #24596) [Link]

It seems unrealistic to expect all daemons to correctly implement all the steps mentioned under "SysV Daemons" here: http://www.freedesktop.org/software/systemd/man/daemon.html; particularly when some of them are written in languages that go out of their way to make such operations difficult or impossible (e.g., Java). Compare that list to the much simpler and easier-to-implement list under "New-Style Daemons".

Poettering: Revisiting how we put together Linux systems

Posted Sep 2, 2014 22:47 UTC (Tue) by jackb (guest, #41909) [Link]

Those are behaviors that shouldn't be in the init system, if you like UNIX philosophical models of "do one thing and one thing well." These complicate an already complex job, better done by task-specific narrowly-scoped tools. Monit, Puppet, Chef, watchdog, and many other programs can do that simply defined task and do it well. And fix those crashing daemons! Crashing should never become accepted, routine program behavior! :)

That's the kind of philosophy that's useless to me.

I have a lot of work to get done. I don't have time to fix all the broken daemons in the world.

I welcome tools that help me get my work done and reject tools that get in my way .

Unfortunately systemd is complicated because it contains a mixture of both so I'm always ambivalent.

Poettering: Revisiting how we put together Linux systems

Posted Sep 3, 2014 10:26 UTC (Wed) by cortana (subscriber, #24596) [Link] (13 responses)

> Monit, Puppet, Chef, watchdog, and many other programs can do that simply defined task and do it well.

If they do it by running '/etc/init.d/foo status' then, no, they can't.

Poettering: Revisiting how we put together Linux systems

Posted Sep 3, 2014 15:26 UTC (Wed) by NightMonkey (subscriber, #23051) [Link] (12 responses)

If your init scripts suck, then sure. In that case, you'd also have Nagios/Icinga checks making sure that N number of processes matching the name of the daemon are running at all times. But, this misses the overarching point I was making that there are better ways to achieve the goal than making your init system more fragile and brittle.

Poettering: Revisiting how we put together Linux systems

Posted Sep 3, 2014 15:46 UTC (Wed) by raven667 (subscriber, #5198) [Link] (11 responses)

The init system is not fragile, it is widely deployed and does not seem to be falling over, most of the ongoing work in in the ancillary tool box, like logind, and not in pid 1. Init scripts universally suck, even compared to service monitors like daemontools from the last century, claiming that periodic nagios checks are a replacement for having a real parent/child relationship with started services is silly, that's the near-beer version of service monitoring, not the real thing. Capturing and logging stdout/stderr and the exit code from a service are great features that don't exist in a standard fashion with init scripts, sure you could hack some of that together on an individual script basis but it's not a feature provided to you.

Poettering: Revisiting how we put together Linux systems

Posted Sep 3, 2014 16:09 UTC (Wed) by NightMonkey (subscriber, #23051) [Link] (10 responses)

Icinga/Nagios is not 'near-beer'. It has parent/child relationships, too. You are shouting "we must have a SYSTEM", when I don't know if that's the answer. People have just not used the tools correctly. This is just One More Thing To Go Wrong. And, yay, there will likely be a LPI exam level for figuring out what went wrong. ;)

More work is needed to make the relationship between users and developers LESS obscured, not more. When I reported a core Apache bug to the Apache developers in 1999, I had a fix in 24 hours, and so did everyone else. Now, if instead I just relied on some system to restart Apache, that bug might have gone unnoticed and unfixed, at least for longer.

Systems like this are a band-aid. Putting more and more complex systems in as substitutes for bug fixing and more human-to-human communication are what are the problem.

Poettering: Revisiting how we put together Linux systems

Posted Sep 3, 2014 16:23 UTC (Wed) by raven667 (subscriber, #5198) [Link] (9 responses)

> Icinga/Nagios is not 'near-beer'. It has parent/child relationships, too

I'm not sure what you are talking about nagios check_procs for eaxmple just shells out to ps to walk /proc, it is not the parent of services and doesn't have the same kind of iron clad handle to their execution state that something like runit or daemontools or systemd has.

> Systems like this are a band-aid.

You are never going to fix every possible piece of software out in the world to never crash, the first step is to admit that such a problem is possible, then you can go about mitigating the risks, not by building fragile systems that pretend the world is perfect and fall apart as soon as something doesn't go right, especially not as some form of self-punishment to cause pain to force bug fixes.

Poettering: Revisiting how we put together Linux systems

Posted Sep 3, 2014 16:34 UTC (Wed) by NightMonkey (subscriber, #23051) [Link] (8 responses)

I am prepared to be wrong about my experience-based opinions. :)

I just think a lot of this is because of decisions made to keep the binary-distro model going.

I'm not interested in fixing all the softwares. :) I am interested in getting the specific software I need, use and am paid to administer in working order. There are certainly many ways to skin that sad, sad cat. ;)

Poettering: Revisiting how we put together Linux systems

Posted Sep 4, 2014 22:05 UTC (Thu) by Wol (subscriber, #4433) [Link] (7 responses)

Well, can I just point you at an example (reported here on LWN first hand ...) of a bog-standard bind install. (Dunno the ref, you'll have to search for it.)

The sysadmin did a "shutdown -r". The system (using init scripts) made the mistake of shutting the network down before it shut bind down. Bind - unable to access the network - got stuck in an infinite loop. OOPS!

The sysadmin, 3000 miles away, couldn't get to the console or the power switch, and with no network couldn't contact the machine ...

If a heavily-used program like bind can have such disasters lurking in its sysvinit scripts ...

And systemd would just have shut it down.

Cheers,
Wol

Poettering: Revisiting how we put together Linux systems

Posted Sep 4, 2014 23:02 UTC (Thu) by NightMonkey (subscriber, #23051) [Link] (6 responses)

That a system architecture problem! Never put a system into production (or, hell, into 'staging') without remote power switches. And practice what happens when critical daemons go down (aka outage drills).

http://www.synaccess-net.com/

I don't care how "robust" the OS is. It's just being cheap that gets your organization into these kinds of situations (and that *is* an organizational problem, not just yours as the sysadmin.)

Poettering: Revisiting how we put together Linux systems

Posted Sep 4, 2014 23:15 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

Yeah, sure. So then you practice everything (and we did test it), and it works fine in testing. You schedule a time when nobody is around to upgrade the system without interruptions.

And then it breaks because of a race condition that happens in only about 10% of cases.

That sysadmin in this dramatization was me, and the offending script was: http://anonscm.debian.org/cgit/users/lamont/bind9.git/tre... near the line 92.

Of course, REAL Unix sysadmins must live in the server room, spending 100% of their time tweaking init scripts and auditing all the system code to make sure it can NEVER hang. NEVER. Also, they disable memory protection because their software doesn't need it.

Poettering: Revisiting how we put together Linux systems

Posted Sep 4, 2014 23:20 UTC (Thu) by NightMonkey (subscriber, #23051) [Link] (4 responses)

No need to take it personally. But, sad to say, you learned a good lesson the hard way in sysadmining - never rely on one system for anything vital. :)

Again, the answer to system hangs (which are *inevitable* - this is *commodity hardware* we're talking about most of the time, not mainframes) is remote power booters. I don't like living in the DC, myself.

Poettering: Revisiting how we put together Linux systems

Posted Sep 4, 2014 23:33 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

Lots of companies can't afford to make a completely redundant infrastructure. Especially if you have a fairly powerful server (that one was handling storage for CCTV streams and lots of other tasks). And anyway, it's non-trivial in any case.

> Again, the answer to system hangs (which are *inevitable* - this is *commodity hardware* we're talking about most of the time, not mainframes) is remote power booters.
Oh, I've witnessed mainframe hangups. Remote reboot is nice, but that server was from around 2006 so it didn't have IPMI and the datacenter where it was hosted offered only manual reboots.

Poettering: Revisiting how we put together Linux systems

Posted Sep 5, 2014 4:09 UTC (Fri) by raven667 (subscriber, #5198) [Link] (2 responses)

> - never rely on one system for anything vital. :)

That sounds dangerously close to internalizing and making excuses for unreliable software rather than engineering better systems that work even in the crazy imperfect world, duct taping and RPS to the side of a machine is in addition to not a replacement for making it work right in the first place.

Poettering: Revisiting how we put together Linux systems

Posted Sep 5, 2014 4:43 UTC (Fri) by NightMonkey (subscriber, #23051) [Link] (1 responses)

It might sound close, but it isn't. :)

I don't think what you are saying are actually separate tasks. All software has bugs.

Poettering: Revisiting how we put together Linux systems

Posted Sep 5, 2014 14:45 UTC (Fri) by raven667 (subscriber, #5198) [Link]

Sure all software has bugs but the risk and impact are not equally distributed.

Poettering: Revisiting how we put together Linux systems

Posted Sep 4, 2014 23:46 UTC (Thu) by flussence (guest, #85566) [Link] (5 responses)

> Except in my case, I really just want the feature of automatically restarting crashed daemons, and automatically funnelling all their console output into the journal.

I've found runit handles both those things flawlessly. Half of my system daemons are running under it — the other half's still on OpenRC, but that's mostly due to laziness.

Poettering: Revisiting how we put together Linux systems

Posted Sep 5, 2014 8:50 UTC (Fri) by cortana (subscriber, #24596) [Link] (4 responses)

Runit fails to deal with daemons that themselves fork and manage their own children.

Poettering: Revisiting how we put together Linux systems

Posted Sep 6, 2014 0:21 UTC (Sat) by flussence (guest, #85566) [Link] (3 responses)

That's true, but I've yet to encounter anything that misbehaves in that way. If and when that happens I'll be sure to complain upstream.

Poettering: Revisiting how we put together Linux systems

Posted Sep 6, 2014 0:47 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

A Python daemon using the multiprocessing module. Yes, I've seen it personally.

Poettering: Revisiting how we put together Linux systems

Posted Sep 6, 2014 9:38 UTC (Sat) by cortana (subscriber, #24596) [Link]

Jenkins and pretty much anything else written in Java.

Poettering: Revisiting how we put together Linux systems

Posted Sep 6, 2014 15:45 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link]

Complaining about this is pointless unless the upstream you are referring to, is the init system itself. There is a ton of software out there that does it. If your init system cannot handle it, it is a failure of the init system.