User: Password:
|
|
Subscribe / Log in / New account

Poettering: The Biggest Myths

Poettering: The Biggest Myths

Posted Jan 27, 2013 7:45 UTC (Sun) by lkundrak (subscriber, #43452)
Parent article: Poettering: The Biggest Myths

I sometimes feel ashamed that I'm not as vocal as people who for some reason oppose systemd while it significantly improves my Linux experience.

With people arguing how awesome, manageable and maintainable old shell-script based init system was I feel like I've been too unlucky to hit every single problem it had. (Or that I was running something else.)

I've seen my system boot take a far longer time than I could bear and seriously harm my experience (what if your TV set took 2 minutes to boot?), yet I still read people deciding for me that I should not care that much.

I've see init scripts shitty beyond belief (and I'm not even taking into account what various third parties ship). Ever seen Fedora's tomcat init script? Most of them are way too long, all of them duplicate mostly all of their code.

Also, many of them are just plain broken and racy -- setup includes external service monitoring (such as monit), it's somehow easy for it to attempt to run the init script at the same time you run it leaving the system in inconsistent state (e.g. daemon running and no pid file). Ironically, systemd not even fixes the problem but also removes the need for such software in some cases.

Besides what's stated above, thing that I like about systemd the most is the inetd-ish service activation. Part of my job is setting up systems consisting of numerous services with rather complex dependencies. Some of the services are in-house grown, most of them are daemons shipped with the platform we use. We run hundreds of nodes for production installation (what we like call that "cloud" so that we're able to sell it). We also perpetually test the thing, which involves creating countless new installations, starting the services, restarting them and shutting them down running various automated tests in between. This is a nightmare with the old init system we use. We can never know when is the service ready to use which is why we ended up with horrible polling shell scripts, often inlined into our test tooling or even in puppet. We need to very carefully order the actions we take and with too many services it's way easy to get wrong. Systemd would do this for us.

I've even seen service stop not work properly (leaving out lurking processes) with old init system for way too many times and services. (I've probably never seen it shut down tomcat properly; it's surprising how a single shitty servlet can get the init system tooling stuck).

And there seem to be more nice things about systemd. The daemon startup messages don't get lost if service misbehaves. Given how broken stock Fedora installations are these days, this has turned out to be a big time-saver for me. Also, it seems to be able to set up control groups for daemons automatically; this could make things simpler for us since we use cgroups for resource management of our services pretty heavily.

I've not been adventurous enough to run Fedora on daily basis, thus I might not be familiar with all the problems people could have with systemd during its early days (and I yet am to see a good example of what has been easy with old init systemd, but is not with systemd), but I do run it regularly to keep myself up to date and I've been very impressed with systemd. We'll celebrate the day RHEL 7 is released and systemd hits our infrastructure.


(Log in to post comments)

Poettering: The Biggest Myths

Posted Jan 27, 2013 8:01 UTC (Sun) by apoelstra (subscriber, #75205) [Link]

> I sometimes feel ashamed that I'm not as vocal as people who for some reason oppose systemd while it significantly improves my Linux experience.

I second this. Being able to throw up a 5-line systemd service file for a new daemon (without trolling through mailing lists looking for prewritten scripts and gotchas) is wonderful.

Poettering: The Biggest Myths

Posted Jan 27, 2013 11:17 UTC (Sun) by alankila (guest, #47141) [Link]

I wrote an upstart script to start some php thing as fastcgi daemon and it sucked. "expect fork" is part of the definition, indicating that the program was expected to fork once before running as a daemon, something upstart needs to be told, but systemd should be able to track the process no matter how many times it forks or doesn't fork.

Either way, I screwed up somewhere and failure at writing the init job definition caused upstart to no longer be able to startup and shutdown that particular service, it seemed to just stall. Killing the php processes didn't help, upstart was still stuck waiting on the service somehow. I couldn't even shutdown the machine anymore because it kept on jamming on that service file forever, forcing a manual power off. Again, on systemd, the fact I killed the process is enough evidence that the service is down, so systemd wouldn't have had the problem in the first place, and if it did, it would have been able to recover because it's actually smart.

I can only hope that I get to enjoy systemd at some point in the future as well. upstart is maybe fine if you know exactly how it works and know how to debug it; systemd seems almost designed for clueless admins like me.

Poettering: The Biggest Myths

Posted Jan 27, 2013 12:20 UTC (Sun) by davidstrauss (subscriber, #85867) [Link]

> systemd should be able to track the process no matter how many times it forks or doesn't fork.

Correct. It's also one of the features that cannot be portable; it relies of cgroups.

> Again, on systemd, the fact I killed the process is enough evidence that the service is down

Ditto.

Poettering: The Biggest Myths

Posted Jan 27, 2013 20:56 UTC (Sun) by cwillu (guest, #67268) [Link]

In fairness to upstart, "expect fork" and "expect daemon" are not something that can be detected by the init system even in principle. Yes, even if that init system is systemd. Which is why http://www.freedesktop.org/software/systemd/man/systemd.s... has a Type=simple|forking option and various forms of PID guessing to distinguish between the cases handled by upstart's "fork" vs "daemon".

Poettering: The Biggest Myths

Posted Jan 28, 2013 0:45 UTC (Mon) by davidstrauss (subscriber, #85867) [Link]

> In fairness to upstart, "expect fork" and "expect daemon" are not something that can be detected by the init system even in principle.

The difference is that Upstart can't reliably detect what's going on, let alone know whether it's correct.

systemd can detect what's going on, regardless of how you have the service configured. The configuration allows systemd to compare what is happening against what it's detecting to know whether the service is in a failed state.

Poettering: The Biggest Myths

Posted Jan 28, 2013 1:55 UTC (Mon) by mezcalero (subscriber, #45103) [Link]

Actually, systemd cannot detect the main process in all cases. If you use Type=forking (without specifying a PID file) and your daemon starts many processes at once before exiting in the parent, then we don't know which one of the remaining ones the "main" one should be. In that case systemd will consider the service exited only after *all* processes are gone, and can't collect as much state info about it... Thankfully, this case is quite seldom, the impact is low, and is easily fixed by specifiying a PID file, or by using Type=notify (after updating the daemon).

Upstart's tracing via ptrace() is one evil feature OTOH. Upstart basically becomes the debugger of the daemons it runs. Really, really evil stuff.

Poettering: The Biggest Myths

Posted Jan 28, 2013 2:53 UTC (Mon) by davidstrauss (subscriber, #85867) [Link]

> Actually, systemd cannot detect the main process in all cases. If you use Type=forking (without specifying a PID file) and your daemon starts many processes at once before exiting in the parent, then we don't know which one of the remaining ones the "main" one should be.

I wasn't speaking to detecting the main PID, only what is happening in terms of all forking behavior. When a process forks into a ton of other processes in the way you describe, there is no amount of collectable data on the system that allows properly identifying the main PID every time.

Poettering: The Biggest Myths

Posted Jan 27, 2013 14:03 UTC (Sun) by pizza (subscriber, #46) [Link]

Third!

systemd is such a vast improvement over what came before that it's not even funny.

Poettering: The Biggest Myths

Posted Jan 27, 2013 18:58 UTC (Sun) by apoelstra (subscriber, #75205) [Link]

Let me give a specific example. A week ago I wanted to set up a hunchentoot (lisp) web server, which involves starting a lisp instance, running some sort of setup lisp code, and leaving it running on some console somewhere.

Various Internet searches gave me 50-60 line init scripts for old versions of Ubuntu. These scripts were long and opaque and probably not even close to something that would run on Fedora 18. They relied on the detachtty tool (a minimal screen-type program) and did some sort of $PID dance around this. I wanted the server to restart whenever I killed it, so that I could reload the setup code that way. (Also I kept accidentally killing it by hitting Ctrl+D into screen.)

There's nothing out there about using hunchentoot with systemd. So, with maybe five minutes of searching manpages, I wrote

#======
User=lisp-web

[Unit]
Description=Run sbcl webserver session on a screen tty
After = syslog.target network.target

[Service]
User=lisp-web
Type=simple
ExecStart=/usr/bin/screen -D -m /bin/sbcl --load-path /home/lisp-web/ --load default-server.lisp
Restart=on-success
#======

Short, simple, clear, works correctly, tells me when things go wrong, etc, etc. I can connect to screen and hit Ctrl+D to reliably restart the server. It never hangs or gets out of sync or leaves ports in use or files locked.

Poettering: The Biggest Myths

Posted Jan 27, 2013 21:11 UTC (Sun) by pizza (subscriber, #46) [Link]

This "thread" finally motivated me to write a systemd unit file for my photo organizer's background threads. I don't know why I put it off so long.

It took me under five minutes. Would have been less time if the Fedora 16 server it's running on had a more modern systemd installed.

Trolling?

Posted Jan 28, 2013 13:23 UTC (Mon) by pboddie (guest, #50784) [Link]

You mean trawling, surely, although I imagine that trolling is a way of getting answers from mailing lists, too.

Sorry for the pedantry, but I keep reading "trolling" used in this sense and assume that it must be either incorrect or American English. At least if it is the latter, the same cannot be said for the epidemic of "loose" (instead of "lose") and the unbearably irritating "choosen" (instead of "chosen").

Trolling?

Posted Jan 28, 2013 13:51 UTC (Mon) by mpr22 (subscriber, #60784) [Link]

Funny thing: I hadn't even noticed the existence of "choosen", and can't imagine it being more irritating than the loose/lose confusion. (Of course, this is what we get for using an orthography invented by non-native speakers 900 years ago, barely upgraded since, and at this point prohibitively expensive to reform.)

Trolling?

Posted Jan 28, 2013 14:35 UTC (Mon) by tnoo (subscriber, #20427) [Link]

Ask Lennard for a new implementation.

Trolling?

Posted Jan 28, 2013 16:27 UTC (Mon) by HelloWorld (guest, #56129) [Link]

> Lennard
Speaking of spelling...

Trolling?

Posted Feb 9, 2013 22:50 UTC (Sat) by cas (subscriber, #52554) [Link]

it's the new and greatly improved way of spelling Lennart and if he doesn't like it, he's free to fork and go his own way.

Trolling?

Posted Jan 28, 2013 15:50 UTC (Mon) by Wol (guest, #4433) [Link]

I've lost context, but both trawling and trolling are means of fishing.

A trawler goes trawling with a trawl-net.

Dunno what the proper name of a troller is, but they go trolling with a troll-line. Most tuna, for example, is caught by trolling.

I've been trolling for mackerel ...

Cheers,
Wol

Trolling?

Posted Jan 29, 2013 14:23 UTC (Tue) by pboddie (guest, #50784) [Link]

I suppose I have learned something from this, never having been very interested in fishing at all. And I imagine that the Internet era meaning of the word is derived from this practice of using a baited line:

http://oxforddictionaries.com/definition/english/troll--2

The interesting thing here is that the example given on the above page is rather similar to the one used earlier, but unless one attracts the catch using bait - putting something out to attract the catch - the trawling metaphor is actually more appropriate, since one is just seeing what gets dragged up. But anyway...

Poettering: The Biggest Myths

Posted Jan 27, 2013 10:31 UTC (Sun) by Klavs (guest, #10563) [Link]

Seconded - I generelly get tired of people feeling the right to continuously critize what other people do - instead of just providing something better themselves. It's much easier to complain, than code!

It's fine to try influcence other developers, and come up with well-argued opinions - but in the end it's the coders that decide what they code and the users the decide what they use.

pls. stop wasting everybodys time with meaningless argueing.

Poettering: The Biggest Myths

Posted Jan 27, 2013 11:53 UTC (Sun) by drago01 (subscriber, #50715) [Link]

The thing is that people are afraid of change because they fear that their knowledge becomes useless. (such a sysadmin is by definition not a sysadmin)

Most of the criticism is based on buzzwords that have no real meaning "this is bloat" ... "xyz has been like this for 30 years" ... "this is not UNIX like" ...

Sure this sounds harsh but it is a good theory to explain the irrational arguments that you keep reading on the internet on such topics.

Poettering: The Biggest Myths

Posted Jan 27, 2013 21:39 UTC (Sun) by misc (subscriber, #73730) [Link]

Not only. While I usually like changes, that's also because I enjoy being a sysadmin. Now, if for some reason, you do not enjoy the job, but someone had to do it, maybe learning your way on Linux was not so great. Maybe you know that you will not have time to learn new ways.

Poettering: The Biggest Myths

Posted Jan 28, 2013 18:19 UTC (Mon) by ThinkRob (subscriber, #64513) [Link]

> The thing is that people are afraid of change because they fear that their knowledge becomes useless. (such a sysadmin is by definition not a sysadmin)

It's not a "fear of change" (which is a ridiculous thing in the first place...) or a "fear that [our] knowledge becomes useless" ('cause if that were the issue we wouldn't be in the IT field). It's just that some of us don't like changes when they bring no benefits that matter to us yet require us to re-learn a bunch of stuff that just generally worked "well enough" for a couple decades.

Yes, it's not a huge deal, but it's still a pain, especially when the previous implementation was satisfactory.

Poettering: The Biggest Myths

Posted Jan 28, 2013 20:58 UTC (Mon) by HelloWorld (guest, #56129) [Link]

Most of what worked with sysvinit still works with systemd. init scripts work, /dev/initctl works, utilities like service(8) from Fedora/RHEL work, even runlevels are emulated to the degree that's possible. So please, where *exactly* is the alleged "pain" you're talking about?

Poettering: The Biggest Myths

Posted Jan 28, 2013 21:13 UTC (Mon) by davidstrauss (subscriber, #85867) [Link]

> It's just that some of us don't like changes when they bring no benefits that matter to us yet require us to re-learn a bunch of stuff that just generally worked "well enough" for a couple decades.

It's not clear what requires immediate re-learning with systemd unless the administrator has a habit of directly altering rc.d symlinks, which hasn't been a best practice for a long time.

Existing init scripts work, unaltered, with systemd. Enabling and disabling services with chkconfig maps transparently to the proper operation in systemd. Mounts configured in fstab come over, automatically, as mount units and can continue to be managed the same way as before.

None of those compatibility features is going away any time soon.

And, for admins wanting to transition to the native tools, extensive documentation and guides [1] exist.

If that's not good enough for you and you can't appreciate the value these tools provide to others, I can only describe your attitude toward keeping SysV init as selfish. The FOSS world doesn't owe you stagnation once it's met your needs.

Poettering: The Biggest Myths

Posted Jan 28, 2013 5:47 UTC (Mon) by tnoo (subscriber, #20427) [Link]

Thanks a lot for this comment. You express my feelings exactly.

And thank you to Lennart and everybody who made systemd happen!


Copyright © 2017, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds