|
|
Subscribe / Log in / New account

Poettering: The Biggest Myths

Lennart Poettering decided to refute a few systemd myths on his blog, where "a few" means "30". "There's certainly some truth in that. systemd's sources do not contain a single line of code originating from original UNIX. However, we drive inspiration from UNIX, and thus there's a ton of UNIX in systemd. For example, the UNIX idea of 'everything is a file' finds reflection in that in systemd all services are exposed at runtime in a kernel file system, the cgroupfs. Then, one of the original features of UNIX was multi-seat support, based on built-in terminal support. Text terminals are hardly the state of the art how you interface with your computer these days however. With systemd we brought native multi-seat support back, but this time with full support for today's hardware, covering graphics, mice, audio, webcams and more, and all that fully automatic, hotplug-capable and without configuration."

to post comments

evolution

Posted Jan 27, 2013 2:01 UTC (Sun) by larryr (guest, #4030) [Link] (23 responses)

I liked Unix system administration because getting the system booted and running was something which demanded no more than some simple, discoverable, static configuration which someone with a basic understanding of the shell and a text editor could figure out and change by editing some files under /etc, and which could be tested by just powering up the system and seeing if it all came up properly. Even with udev the Linux kernel has not been demanding much more than that. More complex and dynamic configuration always required more than just this, but it did not require fundamentally replacing the existing infrastructure, and I am not convinced it needs to.

evolution

Posted Jan 27, 2013 2:12 UTC (Sun) by dashesy (guest, #74652) [Link] (22 responses)

If anything systemd has made things simpler. If you used to have a system with a few simple scripts you will still need a few simple textual files in systemd, and you can still do a lot with shell scripts if you want (Myth 4).

evolution

Posted Jan 27, 2013 2:31 UTC (Sun) by larryr (guest, #4030) [Link] (9 responses)

From the vacuous perspective where all usage of text files and shell scripts is equivalent, it could be speciously argued that systemd is not much more demanding of system administrators than its predecessors.

evolution

Posted Jan 27, 2013 3:14 UTC (Sun) by nirbheek (subscriber, #54111) [Link] (1 responses)

All that systemd demands of system administrators is that they take some time to learn an excellent new piece of software instead of spewing a knee-jerk reaction of "get off my lawn".

At this point in time, I count Pulseaudio and systemd amongst the best pieces of software written for Linux. The no. of features that they expose to ordinary users and sysadmins is staggering—and they do this without removing features or functionality, and while maintaining near-perfect backward-compatibility. I'm shocked that no one realises how hard it is to do this, and what value these add to our ecosystem.

At this point, after dozens of blog posts by Lennart explaining systemd, if I see anyone arguing the way you are, I am compelled to conclude that they never really gave systemd a fair chance; or, worse, they're too angry at its very existence to even try that.

I feel like most of the animosity towards these projects is not because of their (perceived) lack of technical merit, but because of the abrasive personality of their creator.

evolution

Posted Jan 27, 2013 5:51 UTC (Sun) by raven667 (subscriber, #5198) [Link]

> I feel like most of the animosity towards these projects is not because of their (perceived) lack of technical merit, but because of the abrasive personality of their creator.

I think this happens a lot and is also an out of date assessment, Lennart may be matter of fact in a stereotypically Germanic sort of way, which rubs some people the wrong way, but seems to have matured greatly since the flamewars during the PulseAudio days 5+ years ago.

evolution

Posted Jan 27, 2013 3:24 UTC (Sun) by dashesy (guest, #74652) [Link] (4 responses)

The problem with bash scripts is that they are never simple, they are convoluted, use vague idioms to save key strokes, and it takes quite some time for sysadmins to learn them well; I think every one has stories like how she learned to double-quote all variables in if-statement (test not internal) after her hard drive was formatted because of the space in the variable and that bash silently ignored the line.

On the other hand, a simple declarative text file is certainly more elegant. New sysadmins are just lucky, older sysadmins with a good taste will be happier too, after they spend a few hours to learn some new tricks.

evolution

Posted Jan 27, 2013 5:01 UTC (Sun) by Aliasundercover (guest, #69009) [Link]

This is my complaint with scripts as well. Even if well written for clarity they present a hard challenge because they can do anything. I find a plain declarative configuration file easier even if it is one I don't know yet because its scope is bounded.

evolution

Posted Jan 29, 2013 7:26 UTC (Tue) by Seegras (guest, #20463) [Link] (2 responses)

"Some old-school Unix habits have persisted long past
the point that they're even remotely sane. Shell programming at any
volume above a few lines of throwaway code is one of them - it's
*nuts* and we should *stop doing it*." -- ESR on shell scripts.
http://lwn.net/Articles/527308/

ESR does not speak for me

Posted Jan 31, 2013 13:12 UTC (Thu) by ncm (guest, #165) [Link] (1 responses)

This recommendation is almost enough to make me take up shell programming again. Or sed.

ESR does not speak for me

Posted Jan 31, 2013 15:29 UTC (Thu) by HelloWorld (guest, #56129) [Link]

I agree about ESR, but even a broken clock is right twice a day ;-)

evolution

Posted Jan 28, 2013 14:40 UTC (Mon) by drag (guest, #31333) [Link] (1 responses)

* Get me a list of all system services running.
* Get me their status.
* IF they are running then find me the parent PID number and any other processes that belong to the parent. Get me the last few lines of the logs for those services.
* IF it has failed to start up get me the exit code for the service and then the last few lines of the logs relevant for those processes.

All this stuff is deadly simple with systemd.

systemctl status vdsmd

for example.

This is a huge boost in productivity for system administrators to be able to have all the information in one spot.

evolution

Posted Jan 29, 2013 19:02 UTC (Tue) by dfsmith (guest, #20302) [Link]

I'm not sure I like the adverb "deadly" simple.

evolution

Posted Jan 28, 2013 18:05 UTC (Mon) by iabervon (subscriber, #722) [Link] (11 responses)

I'm fine with the system booting regularly with declarative text files that I didn't write instead of shell scripts I didn't write. But I like to be able to boot with init=/bin/bash and fix things, and be able to configure networking (in order to download a non-broken version of a package, for example) with the usual mechanism. While I would ordinarily have systemd running as pid 0, I think it's important that programs (in particular, the other executables in the systemd tarball, which are not supposed to be part of the same monolith) to function if systemd isn't running, or if something else is pid 0. What I've seen from systemd developers suggests that I can't get an environment where services haven't been started in general, but I can interactively start them and also run "avahi-browse | less" and my text editor. (I'd look at the documentation of systemd debugging, but I get a 503 from the link Lennart provided.)

(I don't use an early userspace, so init=/bin/bash actually does work for me presently.)

evolution

Posted Jan 28, 2013 18:22 UTC (Mon) by iabervon (subscriber, #722) [Link] (1 responses)

Actually, now that that page loaded, I see that systemd debugging is actually much like I described. So my complaint is just that the first-level myth busting didn't say that, but hid it behind a flaky link. Lennart says "Some people try to imply that the shell was a good debugger." but (a) the shell actually is a great debugger, when you have good command-line tools and (b) that's what systemd actually provides. I think this exemplifies the biggest myth about systemd, which is that, in order to get any benefit from it, you have to entirely redo your system and learn all new stuff; this mostly comes from systemd developers saying (probably accurately) that each thing in your old system is terrible and you'd be better off replacing it.

evolution

Posted Feb 6, 2013 3:17 UTC (Wed) by vonbrand (subscriber, #4458) [Link]

Sorry, but my extensive experience with assorted shell scripts handling system-y stuff is that they are about as easy to debug as your running kernel, irrespective of tools available (and when things really go south, the requisites for any tools higher up than a well-placed echo just aren't available at all).

evolution

Posted Jan 28, 2013 21:54 UTC (Mon) by smurf (subscriber, #17840) [Link] (8 responses)

Well, with SysV init you remember to type "init=/bin/bash" for a basic root shell with which to attempt to fix stuff.

With systemd it's "systemd.unit=emergency.service". Somewhat more verbose, but I just set my alternate boot command line to it and I'm done.

>> systemd developers suggests that I can't get an environment
>> where services haven't been started in general,
>> but I can interactively start them

Who suggests that, and where? It's patently false. In an emergency.service environment you can "systemctl start foo.service" just like you normally do.

evolution

Posted Jan 28, 2013 23:27 UTC (Mon) by iabervon (subscriber, #722) [Link] (6 responses)

The statement:

"Some people try to imply that the shell was a good debugger. Well, it isn't really. In systemd we provide you with actual debugging features instead. For example: interactive debugging, verbose tracing, the ability to mask any component during boot, and more."

suggests to me that, if you have some problem booting with systemd, you get a special systemd debugger rather than the shell. This makes me worried, because I suspect that I might have problems unrelated to systemd, and might actually have to run the configuration tool for some other service that's needed in order to authenticate users, and a systemd-specific debugger might not make that feasible. However, the truth is that the shell is a good debugger, and it's what systemd actually provides, along with command-line tools to help debug systemd in particular. This is a particularly unfortunate statement, because it plays into the impression people have of systemd that the developers think the tools you understand aren't good and you need to use their new thing instead in order to work with systemd.

evolution

Posted Jan 28, 2013 23:56 UTC (Mon) by HelloWorld (guest, #56129) [Link] (2 responses)

> suggests [...] suspect [...] might [...] impression

Really, whom is this kind of discussion supposed to help? Grown-up people in computing ought to be able to make an argument based on facts.

evolution

Posted Jan 29, 2013 11:37 UTC (Tue) by nix (subscriber, #2304) [Link] (1 responses)

Facts?! In *this* industry?

Come on now. Why dirty a nice flamewar with nasty facts when you can have a stew of speculation and imputations of bad faith and attacks based on myths which are debunked in the very article this is a comment thread for?

The only thing we're missing is a nice car analogy! So let me provide one.

systemd's an Edsel with the trailer and aircraft-carrier-catapult attachments, sysvinit is a Peel Trident. (I just want a Volvo.)

evolution

Posted Jan 31, 2013 13:37 UTC (Thu) by ncm (guest, #165) [Link]

systemd is a flying submarine. sysvinit is a boat anchor with feathers.

evolution

Posted Jan 29, 2013 7:49 UTC (Tue) by smurf (subscriber, #17840) [Link] (2 responses)

>> suggests to me that, if you have some problem booting with systemd,
>> you get a special systemd debugger rather than the shell

Wrong. You get a shell.

Of course you'd then use systemctl commands to figure out what's wrong with your boot (start services individually, etc.), but that's a Good Thing – you want to start your bootup jobs in the same environment as when booting regularly, otherwise you might mask the problem.

evolution

Posted Jan 29, 2013 8:53 UTC (Tue) by iabervon (subscriber, #722) [Link] (1 responses)

That's what I said: "However, the truth is that the shell is a good debugger, and it's what systemd actually provides, along with command-line tools to help debug systemd in particular."

Of your comment, my comment, systemd's actual behavior, and Lennert's original post, the only one that doesn't say that a shell plus systemctl is the right thing is Lennert's original post. That's why I said that *Lennert's writing* gives the wrong impression, and never said that systemd's *actual behavior* was wrong.

evolution

Posted Jan 29, 2013 17:33 UTC (Tue) by raven667 (subscriber, #5198) [Link]

I think the point was supposed to be that there is a bunch of debug logging available in the systemd tool kit that isn't available in shell languages beyond 'set -x', maybe because in shell that doesn't show you the environment or working directory or other things or because debugging shell requires modifying the scripts to print out what they are doing rather than having built-in logging.

evolution

Posted Jan 29, 2013 16:18 UTC (Tue) by s0f4r (guest, #52284) [Link]

Just typing "emergency" will also drop you in emergency mode - it's shorthand for "systemd.unit=emergency.service"

Poettering: The Biggest Myths

Posted Jan 27, 2013 3:41 UTC (Sun) by mgb (guest, #3226) [Link] (286 responses)

UNIX and Linux are evolving tool sets.

Busybox and systemd, though they may ship a billion copies, are monolithic evolutionary dead ends.

Busybox, unlike systemd, is a valuable and portable tool which does not inhibit the continuing evolution of UNIX and Linux.

I see a potential role for a minimal init replacement, but not for a grotesque monolith actively being exploited as a power play by its author.

I am in FLOSS for the long haul. Others may have different priorities.

Poettering: The Biggest Myths

Posted Jan 27, 2013 3:48 UTC (Sun) by nirbheek (subscriber, #54111) [Link] (125 responses)

I feel like I'm reading slashdot comments. Such vitriol and flamebait is very out-of-place for a website like LWN.

Poettering: The Biggest Myths

Posted Jan 27, 2013 3:55 UTC (Sun) by mgb (guest, #3226) [Link] (11 responses)

Permit me to observe that it is not what you have read but what you have written which reminds one of Slashdot.

Perhaps you would care to try again with something which was not an ad-hominem attack and with some technical input beyond "lots of shiny features",

Today's tool-set unixen enable evolution in all directions including into monoliths. Perhaps you would care to explain how future unixen can as easily evolve away from monoliths when a systemd replacement would have to replace so much in one shot?

Poettering: The Biggest Myths

Posted Jan 27, 2013 5:29 UTC (Sun) by imitev (guest, #60045) [Link] (2 responses)

Xorg is being slowly stripped of its features, it's not happening at once: KMS in kernel (talking of a monolith ?), wayland with compat layer, ...

I remember the pain of deploying init scripts to multiple distros so I can only welcome Poettering's efforts to try to improve things. Yes, some shinysmart features happen to come together.
You say you're in FLOSS for the long haul so you know well that nobody is pushing you to use systemd. If your favorite distro chooses to use systemd there might be a good reason, but you could always switch to another one. You may also join force to influence systemd's development, or even write your own init replacement, if it's really good enough people will adopt it for sure. But complaining about systemd becoming a monolith is not true nor fair. Did you really read the article ?

Poettering: The Biggest Myths

Posted Jan 30, 2013 2:55 UTC (Wed) by daniels (subscriber, #16193) [Link] (1 responses)

Xorg is being slowly stripped of its features, it's not happening at once: KMS in kernel (talking of a monolith ?), wayland with compat layer, ...

... what? All the old userspace drivers work just as well in Xorg as they ever used to: full of massive security holes, unable to do proper cross-process zerocopy buffer sharing, not playing well with other Xorg sessions, prone to hard-locking your machine when Xorg crashes, et al. It's just that the newer driver versions now reside in the kernel. Use the old ones if you want to. XWayland is also a new additional feature, rather than feature removal.

Poettering: The Biggest Myths

Posted Jan 30, 2013 21:25 UTC (Wed) by HelloWorld (guest, #56129) [Link]

The old userspace drivers are merely a legacy thing nowadays. In fact, the radeon driver recently dropped userspace mode setting support iirc. So X.org is in fact losing features, because they're being moved to other parts of the graphics stack.

Poettering: The Biggest Myths

Posted Jan 27, 2013 10:28 UTC (Sun) by ovitters (guest, #27950) [Link] (7 responses)

grotesque monolith actively being exploited as a power play by its author

You wrote above, don't complain when people question you about it, or say that this is Slashdot style commentary. I also don't understand your response, is it an attempt at a better comment or not?

Now for what you're suggesting, you argue (as I see it) that 1) systemd is monolithic, 2) in evolution, it needs all to be replaced in one go and #3 adopting something can only be done if it can easily be replaced. For #1, suggest to read the article. For #2, just change individual components as needed. For #3, I disagree :P.

Poettering: The Biggest Myths

Posted Jan 28, 2013 9:02 UTC (Mon) by russell (guest, #10458) [Link] (6 responses)

You really don't understand that the power play comment? Don't be deliberately naive and address the comment directly. He's referring to the way systemd has used it's foot hold as an init system, to then leverage into adjacent software, such as udev and system logging. This is not just creating an integrated system, that would involve just using well defined interfaces. This is a power play to reimplement them "his" way. What is a distro to do now. Take on the new "features" or revert from systemd?

Poettering: The Biggest Myths

Posted Jan 28, 2013 9:17 UTC (Mon) by ovitters (guest, #27950) [Link]

Let's go back to the comment again:

grotesque monolith actively being exploited as a power play by its author

And no, I don't see a) why to use grotesque b) why to ignore monolith as this is addressed in the article c) why to use exploited and d) power play

Now you're trying to change this, question why I don't address the comment. However, I did so as best I understood. Furthermore you've stated that I'm deliberate naive? Right, great suggestion there. If you follow the thread you'd see he wondered why people said it was Slashdot style commentary. Read https://lwn.net/Articles/534230/. I exactly addressed this as well.

Ignoring everything like that and suggesting I'm deliberate naive: whatever. And I don't see this power play happening. Suggest that you read all the comments on LWN and in the article because e.g. I can still run syslog, he discusses things with distributions, etc. If you want to continue please at least show that you read the blog or the comments here.

Poettering: The Biggest Myths

Posted Jan 28, 2013 12:50 UTC (Mon) by smurf (subscriber, #17840) [Link] (1 responses)

>> This is a power play to reimplement them "his" way.
>> What is a distro to do now.

Switch to systemd. Just like, for example, they switched to pulseaudio.

Because, unlike code from certain other people who implemented stuff "their way" (I'm thinking about djb), Lennart's way works. It solves real problems, without stepping on too many other people's feet.

He's matured a lot since the pulseaudio flame wars.

And, hey, OF COURSE it's a powerful ego trip to see your work being used and enhanced and whatnot by so many people. But that's not necessarily bad!

For most people it's a powerful motivator to create even more good stuff.

So what's your problem?

Poettering: The Biggest Myths

Posted Jan 28, 2013 17:06 UTC (Mon) by raven667 (subscriber, #5198) [Link]

> So what's your problem?

Maybe he's a lobster in a bucket, trying to pull the other lobsters down (Thanks Aquabats!)

Poettering: The Biggest Myths

Posted Jan 28, 2013 17:28 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

Yup. And Linux is just a power play to expand into the area of version control systems. Linus hated Subversion so much that he created Linux in order to make git popular.

Right?

Poettering: The Biggest Myths

Posted Jan 28, 2013 18:44 UTC (Mon) by apoelstra (subscriber, #75205) [Link] (1 responses)

> Yup. And Linux is just a power play to expand into the area of version control systems. Linus hated Subversion so much that he created Linux in order to make git popular.

Actually, he probably does hate subversion enough to have done that. :)

Poettering: The Biggest Myths

Posted Jan 29, 2013 11:19 UTC (Tue) by nix (subscriber, #2304) [Link]

No no no, he hated *CVS* enough to do that. And this was an entirely rational attitude.

Poettering: The Biggest Myths

Posted Jan 27, 2013 21:47 UTC (Sun) by Rudd-O (guest, #61155) [Link] (109 responses)

I feel like that too. It is like mgb did not even bother to read the myth list and just vomited the lie that systemd is monolithic. It is really hard not to conclude that he is either dumb or deliberately hateful.

Poettering: The Biggest Myths

Posted Jan 28, 2013 13:34 UTC (Mon) by sorpigal (guest, #36106) [Link] (108 responses)

Monolithic doesn't necessarily mean "one binary" and using multiple binaries doesn't mean systemd easily escapes this accusation.

The very feature of "integration" that is so often touted as a virtue of systemd belies the non-monolithic claim. It is possible to have integrated without monolithic but it is quite a lot more difficult than having it with (and has not been done in the systemd case).

Poettering: The Biggest Myths

Posted Jan 28, 2013 20:09 UTC (Mon) by drag (guest, #31333) [Link]

Depending on sh scripts to do everything seems pretty monolithic to me.

At least when trying to decipher them...

Poettering: The Biggest Myths

Posted Jan 28, 2013 20:31 UTC (Mon) by HelloWorld (guest, #56129) [Link] (106 responses)

> The very feature of "integration" that is so often touted as a virtue of systemd belies the non-monolithic claim.
Oh, so your proposed solution is to simply *not* integrate things at all and leave the resulting hard-to-use-and-configure mess to the users?

Sorry, that's not how things work in 2013.

Poettering: The Biggest Myths

Posted Jan 29, 2013 21:01 UTC (Tue) by aleXXX (subscriber, #2742) [Link] (4 responses)

Yeah, in 2013 all things work in some way automatic, talk with each other using DBUS, and adding abstraction layers where before none were necessary.
I'm running Slackware, which has beautiful simple init scripts. Back in the day, I simply edited my xf86config and entered what video card I have, how my screen looks like, etc, and my system did just that.
I plugged in an audio card, and I had a plain stupid /dev/dsp.
Nowadays I plug stuff together, and hope some magic autodetects it and does what I want it to do.
Yes, we can do more today than 15 years ago, but it has not really become easier to understand and debug.

Alex

Poettering: The Biggest Myths

Posted Jan 30, 2013 2:58 UTC (Wed) by smurf (subscriber, #17840) [Link]

>> Back in the day, I simply edited my xf86config

The word "simply" usually means something else, though.

Yes, X11 debugging has become more complex. *shrug* so has X11. But more often than not, these days, auto setup does it all for you and it simply *works*.

Net win, if you ask me.

Poettering: The Biggest Myths

Posted Jan 30, 2013 3:00 UTC (Wed) by daniels (subscriber, #16193) [Link] (1 responses)

Yes, we can do more today than 15 years ago, but it has not really become easier to understand and debug.

To be fair, modelines and working out which RAMDAC you had wasn't particularly easy to understand, or fun. And I am/was an X developer.

Poettering: The Biggest Myths

Posted Jan 30, 2013 16:29 UTC (Wed) by nix (subscriber, #2304) [Link]

But it was a trial by fire! How can we get rid of such exciting trials by fire?

(And if you had an old enough monitor, a *literal* trial by fire, or sparks and smoke anyway.)

I am very glad that this sort of thing Just Works these days, I must say...

Poettering: The Biggest Myths

Posted Jan 31, 2013 10:31 UTC (Thu) by jospoortvliet (guest, #33164) [Link]

you say "beautiful" and "simple" in conjunction with "init scripts". Unless you mean systemd service files (which are indeed simple and beautiful) I think there is something wrong with your English. Bash is a ridiculous solution for describing the behavior of services and if you've ever compared a 100 line bash script with a 5 line systemd file doing the same thing (just faster and more reliable), you'd never claim a shell script to be 'simple' or 'beautiful'.

Same with xf86config - terrible, terrible example. Sjees, how I hated those files. Now, things just work...

Sound card - yeah, it was beautiful - every time you added an audio device you had to reboot and find out what dspX device Linux had decided to give to your headset and build in audio cards this time. Now, with PulseAudio, the device shows up at runtime and once you've configured it once, next time the settings take effect immediately upon plugging in the headset. How is the old method better, again?

Yes, we can do more today than 15 years ago, AND it is MUCH easier to understand. The ways used to adapt sysVinit to enable faster booting and handling of plugging in devices should give a hint to that - terribly complicated to bend a static system so that it can survive in a dynamic world. It's incredibly nobody made a decent replacement of sysV when USB came on the market - it's crazy it took so long!

Poettering: The Biggest Myths

Posted Jan 30, 2013 12:28 UTC (Wed) by sorpigal (guest, #36106) [Link] (100 responses)

There's a big gap between "Tightly coupled, integrated components" and "Everything is hard to use." I don't buy in to the narrative that you describe, where the only alternative to systemd is a mess.

Poettering: The Biggest Myths

Posted Jan 31, 2013 10:33 UTC (Thu) by jospoortvliet (guest, #33164) [Link] (99 responses)

It is. The static world that sysVinit was build for hasn't existed since USB came on the market - that should have been the day a replacement was written. systemd is looooong overdue... The way sysVinit (and other parts of Linux infrastructure) has been pushed beyond it's limits, into a dynamic world, made it terrible. Maybe, for an experienced sysadmin, it made sense. But for a much less experienced person, systemd is FAR easier and better. And I bet the average experienced sysadmin has less trouble adopting to systemd than I have handling sysV...

Poettering: The Biggest Myths

Posted Jan 31, 2013 16:47 UTC (Thu) by sorpigal (guest, #36106) [Link] (2 responses)

Believe it or not I am a fan of systemd, just not all of it. I am not advocating using sysvinit forever, I am saying that systemd does some things that it doesn't *need* to do and does some things that are undesireable along with all the things that it does which are good. It doesn't have to be that way and the alternative isn't "exactly the same sysv system that was there before." I really like systemd's basic design, but some of the details make me want to scream. The alternative doesn't have to be a mess.

Poettering: The Biggest Myths

Posted Jan 31, 2013 18:59 UTC (Thu) by HelloWorld (guest, #56129) [Link] (1 responses)

Could you please be more specific? First of all, what does "in systemd" mean to you, the code in PID1 or all of the code in the systemd project? What is it that doesn't need to be done in systemd? What would specifically be gained by not doing so, and, most importantly, what are the drawbacks?

I think that many people see the advantages of keeping things separate, such as the possibility to replace parts, but fail to see the disadvantages, such as increased complexity.

Poettering: The Biggest Myths

Posted Jan 31, 2013 21:49 UTC (Thu) by raven667 (subscriber, #5198) [Link]

I don't know what it means to sorpigal but I would think there is a continuum between the core of systemd in PID 1 and the ancillary tools such as udev, dbus, journal and logind and the rest, most of which don't run as daemons and are CLI utilities for managing things.

Poettering: The Biggest Myths

Posted Jan 31, 2013 19:39 UTC (Thu) by dlang (guest, #313) [Link] (95 responses)

> he static world that sysVinit was build for hasn't existed since USB came on the market

Actually, on servers the world is still pretty close to static, you may have a thumb drive plugged in to USB once in a while, but that's aobut it.

> Maybe, for an experienced sysadmin, it made sense. But for a much less experienced person, systemd is FAR easier and better.

Again, for servers things are different, in part due to the legacy of tools and non-systemd systems that are around.

This is like so many other things happening in Linux today, people look at their laptop and say "this is so much easier", or "I don't need that sort of thing for one person"

the vast majority of Linux systems out there are servers (even including Android as Linux systems). It's perfectly legitimate to introduce options for people that make the desktop experience, especially with inexperienced people easier and more automated. But far too frequently, these changes break down when you aren't on a single-user desktop/laptop. If the changes are optional and the admins have the easy option of using the old way at install time, great, go for it. But when the single-user desktop changes are pushed as the only way to work, we have a problem.

The Debian approach of supporting many options avoids this problem (and even though Ubuntu chooses a default, they inherit all the options from Debian so tole old way is not much further than an apt-get away)

Poettering: The Biggest Myths

Posted Jan 31, 2013 20:05 UTC (Thu) by zwenna (guest, #64777) [Link]

> The Debian approach of supporting many options avoids this problem (and even though Ubuntu chooses a default, they inherit all the options from Debian so tole old way is not much further than an apt-get away)

In the case of init systems, Ubuntu does not inherit the options from Debian, neither sysvinit nor systemd are available in the repositories. And since many init scripts have been removed from Ubuntu packages in favor of upstart jobs, your system will almost certainly not boot correctly if you install another init system from source.

Poettering: The Biggest Myths

Posted Jan 31, 2013 22:15 UTC (Thu) by khim (subscriber, #9252) [Link] (91 responses)

Actually, on servers the world is still pretty close to static, you may have a thumb drive plugged in to USB once in a while, but that's aobut it.

Nonetheless on servers systemd makes perfect sense, too: hardware remains static, but services come and go regularly. Today it's solved by a horrible hack (KVM and nested virtual machines) but with systemd it should be possible to develop sane solution.

I've chased around runaways processes from various complex daemons enough to say that even if reliable service stopping is the only advantage systemd will bring to servers it'll be enough to justify all problems it brings. And the fact that I can now reliably limit resources consumption per-service is a nice bonus, too.

the vast majority of Linux systems out there are servers (even including Android as Linux systems).

There are 700 million Android systems out there and less then 100 million of servers (including virtual ones). It's not even a contest.

Poettering: The Biggest Myths

Posted Jan 31, 2013 22:23 UTC (Thu) by raven667 (subscriber, #5198) [Link]

> even if reliable service stopping is the only advantage systemd will bring to servers it'll be enough to justify ...

As someone who used daemontools for years, having reliable service stop/start/status with automatic restart and capturing stdout/stderr for logging are the main features that I need that existing init systems don't do. I'd love to stop requiring third party infrastructure like daemontools/runit or monit or even hackier stuff like cron jobs, to health check and restart critical services.

Poettering: The Biggest Myths

Posted Jan 31, 2013 22:23 UTC (Thu) by dlang (guest, #313) [Link] (89 responses)

> I've chased around runaways processes from various complex daemons enough to say that even if reliable service stopping is the only advantage systemd will bring to servers it'll be enough to justify all problems it brings.

Since systemd doesn't invent the way to do this (it uses cgroups provided by the kernel), why not just create a service launcher command that creates the cgroups as needed instead of taking over everything else that systemd does?

according to you we would gain almost all the benefit while avoiding all the cost.

Poettering: The Biggest Myths

Posted Jan 31, 2013 23:35 UTC (Thu) by khim (subscriber, #9252) [Link] (86 responses)

Since systemd doesn't invent the way to do this (it uses cgroups provided by the kernel), why not just create a service launcher command that creates the cgroups as needed instead of taking over everything else that systemd does?

You assume I know in advance which service will start misbehaving. And I don't. Sure, I have some suspects but in reality all of them can misbehave (I've seed many pretty innocuous services turning to fork bombs because of configuration mistakes). And if you run all your services using such launcher then why do you need to keep useless sysvinit baggage in PID1?

P.S. Cron scripts are especially problematic - that's why I like the fact that systemd offers timer units to replace them.

Poettering: The Biggest Myths

Posted Feb 1, 2013 0:42 UTC (Fri) by dlang (guest, #313) [Link] (85 responses)

anyplace you start services, it would be useful to have this capability.

this includes high availability and load balancing managers.

Should systemd now take over those functions as well?

the "unix way" that you are so dismissive of would be to provide a tool to use when starting things, and then that tool could be used by lots of different users, including ones that you don't think of.

The biggest proof of true 'unix way' is when you show the developer of a component what you are doing with it and they are so startled that they exclaim "I didn't know that was even possible".

I've seen this happen a few times, and heard of it happening more times over the years with Unix/Linux based software.

I'm not sure I've ever heard of it happening with other systems.

Poettering: The Biggest Myths

Posted Feb 1, 2013 1:03 UTC (Fri) by raven667 (subscriber, #5198) [Link] (82 responses)

You seem to be making an argument for systemd as your HA manager should be able to very simply stop and start systemd units to reliably do its job, reducing its complexity and making the system as a whole more reliable. systemd is the tool that's used when starting things that you describe.

Poettering: The Biggest Myths

Posted Feb 1, 2013 1:17 UTC (Fri) by dlang (guest, #313) [Link] (81 responses)

the HA manager also needs to do the monitoring of those systems, are you saying that it now needs to do that through systemd as well?

Poettering: The Biggest Myths

Posted Feb 1, 2013 1:42 UTC (Fri) by khim (subscriber, #9252) [Link] (79 responses)

Obviosly. Systemd is a tool you asked for in your previous message. You can run multiple separate instance of systemd to manage these processes and you can interact with it to know what goes with services which systemd is manages.

It's the same process which manages your PID1 but that's kind of obvious: why create a separate tool for the exact same task?

Poettering: The Biggest Myths

Posted Feb 1, 2013 2:21 UTC (Fri) by mgb (guest, #3226) [Link] (78 responses)

The function being discussed is to run a group of processes within a cgroup.

PID 1 could use such a function, but that is not the function of PID 1.

A properly engineered design would have factored out that functionality into a separate tool.

Which someone working on one of the the less hyper more reliable distros will no doubt do.

Poettering: The Biggest Myths

Posted Feb 1, 2013 8:22 UTC (Fri) by smurf (subscriber, #17840) [Link] (77 responses)

What would be the advantage of having a separate launcher program, besides conforming to the "one tool per job" dogma for no good reason?

How would you make sure that the launcher undoes literally everything you did in your session so far, in order to start the job in a consistent environment? (Hint: This is no longer possible.)

Poettering: The Biggest Myths

Posted Feb 1, 2013 12:02 UTC (Fri) by mgb (guest, #3226) [Link] (76 responses)

Consider how much one-tool-per-job FLOSS world has accomplished compared to the monoliths of M$ with their orders of magnitude more resources.

Cgroups are hierarchical and do not need PID 1. Indeed systemd itself despite all its baggage is sometimes used outside PID 1 to manage a cgroup until a better designed tool comes along.

Monoliths such as BusyBox and systemd can be useful tools but they are evolutionary leaf nodes. The distros which retain their ability to evolve are the future of FLOSS.

Poettering: The Biggest Myths

Posted Feb 1, 2013 13:23 UTC (Fri) by smurf (subscriber, #17840) [Link] (74 responses)

By your argument, Linux itself is a monolith and an evolutionary dead end.

Whether something is a big 400-pound gorilla or a swarm of ants matters much less, in the grand scheme of things, than you think. Your M$ argument is a straw man: one is closed source, the other is not, thus you're comparing apples with rocks.

I would surmise that the ease of adding new and important features (or just fixing bugs, particularly when they require nontrivial changes) is far more important. And that's precisely where having one common repository shines -- you can make one branch with your change, test it, merge it, you're done, instead of coordinating between N repositories and handling the resulting version mismatch.

NB: Please either answer my question, or admit that you can't.

Poettering: The Biggest Myths

Posted Feb 1, 2013 13:47 UTC (Fri) by mgb (guest, #3226) [Link] (73 responses)

The only reason systemd needs to update a massive repository for a simple change is because poor design has resulted in excessive coupling.

Poettering: The Biggest Myths

Posted Feb 1, 2013 16:20 UTC (Fri) by smurf (subscriber, #17840) [Link] (71 responses)

You cannot do all, or even most, of what systemd does in a loosely-coupled way. Even the supposedly-simple job of (re)starting a daemon in a consistent environment (i.e.. one that's the same as being run during system startup) cannot be reliably done any more – unless you're PID 1.

You know … either prove me wrong with actual code, or shut up.

Or at least stop replying selectively. (As if nobody else would notice.)

Seriously.

Poettering: The Biggest Myths

Posted Feb 1, 2013 17:36 UTC (Fri) by mgb (guest, #3226) [Link] (70 responses)

I hope this isn't your homework assignment.

PID 1 can spawn one or more cgroup controller processes where available and if desired, just like it spawns any other process. PID 1 does not need to be dependent upon cgroups, nor indeed any of the other things that Poettering crammed into his monolith.

Happily the UNIXy way, the properly engineered design, and the mechanism most propitious for future FLOSS evolution all coincide in one simple design which just happens to be the opposite of systemd.

Poettering: The Biggest Myths

Posted Feb 1, 2013 18:32 UTC (Fri) by anselm (subscriber, #2796) [Link] (65 responses)

By all means go ahead and implement it then. If it is really so much better than systemd people will be happy to take it on; the major distributions have been switching init systems so much recently that one more time won't matter if – thanks to you – they finally Get It Right.

Poettering: The Biggest Myths

Posted Feb 1, 2013 19:21 UTC (Fri) by mgb (guest, #3226) [Link] (64 responses)

Of the major distros, Debian and Ubuntu have not been and will not be borged. Gentoo has not been borged. Mint has not been borged and probably will not. I doubt Slackware will ever make the mistake of switching. Suse switched but may switch back when Poettering pulls support:

... systemd currently provides compatibility with the special non-standardized "boot" and "S" runlevels covering early boot which are used on Suse and Debian systems. We expect to remove this eventually ...
http://www.freedesktop.org/wiki/Software/systemd/Incompatibilities

In general it's the exciting high-churn buggier distros that switched, not the distros that people use for serious work, although eventually RHEL will probably switch.

Poettering: The Biggest Myths

Posted Feb 1, 2013 19:34 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]

The idea of SUSE switching away from systemd because of that note is just fictional. SUSE agrees with that idea completely.

Poettering: The Biggest Myths

Posted Feb 1, 2013 19:56 UTC (Fri) by anselm (subscriber, #2796) [Link]

Of the major distros, Debian and Ubuntu have not been and will not be borged.

With Debian, I wouldn't be so sure. Debian already offers systemd as an alternative to System-V init; the question is really whether systemd will be made the default init system on Debian GNU/Linux installs. The main counterargument is that systemd isn't available on Debian GNU/kFreeBSD, but there are various conceivable approaches the project could take to sort this out. There are many people within Debian who would like Debian GNU/Linux to default to systemd like most other major distributions do now.

As far as Ubuntu is concerned, the issue isn't whether systemd is a good init system or not; it's that they funded the development of Upstart and seem to find it difficult to go over to something else. Thus, arguably Ubuntu isn't as concerned with saving FLOSS by eschewing systemd as they are with rallying behind Upstart (which isn't all that different from systemd as far as being »Unix« is concerned). Again, systemd is, in fact, available for Ubuntu; it is just not something that Canonical is pushing at the moment but that may well change in the future.

I personally would consider neither Gentoo nor Mint »major distributions«. Once more, Gentoo does offer systemd but not as the default (yet). Mint will probably coast along with whatever Debian or Ubuntu will eventually do, so it is not out of the question that Mint will eventually go over to systemd.

In general it's the exciting high-churn buggier distros that switched, not the distros that people use for serious work, although eventually RHEL will probably switch.

I think we can pretty safely assume that RHEL and SLES – the main »serious-work« distributions – will be switching to systemd in the very foreseeable future (with CentOS &c. tagging along). As you note, OpenSUSE is already using systemd and SLES usually follows OpenSUSE.

Again, let's wait a year or five and see where we stand then. My money is on systemd.

Poettering: The Biggest Myths

Posted Feb 1, 2013 23:01 UTC (Fri) by rgmoore (✭ supporter ✭, #75) [Link]

In general it's the exciting high-churn buggier distros that switched, not the distros that people use for serious work, although eventually RHEL will probably switch.

This could more or less equivalently be described as it being the very conservative, release less than once in a blue moon distributions that haven't switched yet. In those instances, it's not at all clear that they're holding back on systemd for technical reasons. RHEL hasn't made a major release since before systemd was an option, and they're planning on switching for their next release. I think SuSE is in more or less the same boat. Debian stable is having a disagreement based on systemd's cross-kernel compatibility, not its technical suitability. The only big distribution that is clearly rejecting systemd is Ubuntu, and that appears to be more of a NIH syndrome than a technical decision.

Poettering: The Biggest Myths

Posted Feb 2, 2013 13:32 UTC (Sat) by jezuch (subscriber, #52988) [Link] (60 responses)

> Of the major distros, Debian and Ubuntu have not been and will not be borged.

Debian provides systemd; I'm using it on my Debian box and I'm really happy with it. (Except that I have to press CTRL+D on each boot because of some weird interaction with my self-built kernel, but that's a minor annoyance.)

Poettering: The Biggest Myths

Posted Feb 2, 2013 16:19 UTC (Sat) by mgb (guest, #3226) [Link] (59 responses)

Making systemd available is a good thing. Using systemd to block progress or to force gratuitous interface churn is a bad thing.

Debian has not been borged. Debian provides systemd as an "extra" package - the lowest priority tier below the "optional" packages.

This ensures that systemd is unable to bottleneck Debian's progress.

Poettering: The Biggest Myths

Posted Feb 2, 2013 17:05 UTC (Sat) by anselm (subscriber, #2796) [Link] (57 responses)

Debian provides systemd as an "extra" package - the lowest priority tier below the "optional" packages.

This is because it conflicts with the sysvinit package, which is "required", so Debian policy requires that the systemd package be "extra". It does not mean that the Debian project considers systemd unimportant.

It is not entirely unlikely that at some point these priorities will be reversed, at least as far as Debian GNU/Linux is concerned. There are many Debian developers who rather like systemd.

Poettering: The Biggest Myths

Posted Feb 2, 2013 17:51 UTC (Sat) by mgb (guest, #3226) [Link] (56 responses)

Debian developers who rather like systemd
They are of course welcome to use it. They are not welcome to force Poettering's capricious interface churn on everybody else.

Poettering: The Biggest Myths

Posted Feb 2, 2013 20:29 UTC (Sat) by anselm (subscriber, #2796) [Link]

I don't think you get to make that sort of claim on behalf of the Debian project.

If a majority of Debian developers decides that it makes sense for Debian GNU/Linux to default to systemd, then it is a fairly straightforward change to make -- if not now then in a few years' time. The project has made that kind of decision before.

Poettering: The Biggest Myths

Posted Feb 3, 2013 3:44 UTC (Sun) by HelloWorld (guest, #56129) [Link] (54 responses)

There is no "interface churn" in systemd. Most sysvinit interfaces work just as before, i. e. init scripts still work, /dev/initctl (and therefore telinit etc.) still works, /etc/fstab is supported, and there are many other things that work just like they always did. As for systemd's new interfaces, they are covered by the Interface Stability Promise:
http://www.freedesktop.org/wiki/Software/systemd/Interfac...

Poettering: The Biggest Myths

Posted Feb 3, 2013 4:04 UTC (Sun) by mgb (guest, #3226) [Link] (53 responses)

The list of what Poettering says he won't break excludes everything of importance.
... will not accept patches for certain distribution-specific compatibility ...
So if you're not Fedora you're at Poettering's mercy. But whether you're Fedora or not you can't even write a script that configures the NICs in your servers.
Previously it was practically guaranteed that hosts equipped with a single ethernet card only had a single "eth0" interface. With this new scheme in place, an administrator now has to check first what the local interface name is before he can invoke commands on it where previously he had a good chance that "eth0" was the right name.
wwp0s29u1u4i6 is so much easier to remember than eth0, don't you think?

Poettering: The Biggest Myths

Posted Feb 3, 2013 5:04 UTC (Sun) by rahulsundaram (subscriber, #21946) [Link]

You have no idea what you are talking about and merely repeating a misinformed notion. There is zero Fedora specific things in systemd. systemd upstream is entirely distribution neutral.

Poettering: The Biggest Myths

Posted Feb 3, 2013 8:23 UTC (Sun) by smurf (subscriber, #17840) [Link] (1 responses)

> wwp0s29u1u4i6 is so much easier to remember than eth0, don't you think?

You don't remember it. You add it to your system configuration and then forget about it. This is important if you ever add a second interface. For all other uses, there's "ip intf ls" and copy/paste.

You profess to be a sysadmin. You should know that.

Besides, the new scheme can be turned off, so you get a choice.

Stop spreading FUD and stop selectively reading systemd documentation.

Poettering: The Biggest Myths

Posted Feb 3, 2013 15:47 UTC (Sun) by smurf (subscriber, #17840) [Link]

>> ip intf ls

"ip link ls". Duh.

-- me, trying to wean myself from ifconfig

Poettering: The Biggest Myths

Posted Feb 3, 2013 12:21 UTC (Sun) by HelloWorld (guest, #56129) [Link] (48 responses)

> So if you're not Fedora you're at Poettering's mercy.
Many of systemd's "new" configuration files came from Debian, not Fedora. And besides, having the same configuration files across most distros is a good thing in the long term.

As for the network interface name stuff, here's the page your quote is from:
http://www.freedesktop.org/wiki/Software/systemd/Predicta...
So you quoted the paragraph that informs about one trivial disadvantage (the need to type "ifconfig" once to find out the name of the interface) while ignoring all the important stuff, such as consistent interface naming across reboots. Seriously, whom are you trying to shit here?

I don't even know why I'm replying to crazy people like you any more...

Poettering: The Biggest Myths

Posted Feb 3, 2013 12:52 UTC (Sun) by andresfreund (subscriber, #69562) [Link] (47 responses)

> So you quoted the paragraph that informs about one trivial disadvantage (the need to type "ifconfig" once to find out the name of the interface) while ignoring all the important stuff, such as consistent interface naming across reboots. Seriously, whom are you trying to shit here?
Imo the problem is not to have to type ifconfig once, but having to type it all the time. I am completely unashamed to admit that I cannot remember such generated names.
The earlier approach of generating stable ethX style interface name imo worked well enough and generated easier to remember names.

I don't understand why you feel the need to react with that amount of hyperbole. It only reinforces stupid anti-systemd stereotypes.

Poettering: The Biggest Myths

Posted Feb 3, 2013 16:18 UTC (Sun) by smurf (subscriber, #17840) [Link]

>> the problem is not to have to type ifconfig once,
>> but having to type it all the time.

*Sigh*. Did anybody even read the PredictableNetworkInterfaceNames page?
Physical-device-location-based names aren't even the default – they're just a safe fallback, if your BIOS doesn't supply sane interface numbers.
Plus, how to go back to ethX is well-documented.

Further observations:

* If you have to type in the name more than once, even in an emergency situation where you'd have to setup a route by hand, you're doing something wrong. If necessary, I'd type x=ethxMACaddr once, then refer to the thing by $x, which is even less typing than eth0. Or I'd use the shell's command history; even the busybox shell has one, these days.

* What's worse – a bit of an inconvenience, or having your internal network suddenly exposed to the whole world, because a timing quirk re-ordered your interfaces? Give me (and Lennart) a break here.

I do not want an OS which defaults to doing unsafe, random, and/or race-condition-prone things when it boots. Neither WRT my disk drives (remember the switch to UUIDs?) nor my network interfaces.

Poettering: The Biggest Myths

Posted Feb 4, 2013 19:29 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (45 responses)

I use zsh and it tab-completes ifup/ifdown interfaces for me (I don't use NetworkManager even on laptops) for me. A comparable completion function for bash shouldn't be too complicated.

Poettering: The Biggest Myths

Posted Feb 4, 2013 20:32 UTC (Mon) by mgb (guest, #3226) [Link] (44 responses)

When you're managing large numbers of systems in offices and data centers scattered around the globe you can't count on tab completion to ensure your systems reboot online. You need predictable interface names.

eth0 used to be the answer. It was great.

Then along came udev. In solving a rare problem (consistent interface naming in the presence of multiple NICs) it created a much more serious problem (interface names change whenever broken NIC replaced).

Sysadmins have mostly solved this by configuring both eth0 and eth1 even when eth1 doesn't exist yet. It's a PITA but we're ready when udev slams us.

But now with systemd we would have to get out the ouija board to figure out some kind of name like wwp0s29u1u4i6 that's going to take over when a broken NIC is replaced two years hence.

The better solution is to stay with an init system that works well and doesn't get in our way and doesn't cause random problems by starting services in a different order on every boot.

Poettering: The Biggest Myths

Posted Feb 4, 2013 21:03 UTC (Mon) by anselm (subscriber, #2796) [Link] (5 responses)

Read the effing documentation already. A convenient link has been provided to you earlier in this thread.

  • This is not a systemd issue in any way, shape, or form. It is a udev issue. Systemd and udev share maintainers but this is as far as it goes. There are people on System V init who have similar problems with naming their NICs, and they can solve them by installing the appropriate version of udev (out of the systemd tarball) without actually having to move to systemd.
  • Nobody prevents you from staying with eth0 etc. even under the new udev. How to achieve this is documented in excruciating detail in the documentation mentioned earlier. It is very easy.

We understand that you're not so hot on systemd. However your position would be that much more tenable if you had actual valid criticisms informed by facts. Judging from your last few postings this does not appear to be the case.

Poettering: The Biggest Myths

Posted Feb 4, 2013 21:10 UTC (Mon) by dlang (guest, #313) [Link] (2 responses)

you must have missed the memo where the maintainers merged the projects and have made it so that you can't build udev independently of systemd

Yes, the problems he is describing started off as udev problems, but they are now systemd problems

Poettering: The Biggest Myths

Posted Feb 4, 2013 21:21 UTC (Mon) by anselm (subscriber, #2796) [Link]

You don't need to install systemd in order to install udev. If you start from sources you have to build both but it is perfectly possible to install (or package for installation) udev without anything from systemd.

This has all been explained to death here already.

Poettering: The Biggest Myths

Posted Feb 4, 2013 21:23 UTC (Mon) by raven667 (subscriber, #5198) [Link]

They do share build infrastructure but you can run them independently so running udev doesn't imply running systemd. You're not going to get some kind of systemd cooties from building the shared components you don't end up needing if you just want udev. 8-)

Poettering: The Biggest Myths

Posted Feb 6, 2013 5:35 UTC (Wed) by spaetz (guest, #32870) [Link] (1 responses)

> This is not a systemd issue in any way, shape, or form. It is a udev issue. Systemd and udev share maintainers but this is as far as it goes.

Not leaning in on the pro/contra discussion, but this is an exaggeration by far. The announcement (http://lwn.net/Articles/490413/) makes it clear that they "merge the udev sources into the systemd source tree."

To me that indicates strongly that systemd and udev are more than two independent tarballs that can be version mix-and-matched and just happen to have the same maintainers.

Poettering: The Biggest Myths

Posted Feb 6, 2013 12:05 UTC (Wed) by HelloWorld (guest, #56129) [Link]

> Not leaning in on the pro/contra discussion, but this is an exaggeration by far.
No, it's not. udev does the interface naming stuff and systemd has nothing at all to do with it.

Poettering: The Biggest Myths

Posted Feb 4, 2013 21:09 UTC (Mon) by dlang (guest, #313) [Link] (36 responses)

> Then along came udev. In solving a rare problem (consistent interface naming in the presence of multiple NICs) it created a much more serious problem (interface names change whenever broken NIC replaced).

I delete the udev rules that implements this on all my server images, it's just not needed. If an interface fails badly enough that other interfaces get renamed to fill the gap, things don't work (it doesn't cause a security risk as the IPs and routes won't be able to make connections any longer

> The better solution is to stay with an init system that works well and doesn't get in our way and doesn't cause random problems by starting services in a different order on every boot.

On my laptop, I like a fast booting system, but I've been able to do that for a decade by stripping down the boot process to not try a bunch of stuff that I don't need.

On servers, predictability and stability are far more important than boot speed. Boot speed is limited by the hardware initialization anyway, I've got large systems that take so long to go through the hardware initialization that I can hit power on one of the large system at the same time I do on a simple (but fast) system, and boot the simple system off a CD, install the OS, and move the CD to the complex system before it gets around to reading the CD. I've setup demos of doing exactly this to impress manager types about how fast the install process is :-)

If you want a fast boot and don't have to dig into the boot process when something goes wrong, the newer init systems are nice.

But if boot speed is not that important to you, but predictability is, then async device detection, parallelizing the boot, etc add complexity and race conditions that cause more harm than the benefit provided.

Poettering: The Biggest Myths

Posted Feb 4, 2013 22:26 UTC (Mon) by raven667 (subscriber, #5198) [Link] (35 responses)

> If an interface fails badly enough that other interfaces get renamed to fill the gap, things don't work

I think that's the opposite of what happens, udev and the practice of associating interface names to the MAC address is so that the interface names are stable across boots and don't get shuffled around in the manner you describe without the sysadmin taking action. 00:11:22:33:44:55 is always eth0 regardless of detection and module load order, if you put in a different NIC with a new MAC and want it to take over an address then you may want to edit the configs before hand, or from the console if you have OOB access.

> But if boot speed is not that important to you, but predictability is, then async device detection, parallelizing the boot, etc add complexity and race conditions that cause more harm than the benefit provided.

I think the point of socket activation is that its fundamentally not race condition prone, and where you want explicit dependencies you can set Before and After to group things. Not having dependancies and service detection at all is inherently racy. As stated, boot speed wasn't a particular design goal but something that fell out of the design, and is worth mentioning, because it's not doing unnecessary work to get the same end result. Spawning thousands of instances of cut and grep and awk to parse config files and whatnot is much more expensive than just using a normal system programming language.

Poettering: The Biggest Myths

Posted Feb 4, 2013 22:40 UTC (Mon) by dlang (guest, #313) [Link] (34 responses)

I was talking about the problems you can have if you disable the udev interface naming stuff.

socket activation has problems with response time if it takes a noticable amount of time for the service to start and get to steady-state.

Yes, for some things it's great, but for the main service a system is providing, I would not want to use it.

These sorts of things are the difference between desktop use (where you have lots of stuff defined, but seldom use any of it) and servers (where you permanently disable or uninstall what you don't need, and what remains is likely to be hammered)

Poettering: The Biggest Myths

Posted Feb 4, 2013 23:07 UTC (Mon) by raven667 (subscriber, #5198) [Link] (28 responses)

> socket activation has problems with response time if it takes a noticable amount of time for the service to start and get to steady-state.

I don't think thats much of a problem in practice, you can start a service without waiting for a request to come in to activate it, say by having it start after networking and being a requirement for the multi-user target and having it manage its own sockets. Initial request latency on a service which is starting or has just started is a problem that can exist with or without systemd.

Poettering: The Biggest Myths

Posted Feb 4, 2013 23:46 UTC (Mon) by dlang (guest, #313) [Link] (27 responses)

if you have the service start itself, that's not socket activation.

I'm not saying that systemd can't support this either (before someone attacks me, saying that systemd can do this)

Poettering: The Biggest Myths

Posted Feb 5, 2013 0:23 UTC (Tue) by raven667 (subscriber, #5198) [Link]

Of course, but your service can depend on others which are socket activated.

Poettering: The Biggest Myths

Posted Feb 5, 2013 18:53 UTC (Tue) by khim (subscriber, #9252) [Link] (25 responses)

Sure, but you missing the point: when you use systemd you can combine socket activation and other forms of activation.

That's pretty powerfull stuff: your service will start when you system is started and other services can use it even at early boot stages. If your service is somehow started early - not a problem, it'll be used "as is", if it's not yet started - it'll be brought up via socket activation and other services will wait - all transparently and without any fuss or explicit dependencies sorting.

Poettering: The Biggest Myths

Posted Feb 5, 2013 19:24 UTC (Tue) by dlang (guest, #313) [Link] (24 responses)

sigh, that 's why I added the bit about how I wasn't saying that systemd couldn't do this.

I wasn't missing the point, I wasn't saying that systemd can't do this.

I was just saying that there are times when socket activation is not the best thing to be doing.

Poettering: The Biggest Myths

Posted Feb 5, 2013 19:58 UTC (Tue) by khim (subscriber, #9252) [Link] (23 responses)

I was just saying that there are times when socket activation is not the best thing to be doing.

It's never a good idea to disable a socket activation. Socket activation is your safety net. It guarantees that services will be started when they are needed. Everything else is optional.

This is yet another thing which systemd is doing correctly and which was traditionally managed in an an-hoc-kinda-works-in-you-quint-just-right way.

Poettering: The Biggest Myths

Posted Feb 5, 2013 21:55 UTC (Tue) by dlang (guest, #313) [Link] (22 responses)

having a service started multiple times is not always a safe thing to do. In theory the service properly checks and makes sure there's only one copy of it running, in practice you just don't do that, or it _will_ bite you down the road.

Poettering: The Biggest Myths

Posted Feb 5, 2013 22:12 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (20 responses)

Uhm, systemd gurantees that a service will be started only once.

On the other hands, I did have problems when ejabberd started in parallel with the PostgreSQL database - there was no dependency in LSB headers because I was using an optional psql auth module.

This worked just fine until the day it suddenly stopped working because the ordering of services has changed and Postgres moved a little bit later into the boot process.

Poettering: The Biggest Myths

Posted Feb 5, 2013 22:16 UTC (Tue) by dlang (guest, #313) [Link] (19 responses)

If nothing can go wrong, then why would you want to have a service that you configure to start one way also configured for socket based startup?

One of the things you learn after several years of running large numbers of production systems is to not trust claims that a process will always work, whatever that process is.

Failures are generally not that common, but they do happen.

Poettering: The Biggest Myths

Posted Feb 5, 2013 22:24 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (18 responses)

I thought it's obvious - socket-based activation enforces the ordering of services and explicit startup might be required because service might need to do some background stuff.

Also, it might decrease the response time time for the first client.

SystemD nicely allows to do both.

Poettering: The Biggest Myths

Posted Feb 6, 2013 8:14 UTC (Wed) by paulj (subscriber, #341) [Link] (17 responses)

The one downfall to this is that not all dependencies are socket-based. Some dependencies are more complex, and need a more abstract protocol than "open a socket" to express.

Poettering: The Biggest Myths

Posted Feb 6, 2013 9:18 UTC (Wed) by anselm (subscriber, #2796) [Link]

So? Systemd can deal with that, too – at least as well as System V init does. For example, systemd lets you express explicit forward and backward dependencies between services and will automatically construct a starting order based on that. With System V init, you either get to work out any dependencies yourself to set the magic numbers correctly by hand, or you use something like SUSE's insserv based on LSB metadata in the init scripts, where reverse dependencies are not an official feature.

Poettering: The Biggest Myths

Posted Feb 6, 2013 12:21 UTC (Wed) by HelloWorld (guest, #56129) [Link]

> The one downfall to this is that not all dependencies are socket-based.
Systemd offers a lot more than socket-based activation (which btw even supports inetd compatibility). There's also dbus-, device-, timer- and path-based activation and autofs support (which is arguably a service activation scheme as it can be used with FUSE file systems).

Poettering: The Biggest Myths

Posted Feb 6, 2013 16:52 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (14 responses)

If you have some very exotic services that use smoke signals to communicate - go on and add explicit dependencies.

Poettering: The Biggest Myths

Posted Feb 7, 2013 4:34 UTC (Thu) by paulj (subscriber, #341) [Link] (13 responses)

How would you add a dependency on, say, a location? Assume there is some daemon running on the system that can make the current location available over a socket (e.g. GPS co-ordinates, or a more abstract specification). The dependency is not just on the socket, but also on the /content/ of the information sent over that socket.

Hardly smoke signals.

Poettering: The Biggest Myths

Posted Feb 7, 2013 5:38 UTC (Thu) by khim (subscriber, #9252) [Link] (9 responses)

Hardly smoke signals.

It is a smoke signal - from GPS stateliness. It can easily be converted to something systemd understands if you'll create a specialized daemon (in a "true UNIX fashion" people like to preach here so much) which will convert it to signals systemd understands.

And you need such daemon anyway because you need to decide how often you poll, if you use just GPS or if you want to use WiFi, too, etc. It's not as if GPS satellites can pull some trigger on your PC or smartphone which means you'll need some complex logic anyway.

Poettering: The Biggest Myths

Posted Feb 7, 2013 5:50 UTC (Thu) by dlang (guest, #313) [Link] (5 responses)

the GPS daemon does have a socket that systemd understands, what are you suggesting? making a different socket for every possible location so that systemd can set dependencies on if that socket responds???

Just because systemd doesn't handle something doesn't mean that the something is worthless or stupid.

Poettering: The Biggest Myths

Posted Feb 7, 2013 8:00 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (3 responses)

If systemd cannot handle something, is there a bug report? If not, why not?

Poettering: The Biggest Myths

Posted Feb 7, 2013 8:39 UTC (Thu) by paulj (subscriber, #341) [Link] (2 responses)

There'll be no bug report on systemd because the applications involved are likely already using some other system. E.g. DBus IPC. DBus-daemon also can do "socket activation", and did before systemd existed I think.

Poettering: The Biggest Myths

Posted Feb 7, 2013 12:33 UTC (Thu) by khim (subscriber, #9252) [Link] (1 responses)

There'll be no bug report on systemd because the applications involved are likely already using some other system. E.g. DBus IPC.

If it used DBus IPC then it can be easily be used with systemd thus error report will be entirely unnecessary. I've said "convert it to signals systemd understands" and not "convert it to socket activation" exactly for this reason: because systemd handles many different activation requests and makes sure they don't conflict. Socket activation is just one of them (even if one of the most important ones).

Poettering: The Biggest Myths

Posted Feb 7, 2013 17:02 UTC (Thu) by paulj (subscriber, #341) [Link]

Interesting, thanks. :)

Bit of a dance, to have the IPC nexus hand-off this activation. It feels like really these should be part of one thing...

Poettering: The Biggest Myths

Posted Feb 7, 2013 8:25 UTC (Thu) by anselm (subscriber, #2796) [Link]

The GPS daemon provides the current location on a socket. It is probably unreasonable to expect systemd to be able to deal with that sort of information directly (systemd detractors would immediately jump on features such as these and call them out as »bloat«, with some justification).

Hence, a reasonable way of handling this would be to write a (not very complicated) subsidiary daemon that listened to the GPS daemon's output and triggered various systemd actions based on them. This might be a good idea in any case in order to provide »smoothing« of the location data or additional rules (»tell me about bars in the vicinity but only during happy hour«). This daemon itself would of course be managed by systemd.

This approach should also please the Unix traditionalists who insist that programs should »do one job and do it well«.

Poettering: The Biggest Myths

Posted Feb 7, 2013 8:35 UTC (Thu) by paulj (subscriber, #341) [Link] (2 responses)

So, go on, how do you expose that dependency in a way systemd can handle it. Tell me.

(FWIW, I have no opinion generally on systemd - I don't know enough about it. I'm just a tad sceptical of the wonder claims being made for socket activation based dependency resolution).

Poettering: The Biggest Myths

Posted Feb 7, 2013 8:59 UTC (Thu) by cortana (subscriber, #24596) [Link]

Perhaps you could create target units for each location of interest; other units could then be wanted by/conflict with each target in order to be started/stopped when the location is changed. You would need a glue daemon to look at the GPS data and decide when to start/stop the location targets.

Poettering: The Biggest Myths

Posted Feb 7, 2013 12:41 UTC (Thu) by khim (subscriber, #9252) [Link]

So, go on, how do you expose that dependency in a way systemd can handle it. Tell me.

Well, as you've suggested: DBus activation looks like a natural fit for such a use case - and since systemd handles it just fine... I don't see what's your problem.

I'm just a tad sceptical of the wonder claims being made for socket activation based dependency resolution.

Socket activation covers 90% of usecases, but there are other ways to activate service. And the important thing of systemd is that they all can be used simultaneously. You can start some daemon at specific time (using time-based activation) and when your a leaving specific area (D-Bus based activation) and when some other service needs this particular daemon (socket-based activation). They don't conflict and handled correctly in all cases.

Socket activation is there to track dependencies between services on your own system: it's the simplest one to use and most robust one. But there are other to handle "smoke signals", too.

Poettering: The Biggest Myths

Posted Feb 7, 2013 12:19 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

Don't use socket-based activation and do your dependencies manually, just as in the SysV world. Simple.

Additionally, designer of such a crazy interface should be shot immediately to stop propagating bogosity.

Poettering: The Biggest Myths

Posted Feb 7, 2013 15:59 UTC (Thu) by dlang (guest, #313) [Link] (1 responses)

> Don't use socket-based activation and do your dependencies manually, just as in the SysV world. Simple.

Then why is it that when people talk about doing exactly this, they get jumped by systemd people saying things like "why didn't you submit a bug report to get that capibility added to systemd" or "that's a stupid way to do things, you need to re-write your software to use systemd to do it"

This subthread started by the simple statement that socket-based activation was not always appropriate, with the acknowledgement that systemd could support this.

Poettering: The Biggest Myths

Posted Feb 7, 2013 16:03 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

> Then why is it that when people talk about doing exactly this, they get jumped by systemd people saying things like "why didn't you submit a bug report to get that capibility added to systemd" or "that's a stupid way to do things, you need to re-write your software to use systemd to do it"
These advices are not mutually exclusive, you know.

If you have a compelling use-case of some exotic activation system for which it makes sense to add core support then doing a bugreport might be a good idea.

And in other cases it might be a good idea to simply rewrite the offending code.

Poettering: The Biggest Myths

Posted Feb 5, 2013 22:43 UTC (Tue) by raven667 (subscriber, #5198) [Link]

That's not a problem that can happen with systemd though, since it keeps the services on a tight leash and knows what state (running or not) the service is in. Since that part must be self-contained in PID 1 it would seem there is little room for race conditions or errors to exist.

If we had the time we could skim through http://cgit.freedesktop.org/systemd/systemd/tree/src/core/ and see if we can identify where it keeps that state and how it is updated to see if there are any bugs.

Poettering: The Biggest Myths

Posted Feb 5, 2013 2:38 UTC (Tue) by smurf (subscriber, #17840) [Link] (4 responses)

> socket activation has problems with response time if it takes
> a noticable amount of time for the service to start and get to steady-state.

True. On a server, you probably don't want to use socket activation.

But socket activation is only an offshoot of systemd's "let me open all the sockets up front and hand them out to the daemons" idea. And that's useful on servers with their multiple daemons, because you now no longer need 90% of your boot dependencies, which is 90% less stuff to go wrong – esp. since some of these depend on the details in your daemons' config files.

Poettering: The Biggest Myths

Posted Feb 5, 2013 18:55 UTC (Tue) by dgm (subscriber, #49227) [Link] (2 responses)

> that's useful on servers with their multiple daemons, because you now no longer need 90% of your boot dependencies

I'm a bit divided about that, exactly because of this. If you don't need them, they should not be there. I fear that this will lead to distros enabling each and every service under the sun, just because they assume it will not get activated, which may or may not be true.

Poettering: The Biggest Myths

Posted Feb 6, 2013 1:37 UTC (Wed) by HelloWorld (guest, #56129) [Link]

> I fear that this will lead to distros enabling each and every service under the sun
You mean like Debian has been doing for ages?

Poettering: The Biggest Myths

Posted Feb 6, 2013 12:15 UTC (Wed) by smurf (subscriber, #17840) [Link]

You misunderstand.

What I mean is that if Apache needs Mysql, you no longer need to entomb that dependency in your startup scripts.

Poettering: The Biggest Myths

Posted Feb 5, 2013 18:58 UTC (Tue) by khim (subscriber, #9252) [Link]

True. On a server, you probably don't want to use socket activation.

Yes, you absolutely do want to use socket activation on server. It guarantees that service will be brought up if needed. You may want to use other forms of activation, too - but these are optional, if your forgot to explicitly start some service which is needed by other service - socket activation is there to bail you out.

Even if you bring up all the services on server startup you still want socket activation. Without socket activation you need to order them somehow and need to think about dependencies between them, etc but with socket activation you just start them all - and that's it: socket activation will guarantee that nothing will be lost.

Poettering: The Biggest Myths

Posted Feb 6, 2013 1:51 UTC (Wed) by HelloWorld (guest, #56129) [Link]

So you're saying that having multiple NICs is a rare thing when you're managing large numbers of systems in data centers?

And if that weren't enough, you readily admit that you're too incompetent to figure out that the mapping from MAC addresses to interface names is stored in /etc/udev/rules.d/70-persistent-net.rules and can be modified there?

You're a crazy person. Get help. Or don't, I don't care.

Poettering: The Biggest Myths

Posted Feb 3, 2013 22:41 UTC (Sun) by ovitters (guest, #27950) [Link]

Lennart nor systemd force things on others. The systemd developers had a upstream/downstream discussion at FOSDEM. It was announced on their mailing list, it was announced + arranged eventually via Google+. People from various distributions participated, Debian as well.

The seem to be that systemd is a "take it or leave it" and that Lennart somehow makes people do things against their will. While actually there are a lot of people who work within distributions who also do not see the need for so many differences between distributions.

systemd reduces those differences. The work is done by people who agree to that. Those people participate in all the ideas. Lennart & co go to loads of conferences to ensure they reach out to the people who actually make things happen.

It would be nice if you went to FOSDEM and just followed what goes on. Everything is in the open, etc.

Poettering: The Biggest Myths

Posted Feb 2, 2013 20:00 UTC (Sat) by smurf (subscriber, #17840) [Link]

*Yawn* There has been almost no progress, other than (upstart and) systemd, for the last 20 years.

Systemd cannot block progress because (a) nobody prevents you from writing and submitting as many enhancements to it as you like, (b) a modular solution will run into multiple hard-to-solve problems wich are far worse than some nebulous monolithicism. See my other post, which you have conveniently chosen to ignore; I assume because the facts contained therein are incompatible with your world view.

The problem is, computers tend to be not at all interested in world views – unless you can back it up by actual working code.

Thus: put up or shut up.

Poettering: The Biggest Myths

Posted Feb 1, 2013 21:35 UTC (Fri) by smurf (subscriber, #17840) [Link] (3 responses)

OK, just for the sake of argument (and because I admit that I'm having some perverse sort of fun with this) let's treat this as a homework assignment, namely "explain why mgb's idea will crash rather spectacularly".

So let's assume your init will not directly start the daemon (or user session, or …), but spawn a setup-and-exec-daemon program.

And what happens afterwards? The setup-and-exec program then needs to stick around and syslog-and-remember the process output, Oops, can't do that, that'd be too monolithic, so you fork+exec a logger process instead, right?

Said setup-and-exec program also needs to tell init which cgroup the daemon has been spawned into – for the simple reason that if/when it dies, init needs to know how to start another cgroup-aware process which cleans up behind said daemon (i.e. kills any remaining processes, and then tells the logger to exit).

There are a whole lot of problems with this scenario.

* The additional overhead. Running three helpers and a whole lot of interprocess communication in order to start and stop one daemon is … um … rather sub-optimal.

* Either every daemon will have a separate logger running, or you have to do a complicated dance of passing file descriptors between processes. Neither of these looks like a good idea, for rather obvious reasons.

* All of this to-and-fro between processes requires rather tight agreement of what's going on. You can basically forget the idea of a separate repository for part of this song-and-dance. If you disagree, kindly point to a real-world example where this has worked.

* If init does not know about cgroups, it cannot tell the logger that a daemon's last process has died, which it needs to do because part of the interface is /dev/log – which is a datagram interface and thus doesn't get end-of-file notifications. However, there's no other way to discover this because, surprise, PID 1 is the one which gets the SIGCHLD signals.

* No, you cannot get by with a single global /dev/log, because you want the equivalent of "systemctl status X" to show the syslog output from this one daemon X.

* init would need to forward my systemctl request about information about daemon X to the appropriate logger. Yet more complexity.

* All this IPC stuff is at least as complex as using dbus, which is somewhat unfortunate because depending on dbus is evil and much too complex, if I remember previous arguments correctly. But maybe that wasn't yours.

This concludes my part of this homework assignment, using only two aspects of what systemd does, and without reading a single "rationale" document by systemd's authors.

Your part: Either tell me, in as much detail as I just did, how you'd solve all of these problems with the multi-program approach. Or admit that you've been wrong.

Poettering: The Biggest Myths

Posted Feb 1, 2013 21:54 UTC (Fri) by raven667 (subscriber, #5198) [Link] (2 responses)

There is an example of how this can work, although with less features and guarantees (no cgroups, no dependancies), with daemontools. You run the svscan daemon out of /etc/inittab which maintains a parent/child relationship and pipe with PID 1 so that it can be restarted should it fail and svscan fires off a separate supervise process for each daemon you want to manage. supervise maintains a parent/child relationship and pipe with its child process and will restart the child should it go away. If there is a log startup script then that is run as well and anything written to the childs stdout or stderr is forwarded to the stdin of the log daemon (usually multilog).

You can hack around many of the failures (lack of restart throttling for example by using sleep) but it's not feasible to hack around the lack of service dependency resolution and the fact that this is an add-on component and not part of the core OS so can't be relied upon to be there in most cases. There has been development and thought in the area of service management beyond SysV init in the last 15 years or so but there are real reasons why these systems like daemontools and runit haven't gained the traction that systemd has gained, because they are not as comprehensive and are technically inferior.

Poettering: The Biggest Myths

Posted Feb 2, 2013 10:02 UTC (Sat) by jospoortvliet (guest, #33164) [Link]

Now that is a lame reply. "Yeah, there IS a way to do it, it just can't actually do it". Just have the balls to say "ok, you're right". There is nothing wrong with changing your opinion/position based on evidence and good arguments, heck, it's been said that "people who never change their opinion only show that they never learn".

Poettering: The Biggest Myths

Posted Feb 2, 2013 14:17 UTC (Sat) by smurf (subscriber, #17840) [Link]

>> less features and guarantees (no cgroups, no dependancies), with daemontools

Which is why daemontools did not catch on.

(To be fair, daemontools predates cgroups – but then, this list of deficiencies WRT systemd is hardly exhaustive.)

Poettering: The Biggest Myths

Posted Feb 1, 2013 16:44 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

Are we sure that mgb is not just an Eliza-like script?

Poettering: The Biggest Myths

Posted Feb 1, 2013 14:29 UTC (Fri) by anselm (subscriber, #2796) [Link]

Monoliths such as BusyBox and systemd can be useful tools but they are evolutionary leaf nodes. The distros which retain their ability to evolve are the future of FLOSS.

Funnily enough, System V init represents exactly this sort of evolutionary dead end. It has seen essentially no change for the last 25 years or so – in spite of its various glaring problems. There isn't really a lot you can do to System V init in an »evolutionary« way to get it near systemd feature parity. People have been piling stuff on top of it over the years (without doing a lot about important problems like the complete lack of service monitoring) but the basic approach, together with its deficiencies, remains the same. Systemd is a paradigm shift for init systems that relates to System V init like, e.g., Postfix relates to Sendmail.

In addition, the situation for improvements to System V init is now a lot worse than it was a few years ago because the state of the art for init-like systems is no longer defined by System V init itself but by systemd. Any new init system (including one based on evolutionary improvement of System V init) must now be at least as good as systemd – and likely considerably better – to entice the big distributions which are now pushing systemd to convert to the new system (again!). Opinions may differ as to whether systemd is »monolithic« or an »evolutionary leaf node« but as long as nothing does a noticeably better job systemd looks as if is going to be the init system of choice for most major Linux distributions.

To see whether you are right we shall simply have to wait for a few years and then check again. I'll venture the prediction that, by that time, Linux distributions that do not support systemd will be few and far between – much like today we find very few (if any) Linux distributions that still come with XFree86 rather than X.org.

Poettering: The Biggest Myths

Posted Feb 1, 2013 2:47 UTC (Fri) by raven667 (subscriber, #5198) [Link]

Of course the HA manager needs to monitor the HA services, systemd doesn't provide a mechanism for running monitoring scripts, but it doesn't have to maintain a parent/child relationship with them when sufficient tools are provided to reliably start and stop the daemon. That way you can take advantage of cgroups and all the kernel features that are trivially exposed by systemd (man systemd.exec) without duplicating all that code, probably less reliably, in your HA system.

If you aren't using clusters for HA with inter-server failover than systemd does provide several critical features for making a local service highly available such as automatic restarts (Restart=on-failure), hardware watchdog support and a protocol that a daemon can use to be watchdogged by systemd (man systemd.service, I think you can just use systemd-notify --status="WATCHDOG=1"), should one wish to add support for it to their daemon.

Poettering: The Biggest Myths

Posted Feb 1, 2013 3:36 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

>anyplace you start services, it would be useful to have this capability.
>this includes high availability and load balancing managers.
>Should systemd now take over those functions as well?
Sure. We actually rewrote our monitoring scripts to use systemd's DBUS interface to watch for live services. Systemd is used to detect service shutdowns and to do state control.

Works beautifully.

Poettering: The Biggest Myths

Posted Feb 1, 2013 11:10 UTC (Fri) by anselm (subscriber, #2796) [Link]

the "unix way" that you are so dismissive of would be to provide a tool to use when starting things, and then that tool could be used by lots of different users, including ones that you don't think of.

It turns out that systemd is quite useful in a HA/load balancing context – it offers a reliable way of starting and stopping services that can be controlled by an external monitoring infrastructure. It is also possible to run instances of systemd that are not PID 1.

If we stipulate that this is not what Lennart and Kay originally had in mind, then systemd is a good example of the »Unix way«, by your definition.

Poettering: The Biggest Myths

Posted Feb 1, 2013 8:13 UTC (Fri) by smurf (subscriber, #17840) [Link] (1 responses)

> why not just create a service launcher command that creates the cgroups as needed

The word "just" does not apply in this context.

The launcher would still need to tell systemd which cgroup and which main process has been created, so that systemd can take the correct action if/when the thing dies.

In fact, the launcher would have to be *started* from systemd, for the simple reason that you want your daemon to inherit a clean environment. Fixing it all up in a launcher or, worse, in each daemon you start (which is the current method) does not always work and requires additional privileges which you need to guard against being exploited.

Thus, there's no advantage to having a separate launcher, other than giving you a warm fuzzy feeling. Not sufficient to convince me, sorry.

Poettering: The Biggest Myths

Posted Feb 1, 2013 11:15 UTC (Fri) by anselm (subscriber, #2796) [Link]

Presumably, the »service launcher« isn't supposed to be used with systemd – the idea is to hang on to System V init a little longer by having a method to launch services with (some of) the features that systemd would otherwise offer, like per-service cgroups.

In this context the fact that the demise of a service cannot be detected does not matter since System V init can't do it, either, and the proponents of System V init apparently consider this an overrated feature.

Poettering: The Biggest Myths

Posted Feb 6, 2013 3:35 UTC (Wed) by vonbrand (subscriber, #4458) [Link] (1 responses)

My "server" machine running Fedora (18 now) is working fine with systemd and NetworkManager, thank you very much. I'd hate to need to go back fighting stupid, unreadable shell scripts.

Poettering: The Biggest Myths

Posted Feb 6, 2013 5:53 UTC (Wed) by dlang (guest, #313) [Link]

nobody is trying to force you to stop using the tools you like.

Poettering: The Biggest Myths

Posted Jan 28, 2013 9:25 UTC (Mon) by russell (guest, #10458) [Link] (2 responses)

Comments on slashdot would be off topic, trying to be funny, or just plain juvenile. I don't see this as being any of those.

Passionate sure; on topic yes. Free software is full of passionate people. Passion drives debate and causes people to make things better. Lennart is passionate about systemd and pulse audio, just read his posts. And good on him. Not everyone is going to agree, and that's fine. Trying to shut down dissenting opinions by talking about political correctness is OFF TOPIC.

If there is one thing that characterizes the people on LWN, it's that they are political correctness Nazis. Oops I mentions the Nazis, my bad. But I'm sick of the conversation being one sided in controversial topics.

Poettering: The Biggest Myths

Posted Jan 28, 2013 17:14 UTC (Mon) by raven667 (subscriber, #5198) [Link]

> political correctness

I'm not sure how you can characterize being polite and respectful of your fellow human beings as a bad thing. It is not required that one be unpleasant, rude and insulting to have an impassioned argument, good arguments are always welcome here on any side of an issue but impoliteness is not.

Poettering: The Biggest Myths

Posted Jan 28, 2013 17:20 UTC (Mon) by louie (guest, #3285) [Link]

It turns out, after much research, that it is possible for adults to have civil conversations on controversial, many-sided topics. Asking for others to have civil conversations is not being a "political correctness Nazi", it's being a mature human being who has learned the necessary social skills to not be an asshole, and expects that same non-asshole-ness from others. For more detail, see rule #8 of Scalzi's how to be a good commenter.

Poettering: The Biggest Myths

Posted Jan 27, 2013 4:25 UTC (Sun) by rgmoore (✭ supporter ✭, #75) [Link] (94 responses)

UNIX and Linux are evolving tool sets.

Yes, and part of that evolution is replacement of components of the system with new components as the demands of the system change. It seems to me that the people screaming that it's unnecessary to replace a 30 year old component that is obviously crufty and full of hacks are the ones standing in the way of Unix and Linux evolution, not the ones who are proposing a replacement.

Poettering: The Biggest Myths

Posted Jan 27, 2013 4:39 UTC (Sun) by prometheanfire (subscriber, #65683) [Link] (93 responses)

It is not that the old system should not be overhauled (see openrc for a better init system imo). It's more that systemd aims to do too much, I want an init system and that's it, init...

Poettering: The Biggest Myths

Posted Jan 27, 2013 5:48 UTC (Sun) by rgmoore (✭ supporter ✭, #75) [Link] (91 responses)

I want an init system and that's it, init...

I guess I understand the motivation, but it doesn't seem that it's necessarily a definitive answer. Yes, I understand the desire to keep traditionally separate components separate, but I also think that it's important to challenge assumptions about the way things ought to be from time to time. Replacing individual components can improve those individual components, but it can't fix design flaws about how the jobs were divided between components. That can only be fixed by changing the underlying architecture to divide the task differently. Bigger changes give the potential for bigger rewards.

I won't claim to be a guru, but the explanations Mr. Poettering has given for his architectural changes seem reasonable to me. The changes he is making seem to be adding valuable functionality that is not available when components are kept separate. Given that he has plausible arguments for why he's breaking down traditional boundaries, I'd like to see counter-arguments that focus on the technical points he's making, e.g. how the new functionality he's adding could be produced with traditional modular tools and/or why it's insufficiently important to justify the changes he's making. Instead, I see a lot of criticism of his judgment and motivation but very little that directly challenges his technical decisions.

Poettering: The Biggest Myths

Posted Jan 27, 2013 6:15 UTC (Sun) by prometheanfire (subscriber, #65683) [Link] (90 responses)

I see the personal attacks as a bad thing too. Forgot to add to my previous comment, but I am also interested in cross OS support, which is one thing that he stated not to support (which is too bad).

Poettering: The Biggest Myths

Posted Jan 27, 2013 6:47 UTC (Sun) by raven667 (subscriber, #5198) [Link] (8 responses)

systemd isn't the largest body of code out there, if you wanted its features on another platform it would probably be best to use it as a guidepost and then re-implement it using your preferred OS native APIs for doing so.

Poettering: The Biggest Myths

Posted Jan 27, 2013 6:57 UTC (Sun) by prometheanfire (subscriber, #65683) [Link] (7 responses)

I somehow doubt upstream would be receptive to that type of code (see eudev...)

Poettering: The Biggest Myths

Posted Jan 27, 2013 7:08 UTC (Sun) by raven667 (subscriber, #5198) [Link] (6 responses)

That was my point, make a new project, say 'startupd', for your favorite OS using systemd as a design guide but not necessarily re-using code from it. Then you too can support your favorite OS security and container frameworks as well as process information, like/usr/bin/top which is different on every OS.

Poettering: The Biggest Myths

Posted Jan 27, 2013 7:11 UTC (Sun) by prometheanfire (subscriber, #65683) [Link] (5 responses)

passive aggressiveness?

Poettering: The Biggest Myths

Posted Jan 27, 2013 12:35 UTC (Sun) by smurf (subscriber, #17840) [Link] (4 responses)

What's "passive aggressive" about this?

Why should systemd eschew a whole lot of features of Linux which are simply not available elsewhere? More so if he cannot do half the ob he set out to do without them?

If you think he's wrong, and if you think something like systemd would be way cool to have on *BSD, write it yourself. I doubt that it'd be possible without kernel changes; systemd wasn't either.

NB: eudev is a rather bad example. I still fail to see which problem the eudev fork is supposed to solve.

Poettering: The Biggest Myths

Posted Jan 27, 2013 23:17 UTC (Sun) by ewan (guest, #5533) [Link] (3 responses)

Indeed, some ideas in systemd are similar to things done in Apple's launchd - you don't need cross platform code or a friendly 'upstream' to be able to nick good ideas from where ever you find them.

Poettering: The Biggest Myths

Posted Jan 28, 2013 5:21 UTC (Mon) by rsidd (subscriber, #2582) [Link] (2 responses)

Actually, launchd is under the Apache licence -- why didn't Red Hat just take it? (There are claims that Ubuntu considered launchd in 2006 and rejected it because of its licence -- APSL at that time. Apple subsequently relicensed it.)

Interestingly, one complaint about launchd I see from a Unix-loving mac user sounds exactly like some of the systemd complaints here:

Reasonable people can argue the technical merits of launchd, and I think a case can be made for improving the original init system. However, I have a deeper philosophical problem with all the functionality that has been rolled into it: One of the core principles of Unix programing is do one thing and do it well. Another is keep things as simple as possible. From that perspective, launchd tries to do too many things, creating needless risk and complexity. Apple sees launchd as a natural improvement to traditional Unix; I view it as an anti-Unix monolith.

Poettering: The Biggest Myths

Posted Jan 28, 2013 5:40 UTC (Mon) by raven667 (subscriber, #5198) [Link] (1 responses)

In a way they did take it, not the implementation but certainly the good ideas. They did research other systems including Solaris SMF and launchd as well as Upstart before picking out the best of all of them before implementing it the Linux way.

Poettering: The Biggest Myths

Posted Jan 28, 2013 19:34 UTC (Mon) by k8to (guest, #15413) [Link]

Mentioning launchd is a good way to make me feel better about systemd. At least systemd has some kind of useful documentation. launchd leaves many aspects and values of its baroque xml completely undefined.

Poettering: The Biggest Myths

Posted Jan 27, 2013 10:39 UTC (Sun) by ovitters (guest, #27950) [Link] (78 responses)

The entire boot process between distributions was already largely different. I've used Mandriva and now Mageia. The initscripts and more is often taken from Fedora, then adjusted. Somehow sysvinit has the reputation of being the same and portable. But in practice loads of things were different between distributions. The way Mandriva+Mageia booted was already hugely different than Fedora, despite that large parts were taken from Fedora.

Practical thing I'd like to do on a server is ensure via Puppet that the timezone is in UTC (not just the current timezone, also the one for all services, some of which have been started already). Now if you have servers running RHEL, Ubuntu, Debian, Cent OS and Fedora, you'll discover that the only easy way to do this is to rely on Google+hope someone had the same problem.

systemd more and more makes unneeded differences between distributions go away.

Poettering: The Biggest Myths

Posted Jan 27, 2013 10:48 UTC (Sun) by prometheanfire (subscriber, #65683) [Link] (6 responses)

That's true, but when I say cross OS, I mean the bsds as well :D

Poettering: The Biggest Myths

Posted Jan 27, 2013 13:00 UTC (Sun) by ovitters (guest, #27950) [Link] (5 responses)

That would be nice, but systemd uses lots of (kernel) features that are not available on *BSD. I also don't see *BSD using software that is not BSD licensed. At least that is what I understood from that OpenBSD developer who complained about GNU tools being different and that standards are updated to include those GNU features. If something like 'sed' is a problem, I doubt that relying on a non-GPL init system is acceptable.

Poettering: The Biggest Myths

Posted Jan 27, 2013 16:40 UTC (Sun) by smurf (subscriber, #17840) [Link] (4 responses)

Perhaps those features just make sense, and they complain because they didn't think of them first?

Anyway, nothing prevents them from implementing what's missing on their side.

In related news, *bsd came up with strlcpy(). Do we go to them and complain that it's non-standard? No, we go to our libc maintainer and complain that he doesn't want to include that in the standard library.

I see a fundamental asymmetry here. Which might just be part of the reason there are >=3 BSDs out there, but just one single Linux. :-P

Poettering: The Biggest Myths

Posted Jan 28, 2013 1:05 UTC (Mon) by mezcalero (subscriber, #45103) [Link] (3 responses)

Actually, I find the discussion around strlcpy() interesting, because I can actually understand the points Drepper makes about it, and I think I side with him on it on the opinion that it doesn't really lead to better code... (I know for sure at least that I never needed a call like that when hacking on systemd.)

The thing is simply that strlcpy() leads people to use fixed size arrays to store possibly much longer data in it, so they want the truncation. But if you do this, then you tend to hide a problem with it, and instead you should have the checked the length of the argument in the first place and returned an error (after all the general rule is to always check your input!). Implicit truncation without error condition is almost never a good idea. And if you didn't check explicitly for the overly long string, then you should handle that string in its entirety properly, which generally means dynamically allocated memory of the correct size, which still not requires strlcpy().

I do see a valid usecase for some ways to call strncpy() (to fill in fixed size strings in structs where you usually want to zero out the rest of the string), and there's a very useful usecase for strdup() -- but strlcpy(), not so sure.

But, then again, given how trivial the call is, and how many people want it, if I was glibc hacker I'd still merge it... This shouldn't be fought with the fervour that people fight it with.

Poettering: The Biggest Myths

Posted Jan 28, 2013 10:50 UTC (Mon) by smurf (subscriber, #17840) [Link]

It's in libbsd, so if you want it it's there.

Anyway, strlcpy was intended to be an example of the different styles of handling NIH situations in Linux vs. *BSD land. I didn't exactly intend to trigger yet another strlcpy sub-discussion. ;-)

Poettering: The Biggest Myths

Posted Jan 28, 2013 15:59 UTC (Mon) by epa (subscriber, #39769) [Link] (1 responses)

Unfortunately the kind of programmers who use fixed size arrays will do so with or without strlcpy() being in the standard library. Given that, you might as well provide the tools to do so safely rather than push people towards the more dangerous strcpy(). And there is no harm in a belt-and-braces approach, where you check the length of every input string and return a sensible failure message, but then use strlcpy() and other safe functions *just in case* you accidentally missed a check somewhere. Silently truncating is bad, but it's much the lesser evil compared to a buffer overrun.

Poettering: The Biggest Myths

Posted Jan 28, 2013 19:42 UTC (Mon) by k8to (guest, #15413) [Link]

I work on various code written in various styles.

Sometimes I work on code that is assembling various things into buffers in order to generate a representation out-of-program, like on disk or on the wire. These things often work with fixed sized buffers into which they are assembling data, because the data representation has a fixed size.

In this case, changing codepaths to use strlcopy gives SIGNIFICANT improvements in safety. strncpy is not acceptable in these cases because it will incur large performance penalties with the null padding. Yes, we try to have sane and well placed logic to enforce the sizing, but putting the logic in each callsite is just begging for overruns which can be unacceptable in some use cases, so strlcpy gives us some safeguard, despite being careful in the logic.

Poettering: The Biggest Myths

Posted Jan 27, 2013 13:43 UTC (Sun) by man_ls (guest, #15091) [Link] (68 responses)

Now if you have servers running RHEL, Ubuntu, Debian, Cent OS and Fedora [...] systemd more and more makes unneeded differences between distributions go away.
In fact the problems are exactly the same, since neither Ubuntu or Debian have systemd and the others are a monoculture. Even if Debian accepts systemd as an alternative it will not be the default. As many people have said before, it would be different if Red Hat (or Fedora) had adopted Upstart which was already available, perhaps improving on it along the way. As it is, you will need two or three sets of init scripts -- the best shot seems to be a single tweaked SysV init script. Which is sad.

Poettering: The Biggest Myths

Posted Jan 27, 2013 13:58 UTC (Sun) by pizza (subscriber, #46) [Link] (64 responses)

Have you actually tried to maintain a meaningful cross-distro init script? Judging by your comments (ie "a single tweaked SysV init script") I'd say you haven't.

Those "tweaks" are where nearly all of your pain comes from. Hell, even in the same distro, one release to the next can require different tweaks. And of course they all have to be simultaneously supported.

Myths not debunked but confirmed

Posted Jan 27, 2013 15:22 UTC (Sun) by man_ls (guest, #15091) [Link] (61 responses)

No, I haven't had that bad luck.

I have recently used Upstart and I have to say that it is very nice; I cannot see why Poettering et al couldn't work with Ubuntu despite what the article says. Dependency management? Surely that could have been solved differently and deprecated with time. Contributor agreement is a problem for six full time Red Hat developers? Sounds like NIH in disguise: leaving a strategic component in the hands of a rival.

Having Upstart transform the face of Linux init would have been great. Even a fork of Upstart might have been tolerable. As it is, they have created the infamous (n+1)th standard -- and are even proud of it. Just for that they should be ostracized by the community.

Myths not debunked but confirmed

Posted Jan 27, 2013 15:42 UTC (Sun) by cortana (subscriber, #24596) [Link]

I dunno, their (n+1)th standard has been adopted by several distributions, which I assume previously all had their own home grown startup scripts and config files. I think the post-systemd world has fewer standards than before.

Myths not debunked but confirmed

Posted Jan 27, 2013 15:55 UTC (Sun) by niner (subscriber, #26151) [Link] (34 responses)

If Upstart is so great, why is it only used by Ubuntu and a few small others while systemd got adopted by openSUSE, Mageia, Arch, Fedora and others. Except for Fedora, NIH cannot explain any of these. Noone at openSUSE was enthusiastic about including Upstart, probably simply because Upstart doesn't solve any problem. SUSE already had parallel startup for a decade. systemd on the other hand solves real problems and helps getting rid of a really overcomplicated and fragile part of the system.

Myths not debunked but confirmed

Posted Jan 27, 2013 16:10 UTC (Sun) by man_ls (guest, #15091) [Link] (32 responses)

So apparently Fedora, Red Hat (plus all derivatives), Debian, Maemo, ChromeOS, NixOS, WebOS are just "a few small others"; while OpenSuse Mageia, Arch are "major ones"?

Allow me to reverse your argument: if Upstart was so lousy, why was it adopted by Fedora, Red Hat, Debian and Ubuntu at all?

Myths not debunked but confirmed

Posted Jan 27, 2013 16:26 UTC (Sun) by niner (subscriber, #26151) [Link] (22 responses)

Please read the article these comments are about. They used it, because it seemed to be better than what they had, but it was not good enough to be the solution they were really looking for.

Fedora does not use Upstart anymore. Red Hat will follow. AFAIK Debian still uses SysV init.

Myths not debunked but confirmed

Posted Jan 27, 2013 18:30 UTC (Sun) by smurf (subscriber, #17840) [Link] (10 responses)

Debian has all three of them. They think about switching to systemd and auto-generate the others' config files (which requires limiting the alowed stanzas to some subset). After the Wheezy release. Whenever *that* is.

Good idea if you ask me.

Myths not debunked but confirmed

Posted Jan 28, 2013 20:25 UTC (Mon) by drag (guest, #31333) [Link] (9 responses)

No.

It's a _TERRIBLE_ idea.

If Systemd is such a burden, then why the hell is maintaining no less then three different init systems better?!

Between sticking with sysvinit, going upstart, or going systemd, Debian has choosen the worst possible outcome. I mean this is just mind boggling terrible decision making on the part of Debian. Choosing to do _anything_ else would of been a better move. A drunk hobo rolling dice would of done a better job.

And this is really sad thing for me to say, because Debian is one of the few distros that I actually like using.

Init systems in Debian

Posted Jan 28, 2013 21:23 UTC (Mon) by man_ls (guest, #15091) [Link] (5 responses)

Well, if things work well you will not feel the burden of maintenance, any more than with the myriad of desktop environments that Debian supports. You may argue that it is a waste since (GNOME|KDE|XFCE|...) provides anything that a grown adult may desire, but proponents of the others will not agree.

And before you say that an init system is a core component... Debian maintains three different kernels, and one of them is not even widely used.

Users familiar with one of the init systems will appreciate its existence. Interested users will be able to compare init systems side by side. The rest will be grateful for being able to migrate at their own pace. That for me is enough reason to have all three.

Init systems in Debian

Posted Jan 28, 2013 21:42 UTC (Mon) by drag (guest, #31333) [Link] (4 responses)

I'd rather just have a system that actually works. I don't care about the implementation details as long as I don't have to deal with the implementation details.

By not choosing Debian has more then tripled the amount of workload they have to deal with to make the system function properly. This not only increases their workload, it exponentially increases the amount of bugs a person is likely to run into. It makes it harder to write software for the system because now you have at least 3 different operating systems personalities you have to deal with.

Throwing scripts at it to try to automate the creation of other scripts for upstart and sysinv just means that now instead of just supporting 3 different init systems they have to support 3 different init systems and a bunch of new scripts.

To put it another way:

* It makes the workload they have to deal with much higher. This is Debian's developers problem. Maybe they prefer to play around with init systems rather then having a working OS. I can't fault them for how they want to spend their time, but... if their goal is to have a stable working OS with lots of software and third party support it's not something that is going to help them any meaningful respect in that regard.

* It increases the burden of third parties writing and maintaining services much harder. This is something that anybody trying to use or support Debian is now forced to deal with.

* It exponentially increases the amounts of bugs any person is likely to run into with Debian init's system. This is something that will bite end users and will likely contribute to issues for people that try to use Debian in production.

This is why it's a bad decision. It makes the OS worse, not better. Maintaining different kernels is very silly also if your goal is to have anything other then a toy OS, but at least the non-linux-kernel-debian folks can be marginalized by the fact that nobody actually uses that stuff. Unlike KFreeBSD I expect that they will have users that actually expect that sysvinit, systemd and/or upstart works.

And what is the benefit? So I can compare init systems side by side? Doesn't seem like much of a return of on investment.

Init systems in Debian

Posted Jan 28, 2013 22:05 UTC (Mon) by smurf (subscriber, #17840) [Link]

Debian didn't decide. Or rather, they discovered that any attempt to force a decision would fail, so they decided not to decide.

In any case, I am fairly optimistic that post-Wheezy the "best practice" way to start a service in Debian will be a native systemd script, with auto-generated SysV-ish shell scripts and/or upstart-esque config files, whenever there are no special requirements.

How much time will elapse until then, I have no idea and refuse to speculate about. This is Debian, after all. :-/

Init systems in Debian

Posted Jan 28, 2013 22:08 UTC (Mon) by man_ls (guest, #15091) [Link] (2 responses)

As stated above, the same argument could be made for desktop environments, text editors, graphical editors, spreadsheets and every other redundant set of programs; not only for kernels. Perhaps you are interested in a distribution that offers just one set of integrated, polished components. That is not Debian in my experience. And that is its strength, despite the burden of development.

As to that burden, I expect that it will not be so difficult to develop for three init systems given that both Upstart and systemd claim to be SysV init-compatible. Even if the script automation script does not work out. I have had to customize init scripts every once in a while, this should not be too different.

Init systems in Debian

Posted Jan 28, 2013 23:00 UTC (Mon) by ovitters (guest, #27950) [Link]

Various components have support for systemd. Some still fall back to other methods at runtime, some do not. This does add a lot of complexity, potential bugs, etc. I noticed this when Mageia switched to fully systemd in the development version. That just took a few days, because quite easy if the goal is systemd only. If you want to support all, I guess you might need to write some code because not every component (package) compiled with systemd support has that support as a runtime extra feature.

Init systems in Debian

Posted Jan 29, 2013 16:51 UTC (Tue) by drag (guest, #31333) [Link]

The goal of the operating system is to make it easier to write and run applications/services. Anything that is done to make those activities more difficult then necessary is full of fail.

Myths not debunked but confirmed

Posted Jan 29, 2013 0:01 UTC (Tue) by rahvin (guest, #16953) [Link]

As long as I've been using Debian there has been a default set of install options, but ultimately it was possible to replace nearly everything with "non-standard" components using the Debian archives.

In other words, even though they pick a "default" they understand that every user will have different preferences and where there is developer support for it they provide the options. Honestly if they had more volunteers they'd provide even more options. Hell, they even have a hurd port and no one has a hurd port.

That's the beauty of Debian, they don't pick sides, they pick a default for a base install then allow the user to rip out and replaces with options fairly painlessly. That's the Debian way I'm familiar with, not the Debian way you are advocating.

Myths not debunked but confirmed

Posted Jan 29, 2013 17:56 UTC (Tue) by Wol (subscriber, #4433) [Link] (1 responses)

Going systemd is not an option. It only works on a subset of the systems supported by Debian.

Dunno about upstart - that may have the same problem.

So the only thing that will actually work across ALL supported platforms is SysV. But you've got to give people the OPTION of the others.

Cheers,
Wol

Myths not debunked but confirmed

Posted Jan 30, 2013 8:36 UTC (Wed) by micka (subscriber, #38720) [Link]

Different systems could have a different default init system. After all, they already have a different default kernel.

Myths not debunked but confirmed

Posted Jan 27, 2013 19:13 UTC (Sun) by man_ls (guest, #15091) [Link] (10 responses)

I read the fine article, thanks. If Upstart was not good enough, why not improve upon it instead of building their own init system? The explanation in the "debunking" is not good enough; manual dependencies might be substituted with automatic dependencies, I don't see any technical hurdles there. The general ideas behind Upstart are still good. At the same time I don't see Upstart gaining all the "features" going on to span through 69 binaries.

Debian has Upstart in testing right now, though not as the default.

Myths not debunked but confirmed

Posted Jan 27, 2013 20:03 UTC (Sun) by Jonno (subscriber, #49613) [Link]

> If Upstart was not good enough, why not improve upon it instead of building their own init system?
They did for a few years, until they realised actually fixing some of their issues would have meant rewriting upstart from scratch, breaking compatibility with the original upstart in subtle ways. That would have been systemd in all but name anyway...

> Debian has Upstart in testing right now, though not as the default.
Yes, and just like systemd...

Myths not debunked but confirmed

Posted Jan 27, 2013 23:39 UTC (Sun) by anselm (subscriber, #2796) [Link] (8 responses)

Systemd : Upstart : SysV-Init = Git : Subversion : CVS.

Myths not debunked but confirmed

Posted Jan 28, 2013 9:45 UTC (Mon) by danpb (subscriber, #4831) [Link] (4 responses)

Couldn't agree more, i've frequently used this analogy myself when talking about SystemD to people. Just like GIT there's a bit of a learning curve because of the new concepts involved, but once you're past that, you'll never want to go back to the old ways.

Myths not debunked but confirmed

Posted Jan 28, 2013 18:10 UTC (Mon) by ThinkRob (guest, #64513) [Link] (3 responses)

This would be a better analogy if CVS and Subversion were portable and Git was not, and the standard response to complaints about the complete inattention to portability were "yeah, well just use Linux because I don't want to hold Git back by making it run on obsolete OSs -- besides, you can always just keep using SVN".

Myths not debunked but confirmed

Posted Jan 28, 2013 18:44 UTC (Mon) by apoelstra (subscriber, #75205) [Link]

> This would be a better analogy if CVS and Subversion were portable and Git was not, and the standard response to complaints about the complete inattention to portability were "yeah, well just use Linux because I don't want to hold Git back by making it run on obsolete OSs -- besides, you can always just keep using SVN".

Git technically works on non-POSIX systems, but it's painfully slow. CVS and Subversion were equal-opportunity crap.

Myths not debunked but confirmed

Posted Jan 28, 2013 20:30 UTC (Mon) by drag (guest, #31333) [Link]

> This would be a better analogy if CVS and Subversion were portable and Git was not,

Sysvinit scripts are _NOT_ portable.

Myths not debunked but confirmed

Posted Jan 28, 2013 20:39 UTC (Mon) by HelloWorld (guest, #56129) [Link]

Making systemd portable is essentially impossible (myth 16) and if it weren't, it'd be pointless (myth 13). So will you *please* stop bringing up this nonsense?

Myths not debunked but confirmed

Posted Jan 28, 2013 12:13 UTC (Mon) by dgm (subscriber, #49227) [Link] (2 responses)

Well, I'm not so sure the analogy is correct. Git is build on a very simple core, with cleverly simple data structures, even if the porcelain is complex. Could it be more like:

SystemD : Upstart : SysV-Init =? Emacs : Vim : ed

Myths not debunked but confirmed

Posted Jan 28, 2013 13:38 UTC (Mon) by anselm (subscriber, #2796) [Link]

The main point of the analogy is that, just as Subversion is basically CVS with the most glaring problems papered over, Upstart is basically SysV-Init with the most glaring problems papered over.

On the other hand, like Git, systemd represents a completely different way of attacking the problem at hand.

Myths not debunked but confirmed

Posted Feb 2, 2013 10:04 UTC (Sat) by Kwpolska (guest, #89145) [Link]

Vim is better than Emacs, but Upstart is worse than systemd.

systemd/upstart adoption

Posted Jan 27, 2013 18:18 UTC (Sun) by HelloWorld (guest, #56129) [Link] (3 responses)

Maemo and WebOS are dead, Red Hat will switch to systemd with RHEL 7, Fedora already did, as did NixOS just days ago (http://www.mail-archive.com/nix-dev@lists.science.uu.nl/m...). Debian offers systemd in its unstable branch (they also have to ship sysvinit for the time being because of kFreeBSD). What does that leave? ChromeOS, because Scott James Remnant works there. Surprise, surprise...

systemd/upstart adoption

Posted Jan 29, 2013 11:53 UTC (Tue) by oak (guest, #2786) [Link]

Systemd wasn't an option for Maemo because Systemd came after Maemo.

systemd/upstart adoption

Posted Feb 5, 2013 0:55 UTC (Tue) by sonnyrao (subscriber, #11351) [Link] (1 responses)

I think that's pretty unfair to ChromeOS. Scott doesn't even really work on the system initialization part of ChromeOS, he's been working on Bluetooth support for quite a while. I think the reason systemd hasn't been investigated is more likely because the current system works and making system services boot faster hasn't been one of the most pressing problems for the project.

systemd/upstart adoption

Posted Feb 5, 2013 19:00 UTC (Tue) by khim (subscriber, #9252) [Link]

Fast boot is one of the important goals of ChromeOS. Systemd was not investigated because it will needs fine-tuning to work faster then the existing optimized boot and nobody have done the required work.

Myths not debunked but confirmed

Posted Jan 27, 2013 21:10 UTC (Sun) by misc (subscriber, #73730) [Link] (3 responses)

Upstart is not bad, but not great. There is some design issues.

For one, the way it track process is by using ptrace. That's not a bad idea, but this doesn't work great for some kind of software. There is 2 bugs opened since 4 years about that ( https://bugs.launchpad.net/upstart/+bug/406397 and https://bugs.launchpad.net/upstart/+bug/703800 ).

Then upstart do not permit to have a daemon installed but not started. The state and the config file are conflated. See http://utcc.utoronto.ca/~cks/space/blog/linux/UpstartCoup... on the details.

On the same type of issue, there is this http://utcc.utoronto.ca/~cks/space/blog/linux/UpstartDepe... , ie, upstart config are just script, so that's the same problem.

The root cause is something solved by systemd, with the .include and the new directory system ( ie, using /etc/system/foo.unit.d/foo ).

Upstart config file are also harder to parse ( ie, this is a full blown DSL ), which doesn't help to create a tool to audit or modify them. While auditing is not something I ever seen, I know that some organisations requires this, so every roadblock that can be removed is better. And as someone involved into automated package checking, having a parsable format is better.

One last issue is the whole copyright assignation to Canonical, that AFAIK is always annoying since company exec tend to avoid letting people work on such projects ( and IMHO for good reasons ).

RHEL and Fedora did used Upstart for a time, Opensuse and Debian also looked at it at the same time. So if they collectively decided to not switch or to use something else, there is likely a technical reason.

Perhaps a myth after all

Posted Jan 28, 2013 0:00 UTC (Mon) by man_ls (guest, #15091) [Link]

Thanks for a very civil and informative answer. It makes me think that myth #9 is perhaps a myth after all.

Myths not debunked but confirmed

Posted Jan 28, 2013 1:25 UTC (Mon) by mezcalero (subscriber, #45103) [Link]

To underline this, I think the sources of Upstart are really nice. It's very readable, well documented, has many tests. In many ways it's exemplary.

We didn't have any problems with the quality of code or anything. Heck, we have so much more bad code in our stack, Upstart totally stands out in quality. So, the implementation is fine, however the overall design was not, in our eyes, and that's very hard to fix without resulting in well, basically a pretty complete rewrite, and that's what we then did, after much consideration.

Myths not debunked but confirmed

Posted Jan 28, 2013 10:23 UTC (Mon) by xnox (guest, #63320) [Link]

In upstart, there is now support for /etc/init/foo.override file.
It's an overlay which can shadow any part of the job.
E.g. echo "manual" >> /etc/init/foo.override
will make sure that foo is never automatically started, only manually when a user does it explicitly.

For the NFS related dependency problem you outlined, sounds like you need a couple .overrides with modified "start on" conditions.

Myths not debunked but confirmed

Posted Jan 28, 2013 10:15 UTC (Mon) by regala (guest, #15745) [Link]

you don't seem to even remotely know what you are talking about... Fedora, upstart ? Debian, upstart ? mmmm, 2013, man, 2013.

Myths not debunked but confirmed

Posted Jan 27, 2013 20:44 UTC (Sun) by jubal (subscriber, #67202) [Link]

Upstart is used in ChromeOS too.

Nonsense.

Posted Jan 27, 2013 16:28 UTC (Sun) by HelloWorld (guest, #56129) [Link] (1 responses)

> I cannot see why Poettering et al couldn't work with Ubuntu despite what the article says.
Then you probably you don't want to.

> Dependency management? Surely that could have been solved differently and deprecated with time.
This sort of design decision is at the very core of upstart. Saying that they should have modified upstart to do what they want is like saying that one should build a car by taking a ship and replacing parts because after all, they're both means of transportation. How is that supposed to be useful? And besides, judging by the way Scott James Remnant defended his design decisions after systemd had launched (to the point where he'd refuse to adopt systemd's socket activation scheme and come up with his own design that doesn't offer any advantages), I don't think he would have agreed with such far-reaching changes.

> Contributor agreement is a problem for six full time Red Hat developers?
I don't see what Poettering et al. being a Red Hat employees has to do with accepting copyright assignment. And besides, even if these things had anything to do with each other, you'd still be ignoring at least 10 other systemd developers with commit rights.

> Sounds like NIH in disguise: leaving a strategic component in the hands of a rival.
If being in control were the motivation, the systemd project would require copyright assignment itself. It doesn't, and if Ubuntu/Canonical were interested in being a good open source citizen, they wouldn't either.

> Having Upstart transform the face of Linux init would have been great. Even a fork of Upstart might have been tolerable. As it is, they have created the infamous (n+1)th standard -- and are even proud of it.
Uh, no they didn't. systemd replaced many other init schemes in different distributions, and systemd unit files, unlike SysV init scripts, can trivially be shared between distros. Upstart never saw nearly as much adoption, and every distro except Ubuntu dumped it as soon as a sane alternative was available (this alone speaks volumes about Upstart).

Nonsense.

Posted Jan 28, 2013 1:29 UTC (Mon) by mezcalero (subscriber, #45103) [Link]

Actually, don't blame Scott here, he's not involved since quite some time, and some things you attribute to him are probably things that came after his departure.

Myths not debunked but confirmed

Posted Jan 27, 2013 18:20 UTC (Sun) by raven667 (subscriber, #5198) [Link] (3 responses)

They did work with Upstart, did all the work to integrate it with Fedora and RHEL, and have commented that they thought Upstart was very professionally written and reliable. The problem they found after implementing it is that its event based framework is fragile and doesn't solve the right problem. When they started looking around at other systems, such as launchd which runs the BSD-based Mac OS X, the found that using socket activation to allow services to be implicitly ordered automatically without needing admin intervention was a superior paradigm to the event framework that Upstart has. I believe they even looked at what it would take to modify Upstart to do auto-dependency resolution by socket activation and found that it would be too invasive and different and would break compatibility.

You can see the rationale yourself by reading blog entries by the creators or looking at recorded conference presentations on YouTube. systemd wasn't created on a whim, Lennart et. al. thought long and hard about creating N+1 init systems before tackling this problem.

Myths not debunked but confirmed

Posted Jan 27, 2013 21:15 UTC (Sun) by misc (subscriber, #73730) [Link] (2 responses)

Having a list of such conference and articles would be helpful IMHO, only for documenting the past once for all.

Myths not debunked but confirmed

Posted Jan 28, 2013 1:31 UTC (Mon) by mezcalero (subscriber, #45103) [Link] (1 responses)

Humm, we did not work on the integration of Upstart into Fedora. Neither Kay nor me should take credit for that work.

Myths not debunked but confirmed

Posted Jan 28, 2013 4:57 UTC (Mon) by raven667 (subscriber, #5198) [Link]

My mistake, I misread and assumed.

I have seen recordings of some of your presentations, I'm not sure which one where you mentioned looking at Upstart, and praising its code quality, it may have been Pushing Big Changes at foss.in 2012. There are several conference videos on YouTube but I'm not going to re-watch them all just to get proper attribution.

I was skeptical about the Journal until I saw the practical demonstration video from T-DOSE 2012 on YouTube. It might be worthwhile to include links to these various conference videos on your blog in addition to the wonderful "systemd for Administrators" series, maybe the personal presentation will slow the tide of Internet hatemail, so much is lost when your only interaction is the written word.

Myths not debunked but confirmed

Posted Jan 28, 2013 0:43 UTC (Mon) by alan (subscriber, #4018) [Link] (18 responses)

Creating an alternative design should never be ostracized. You happen to like Upstart. I happen to like systemd. We should have reasoned discussion about the pros and cons, which we will probably also see differently. Since when is diversity something to be criticized?

Myths not debunked but confirmed

Posted Jan 28, 2013 2:21 UTC (Mon) by dlang (guest, #313) [Link] (17 responses)

actually, very few people are saying that systemd should not exist.

What they are upset about is the push to make systemd mandatory (as we saw by the statements about how ubuntu is fragmenting linux by not switching to systemd)

RHEL is the 800 pound gorilla distro, if it does something, it impacts all other distros.

At this point we don't _know_ that RHEL is going to switch to systemd in the next release but the fact that systemd is being pushed by RedHat employees says that it's a real possibility.

I think it's a mistake to do so, RHEL customers have lots of sysadmins with lots of experience with sysV init, and many of those customers have lots of non-linux systems. They also have lots of old RHEL systems (many of them are still running RHEL4 and older)

Making the new RHEL systems have a completely different init system than the rest of the systems they run is not going to lead to happy admins, and taking the attitude that any admins who don't embrace having to re-learn the init system (in addition to all the other things they have to be learning to keep up) are fools who should loose their jobs is not going to improve things.

Many of these admins are working way over 40 hours/week as it is, and have a list of things they want to do (including many that are really rather important) that they don't have time to do that's months, if not years long.

it's not that they aren't willing to learn, it's that they don't see the value of disrupting this particular core piece and having to track multiple ways of doing the same thing, and remember to check which way the box they are logged into at the moment does things.

In many ways, this parallels the Gnome complaints. It really doesn't matter if the new way is better, the fact that it's so different, and therefor disruptive is a large part of the problem

Myths not debunked but confirmed

Posted Jan 28, 2013 2:39 UTC (Mon) by rahulsundaram (subscriber, #21946) [Link]

As you been informed with authoritative sources several times, we do know that RHEL is going to switch to systemd and btw, the switch is not from sys v but from upstart and both of them have sysv init compatibility and that weakens your arguments drastically.

Myths not debunked but confirmed

Posted Jan 28, 2013 12:37 UTC (Mon) by HelloWorld (guest, #56129) [Link] (15 responses)

systemd offers a high degree of backward compatibility. If you want to use SysV style init scripts, nothing will stop you. The service(8) utility still works the way it always did on RHEL, as does /dev/initctl. And even if you use these legacy tools and interfaces, you still get new features such as service management with cgroups. If this isn't enough for the people you describe, then I indeed think that they should get another job. Preferably one where they get paid for pointless whining.

Myths not debunked but confirmed

Posted Jan 28, 2013 19:51 UTC (Mon) by k8to (guest, #15413) [Link] (14 responses)

Prophecy of blaming the victim is fulfilled.

Myths not debunked but confirmed

Posted Jan 28, 2013 20:34 UTC (Mon) by HelloWorld (guest, #56129) [Link] (13 responses)

Speaking of victims, have you stopped beating your wife yet?

Myths not debunked but confirmed

Posted Jan 29, 2013 8:13 UTC (Tue) by k8to (guest, #15413) [Link] (12 responses)

Always raising the bar, aren't you.

Myths not debunked but confirmed

Posted Jan 29, 2013 12:05 UTC (Tue) by HelloWorld (guest, #56129) [Link] (11 responses)

OK, since you obviously didn't get it: I don't actually think you're beating your wife. I was merely trying to point out that the kind of loaded language you use isn't helpful (specifically, the use of the word "victim" when there's nothing to be a victim of).

cf. http://en.wikipedia.org/wiki/loaded_question

Myths not debunked but confirmed

Posted Jan 29, 2013 20:37 UTC (Tue) by k8to (guest, #15413) [Link] (9 responses)

No, I got it. You were making a false accusation using rude language.

The victims here are sysadmins who have to deal with a variety of systems who a new thing dumped on them.

Pro: systemd might be better for the subset of systems that use it

Con: most systems don't use it

Sysadmins have many things to learn and accomplish and not welcoming this relatively useless (to them) change is not a sign of incompetence but rather sanity.

That doesn't mean that they should get to dictate how the systems are architected, but they are the ones who get dumped on by this particular set of changes.

So yes, you were blaming the victims. Don't do that.

Myths not debunked but confirmed

Posted Jan 29, 2013 21:38 UTC (Tue) by ovitters (guest, #27950) [Link]

Why complain about wording and also say sysadmins are victims? I'm looking forward to RHEL7!

Myths not debunked but confirmed

Posted Jan 29, 2013 22:12 UTC (Tue) by HelloWorld (guest, #56129) [Link] (1 responses)

As evidenced by many comments here and elsewhere, systemd is *not* a relatively useless change, most people welcome it because it's easier to configure, more reliable and more featureful than what came before it. So please stop spreading this kind of FUD. Systemd makes life easier for most people, not harder, which is why most significant distros adopted it by now.

And besides, most systems don't use SysVinit either, and even those who do often bear no resemblance to each other as pretty much everything is done by system-specific scripts. The introduction of systemd actually led to an *increase* in uniformity as distros share most of the configuration files.

Myths not debunked but confirmed

Posted Jan 30, 2013 4:56 UTC (Wed) by k8to (guest, #15413) [Link]

So now you chop and change as well.

Every response has been an exercise in mischaracterization.

Myths not debunked but confirmed

Posted Jan 29, 2013 22:15 UTC (Tue) by smurf (subscriber, #17840) [Link] (5 responses)

>> Sysadmins have many things to learn and accomplish and not welcoming
>> this relatively useless (to them) change is not a sign of
>> incompetence but rather sanity.

If you truly think systemd is "relatively useless" for sysadmins, you should re-read this discussion.

Sysadmins need to be able to correctly and efficiently deal with failure situations. Once you use "systemctl status" and see instantly what the problem is, as opposed to grepping through heaps of ps and syslog output, you will not want to go back.

A more obscure, yet incredibly useful, feature of systemd is to start services in a consistent environment. No root login whose strange environment settings can contaminate the daemon and render its messages useless. No tty which can randomly block or vanish. No job control that can block your program or send it strange signals. And so on.

Myths not debunked but confirmed

Posted Jan 30, 2013 4:57 UTC (Wed) by k8to (guest, #15413) [Link] (4 responses)

If you're a sysadmin of a monoculture, sure it will eventually be useful.

AFAICT, those people are now called "devops", and they may love it.

Myths not debunked but confirmed

Posted Jan 30, 2013 7:19 UTC (Wed) by smurf (subscriber, #17840) [Link] (3 responses)

Translation: "Incremental progress is bad."

You seem not to notice that you contradict yourself.

Oh well.

Myths not debunked but confirmed

Posted Jan 30, 2013 12:16 UTC (Wed) by mgb (guest, #3226) [Link] (2 responses)

@smurf - The words which k8to wrote have no need of your "translation".

Incremental progress is good.

systemd is bad because its poor design and premature optimization create unnecessary coupling which inhibits incremental progress across a critical mass of system software.

Nevertheless it appears that the systemd problem has been solved. Fedora and Redhat and some other distros have been borged but Debian and Ubuntu and many other distros have stood firm.

FLOSS will simply evolve to leave the systemd-limited distros behind. Debian and Ubuntu and other distros will offer parallel boot options and cgroups options to their users without compromising FLOSS evolution.

Myths not debunked but confirmed

Posted Jan 30, 2013 13:28 UTC (Wed) by anselm (subscriber, #2796) [Link]

FLOSS will simply evolve to leave the systemd-limited distros behind.

I think that, given time, the problem will take care of itself since nobody will be able to come up with anything that is both enough of a significant improvement on System V init to get System V init anywhere near what systemd does even today, compatible with tradition enough so the die-hard System V init fans won't complain, and gets traction in enough distributions so people will actually be interested. (The ones which have subscribed to systemd already aren't going back unless whatever the System V init camp has to propose is really a lot better than systemd, which would be quite surprising.) This is very unlikely to actually happen since historically the distributions didn't even seem to be able to agree on a standard for init scripts, let alone all of System V init or indeed an evolutionarily improved System V init.

In the meantime, systemd will improve even further and, with the major Linux distributions behind it, will become even more compelling. In effect, System V init will leave systemd behind in exactly the way that CVS left Git behind.

Feel free to prove me wrong.

Myths not debunked but confirmed

Posted Jan 30, 2013 14:47 UTC (Wed) by mpr22 (subscriber, #60784) [Link]

Somehow, "FLOSS will simply evolve to leave the systemd-limited distros behind." makes me think "What, like XFree86 evolved to leave the X.org-limited distros behind?"

Myths not debunked but confirmed

Posted Jan 29, 2013 20:42 UTC (Tue) by k8to (guest, #15413) [Link]

Nevermind that the whole IDEA of loaded questions is that you ask a question and presuppose the framing is accurate in a weasely way. There was no *question* so the linguistic trick of a loaded question cannot even apply.

I made a statement which linguistically a clear thing which can be accepted or rejected by the reader. So your attempt to imply such underhanded tactics is itself underhanded. Which, if you didn't realize, is why you were being unreasonably rude.

Poettering: The Biggest Myths

Posted Jan 27, 2013 16:20 UTC (Sun) by dskoll (subscriber, #1630) [Link] (1 responses)

Have you actually tried to maintain a meaningful cross-distro init script?

Why, yes. Yes, I have. Our (commercial) product runs on all flavors of Linux as well as any UNIX-like system (the BSDs, Solaris, etc.)

I did the rather heretical thing of writing our init script in Perl. Our product requires Perl anyway, so we know that Perl is going to be available on the system. And it was quite easy.

I'm not against systemd. I think it has some very nice features and probably is the way of the future; I just wanted to point out that writing portable init scripts is not that daunting.

Poettering: The Biggest Myths

Posted Jan 27, 2013 20:59 UTC (Sun) by pizza (subscriber, #46) [Link]

>>Have you actually tried to maintain a meaningful cross-distro init script?

>Why, yes. Yes, I have. Our (commercial) product runs on all flavors of Linux as well as any UNIX-like system (the BSDs, Solaris, etc.)

Yay!

>I did the rather heretical thing of writing our init script in Perl. Our product requires Perl anyway, so we know that Perl is going to be available on the system. And it was quite easy.

Perl makes some things easier (if nothing else, it has real data structures!) but it doesn't simplify integrating into the rest of the system.

Indeed, I couldn't even rely on having *bash* because among the targets for aforementioned init scripts were embedded systems with 4MB flash -- certainly no room for perl either)

In my case I had to integrate into/with distro networking layers, which was about as insane as things could get.

Poettering: The Biggest Myths

Posted Jan 27, 2013 17:37 UTC (Sun) by HelloWorld (guest, #56129) [Link]

> As many people have said before, it would be different if Red Hat (or Fedora) had adopted Upstart which was already available,
Fedora *did* adopt Upstart, and it sucked so hard and fundamentally that they decided to drop it at the first viable oppotunity.

Poettering: The Biggest Myths

Posted Jan 27, 2013 23:49 UTC (Sun) by shmerl (guest, #65921) [Link] (1 responses)

I'm running systemd on my Debian testing.

http://wiki.debian.org/systemd

systemd on debian

Posted Jan 28, 2013 10:45 UTC (Mon) by smurf (subscriber, #17840) [Link]

So am I. The main problem is that it's an ancient version; re-packaging dbus is unlikely to happen before the Wheezy release.

Poettering: The Biggest Myths

Posted Jan 27, 2013 21:16 UTC (Sun) by misc (subscriber, #73730) [Link] (1 responses)

timezone is the wrong example, because posix kinda mandate to have /etc/localtime to be the timezone information.

I do manage with puppet several server on various distributions and I do it like this.

Poettering: The Biggest Myths

Posted Jan 27, 2013 23:03 UTC (Sun) by ovitters (guest, #27950) [Link]

That is what I initially did, change that file. But then some distributions reverted that change sometimes (IIRC). You had to change the config which they looked at, otherwise you'd start the server with one timezone. Then once puppet service did its thing, /etc/localtime would change. However, all started services are still running under the different timezone.

I gave this example due to experience I had. You expect at least this to be easy, in practice it was not.

Poettering: The Biggest Myths

Posted Jan 27, 2013 13:04 UTC (Sun) by robert_s (subscriber, #42402) [Link] (1 responses)

> but I am also interested in cross OS support

Why?

Why should we all be stuck using a lowest common denominator feature set?

Poettering: The Biggest Myths

Posted Jan 28, 2013 16:18 UTC (Mon) by misc (subscriber, #73730) [Link]

It is kinda important to have it running on windows, they have a rather huge market share. Unless when people say "cross os support", they mean "except windows" :)

Openrc a better init system?

Posted Jan 28, 2013 10:51 UTC (Mon) by Wol (subscriber, #4433) [Link]

Well, I'm seriously considering converting my gentoo system from openrc to systemd. I'm just worried what I'm going to break.

On the other hand, however, what I would GAIN from converting to systemd is obvious - a proper, working, multihead system.

At the moment, my X server is configured to enable remote login, from my ancient 2nd PC and via Cygwin from the Win7 laptop. And using it is dire :-( I'd much rather just plug in my 2nd monitor and keyboard, and have a proper, working, multi-user system, just like Unix used to be :-)

Cheers,
Wol

Poettering: The Biggest Myths

Posted Jan 27, 2013 4:57 UTC (Sun) by imgx64 (guest, #78590) [Link] (11 responses)

>UNIX and Linux are evolving tool sets.

Myth 10.

>Busybox and systemd, though they may ship a billion copies, are monolithic evolutionary dead ends.

Myth 1.

>Busybox, unlike systemd, is a valuable and portable tool which does not inhibit the continuing evolution of UNIX and Linux.

Myth 16.

>I see a potential role for a minimal init replacement, but not for a grotesque monolith actively being exploited as a power play by its author.

Myth 27.

>I am in FLOSS for the long haul. Others may have different priorities.

Myth 19.

Serious question: did you read the article?

Poettering: The Biggest Myths

Posted Jan 27, 2013 7:40 UTC (Sun) by mgb (guest, #3226) [Link] (10 responses)

> Myth 10.

(sigh) Wrong. Neither in myth 10 nor anywhere else does Lennart dispute that "UNIX and Linux are evolving tool sets".

> Myth 1.

Systemd makes 69 binaries interact so tightly that it is beyond the political ability of most competent programmers to replace a portion with a better solution. That is monolithic even though Lennart purports to define monolithic otherwise. Systemd is harmless if used on leaf nodes as BusyBox is. If used on a main trunk of Linux, systemd will cause serious harm.

> Myth 16.

Lennart admits that systemd is not portable. As such it is harmful. All Lennart claims in his myth 16 is that there is a reason for his harming Linux.

> Myth 27.

Lennart most definitely exploits systemd as a power play. We have seen this numerous times. A recent instance was "support for some distribution specific legacy configuration file formats has been dropped. We recommend distributions to simply adopt the configuration files everybody else uses now", which translated means that he's using systemd as a power play to try to make the overwhelming majority of distros change configuration files to match those of his tiny minority.

> Myth 19.

Lennart is correct. Lennart's cabal cannot force people with their eyes open to fall into his systemd trap. It is important that balanced information be available for consideration, not just Lennart's manifestos.

Serious suggestion: perhaps you'd do better to author a considered comment appropriate to the situation rather than relying on a monolithic manifesto to do your job badly.

Poettering: The Biggest Myths

Posted Jan 27, 2013 8:37 UTC (Sun) by imgx64 (guest, #78590) [Link]

> (sigh) Wrong. Neither in myth 10 nor anywhere else does Lennart dispute that "UNIX and Linux are evolving tool sets".

Your statement implies that systemd is un-UNIX-like. Myth 10 answers that. If you didn't mean to imply that, then I'm sorry.

> Serious suggestion: perhaps you'd do better to author a considered comment appropriate to the situation rather than relying on a monolithic manifesto to do your job badly.

I don't see why. All of your arguments (in both comments) are answered right there in the article.

Poettering: The Biggest Myths

Posted Jan 27, 2013 13:02 UTC (Sun) by robert_s (subscriber, #42402) [Link]

> Lennart's cabal

*rolleyes* here we go.

Was it you that was complaining about "ad-hominem" attacks above?

Poettering: The Biggest Myths

Posted Jan 27, 2013 17:15 UTC (Sun) by HelloWorld (guest, #56129) [Link]

> Systemd makes 69 binaries interact so tightly that it is beyond the political ability of most competent programmers to replace a portion with a better solution.
This simply isn't true. Please stop spreading lies like these.

> Lennart admits that systemd is not portable. As such it is harmful.
It's not, see myth 13.

> All Lennart claims in his myth 16 is that there is a reason for his harming Linux.
How is an alleged problem that doesn't affect Linux at all supposed to harm it? You're not making any sense.

> Lennart most definitely exploits systemd as a power play. We have seen this numerous times. A recent instance was "support for some distribution specific legacy configuration file formats has been dropped. We recommend distributions to simply adopt the configuration files everybody else uses now", which translated means that he's using systemd as a power play to try to make the overwhelming majority of distros change configuration files to match those of his tiny minority.
Systemd is free software, if some distro wants to keep using these configuration files, they can maintain the code required to do so themselves -- in fact, this very possibility is mentioned in the NEWS file. systemd doesn't stop you from doing anything. It just won't help you to do certain things, and that's something *completely* different.

Poettering: The Biggest Myths

Posted Jan 27, 2013 19:46 UTC (Sun) by rgmoore (✭ supporter ✭, #75) [Link]

A recent instance was "support for some distribution specific legacy configuration file formats has been dropped. We recommend distributions to simply adopt the configuration files everybody else uses now", which translated means that he's using systemd as a power play to try to make the overwhelming majority of distros change configuration files to match those of his tiny minority.

This would be a more plausible theory if they had enforced the Fedora/RedHat filenames and locations in all cases, but they didn't. And, for what it's worth, this was probably a worthwhile effort regardless of whether it was associated with systemd. There is no particularly good reason for each distribution to keep the same information in different formats and locations. All that doing so achieves is confusion, fragmentation, and needless duplication of effort. This kind of standardization creates some short-term pain, but it will make everyone's life easier in the long run.

Poettering: The Biggest Myths

Posted Jan 28, 2013 10:34 UTC (Mon) by andreashappe (subscriber, #4810) [Link] (2 responses)

> Serious suggestion: perhaps you'd do better to author a considered comment appropriate to the situation rather than relying on a monolithic manifesto to do your job badly.

*sigh* comments like that make me wish for killfile support in lwn.

Poettering: The Biggest Myths

Posted Jan 28, 2013 11:31 UTC (Mon) by cladisch (✭ supporter ✭, #50193) [Link] (1 responses)

> killfile support in lwn

… is there for subscriptions at “professional hacker” and higher levels.

Poettering: The Biggest Myths

Posted Jan 29, 2013 17:07 UTC (Tue) by mgedmin (subscriber, #34497) [Link]

What a timely comment!

My subscription has expired a few days ago (and I noticed only now when I was trying to check what my level was), and your comment has convinced me to upgrade from starving to professional hacker.

Poettering: The Biggest Myths

Posted Jan 28, 2013 17:13 UTC (Mon) by malor (guest, #2973) [Link] (2 responses)

which translated means that he's using systemd as a power play to try to make the overwhelming majority of distros change configuration files to match those of his tiny minority.

You're characterizing this as a 'power play', but I don't see how he would particularly benefit. Config files aren't that big a deal. As long as they're still flat text files, and not some weird binary format like the Windows Registry, then it's usually pretty easy to adapt tools to read and generate a little different format.

It just doesn't seem to me that config files are worth getting het up about. As long as they're flat files, they're easy to translate and move around. Yeah, it might be a little more work, but if you're on a really oddball distro with a strange format for configuration, well, that's the price you'll have to pay if you want to run systemd. Maybe it's worth it to you, maybe it isn't, but it doesn't appear likely that any other software is going to stop working if you don't have it.

At least at present, the only benefit I see to Lennart in dropping minority config files is saving work, and I don't particularly begrudge him that.

As a thought experiment, if every Linux distro, everywhere, switches to systemd format for configuration files, do we really lose anything? Does it actually matter, or is it just resistance to change? Perhaps a feeling of losing a competition?

tl;dr version: I don't see any reason for a big emotional investment in config files. Am I missing something?

Poettering: The Biggest Myths

Posted Jan 28, 2013 17:44 UTC (Mon) by sorpigal (guest, #36106) [Link] (1 responses)

From systemd's point of view it's rational, but let me point out that what you call "Minority config files" I call "Consistency." Example: Debian stores defaults in /etc/default/ pretty consistently, moving locale to /etc/locale.conf makes the system less coherent--unless you move *ALL* of the defaults similarly, and systemd has not (yet) expanded to be a general settings manager. It just doesn't make sense for Debian to move backwards like this in order to move forwards (or sideways, or however you want to see it) with systemd. It's not up to Debian to sway because other distributions didn't have a nicer system for setting this before (but IANADD, so perhaps they're already adopting the change.)

The same argument can be made from other distro viewpoints regarding /etc/syscofnig/, I imagine, but I know Debian and used it for my example.

Some of the emotional reaction comes, I think, from a feeling of being put upon by a heavy handed outsider. Who is Lennart (or what is systemd, if you prefer to say it that way) to come in to my distro and declare that my distro's config files are "old" and in need of replacement when I and my cohorts carefully chose a good way of solving the problem that has now been working for years?

"You are nonstandard, you will be assimilated. Your technological distinctiveness will be added to our own." --systemd, addressing udev, cron, initd, Debian, Suse, etc.

Poettering: The Biggest Myths

Posted Jan 28, 2013 18:07 UTC (Mon) by malor (guest, #2973) [Link]

Well, I'm a big Debian user too. I don't care what the actual configuration format is, but I do like stuff in /etc/default.

That said, I don't think putting locale.conf in /etc is that big a deal. There's plenty of other system-wide conf files in there; there's never been a full transition. The timezone file is in /etc, as an example, and I'd call that almost exactly identical in scope. I see no logical reason for timezone to be in /etc, with locale in /etc/default, but that's how it presently is. It seems likely to be backward compatibility driving that choice, rather than technical merit or logical consistency.

Like all other flavors of Unix, Debian is sloppy about where it puts configuration.... look at all the crap under /var/run, as an example. As much as I like it, and as much as I use it, I don't think we should try to paint it as the shining city on the hill. It's actually quite messy.

In this particular case, I guess I'd just sort of shrug and use symlinks. We can either stick them in default and link to them from /etc, or the other way around. I don't see it as being that big a deal.

Yeah, maybe it's a little rude on Poettering's part, but meh, not worth worrying about that much. Adding one more tiny inconsistency to a distro that already has so many doesn't strike me as a major aesthetic affront. You can argue that we'll never get to the hill if we keep moving away from it, but I don't think a symlink is really moving away.

That said, it seems a little silly for Poettering to refuse a patch to look in one more directory.

Poettering: The Biggest Myths

Posted Jan 28, 2013 8:28 UTC (Mon) by sbakker (subscriber, #58443) [Link] (52 responses)

I see a potential role for a minimal init replacement, but not for a grotesque monolith ...
Uhm, if you are so wary of monolithic pieces of software, why are you using the Linux kernel in the first place...?

Poettering: The Biggest Myths

Posted Jan 28, 2013 12:46 UTC (Mon) by HelloWorld (guest, #56129) [Link]

Poettering: The Biggest Myths

Posted Jan 28, 2013 13:03 UTC (Mon) by davidstrauss (guest, #85867) [Link] (50 responses)

> Uhm, if you are so wary of monolithic pieces of software, why are you using the Linux kernel in the first place...?

I don't think that's a valid critique. Perfectly rational people can have preferences in the design of the tools they use and not match all preferences every time.

Being wary of monolithic software (which I'm not says systemd is) means that: wary. It means, on a case-by-case basis, you're skeptical of monolithic design and prefer things not be that way. It's not hypocritical to challenge software project A on being monolithic but use monolithic software project B when project B is proven and well-supported.

You can't possibly expect that the only way to challenge monolithic design is to run Hurd or some other microkernel.

Poettering: The Biggest Myths

Posted Jan 28, 2013 21:27 UTC (Mon) by khim (subscriber, #9252) [Link] (49 responses)

You can't possibly expect that the only way to challenge monolithic design is to run Hurd or some other microkernel.

The only way to show that monolithic design is somehow bad is to show that alternative exist which is non-monolithic and better.

There are exactly two classes of systems:
1. Monolithic ones (as per sorpigal's insane definition), and
2. Failed ones.

Most popular OSes are all almost completely monolithic per this definition: Android, Busybox, BSDs. Only GNU/Linux is weird creature which includes robust and reliable monolithic part (GNU components: GCC, GLibC, binutils, coreutils, etc - they are not really supposed to be used with anything else) and loose and flaky things like sysvinit with it's plethora of weird scripts.

Frankly: init is supposed to be small, simple and reliable part (it's tiny in comparison, to e.g. GLibC) but in reality it's least reliable part - and if you'll think this is exactly because it's not "monolithic". Systemd fixes that and it's a good thing.

People have preached "write programs that do one thing and do it well" that they forgot the whole thing which says instead Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

How exactly can you convince programs to work together if you don't write them together? Unix was always created as a large cohesive set of tools which work together, not as a bazillion pieces to be assembled by the end user. Systemd does the same thing - yet somehow people try to say that's un-UNIX-like. How can the basic principle used by Unix for the last fourty years be un-UNIX-like?

you do NOT need to write all your programs together to make them work together.

Posted Jan 28, 2013 21:47 UTC (Mon) by dlang (guest, #313) [Link] (37 responses)

> Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

This is exactly what is not happening. Instead systems are being written that try very hard to force you to replace many tools with this new system, and the components of the new system do not maintain compatibility with the old interfaces.

> How exactly can you convince programs to work together if you don't write them together?

this is a core part of the problem

you make programs work together, not by writing them together, but by defining the interface between them. That way the different programs can be written by different people at different times, replaced one by one, etc

That is the way to have a long-term robust solution.

If you consider that all the programs need to be written together, it's a very small step to consider that you can break all your old interfaces because you ship all new versions of everything.

Linux desktop environment projects are following this logic, and users quite rightly complain when major changes happen that break how they used to work, and make it so that you can't use old components that work great for you with the new version.

APIs need to be defined, and you need to support old ones going forward.

you do NOT need to write all your programs together

you do NOT need to write all your programs together to make them work together.

Posted Jan 28, 2013 21:56 UTC (Mon) by raven667 (subscriber, #5198) [Link] (7 responses)

> This is exactly what is not happening. Instead systems are being written that try very hard to force you to replace many tools with this new system, and the components of the new system do not maintain compatibility with the old interfaces.

That's not something that's happening with systemd though as the developers have worked very hard on backwards compatibility including automatically importing /etc/init.d, parsing /etc/fstab, supporting initctl, supporting tools such as chkconfig. I'm not sure what more could possibly be done.

you do NOT need to write all your programs together to make them work together.

Posted Jan 28, 2013 22:14 UTC (Mon) by dlang (guest, #313) [Link] (6 responses)

> I'm not sure what more could possibly be done.

don't merge udev, go ahead and enhance udev to provide whatever service you are wanting to use, but keep udev a separate project so that people who don't want systemd continue to be first-class users

don't reinvent syslog, if there is something that you need, contact the syslog maintainers and ask for the feature if you want it. Or if they don't have time to implement the feature, write it and contribute it (all of the syslog daemons accept outside contributions, and rsyslog, the default is very accepting, and supports sponsoring new features if you can't just convince them it's a good idea)

etc

for the most part it's not that systemd needed to do more, it's that they should have done less, writing utilities and working with existing tools rather than reinventing them or re-implementing part of the existing functionality inside systemd.

The latest thing with CRON is another example, the argument seems to be that they need to take over cron because it's another way to start a process, and they need to know about all process starts to 'manage' them.

How about instead defining a 'startup' program that starts jobs in their own cgroup, notifying interested programs through some mechanism (init being one of them, but probably not the only one) and then get CRON modified to use this program to fire off the apps.

This program could then be used by other things, desktop environments could use it to start apps, user scripts could use it to launch apps. Things besides init could watch for processes to start (such as high availability and other monitoring software)

This would be "the unix way", a tool that does one thing "launch an app with appropriate isolation and notification) that could be re-used by many other people

you do NOT need to write all your programs together to make them work together.

Posted Jan 28, 2013 22:39 UTC (Mon) by davidstrauss (guest, #85867) [Link] (3 responses)

> don't merge udev, go ahead and enhance udev to provide whatever service you are wanting to use, but keep udev a separate project so that people who don't want systemd continue to be first-class users

It's sort of touched on in myth #10. Separate git repositories is not the only path to modularity.

> don't reinvent syslog, if there is something that you need, contact the syslog maintainers and ask for the feature if you want it. Or if they don't have time to implement the feature, write it and contribute it (all of the syslog daemons accept outside contributions, and rsyslog, the default is very accepting, and supports sponsoring new features if you can't just convince them it's a good idea)

Projects don't own turf just because they provide the current implementation. Contributing to existing projects is certainly a good method when possible, but I don't think you'd get very far asking MongoDB developers why they didn't try to get their changes progressively into PostgreSQL or MariaDB.

> The latest thing with CRON is another example, the argument seems to be that they need to take over cron because it's another way to start a process, and they need to know about all process starts to 'manage' them.

My same thoughts here. anacron didn't integrate their features into cron. Nor did chrony seek to slowly improve ntpd. Nor did git seek to build on Subversion. (And there are many more examples.) Why does systemd have a unique responsibility to improve other implementations rather than providing capable replacements?

> How about instead defining a 'startup' program that starts jobs in their own cgroup, notifying interested programs through some mechanism (init being one of them, but probably not the only one) and then get CRON modified to use this program to fire off the apps.

That sounds like a reasonable capability to me, but it doesn't have any bearing on whether systemd should implement other features like a cron equivalent and the journal.

you do NOT need to write all your programs together to make them work together.

Posted Jan 28, 2013 23:17 UTC (Mon) by dlang (guest, #313) [Link] (2 responses)

> Separate git repositories is not the only path to modularity.

no, they are not. And separate git repositories do not guarantee modularity either (as an example, see the historic problems that have existed with ALSA user space vs kernel code)

There is a mindset required.

1. you have to accept that there will be other users of your interfaces

2. you have to accept that there will be people who don't use your favorite components

3. you have to define your interfaces

4. you have to maintain backwards compatibility for your interfaces (no dropping features just because it's convenient for you and you don't need them any longer)

> Projects don't own turf just because they provide the current implementation. Contributing to existing projects is certainly a good method when possible, but I don't think you'd get very far asking MongoDB developers why they didn't try to get their changes progressively into PostgreSQL or MariaDB.

did they even _try_ to work with existing projects? in the case of journal/syslog we know that they didn't, from the simple fact that a lot of the things that they claimed syslog couldn't do (and therefor justified the journal), were in fact easily done with syslog, and some of the other things were added within days of the journal announcement.

It's not like the journal added capabilities that are so shocking and new that they can't possibly work in standard syslog daemons.

> My same thoughts here. anacron didn't integrate their features into cron...Nor did git seek to build on Subversion....Why does systemd have a unique responsibility to improve other implementations rather than providing capable replacements?

git doesn't force you to not run subversion, being a system tool means that systemd needs to be more accomodating than user tools where multiple ones can be used.

in the case of anacron, you have a single purpose tool that replaced another single purpose tool, and did so in a way that supported existing configs. If systemd was just an init that fully supported everything that was in sysV init (it supports some, but not all, inittab for example) and was presented as "in this version we switch from init to systemd and you don't see any changes if you aren't using new features", and then added new ways of streamlining the things that init has been responsible for, you would not have nearly the pushback that you have now.

> That sounds like a reasonable capability to me, but it doesn't have any bearing on whether systemd should implement other features like a cron equivalent

in the case of CRON, if you were to take away the justification of needing to know about all the processes by having a 'startup' program, what's the justification for replacing a established, well known tool with a new implementation inside systemd?

you do NOT need to write all your programs together to make them work together.

Posted Jan 28, 2013 23:51 UTC (Mon) by davidstrauss (guest, #85867) [Link]

> 1. you have to accept that there will be other users of your interfaces

Multiple external projects use the systemd interfaces. We've built a lot of them here at Pantheon. Some, like the Python journal API, have been moved to the systemd core repository. Some interfaces, like the PHP, Lua, and node.js interfaces, remain external.

Projects like the journal gateway are *entirely* APIs for external use and were built to improve inter-tool operation.

> 2. you have to accept that there will be people who don't use your favorite components

In the case of cron, there's no conflict at all. Keep running cron if you want to. It might eventually not be installed by default, but I'm sure it will be packaged and maintained for the next decade or so.

In the case of the journal, messages collected by systemd do have to travel through it but can be forwarded to syslog. syslog can also forward to the journal. The journal, by default, runs as an in-memory-only pass-through for messages to syslog.

In the case of init itself, you obviously can't have two co-existing as PID 1 on the same system, but systemd support init scripts, LSB headers, and the standard, traditional tools for managing services.

> 3. you have to define your interfaces

They're either plain text with key/value pairs (the journal interface), documented CLI tools, or metadata-defined D-Bus serializations.

> 4. you have to maintain backwards compatibility for your interfaces (no dropping features just because it's convenient for you and you don't need them any longer)

You can generally mix and match new and old API invocations without much trouble. For example, journalctl runs fine with newer and older journald implementations. The CLI tools have been consistent enough that we haven't had to alter Chef's systemd service support since it was implemented over a year ago. I'm not familiar enough with the D-Bus interfaces to comment on those.

> did they even _try_ to work with existing projects? in the case of journal/syslog we know that they didn't, from the simple fact that a lot of the things that they claimed syslog couldn't do (and therefor justified the journal), were in fact easily done with syslog, and some of the other things were added within days of the journal announcement.

First, you need to cite the things that were supposedly claimed impossible with syslog but were actually easy, including where systemd maintainers made the claim.

Second, a project doing something within days of a fork is not evidence they were receptive to the change before. The Linux kernel stonewalled Android-style sleep-as-default (I forget the exact name) until Android forked the kernel and demonstrated a successful architecture for it. It took more than a few days, but it's still clear that competitive pressure is quite different from lobbying a project.

> It's not like the journal added capabilities that are so shocking and new that they can't possibly work in standard syslog daemons.

Maybe, but nothing comparable (even in terms of major changes unrelated to the journal's goals) has happened with syslog daemons in the last decade of running Unix logging. Until the journal, I still couldn't log a PHP backtrace in any way that kept the lines all together.

> git doesn't force you to not run subversion, being a system tool means that systemd needs to be more accomodating than user tools where multiple ones can be used.

systemd timer units don't force you to not run cron. Stop spreading obvious FUD.

> "in this version we switch from init to systemd and you don't see any changes if you aren't using new features"

That was basically the case for Fedora aside from dropping rc.d symlinks. The journal only came later, and it runs as a completely separate daemon and CLI toolset.

> what's the justification for replacing a established, well known tool with a new implementation inside systemd?

Consistent configuration of executable units (whether it's a service or a periodic job) and consistent administrative tools for logging and monitoring success/failure. It's unclear to me why anyone would want to configure things the old cron way once they're accustomed to writing systemd units.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 2:54 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

> There is a mindset required.
> 1. you have to accept that there will be other users of your interfaces
> 2. you have to accept that there will be people who don't use your favorite components
> 3. you have to define your interfaces
> 4. you have to maintain backwards compatibility for your interfaces (no dropping features just because it's convenient for you and you don't need them any longer)

Well, systemd does that[1].

[1]http://www.freedesktop.org/wiki/Software/systemd/Interfac...

you do NOT need to write all your programs together to make them work together.

Posted Jan 28, 2013 23:17 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

> How about instead defining a 'startup' program that starts jobs in their own cgroup, notifying interested programs through some mechanism (init being one of them, but probably not the only one) and then get CRON modified to use this program to fire off the apps.
Internally "cron" subsystem is implemented by a separate utility. It just uses the common systemd infrastructure and can be easily developed separately.

It's certainly possible to hook up systemd to launch services using DBUS interface. And you can then use it with any kind of triggers - I've actually used good old cron to start systemd tasks for a while.

you do NOT need to write all your programs together to make them work together.

Posted Jan 28, 2013 23:22 UTC (Mon) by raven667 (subscriber, #5198) [Link]

> don't merge udev, go ahead and enhance udev to provide whatever service you are wanting to use, but keep udev a separate project so that people who don't want systemd continue to be first-class users

I'm not sure that this has anything to do with what we were discussion about systemd compatibility with existing management methods but I should point out that in this case udev is still very much supported as a stand-alone component although it is fair to mention that because it shares build scripts with the rest of systemd so there there are additional build time dependancies that aren't explicitly needed by udev alone. This is of interest to developers and Gentoo users but is of no practical effect to anyone else.

> don't reinvent syslog, if there is something that you need, contact the syslog maintainers and ask for the feature if you want it.

http://blog.gerhards.net/2011/11/journald-and-rsyslog.html

Some thoughts on the Journal by the main rsyslogd maintainer. What might be interesting is if a tool like rsyslog would talk the Journal protocol natively, possibly replacing the systemd included implementation for large environments. In the blog post there are several examples given as to why building a new component might be a better approach, because of the things that you can't change for compatibility reasons, even if you improve them in rsyslog.

you do NOT need to write all your programs together to make them work together.

Posted Jan 28, 2013 22:20 UTC (Mon) by davidstrauss (guest, #85867) [Link] (16 responses)

> This is exactly what is not happening. Instead systems are being written that try very hard to force you to replace many tools with this new system, and the components of the new system do not maintain compatibility with the old interfaces.

First, it's not clear what loss of interface compatibility you're lamenting. systemd supports most of the same traditional CLI utilities, syslog protocol (both in and out of the journal), inetd-style service activation, and SysV-style init scripts (including the LSB metadata). The journal's CLI supports text streams compatible with all the standard shell tools.

That said, needing to replace many tools at once isn't inherently evidence of a monolithic design. It can also be a sign that the boundaries of functionality or interfaces between tools have changed.

For example, we added the journal with a new API and a compact on-disk format. Necessarily, we have to have a new daemon to receive messages (journald), a new tool for browsing the files as text streams (journalctl), and a new C header for doing the structured logging. The journal would be rather useless for a long time if added in program by program over years. Nothing in this design is more monolithic or less Unix-y than what it replaced, but it did require replacing all of those parts at once to get end-user value.

There is nothing in this statement about how many programs you should change at once:
"Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface."

systemd is not a single "program," nor is splitting an implementation over many version control repositories (a thing many people seem to fixate one) a sign of good, modular design.

> That way the different programs can be written by different people at different times, replaced one by one, etc

Many of the interfaces for systemd are well-defined: the D-Bus API, the native journal field format, discovery of inherited sockets for activation. Those latter two already have multiple implementations talking over the defined interfaces, often for scripting languages that don't want to link to the C functions systemd exports or for projects outside the main systemd repository. We're currently looking at having scripting languages like Python manage services directly over D-Bus as well to enforce stability and completeness for those interfaces.

The Python-based boot graph analysis tool that talks to systemd over D-Bus is about to get replaced with a completely new implementation in C by a different author.

So, systemd has already demonstrated the ability to substitute implementations interfacing over well-defined boundaries.

Therefore, I think it's unnecessary to replace programs one by one rather than as a suite. Nor does the Unix philosophy you quoted say anything about this concern.

you do NOT need to write all your programs together to make them work together.

Posted Jan 28, 2013 23:23 UTC (Mon) by dlang (guest, #313) [Link] (15 responses)

> For example, we added the journal with a new API and a compact on-disk format. Necessarily, we have to have a new daemon to receive messages (journald), a new tool for browsing the files as text streams (journalctl), and a new C header for doing the structured logging. The journal would be rather useless for a long time if added in program by program over years. Nothing in this design is more monolithic or less Unix-y than what it replaced, but it did require replacing all of those parts at once to get end-user value.

and why did you need to change all of those things?

syslog daemons already supported outputting to many different on-disk formats, why should you be requiring a different format?

Why invent a new way of doing structured logging? there were already N standard ways of doing structured logging, why did you have to invent a new one for systemd?

etc....

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 0:00 UTC (Tue) by davidstrauss (guest, #85867) [Link] (14 responses)

> syslog daemons already supported outputting to many different on-disk formats, why should you be requiring a different format?

None of those approaches supports field-structured logging because syslog has no method for handling such messages being sent to the daemon, let alone preserving the data on disk after receipt. You could have syslog write to an RDBMS like SQLite for all it matters, but if the structure's already gone, the structure's already gone.

The journal format also has a number of nice attributes like FSS, compression, atomic rotation, and indexing. And it's all available as friendly text streams in many formats using journalctl.

> Why invent a new way of doing structured logging? there were already N standard ways of doing structured logging, why did you have to invent a new one for systemd?

What are those N ways? Certainly, syslog doesn't support field-structured logging. Protocols like GELF are asynchronous and lossy by design. Java's logging tools have obvious lack of usability on many embedded systems. Tools like Flume are designed to harvest logs off disk after they've already been written.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 1:16 UTC (Tue) by HelloWorld (guest, #56129) [Link]

dlang has been spreading falsehoods and refused to educate himself about the design rationale of systemd, the journal etc. so many times that I suggest you stop wasting your time.
Haters gonna hate, that's life.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 1:30 UTC (Tue) by mgb (guest, #3226) [Link] (12 responses)

Field structured logging might be useful for some people.

So write a field-structured logger and allow people to use it. That's the UNIX way. That's the way that allows UNIX to progress.

Don't leverage your control of pid 1 to force people to use your logger. That's the Poettering way. That's the way that stifles UNIX progress.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 2:06 UTC (Tue) by raven667 (subscriber, #5198) [Link] (8 responses)

> Field structured logging might be useful for some people.

True

> So write a field-structured logger and allow people to use it. That's the UNIX way. That's the way that allows UNIX to progress.

True, and that well describes the systemd approach

> Don't leverage your control of pid 1 to force people to use your logger. That's the Poettering way. That's the way that stifles UNIX progress.

False! I don't know how you got here, no one is forced to use the Journal, it does get a copy of the local logs but they go to the normal syslog daemon too and are written out normally or sent across the network normally.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 2:34 UTC (Tue) by mgb (guest, #3226) [Link] (7 responses)

See e.g. http://rpmfind.net//linux/RPM/fedora/18/i386/s/systemd-19...

systemd REQUIRES libsystemd-journal.so.0

You are Lennart Poettering and I claim my £5.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 3:59 UTC (Tue) by raven667 (subscriber, #5198) [Link] (6 responses)

I am not Lennart Poettering but I'm flattered you think so 8-) No £5 for you. Pro tip: Poettering is mezcalero on LWN.

In any event I didn't dispute that systemd logs to the Journal, just that _YOU_ aren't required to use it because everything still goes to traditional syslog. There is no credible way you could claim that anything is being stifled when compatibility is such an obvious priority.

I'm not sure we are really communicating effectively because you don't seem to be understanding what I am saying and your responses don't make any sense to me. I will assume that you are operating in good faith, bid you a good day and thank you for your time, I don't have anything else to say.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 11:17 UTC (Tue) by nix (subscriber, #2304) [Link] (5 responses)

To be honest, if the default journal is an in-memory passthrough, I don't see why anyone would care, anymore than anyone says that syslog-ng has an 'extra useless log' because it does in-memory buffering to keep disk I/O down.

If one said that systemd had an *optional* field-structured message log, would this suddenly make this problem go away, even though the only difference is to change terminology and declare the journal 'off' rather than 'in-memory'?

(The only thing I really disliked about the journal was the explicit lack of definition of the binary journal format. Since this has now been rectified, my complaints about it are gone.)

you do NOT need to write all your programs together to make them work together.

Posted Jan 31, 2013 19:07 UTC (Thu) by sorpigal (guest, #36106) [Link] (4 responses)

> If one said that systemd had an *optional* field-structured message log, would this suddenly make this problem go away, even though the only difference is to change terminology and declare the journal 'off' rather than 'in-memory'?

I suspect a great deal of the push-back against systemd comes from this very thing. Given the option of talking about it in a way that doesn't rile people up and talking about it in a way that does, the latter is what happens. If the story were "systemd internally uses its own field-structured log format but still outputs logs to syslog by default (or you can just use the internal format directly)" then you'd see more acceptance, but the story was "systemd has replaced syslog with a new, undocumented, non-text log format." If the goal wasn't to annoy people then I can only say that there exists an incredible natural talent for annoyance which is being wasted in the field of software engineering.

you do NOT need to write all your programs together to make them work together.

Posted Jan 31, 2013 19:41 UTC (Thu) by dlang (guest, #313) [Link] (3 responses)

remember, this isn't just systemd's internal logs, systemd intercepts logs that the apps write to syslog and turn them into this new structure as well.

you do NOT need to write all your programs together to make them work together.

Posted Jan 31, 2013 20:42 UTC (Thu) by davidstrauss (guest, #85867) [Link] (2 responses)

> remember, this isn't just systemd's internal logs, systemd intercepts logs that the apps write to syslog and turn them into this new structure as well.

Just keep omitting that systemd's journal forwards the entries *unaltered as syslog*, and you might convince some people that the journal breaks their traditional syslog toolchain.

you do NOT need to write all your programs together to make them work together.

Posted Jan 31, 2013 20:46 UTC (Thu) by dlang (guest, #313) [Link] (1 responses)

look at the context, I am replying to someone explaining how presenting how the journal deals with logs differently could avoid antagonizing people. I am just pointing out that his statement is not complete, it's not just the internal journald logs that are involved.

you do NOT need to write all your programs together to make them work together.

Posted Feb 1, 2013 1:21 UTC (Fri) by davidstrauss (guest, #85867) [Link]

> look at the context, I am replying to someone explaining how presenting how the journal deals with logs differently could avoid antagonizing people. I am just pointing out that his statement is not complete, it's not just the internal journald logs that are involved.

I'm aware of the context. I just think it's disingenuous to use how the systemd's journal maintains its own data structures to imply some change happens that creates complexity for syslog users.

Use of internal data structures is true of any log daemon, including rsyslog. Saying that the journal "turn[s] them into this new structure as well" implies that the syslog messages emitted by the journal are not identical to the messages sent in, which is false.

The journal goes beyond syslog, but it does not get in its way or force users to change any existing applications, monitoring, or workflows.

When we updated our configurations for Fedora 17 (the first release with the journal), we didn't have to touch anything to continue getting the same syslog messages we got before in the same places we had always looked for them.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 7:59 UTC (Tue) by smurf (subscriber, #17840) [Link] (2 responses)

Oh come on.

So you want to force the systemd people to support writing two log streams? One for traditional syslog and one for field-structured logging? In what way does that fit your UNIXy "do one job well" definition?

Instead, systemd spits out a nice structured log, journald reads that and translates it to "standard" syslog, other programs can be switched over to structured logging once journald can be assumed to be available (or they can support both if they want the compatibility, or somebody can write a compatibility library with a journal API and syslog output), everybody is happy.

Why the hell should it matter that systemd and journald are maintained in a single repository?

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 17:35 UTC (Tue) by raven667 (subscriber, #5198) [Link] (1 responses)

I'm guessing that most software never will switch to using the Journal and will continue using standard syslog but that's fine and is supported.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 20:43 UTC (Tue) by khim (subscriber, #9252) [Link]

Yes, it is supported but in sane manner: systemd only creates log in journald format and journald converts them. Somehow people who like to raise racket about "monolithic design" and "lack of modularity" simultaneously find this solution problematic and want to stuff syslogd format support in systemd itself. Talk about consistency.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 12:07 UTC (Tue) by khim (subscriber, #9252) [Link] (11 responses)

you make programs work together, not by writing them together, but by defining the interface between them. That way the different programs can be written by different people at different times, replaced one by one, etc

That's nice theory but it does not work in practice, sadly.

That is the way to have a long-term robust solution.

It does not work this way. Either stuff is developed in lockstep (to use latest version of GLibC you need latest version of GCC and to use latest version of GCC you need not-yet-released-version of binutils), or there are bunch of problems when one component or another does not adhere to the specs.

And if you develop bunch of small programs in lockstep then why don't develop them together in a single repository? That's how UNIX was traditionally developed after all!

If you consider that all the programs need to be written together, it's a very small step to consider that you can break all your old interfaces because you ship all new versions of everything.

Yup. That's the point: if the interfaces only tie these programs together and are not exposed outside then why do you need to keep them static?

Practice shows that it's impossible to keep bazillion interfaces robust and stable. Only two possibilities can be realized by real people in real life:
1. Large amount of interfaces are in constant flux but some are declared "sacred" and kept around forever.
2. Large amount of interfaces are declared "stable" yet in practice they all evolve in some way and you need to patch system all the time to keep it running.

APIs need to be defined, and you need to support old ones going forward.

Yes. And this is only possible if scope of these APIs are kept limited.

you do NOT need to write all your programs together

If you want stable ABIs? Yes, you do. It's impossible to keep all the ABIs stable which means that either you have some ABIs "set in stone" and most in "we don't care because these are iternal ABIs" state or you have lots of ABIs in "permanently broken" state.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 12:43 UTC (Tue) by mgb (guest, #3226) [Link] (5 responses)

@khim - Software engineers have been doing what you consider impossible for many decades.

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
Doug McIlroy

Text interfaces are more resilient, more flexible, and more readily debuggable than binary interfaces. Inappropriate binary interfaces are a hallmark of the premature optimizer.

Premature optimization is the root of all evil.
Donald Knuth

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 13:44 UTC (Tue) by khim (subscriber, #9252) [Link]

Software engineers have been doing what you consider impossible for many decades.

Nope. They claimed they do that, sure. But did? Examples, please. Some popular thing which is built a way you claim it should be built.

I know of many things which claim to be built from tiny independent components in a fashion you preach. I also know that any small change tend to get a lukewarm response and request to stop using non-standard components.

And if we can only use one particular version of component A and one particular version of component B with one particular version of component C then why not develop them together?

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
Doug McIlroy

Nice quote. Yet somehow exactly zero SQL databases (and Unix was big in SQL world till it died) use text interfaces. Says something about applicability of said principle, isn't it?

Text interfaces are more resilient, more flexible, and more readily debuggable than binary interfaces. Inappropriate binary interfaces are a hallmark of the premature optimizer.
Premature optimization is the root of all evil.
Donald Knuth

Nice, but incomplete quote. The full one says something different: "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil" and when compounded by a different quite from the same guy (In established engineering disciplines a 12 % improvement, easily obtained, is never considered marginal and I believe the same viewpoint should prevail in software engineering) nicely explains why journald makes sense, isn't it? Yes, it makes sense because it's in critical path for many workloads, it speedups things by more then 12% and thus it fits nicely in this 3% niche.

If you try to counter facts with quotes then at least make sure you understand what the quoted documents say in reality.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 13:59 UTC (Tue) by HelloWorld (guest, #56129) [Link] (3 responses)

The funny thing about this "text streams" meme is that Unix doesn't actually use text streams, yet nobody seems to realise that. A Unix pipe is a byte stream, not a text stream. To interpret a byte stream as text, one needs to negotiate an encoding, and Unix pipes don't offer a means to do that.

And besides, even if Unix did offer text streams, I'd still disagree. Text is not a universal interface, there's a *lot* of data where a text format doesn't make any sense at all, like videos, audio, images, and many many others. And in fact, most programs today don't mess around with text files any longer, but use structured data representations and libraries to (de)serialise them. DBus is a binary protocol because it's faster, but also because nobody cares as everyone uses it via a library (and code generators) anyway.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 17:24 UTC (Tue) by smurf (subscriber, #17840) [Link] (1 responses)

> one needs to negotiate an encoding,
> and Unix pipes don't offer a means to do that.

One might argue that the negotiation is implied by the LOCALE setting.
Or that it is not necessary these days, because anybody who does not use UTF-8 deserves to lose. :-P

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 18:17 UTC (Tue) by HelloWorld (guest, #56129) [Link]

> One might argue that the negotiation is implied by the LOCALE setting.
The LOCALE setting is just a way to specify manually the information that can't be negotiated through the pipe. If pipes were actually a text stream, there'd be no need to do with manually and things would just work.

Anyway, I don't think such a design would be desirable, because as I said before, text is in fact not a universal interface. Which is of course why many programs today communicate with much more structured protocols like D-Bus.

While I do sympathise with your views about UTF-8, there's a large amount of data stored in legacy encodings, and it's not going away any time soon.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 20:28 UTC (Tue) by anselm (subscriber, #2796) [Link]

A Unix pipe is a byte stream, not a text stream. To interpret a byte stream as text, one needs to negotiate an encoding, and Unix pipes don't offer a means to do that.

In the interest of fairness it should be mentioned that, at the time Doug McIlroy made the quoted statement, ASCII was still the text encoding of choice (at least if you were in the US). The idea that an encoding might have to be »negotiated« before a pipe could be useful didn't really show up on anyone's radar.

Also, most of the usual Unix tools assume that their input consists of lines of text separated by newline characters, rather than arbitrary »byte streams«, and generate output to suit this assumption. Note that UTF-8 was carefully defined (by Ken Thompson, no less) to fit the common Unix notion of »text« – up to a point where many Unix tools will do »reasonable« things when fed UTF-8 data even if they do not include explicit support for UTF-8.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 16:22 UTC (Tue) by nix (subscriber, #2304) [Link] (4 responses)

to use latest version of GLibC you need latest version of GCC and to use latest version of GCC you need not-yet-released-version of binutils
Neither of these things are true. What is true is that older versions are likely less tested than newer ones, and there is a fuzzy line who knows how far back (probably less far than configure tests for) where things stop working entirely.

But neither of the constraints you suggest are true, not even slightly. You can build the latest glibc with GCC 4.4 and you can build GCC with really rather aged binutils (2.19 or something, maybe even older).

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 20:35 UTC (Tue) by khim (subscriber, #9252) [Link] (3 responses)

But neither of the constraints you suggest are true, not even slightly.

This may be true right now, today, but that's because there are no need to change anything rapidly (last major architecture was x86-64 and it was introduced years ago). When new platform is introduced you see things like "Binutils 2.16.91 or newer are required" quite regularly. And things often breaks even if these requirements are not written explicitly.

We'll see how AArch64 will affect all that - but I doubt it'll be possible to use binutils "2.19 or something, maybe even older" for that.

P.S. Note: I'm not saying it's somehow bad that it's done that way. In fact that's the right thing to do. But if GLibC is tightly tied GCC and GCC is tightly tied to binutils then why pretend that they are different projects? They are not: they are parts of larger project: GNU.

you do NOT need to write all your programs together to make them work together.

Posted Jan 30, 2013 16:24 UTC (Wed) by nix (subscriber, #2304) [Link] (2 responses)

Well, yes, of *course* if a new architecture is introduced then you have to use a toolchain new enough to support that architecture! That doesn't mean that all projects that know about that architecture are the same project! Nor does it suddenly imply a tight tying among the projects.

Even among the toolchain-related projects, they have some different maintainers (though some in common), different governance structures, different mailing lists, different release schedules, different source trees and not much shared code infrastructure (the only bits I can think of shared between GNU toolchain components are libiberty and libbfd, though the latter is quite sizeable).

you do NOT need to write all your programs together to make them work together.

Posted Jan 31, 2013 22:52 UTC (Thu) by khim (subscriber, #9252) [Link] (1 responses)

Even among the toolchain-related projects, they have some different maintainers (though some in common), different governance structures, different mailing lists, different release schedules, different source trees and not much shared code infrastructure (the only bits I can think of shared between GNU toolchain components are libiberty and libbfd, though the latter is quite sizeable).

This is true but I'm interaction with these people enough to notice that despite all that development happens in all these projects simultaneously. When you need to add, e.g. something like ifunc you need to change GLibC, binutils, and GCC in a lockstep - and this is done by the same people without regard to any other projects. To me it looks more like a large single project which is complicated by artificial division between glibc/binutils/gcc rather then three separate project: you still need to introduce changes in all these places but additionally you need to write code which will detect version skew and disable these features appropriately.

The division lies at the different level: in GCC there are few layers (SSA, RTL, etc) and developers who work with different layers know less about other layers then "low-level GCC people" know about binutils and glibc!

you do NOT need to write all your programs together to make them work together.

Posted Feb 2, 2013 19:17 UTC (Sat) by nix (subscriber, #2304) [Link]

All you say there is true -- some changes must affect several layers at once. However, a lot of changes affect glibc and the kernel at once, too -- does that mean that glibc and the kernel are the same project? (Note that for quite a long time the founder and co-maintainer of glibc was also one of the maintainers for one of the nastiest parts of the core kernel, ptrace() and signal handling, so even the people-in-common rule is true!)

I don't think they're the same project -- and neither are the various toolchain projects the same project, anymore than gnulib is the same project as coreutils merely because changes to both often happen in synchrony. They are simply projects with *some* close coupling between them -- but even that coupling is optional (you can build GCC and perhaps even glibc with a binutils that doesn't support IFUNC, and nothing goes wrong).

Poettering: The Biggest Myths

Posted Jan 29, 2013 9:39 UTC (Tue) by nix (subscriber, #2304) [Link] (10 responses)

I like the idea of systemd -- I'm avoiding it for now simply because I'm a stability freak whose boot process is still stuck using BSD-style monolithic shell scripts, which are really simple and easy to understand, and because I'd need some time to migrate when I didn't need my system to do other stuff. But this is just nonsense:
reliable monolithic part (GNU components: GCC, GLibC, binutils, coreutils, etc - they are not really supposed to be used with anything else)
GCC, binutils and coreutils are insanely portable. They work on virtually everything and (in the case of GCC and binutils) target virtually everything, though the recent switch of GCC to C++ will restrict the set of bootstrap compilers somewhat. Heck, until very recently there were a bunch of targets on which you could run GCC but could not use binutils -- your claim is false on its face.

glibc is also portable, in theory -- it's just that all the ported bits have fallen into disuse and disrepair. The portability framework is unlikely to rot anytime soon because it is also the system library for the Hurd, which at that level is quite thoroughly unlike Linux in some ways. (Also, that framework is also used for cross-arch portability, which is definitely not going away now that glibc's maintainers do not have Uli's charming contempt for all non-x86 architectures.)

Poettering: The Biggest Myths

Posted Jan 29, 2013 12:49 UTC (Tue) by khim (subscriber, #9252) [Link] (9 responses)

glibc is also portable, in theory -- it's just that all the ported bits have fallen into disuse and disrepair.

This is nice theory but it's not supported by practice. You see, just recently we've needed to port GLibC to ARM for PNaCl-arm. We've talked about about this project with Roland and eventually the resolution was to add support for NaCl-arm to GCC. If Roland feels it's easier to add support for the new architecture to GCC rather then port GLibC to Clang I tend to think that it's basically impossible to do in practice.

The portability framework is unlikely to rot anytime soon because it is also the system library for the Hurd, which at that level is quite thoroughly unlike Linux in some ways.

Sure. You can use GLibC with two different kernels (and soon with third one if you'll count NaCl as separate "kernel") but it can only be used with GCC and binutils - and you need quite recent versions of both.

Kinda like systemd with udev, journald and others: you can use udev separately but systemd can only work with one particular version of udev and one particular version of journald.

Heck, until very recently there were a bunch of targets on which you could run GCC but could not use binutils -- your claim is false on its face.

And there are bunch of targets where you must use GNU ld and GNU as and can not use system-provided versions. Basically it's all very simple: GNU tools are supposed to be used with other GNU tools only but where feasible to use system-tools provided by different platforms they do that. This is very explicitly not some nirvana of free-for-all pick-and-choose, but a set of decisions made by sane people: they used system versions of tools where it was harder to port binutils to the new platform and the forced GNU tools in most other cases.

Poettering: The Biggest Myths

Posted Jan 29, 2013 16:14 UTC (Tue) by nix (subscriber, #2304) [Link] (4 responses)

One bit of portability you will *not* generally find is portability for lowlevel components like C libraries to different compilers. I am not at all surprised glibc only works with GCC: it's stuffed with GCC-specific stuff, just as its makefiles are very heavily GNU make dependent. (I'm amazed you thought it might have been worth trying to make it work with something else.)

Poettering: The Biggest Myths

Posted Jan 29, 2013 20:22 UTC (Tue) by khim (subscriber, #9252) [Link] (3 responses)

One bit of portability you will *not* generally find is portability for lowlevel components like C libraries to different compilers.

Many C libraries can be compiled with different compilers. Including GLibC 1.x. But version 2.x was redone to consciously use features of GCC - just like systemd was consciously redone to use features of Linux. Somehow it's ok for GLibC but not for systemd? Why?

Poettering: The Biggest Myths

Posted Jan 30, 2013 16:21 UTC (Wed) by nix (subscriber, #2304) [Link] (2 responses)

Well, I wasn't one of those arguing that systemd should be portable across OSes: init systems generally aren't, not even the simple ones. I was arguing against your claim that this is universal in other sorts of software: it is not. Lots of software supports portability in various ways, be that compilation with different compilers (binutils, GCC), or running on different platforms (glibc, to an extent)... and huge amounts of software doesn't care e.g. what glibc or binutils or kernel version it was built with. The coupling is not as tight as you suggest.

Poettering: The Biggest Myths

Posted Jan 31, 2013 22:43 UTC (Thu) by khim (subscriber, #9252) [Link] (1 responses)

I was arguing against your claim that this is universal in other sorts of software.

Please remind me where I've made such claim.

Lots of software supports portability in various ways, be that compilation with different compilers (binutils, GCC), or running on different platforms (glibc, to an extent)... and huge amounts of software doesn't care e.g. what glibc or binutils or kernel version it was built with.

Lots of peripheral software is portable. The core is not. And this is exactly what I've said in the very beginning: Most popular OSes are all almost completely monolithic per this definition: Android, Busybox, BSDs. Only GNU/Linux is weird creature which includes robust and reliable monolithic part (GNU components: GCC, GLibC, binutils, coreutils, etc - they are not really supposed to be used with anything else) and loose and flaky things like sysvinit with it's plethora of weird scripts.

Take any non-Linux-based OS you want - and you'll find out that he core is built from a single repo using a single build script. Kernel, libc, some core utilities. The next ring is more portable: BSD make itself may be ported (and was, in fact, ported) to other platforms but you can not use anything else to build, say, FreeBSD's kernel or libc. The outermost ring is the most portable: these are applications which can be ported to other platforms.

But a quirk of history made Linux an exception from this rule and, surprise, surprise, this exactly where a lot problems originates: various tiny core components don't cooperate well enough and require plethora if work from distribution makers to somehow create something usable. And even then they often fail. Note that usually you can not even replace these core components: even distributions which formally offer such ability (like Debian or Gentoo) crumble and start misbehaving when you actually try to use such "freedom".

The coupling is not as tight as you suggest.

The coupling is as tight as I suggest and to me systemd looks like a try to finally bring this approach to desktop and server linux. Note that places where Linux dominates - such as routers and mobile phones - already use this approach.

P.S. It's also looks funny to me when people try to say that what systemd developers are doing is somehow against the "UNIX philosophy" when, in fact, they are doing what UNIX did from a day one and what all direct descendants of UNIX are still doing today.

Poettering: The Biggest Myths

Posted Feb 2, 2013 19:14 UTC (Sat) by nix (subscriber, #2304) [Link]

I'm not even going to bother arguing with this. If you want to keep arguing that black is white and that GNU binutils, GCC, and the other parts of the toolchain are nonportable, that's fine. You're wrong, and spouting nonsense, but you're allowed to do that.

Poettering: The Biggest Myths

Posted Jan 29, 2013 16:18 UTC (Tue) by nix (subscriber, #2304) [Link] (2 responses)

This is very explicitly not some nirvana of free-for-all pick-and-choose, but a set of decisions made by sane people: they used system versions of tools where it was harder to port binutils to the new platform and the forced GNU tools in most other cases.
This is entirely incorrect. System versions of tools are supported for toolchain building where they are not too broken to work reliably, particularly if it seems likely that people would otherwise need to resort to cross-compilation to get the thing built on that platform. Heck, the horrible bundled K&R HP-UX 10.x C compiler was supported for many many years, despite barely grasping advanced concepts like functions: if your allegations were true it would never have been supported at all. (It is true that, these days, it is much easier to cross-compile from a random Linux system than it ever was before, so tolerance for totally broken crap like that is going down. On the gnulib list, they say "we are not running a software archaeology project", and they are quite right.)

Unfortunately there are a lot of broken assemblers, linkers and the like out there. (Oddly, one of the most often broken components is sed. sed! How hard is it to write a working sed, I ask you...)

Poettering: The Biggest Myths

Posted Jan 30, 2013 0:48 UTC (Wed) by malor (guest, #2973) [Link] (1 responses)

(Oddly, one of the most often broken components is sed. sed! How hard is it to write a working sed, I ask you...)

Heh, I bet most of the broken implementations started out with that exact thinking..."how hard could this be?"

Poettering: The Biggest Myths

Posted Jan 30, 2013 16:30 UTC (Wed) by nix (subscriber, #2304) [Link]

Most of it seemed to be classic Unix thinking, "we can just use a fixed N-character buffer here, nobody will want more than X of these". So you get fun like seds that coredump if you use more than 512 characters in an s// expression's regexp part, and that sort of thing. A crashing sed tends to annoy configure scripts...

Poettering: The Biggest Myths

Posted Jan 31, 2013 21:45 UTC (Thu) by dirtyepic (guest, #30178) [Link]

>Sure. You can use GLibC with two different kernels (and soon with third one if you'll count NaCl as separate "kernel") but it can only be used with GCC and binutils - and you need quite recent versions of both.

I'm not sure what you define as quite recent, but as of last month the minimum requirements are binutils-2.20 and gcc-4.3, which are both 3+ years old. Before that you could still build glibc with gcc-2.95.3 with a bit of futzing.

Oh, it also requires a kernel later than 2.6.16 (or 3.7 if you want AArch64 support which is obviously a conspiracy) so I guess you'll have to make some room for it under your tightly-knit GNU umbrella.

Poettering: The Biggest Myths

Posted Jan 27, 2013 6:51 UTC (Sun) by dowdle (subscriber, #659) [Link]

Represent!

This is like West Side Story. Some have already started dancing and singing. Everyone else is here for the blood.

Poettering: The Biggest Myths

Posted Jan 27, 2013 7:45 UTC (Sun) by lkundrak (subscriber, #43452) [Link] (24 responses)

I sometimes feel ashamed that I'm not as vocal as people who for some reason oppose systemd while it significantly improves my Linux experience.

With people arguing how awesome, manageable and maintainable old shell-script based init system was I feel like I've been too unlucky to hit every single problem it had. (Or that I was running something else.)

I've seen my system boot take a far longer time than I could bear and seriously harm my experience (what if your TV set took 2 minutes to boot?), yet I still read people deciding for me that I should not care that much.

I've see init scripts shitty beyond belief (and I'm not even taking into account what various third parties ship). Ever seen Fedora's tomcat init script? Most of them are way too long, all of them duplicate mostly all of their code.

Also, many of them are just plain broken and racy -- setup includes external service monitoring (such as monit), it's somehow easy for it to attempt to run the init script at the same time you run it leaving the system in inconsistent state (e.g. daemon running and no pid file). Ironically, systemd not even fixes the problem but also removes the need for such software in some cases.

Besides what's stated above, thing that I like about systemd the most is the inetd-ish service activation. Part of my job is setting up systems consisting of numerous services with rather complex dependencies. Some of the services are in-house grown, most of them are daemons shipped with the platform we use. We run hundreds of nodes for production installation (what we like call that "cloud" so that we're able to sell it). We also perpetually test the thing, which involves creating countless new installations, starting the services, restarting them and shutting them down running various automated tests in between. This is a nightmare with the old init system we use. We can never know when is the service ready to use which is why we ended up with horrible polling shell scripts, often inlined into our test tooling or even in puppet. We need to very carefully order the actions we take and with too many services it's way easy to get wrong. Systemd would do this for us.

I've even seen service stop not work properly (leaving out lurking processes) with old init system for way too many times and services. (I've probably never seen it shut down tomcat properly; it's surprising how a single shitty servlet can get the init system tooling stuck).

And there seem to be more nice things about systemd. The daemon startup messages don't get lost if service misbehaves. Given how broken stock Fedora installations are these days, this has turned out to be a big time-saver for me. Also, it seems to be able to set up control groups for daemons automatically; this could make things simpler for us since we use cgroups for resource management of our services pretty heavily.

I've not been adventurous enough to run Fedora on daily basis, thus I might not be familiar with all the problems people could have with systemd during its early days (and I yet am to see a good example of what has been easy with old init systemd, but is not with systemd), but I do run it regularly to keep myself up to date and I've been very impressed with systemd. We'll celebrate the day RHEL 7 is released and systemd hits our infrastructure.

Poettering: The Biggest Myths

Posted Jan 27, 2013 8:01 UTC (Sun) by apoelstra (subscriber, #75205) [Link] (16 responses)

> I sometimes feel ashamed that I'm not as vocal as people who for some reason oppose systemd while it significantly improves my Linux experience.

I second this. Being able to throw up a 5-line systemd service file for a new daemon (without trolling through mailing lists looking for prewritten scripts and gotchas) is wonderful.

Poettering: The Biggest Myths

Posted Jan 27, 2013 11:17 UTC (Sun) by alankila (guest, #47141) [Link] (5 responses)

I wrote an upstart script to start some php thing as fastcgi daemon and it sucked. "expect fork" is part of the definition, indicating that the program was expected to fork once before running as a daemon, something upstart needs to be told, but systemd should be able to track the process no matter how many times it forks or doesn't fork.

Either way, I screwed up somewhere and failure at writing the init job definition caused upstart to no longer be able to startup and shutdown that particular service, it seemed to just stall. Killing the php processes didn't help, upstart was still stuck waiting on the service somehow. I couldn't even shutdown the machine anymore because it kept on jamming on that service file forever, forcing a manual power off. Again, on systemd, the fact I killed the process is enough evidence that the service is down, so systemd wouldn't have had the problem in the first place, and if it did, it would have been able to recover because it's actually smart.

I can only hope that I get to enjoy systemd at some point in the future as well. upstart is maybe fine if you know exactly how it works and know how to debug it; systemd seems almost designed for clueless admins like me.

Poettering: The Biggest Myths

Posted Jan 27, 2013 12:20 UTC (Sun) by davidstrauss (guest, #85867) [Link]

> systemd should be able to track the process no matter how many times it forks or doesn't fork.

Correct. It's also one of the features that cannot be portable; it relies of cgroups.

> Again, on systemd, the fact I killed the process is enough evidence that the service is down

Ditto.

Poettering: The Biggest Myths

Posted Jan 27, 2013 20:56 UTC (Sun) by cwillu (guest, #67268) [Link] (3 responses)

In fairness to upstart, "expect fork" and "expect daemon" are not something that can be detected by the init system even in principle. Yes, even if that init system is systemd. Which is why http://www.freedesktop.org/software/systemd/man/systemd.s... has a Type=simple|forking option and various forms of PID guessing to distinguish between the cases handled by upstart's "fork" vs "daemon".

Poettering: The Biggest Myths

Posted Jan 28, 2013 0:45 UTC (Mon) by davidstrauss (guest, #85867) [Link] (2 responses)

> In fairness to upstart, "expect fork" and "expect daemon" are not something that can be detected by the init system even in principle.

The difference is that Upstart can't reliably detect what's going on, let alone know whether it's correct.

systemd can detect what's going on, regardless of how you have the service configured. The configuration allows systemd to compare what is happening against what it's detecting to know whether the service is in a failed state.

Poettering: The Biggest Myths

Posted Jan 28, 2013 1:55 UTC (Mon) by mezcalero (subscriber, #45103) [Link] (1 responses)

Actually, systemd cannot detect the main process in all cases. If you use Type=forking (without specifying a PID file) and your daemon starts many processes at once before exiting in the parent, then we don't know which one of the remaining ones the "main" one should be. In that case systemd will consider the service exited only after *all* processes are gone, and can't collect as much state info about it... Thankfully, this case is quite seldom, the impact is low, and is easily fixed by specifiying a PID file, or by using Type=notify (after updating the daemon).

Upstart's tracing via ptrace() is one evil feature OTOH. Upstart basically becomes the debugger of the daemons it runs. Really, really evil stuff.

Poettering: The Biggest Myths

Posted Jan 28, 2013 2:53 UTC (Mon) by davidstrauss (guest, #85867) [Link]

> Actually, systemd cannot detect the main process in all cases. If you use Type=forking (without specifying a PID file) and your daemon starts many processes at once before exiting in the parent, then we don't know which one of the remaining ones the "main" one should be.

I wasn't speaking to detecting the main PID, only what is happening in terms of all forking behavior. When a process forks into a ton of other processes in the way you describe, there is no amount of collectable data on the system that allows properly identifying the main PID every time.

Poettering: The Biggest Myths

Posted Jan 27, 2013 14:03 UTC (Sun) by pizza (subscriber, #46) [Link]

Third!

systemd is such a vast improvement over what came before that it's not even funny.

Poettering: The Biggest Myths

Posted Jan 27, 2013 18:58 UTC (Sun) by apoelstra (subscriber, #75205) [Link] (1 responses)

Let me give a specific example. A week ago I wanted to set up a hunchentoot (lisp) web server, which involves starting a lisp instance, running some sort of setup lisp code, and leaving it running on some console somewhere.

Various Internet searches gave me 50-60 line init scripts for old versions of Ubuntu. These scripts were long and opaque and probably not even close to something that would run on Fedora 18. They relied on the detachtty tool (a minimal screen-type program) and did some sort of $PID dance around this. I wanted the server to restart whenever I killed it, so that I could reload the setup code that way. (Also I kept accidentally killing it by hitting Ctrl+D into screen.)

There's nothing out there about using hunchentoot with systemd. So, with maybe five minutes of searching manpages, I wrote

#======
User=lisp-web

[Unit]
Description=Run sbcl webserver session on a screen tty
After = syslog.target network.target

[Service]
User=lisp-web
Type=simple
ExecStart=/usr/bin/screen -D -m /bin/sbcl --load-path /home/lisp-web/ --load default-server.lisp
Restart=on-success
#======

Short, simple, clear, works correctly, tells me when things go wrong, etc, etc. I can connect to screen and hit Ctrl+D to reliably restart the server. It never hangs or gets out of sync or leaves ports in use or files locked.

Poettering: The Biggest Myths

Posted Jan 27, 2013 21:11 UTC (Sun) by pizza (subscriber, #46) [Link]

This "thread" finally motivated me to write a systemd unit file for my photo organizer's background threads. I don't know why I put it off so long.

It took me under five minutes. Would have been less time if the Fedora 16 server it's running on had a more modern systemd installed.

Trolling?

Posted Jan 28, 2013 13:23 UTC (Mon) by pboddie (guest, #50784) [Link] (6 responses)

You mean trawling, surely, although I imagine that trolling is a way of getting answers from mailing lists, too.

Sorry for the pedantry, but I keep reading "trolling" used in this sense and assume that it must be either incorrect or American English. At least if it is the latter, the same cannot be said for the epidemic of "loose" (instead of "lose") and the unbearably irritating "choosen" (instead of "chosen").

Trolling?

Posted Jan 28, 2013 13:51 UTC (Mon) by mpr22 (subscriber, #60784) [Link] (3 responses)

Funny thing: I hadn't even noticed the existence of "choosen", and can't imagine it being more irritating than the loose/lose confusion. (Of course, this is what we get for using an orthography invented by non-native speakers 900 years ago, barely upgraded since, and at this point prohibitively expensive to reform.)

Trolling?

Posted Jan 28, 2013 14:35 UTC (Mon) by tnoo (subscriber, #20427) [Link] (2 responses)

Ask Lennard for a new implementation.

Trolling?

Posted Jan 28, 2013 16:27 UTC (Mon) by HelloWorld (guest, #56129) [Link] (1 responses)

> Lennard
Speaking of spelling...

Trolling?

Posted Feb 9, 2013 22:50 UTC (Sat) by cas (guest, #52554) [Link]

it's the new and greatly improved way of spelling Lennart and if he doesn't like it, he's free to fork and go his own way.

Trolling?

Posted Jan 28, 2013 15:50 UTC (Mon) by Wol (subscriber, #4433) [Link] (1 responses)

I've lost context, but both trawling and trolling are means of fishing.

A trawler goes trawling with a trawl-net.

Dunno what the proper name of a troller is, but they go trolling with a troll-line. Most tuna, for example, is caught by trolling.

I've been trolling for mackerel ...

Cheers,
Wol

Trolling?

Posted Jan 29, 2013 14:23 UTC (Tue) by pboddie (guest, #50784) [Link]

I suppose I have learned something from this, never having been very interested in fishing at all. And I imagine that the Internet era meaning of the word is derived from this practice of using a baited line:

http://oxforddictionaries.com/definition/english/troll--2

The interesting thing here is that the example given on the above page is rather similar to the one used earlier, but unless one attracts the catch using bait - putting something out to attract the catch - the trawling metaphor is actually more appropriate, since one is just seeing what gets dragged up. But anyway...

Poettering: The Biggest Myths

Posted Jan 27, 2013 10:31 UTC (Sun) by Klavs (guest, #10563) [Link] (5 responses)

Seconded - I generelly get tired of people feeling the right to continuously critize what other people do - instead of just providing something better themselves. It's much easier to complain, than code!

It's fine to try influcence other developers, and come up with well-argued opinions - but in the end it's the coders that decide what they code and the users the decide what they use.

pls. stop wasting everybodys time with meaningless argueing.

Poettering: The Biggest Myths

Posted Jan 27, 2013 11:53 UTC (Sun) by drago01 (subscriber, #50715) [Link] (4 responses)

The thing is that people are afraid of change because they fear that their knowledge becomes useless. (such a sysadmin is by definition not a sysadmin)

Most of the criticism is based on buzzwords that have no real meaning "this is bloat" ... "xyz has been like this for 30 years" ... "this is not UNIX like" ...

Sure this sounds harsh but it is a good theory to explain the irrational arguments that you keep reading on the internet on such topics.

Poettering: The Biggest Myths

Posted Jan 27, 2013 21:39 UTC (Sun) by misc (subscriber, #73730) [Link]

Not only. While I usually like changes, that's also because I enjoy being a sysadmin. Now, if for some reason, you do not enjoy the job, but someone had to do it, maybe learning your way on Linux was not so great. Maybe you know that you will not have time to learn new ways.

Poettering: The Biggest Myths

Posted Jan 28, 2013 18:19 UTC (Mon) by ThinkRob (guest, #64513) [Link] (2 responses)

> The thing is that people are afraid of change because they fear that their knowledge becomes useless. (such a sysadmin is by definition not a sysadmin)

It's not a "fear of change" (which is a ridiculous thing in the first place...) or a "fear that [our] knowledge becomes useless" ('cause if that were the issue we wouldn't be in the IT field). It's just that some of us don't like changes when they bring no benefits that matter to us yet require us to re-learn a bunch of stuff that just generally worked "well enough" for a couple decades.

Yes, it's not a huge deal, but it's still a pain, especially when the previous implementation was satisfactory.

Poettering: The Biggest Myths

Posted Jan 28, 2013 20:58 UTC (Mon) by HelloWorld (guest, #56129) [Link]

Most of what worked with sysvinit still works with systemd. init scripts work, /dev/initctl works, utilities like service(8) from Fedora/RHEL work, even runlevels are emulated to the degree that's possible. So please, where *exactly* is the alleged "pain" you're talking about?

Poettering: The Biggest Myths

Posted Jan 28, 2013 21:13 UTC (Mon) by davidstrauss (guest, #85867) [Link]

> It's just that some of us don't like changes when they bring no benefits that matter to us yet require us to re-learn a bunch of stuff that just generally worked "well enough" for a couple decades.

It's not clear what requires immediate re-learning with systemd unless the administrator has a habit of directly altering rc.d symlinks, which hasn't been a best practice for a long time.

Existing init scripts work, unaltered, with systemd. Enabling and disabling services with chkconfig maps transparently to the proper operation in systemd. Mounts configured in fstab come over, automatically, as mount units and can continue to be managed the same way as before.

None of those compatibility features is going away any time soon.

And, for admins wanting to transition to the native tools, extensive documentation and guides [1] exist.

If that's not good enough for you and you can't appreciate the value these tools provide to others, I can only describe your attitude toward keeping SysV init as selfish. The FOSS world doesn't owe you stagnation once it's met your needs.

Poettering: The Biggest Myths

Posted Jan 28, 2013 5:47 UTC (Mon) by tnoo (subscriber, #20427) [Link]

Thanks a lot for this comment. You express my feelings exactly.

And thank you to Lennart and everybody who made systemd happen!

Poettering: The Biggest Myths

Posted Jan 27, 2013 10:33 UTC (Sun) by hazard (guest, #3695) [Link] (56 responses)

My biggest sources of dissatisfaction with systemd are not really addressed in this article.

1) systemd is a solution for a problem that doesn't exist for me. I've been using Linux in server, PC, embedded setup since 1995 and never in my life I have experienced a situation where I couldn't implement something by modifying existing SysV-based based setup. I'm not alone, I heard similar sentiments from others. Yeah, boot time is a bit better. I don't care, my systems have months and years of uptime.

2) Entire SysV based boot can be understood on the spot by reading the code of the scripts. Compared to this, systemd is a black box, to fully understand its internals you have to go outside the command line: hunt for the docs on the Web, download the source code.

3) systemd makes the boot process non-linear and complex to comprehend and predict. I cannot easily see the boot process visually as it is happening and I have to rely on external utilities to understand the order in which everything will be started. With systemd, I get a feeling that I'm booting Windows - I don't have a feeling that I'm in control anymore.

4) systemd is more difficult to troubleshoot. Starting a SysV script, I get output and errors straightaway on the screen. With systemd, at best this information is hidden somewhere in the logs which I have to hunt for.

5) systemd has a steep learning curve. There are lots of config files and interaction between them is not obvious. It'll be quite difficult to explain it to a junior system administrator.

PS: My experiences are based on FC16. Perhaps these issues have been addressed in newer Fedora releases - I'll be happy to hear about that.

Poettering: The Biggest Myths

Posted Jan 27, 2013 11:27 UTC (Sun) by pebolle (guest, #35204) [Link]

> 1) systemd is a solution for a problem that doesn't exist for me. I've
> been using Linux in server, PC, embedded setup since 1995 and never in
> my life I have experienced a situation where I couldn't implement
> something by modifying existing SysV-based based setup. I'm not alone, I
> heard similar sentiments from others.

How about myth 19:
[Y]ou can do with it whatever you want, and that includes not using it.

> Yeah, boot time is a bit better. I don't care, my systems have months
> and years of uptime.

Myth 2 and myth 3?

> 2) Entire SysV based boot can be understood on the spot by reading the
> code of the scripts. Compared to this, systemd is a black box, to fully
> understand its internals you have to go outside the command line: hunt
> for the docs on the Web, download the source code.

Myths 5 and 11, I guess.

> 3) systemd makes the boot process non-linear and complex to comprehend
> and predict. I cannot easily see the boot process visually as it is
> happening and I have to rely on external utilities to understand the
> order in which everything will be started.

Myths 5 and 11 again.

> With systemd, I get a feeling that I'm booting Windows - I don't have a
> feeling that I'm in control anymore.

Other peoples feelings are bit hard to discuss, but myth 10 comes close.

> 4) systemd is more difficult to troubleshoot. Starting a SysV script, I
> get output and errors straightaway on the screen. With systemd, at best
> this information is hidden somewhere in the logs which I have to hunt
> for.

Myth 25.

> 5) systemd has a steep learning curve. There are lots of config files
> and interaction between them is not obvious. It'll be quite difficult to
> explain it to a junior system administrator.

Myths 5 and 11 again.

So to me it seems your issues are basically addressed. You may disagree with the author's view. That's another story.

Poettering: The Biggest Myths

Posted Jan 27, 2013 12:11 UTC (Sun) by lkundrak (subscriber, #43452) [Link] (1 responses)

1) systemd is a solution for a problem that doesn't exist for me.
Be very thankful you're not me then!
root@megalodon# service postgresql start
Starting postgresql service: [  OK  ]
root@megalodon# su postgres -c "psql -c 'select 1;'"
psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
For a good explanation how systemd solved problems that bothered me for years please refer to my posting above. Also, argument that most issues probably could have been solved by hacking the shell init scripts is indeed valid (some couldn't, like reliable shutdown via freezer which needs init system assistance at service startup time). Problem is that it would have been rather complex for certain problems, way more than can be legibly expressed in a shell script and leading to fragile and slow solutions (I'm pretty sure, I've seen too many of those).
2) Entire SysV based boot can be understood on the spot by reading the code of the scripts. Compared to this, systemd is a black box, to fully understand its internals you have to go outside the command line: hunt for the docs on the Web, download the source code.
Which system do you use, out of curiosity? I just tried to understand what could go wrong during a failed openldap startup by peeking at /etc/rc.d/init.d/slapd and found out that it seems to be a startup script for a nuclear power plant instead of a directory service.
3) systemd makes the boot process non-linear and complex to comprehend and predict. I cannot easily see the boot process visually as it is happening and I have to rely on external utilities to understand the order in which everything will be started. With systemd, I get a feeling that I'm booting Windows - I don't have a feeling that I'm in control anymore.
This is probably the same issue as (5): the learning curve. Speaking purely for myself, it does not take a lot of time to get comfortable with the utilities and the utilities are superb in cause something starts in an incorrect order (due to a missing dependency or something). Also, due to lazy activation it doesn't happen very frequently that things mess up.
4) systemd is more difficult to troubleshoot. Starting a SysV script, I get output and errors straightaway on the screen. With systemd, at best this information is hidden somewhere in the logs which I have to hunt for.
This is not true, maybe it improved significantly since you've tried systemd? Systemd now catches stderr output and you can easily inspect it with systemctl status when things fail, without fear that it will scroll away from your screen. This pleased me very much when I found out about it :)
5) systemd has a steep learning curve. There are lots of config files and interaction between them is not obvious. It'll be quite difficult to explain it to a junior system administrator.
It's probably easier to get a junior administrator write a robust and correct service file than write a robust and correct sysvinit script. He may probably not understand the inner working of the init system, but most juniors probably never peek at /etc/rc* either. Moreover, most juniors probably are not terribly curious about kernel internals, yet kernel delivers satisfactory "just works" experience to them. If systemd could do the same (it does for me!) it would probably be fine.

Poettering: The Biggest Myths

Posted Jan 28, 2013 17:31 UTC (Mon) by malor (guest, #2973) [Link]

It's probably easier to get a junior administrator write a robust and correct service file than write a robust and correct sysvinit script.

I haven't experimented yet with systemd, so I'm speaking from 50% ignorance, but sysvinit scripts are not simple things. We think of them as simple because we've invested the very large amount of time it takes to learn them well enough to write them properly. I think many of us have forgotten how hard that system was, back when we all started.

There is a lot to know to write a startup script from scratch. Even after all these years, I usually copy other scripts and adapt them, rather than writing them from whole cloth, because there's so much to get exactly correct. From the examples in this thread, at least, it looks like systemd would be much easier. The major downside is that admins may not understand their systems as well, as more is being abstracted away, but that's been true of every abstraction that's ever been invented. Part of OS evolution is moving further and further from the bare metal, and I don't think that process is likely to stop anytime soon.

The question is: are systemd's abstractions good ones? It seems like most folks think that they are.

Poettering: The Biggest Myths

Posted Jan 27, 2013 12:13 UTC (Sun) by davidstrauss (guest, #85867) [Link] (50 responses)

> 1) systemd is a solution for a problem that doesn't exist for me. I've been using Linux in server, PC, embedded setup since 1995 and never in my life I have experienced a situation where I couldn't implement something by modifying existing SysV-based based setup. I'm not alone, I heard similar sentiments from others. Yeah, boot time is a bit better. I don't care, my systems have months and years of uptime.

You don't seem to dispute that systemd is valuable for others. Some of the ways it's valuable to others, like fast server boot times, even appear in the article. You're not even making the case here that systemd makes things *worse* for you in terms of capability or performance. Certainly, you understand that not every change in Fedora or other FOSS projects has to personally benefit you.

> 2) Entire SysV based boot can be understood on the spot by reading the code of the scripts. Compared to this, systemd is a black box, to fully understand its internals you have to go outside the command line: hunt for the docs on the Web, download the source code.

Your experience with SysV init seems to obscure its rougher edges to you. There's nothing obvious at all about SysV init in terms of the metadata-in-comments at the top of init scripts, run levels, rc.d, and the inconsistent suite of service-management utilities like update-rc.d and chkconfig. There's also the litany of executables invoked from within the init scripts like daemonize, start-stop-daemon, and runsv. Subtle errors in using these tools can open many security errors related to improper privilege dropping and inheriting too much of the admin's session.

As for needing to leave the CLI, systemd's docs are fully available as man pages. There's even an index of topics in those man pages. There is no equivalent documentation for someone on the CLI of a system running SysV init. You'd have to already know which utilities you'd need to invoke before being able to find the man pages.

> 3) systemd makes the boot process non-linear and complex to comprehend and predict. I cannot easily see the boot process visually as it is happening

systemd shows service startup the same way most init scripts always have. It's actually *more* consistent because systemd handles the formatting rather than a bunch of init scripts that have more-or-less converged on standard startup output. systemd even includes a "notify" API for services to inform systemd, semantically, from within each service that startup was successful and the daemon is fully online. There is no equivalent for SysV init; output generally stops after the service daemonizes but before it's fully operational.

The problem here seems to be with tools like Plymouth hiding text startup information. That is not a systemd issue.

> and I have to rely on external utilities to understand the order in which everything will be started. With systemd, I get a feeling that I'm booting Windows - I don't have a feeling that I'm in control anymore.

It's actually pretty simple once you dive in:
* If any two services don't have an ordering dependency, start and stop them in parallel.
* If any two services do have an ordering dependency, start and stop them in a way respectful of the specified Before= or After= ordering.

This is all configurable by the packager or admin. Usually, an admin shouldn't have to worry about ordering and should just be able to enable or disable services.

The vast majority of services in systemd either start on-demand (like CUPS) or in parallel with the multiuser target, which is equivalent to the highest few traditional runlevels.

systemd provides a superset of the ordering control present in SysV init. Any lack of control you feel probably comes from unfamiliarity with systemd rather than any actual lack of control. I learned everything I know about systemd's service ordering from the man pages; I have not browsed the source code in that area.

> 4) systemd is more difficult to troubleshoot. Starting a SysV script, I get output and errors straightaway on the screen. With systemd, at best this information is hidden somewhere in the logs which I have to hunt for.

That SysV init output is visible to the admin manually running the script, but it is pretty hard for any other admin or automated tool (that isn't directly running the scripts) to view or collect.

The systemd journal, which has been included since Fedora 17, has made structured logging associated with services more streamlined than it's ever been. Running "systemctl status" on a service provides a quick last-ten-entries log output while providing many more options to filter, tail, and view the data.

There's even work on the selinux side to push access auditing entries into the journal associated with the service, making it far less of a mystery why a service is failing when the cause is selinux.

> 5) systemd has a steep learning curve. There are lots of config files and interaction between them is not obvious. It'll be quite difficult to explain it to a junior system administrator.

This has been improved a bit in later releases with more built-in help in systemctl output, more natural defaults (assuming a .service suffix to most systemctl calls), and ever-improving man pages.

systemd's documentation is remarkably well-maintained and extensive. It's certainly better than you'll find for any distro on any other init system, especially SysV init. For example, compare systemctl's man page to documentation for chkconfig.

I think the "steep learning curve" is also an illusion created by experience with SysV. Basic service management is no more complex with systemd than the bundle of distro-specific SysV helpers. Creating new services and performing advanced tasks is considerably more straightforward and less error-prone with systemd than writing new init scripts or hacking in non-standard runlevels.

That said, we're always open to suggestions of how to make the tools more comfortable for new users. We do value capability and correctness over SysV init familiarity, though.

> PS: My experiences are based on FC16. Perhaps these issues have been addressed in newer Fedora releases - I'll be happy to hear about that.

I've included notes on this with my other responses.

Poettering: The Biggest Myths

Posted Jan 27, 2013 13:06 UTC (Sun) by hazard (guest, #3695) [Link] (49 responses)

Hi,

Thanks for the well-argumented and informative response, appreciated.

> Your experience with SysV init seems to obscure its rougher edges to you. There's nothing obvious at all about SysV init in terms of the metadata-in-comments at the top of init scripts, run levels, rc.d, and the inconsistent suite of service-management utilities like update-rc.d and chkconfig. There's also the litany of executables invoked from within the init scripts like daemonize, start-stop-daemon, and runsv. Subtle errors in using these tools can open many security errors related to improper privilege dropping and inheriting too much of the admin's session.

You're right, with my 17-year experience, the internals and workarounds for particular flavors of SysV are very well known to me by now.

Actually, this is the core issue: systemd, as a complete replacement, makes all these years of experience obsolete and irrelevant. However, if we take commercial deployments, who is systemd's target audience? Existing sysadmins, who know SysV already.

If you force these people to learn something new, they would expect something intuitive and easy to understand based on their existing SysV experience. They would expect some "upgrade" based on the core SysV principles: they'd have to learn new things, but not start from scratch.

The way I see it, most of the new functionality could be implemented without a complete eradication of SysV concepts.

Unfortunately this is not the case and systemd designers preferred to throw out anything that reminds of SysV boot process. People like me simply don't understand why they have to spend hours learning new system when the old one wasn't "broken enough" for them. Once RHEL7 comes out, they'd just plainly reject it and look for a CentOS fork that keeps SysV in.

Small real-life example: About a year ago, I was given a Fedora laptop. I needed to configure it so that a specific application automatically starts on tty1 after booting (in order to show measurement results on screen). Unfortunately IT techs didn't heed to my advice to use older Fedora and I got a new one with systemd in. I planned to perform this configuration in 5-10 minutes, based on my existing knowledge, but it took me close to one hour and numerous searches on Google to figure out how to do it. Moreover, I had to touch like 3-4 config files in order to implement it.

I definitely didn't feel that systemd made it more easy for me to perform this task, I just felt that systemd makes my years of experience with SysV irrelevant and we should be careful not to install systemd on production systems, since at least due to lack of knowledge it'll take our sysadmins much longer to troubleshoot.

Poettering: The Biggest Myths

Posted Jan 27, 2013 14:01 UTC (Sun) by niner (subscriber, #26151) [Link] (31 responses)

So your complaint is that it's different and you have to learn something new. Well if you don't like learning, you should look for a different job. If I complained loudly every time something I knew in technology became obsolete, noone would like to talk to me anymore, since all I'd do was complaining.

I would love to know, how you would implement systemd's features without departing significantly from SysV init. Especially systemd's ability to reliably kill a daemon and all it's child processes. This feature alone would make me switch to systemd immediately, since it makes cluster failover much more reliable. File systems can only be unmounted if all processes accessing them are killed. So we need reliable killing to unmount and switch the drbd node to secondary.

I will not for one second miss the whole "oh the daemon could not write it's PID file, but it started anyway and now the init script cannot stop it anymore" mess... Or the init script shows a green "done" but it didn't notice that apache died immediately afterwards.

systemd solves real problems for real users. It does so at the cost of and hour or two of learning how to use it versus the 17 years of experience one needed to get SysV based systems to run reliably.

Poettering: The Biggest Myths

Posted Jan 27, 2013 14:32 UTC (Sun) by hazard (guest, #3695) [Link] (30 responses)

> So your complaint is that it's different and you have to learn something new. Well if you don't like learning, you should look for a different job.

I expected that someone will appear and oversimplify my thoughts down to this.

There are reasons why Cisco keeps IOS syntax unchanged, why GUI apps keep using "floppy disk" icon to save files and why Linus insists to keep kernel's user-level API unchanged.

And these reasons are: to preserve compatibility and to preserve existing knowledge.

Try coming to any serious enterprise and tell them to switch all servers to Linux, just because "it's better". And if their IT responds that they don't know Linux, tell them that they should look for a different job if they don't like to learn.

Sorry, real life doesn't work this way. What people know DOES matter, what your existing customer base wants also DOES matter.

I may learn systemd; or I may as well switch to some other distro which listens to its user-base.

Poettering: The Biggest Myths

Posted Jan 27, 2013 14:50 UTC (Sun) by hazard (guest, #3695) [Link] (18 responses)

To make it clear, I never said that systemd doesn't bring improvements. systemd does offer numerous quite-nice-features.

However, in my opinion it was completely unnecessary to wipe out existing SysV principles in order to implement those quite-nice-features.

Actually it seems to me that the main reasons to throw SysV concepts out were to implement features which many users don't really care for, like use of targets instead of runlevels.

You can implement a daemon which would launch scripts from /etc/init.d/rcS.N and use /etc/inittab, you can introduce special functions in the scripts to support reliable service launch and PID capture, you can still have this daemon to listen for callbacks from the services to indicate succesful startup, catch service startup output in a structured log, etc. You can even provide OPTIONAL ability for parallel startup for those who want it, while keeping the traditional startup sequence by default.

Poettering: The Biggest Myths

Posted Jan 27, 2013 15:13 UTC (Sun) by niner (subscriber, #26151) [Link] (17 responses)

If you want, you _can_ still use systemd as if it was just a SysV implementation and have scripts in /etc/initd (or was it /etc/rc.d or /etc/init.d/rc.d on your distro?). I actually still use them (or rather the rc<daemonname> symlinks in /usr/sbin to them) out of habit. But most of them are nice and redirect to systemd to do what I mean.

"You can implement a daemon which would launch scripts from /etc/init.d/rcS.N and use /etc/inittab, you can introduce special functions in the scripts to support reliable service launch and PID capture, you can still have this daemon to listen for callbacks from the services to indicate succesful startup, catch service startup output in a structured log, etc."

In other words, one could create new infrastructure to support those features and rewrite every single init script to use it. Administrtors would have to learn a lot again about what new magical commands one would have to use to start a daemon.

But why not use this chance to go a little further and make another leap in usability and features? I still remember the WTF look on my dad's face, when I showed him how to fix the ordering of the init scripts on his server by editing some magical comments in bash scripts.

"You can even provide OPTIONAL ability for parallel startup for those who want it, while keeping the traditional startup sequence by default."

I have not had a purely sequential startup on my systems for a decade, since SUSE's init already started init scripts with the same number in parallel. So systemd does not even deviate here from SysV. It just fixes the mentioned WTF problems.

Poettering: The Biggest Myths

Posted Jan 27, 2013 15:30 UTC (Sun) by hazard (guest, #3695) [Link] (16 responses)

> In other words, one could create new infrastructure to support those features and rewrite every single init script to use it. Administrtors would have to learn a lot again about what new magical commands one would have to use to start a daemon.

Yes, but that can be done as a limited incremental change in the init scripts. Users would still know where to look and what to expect. In case of systemd, the approach is to wipe out everything and replace it with something completely different.

> But why not use this chance to go a little further and make another leap in usability and features?

Very simply, to allow people to use their existing knowledge and preserve backwards compatibility.

Too big a leap and you get your customers against you. Remember GNOME 3, Windows 8, etc.

> I still remember the WTF look on my dad's face, when I showed him how to fix the ordering of the init scripts

That's not a big problem. A big problem is when I get WTF face from my sysadmins. :)

Poettering: The Biggest Myths

Posted Jan 27, 2013 15:50 UTC (Sun) by niner (subscriber, #26151) [Link]

"In case of systemd, the approach is to wipe out everything and replace it with something completely different."

Which is simply not true. On my systems there are still a lot of init scripts still in use despite using systemd as init. None of our own custom init scripts needed even a single change. But of course we will replace them by real systemd unit files at some convenient time.

"That's not a big problem. A big problem is when I get WTF face from my sysadmins. :)"

Well my dad _is_ a sysadmin with decades of expierience with Netware and Windows. Stupidly overcomplicated things like SysV init scripts almost made him give up on Linux and stay with those systems. Because really, that 17 years of experience with something so mundane like the way to start and stop services on a server actually are worth something shouts that there's somthing wrong with the system.

Poettering: The Biggest Myths

Posted Jan 27, 2013 16:04 UTC (Sun) by drago01 (subscriber, #50715) [Link] (14 responses)

> That's not a big problem. A big problem is when I get WTF face from my sysadmins. :)

Go fire them and hire new ones. Seriously technology is not static a thing ... a sysadmin that cannot spend a few hours (at most) to learn a new system that is actually way easier to use then the old one has chosen the wrong job.

Poettering: The Biggest Myths

Posted Jan 31, 2013 15:54 UTC (Thu) by nye (subscriber, #51576) [Link] (13 responses)

Conversely it could be said that a sysadmin who is happy to spend a few hours here and there learning to use something new that solves problems *they don't have* is too ready to waste time.

Poettering: The Biggest Myths

Posted Jan 31, 2013 17:02 UTC (Thu) by davidstrauss (guest, #85867) [Link] (6 responses)

> Conversely it could be said that a sysadmin who is happy to spend a few hours here and there learning to use something new that solves problems *they don't have* is too ready to waste time.

A boss managing sysadmins with that attitude has zero respect for employee skill development or finding improved ways to do even the company's own work. It creates a workplace that hires people and spits them out ten years later with no recent skill development. Around here, the Bay Area, that approach couldn't even retain a good sysadmin. They'd quit for a company that shows them more respect.

But, maybe you don't have the experience to understand that. I've actually owned companies employing engineers and sysadmins for the last eight years, both here in and in Austin. The way to keep good people is to go out of your way as an employer to foster careers and skill development through side projects, training, books, conferences, and community involvement.

Poettering: The Biggest Myths

Posted Feb 2, 2013 19:11 UTC (Sat) by nix (subscriber, #2304) [Link] (1 responses)

The Bay Area is unusual. I was considered almost freakish in the City of London for wanting to keep my (software development) skills up to date. Just use what everyone else is using! Follow the herd and learn the hot stuff only when everyone else learns it! Baaaa.

Poettering: The Biggest Myths

Posted Feb 2, 2013 22:04 UTC (Sat) by davidstrauss (guest, #85867) [Link]

> The Bay Area is unusual. I was considered almost freakish in the City of London for wanting to keep my (software development) skills up to date. Just use what everyone else is using! Follow the herd and learn the hot stuff only when everyone else learns it!

I've done lots of work in London for The Economist and Cap Gemini. While they're both conservative in their technology choices, I'd have trouble generalizing it to the London area as a whole, mostly based on me having such a small sample size of organizations.

I've already mentioned Austin in my post, too, but devops culture aggressively focused on improving sysadmin work also extends to Portland (Puppet Labs), Seattle (Opscode), Austin/Chicago (bcfg2 and devops events), Berlin (systemd), New York, Boston, and a host of other cities with major projects or conferences devoted to improving the the way we do things.

Poettering: The Biggest Myths

Posted Feb 4, 2013 13:09 UTC (Mon) by nye (subscriber, #51576) [Link] (3 responses)

The point is, I'm sure you're not seriously suggesting that every time some new idea comes along, every sysadmin in the world should either be chomping at the bit to spend hours learning about it or be labelled as 'incompetent' and fired.

That's what I was responding to, and it's absurd. Nobody would ever get anything done.

I was attempting to make a rhetorical point, and clearly didn't do very well. I'm not suggesting that wanting to improve one's skills is a bad thing, but characterising anyone who doesn't immediately want to jump onto every new bandwagon as lazy and incompetent is appalling.

Poettering: The Biggest Myths

Posted Feb 4, 2013 14:28 UTC (Mon) by anselm (subscriber, #2796) [Link]

I'm not suggesting that wanting to improve one's skills is a bad thing, but characterising anyone who doesn't immediately want to jump onto every new bandwagon as lazy and incompetent is appalling.

I think that with systemd we're quickly getting past the »new bandwagon« stage. With the major enterprise-type distributions (RHEL and SLES) slated to replace System V init by systemd in the foreseeable future, administrators of machines based on these distributions, along with their spinoffs such as CentOS or Scientific Linux, could do worse than start getting used to systemd. On many other popular distributions, systemd is at least an option that is available to those interested in it, and it is quite likely that most of these distributions will also move over in time.

The nice thing is that systemd, for all it is being dissed by the traditionalists, is in fact in many ways an improvement on the tangle of »System V init plus distribution-specific init scripts plus various distribution-specific bits of infrastructure not actually part of System V init but tacked on to make it work in practice« that many system administrators are saddled with. In my experience, it doesn't take long for somebody who approaches systemd with an open mind to see that there actually might be something to it after all. Also, using systemd in place of System V init isn't exactly rocket science; it's not as if system administrators were suddenly forced to learn Mandarin in order to be able to keep their systems running.

Poettering: The Biggest Myths

Posted Feb 4, 2013 19:16 UTC (Mon) by davidstrauss (guest, #85867) [Link] (1 responses)

> I'm sure you're not seriously suggesting that every time some new idea comes along, every sysadmin in the world should either be chomping at the bit to spend hours learning about it or be labelled as 'incompetent' and fired.

No, I'm saying that "a few hours here and there learning to use something new" isn't a bad thing. I put that in quotes because that's what you actually said was bad.

Arguing that's valuable is hardly saying that "every time some new idea comes along, every sysadmin in the world should either be chomping at the bit to spend hours learning about it or be labelled as 'incompetent' and fired."

Don't create straw men out of my arguments.

Poettering: The Biggest Myths

Posted Feb 6, 2013 12:08 UTC (Wed) by nye (subscriber, #51576) [Link]

>Don't create straw men out of my arguments.

I didn't. That's why I said "I'm sure you're not" before paraphrasing the argument that the other guy made, specifically to point out that what you're arguing for is *not* what I'm arguing against.

I'm trying to say that you can support the idea that people might want to educate themselves without jumping right to "and anyone who doesn't always want to do that is incompetent".

Poettering: The Biggest Myths

Posted Jan 31, 2013 18:51 UTC (Thu) by smurf (subscriber, #17840) [Link] (5 responses)

Aww. So what you're saying is that there actually _is_ a sysadmin out there who never needs to figure out whether a given service is running, and if so, why not?
And thus, a less cumbersome/error-prone way to do that won't be of _any_ use to said sysadmin? Not even if … oh well, I'm not going to repeat myself again.

Anyway. I feel for them. I really do.

You see, in _my_ company sysadmins with _that_ attitude will quickly find themselves with exactly one remaining task: clean out their desk.

Poettering: The Biggest Myths

Posted Feb 4, 2013 13:13 UTC (Mon) by nye (subscriber, #51576) [Link] (4 responses)

>Aww. So what you're saying is that there actually _is_ a sysadmin out there who never needs to figure out whether a given service is running, and if so, why not?

I'm saying that there are loads of people who keep saying that they don't need systemd because they have no need for the facilities it provides, and that maybe - just maybe - they might not be lying.

>You see, in _my_ company sysadmins with _that_ attitude will quickly find themselves with exactly one remaining task: clean out their desk.

If this childish posturing makes you feel important, go right ahead.

Poettering: The Biggest Myths

Posted Feb 4, 2013 14:35 UTC (Mon) by anselm (subscriber, #2796) [Link]

I'm saying that there are loads of people who keep saying that they don't need systemd because they have no need for the facilities it provides, and that maybe - just maybe - they might not be lying.

They may not desperately need systemd's facilities but they might actually appreciate them once they have them available.

This »We don't need this« attitude reminds me of some of the Windows administrators I encounter in the Linux workshops I teach. These often refuse to have anything to do with pipelines, regular expressions, etc. on the grounds that they »don't need this« on their Windows machines, so these features can be of no conceivable worth whatsoever. Some of them are of a sufficiently agile mind to see that using these facilities on Linux does in fact make some things easier, even to a point where they find they have less drudge work to do on Linux than on Windows, but others never quite seem to get it.

Poettering: The Biggest Myths

Posted Feb 4, 2013 16:12 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link] (2 responses)

I'm saying that there are loads of people who keep saying that they don't need systemd because they have no need for the facilities it provides, and that maybe - just maybe - they might not be lying.

And they might be more believable if they appeared to be more familiar with exactly what facilities systemd provides. It appears to me that a substantial fraction of the resistance to systemd is based around ignorance of what it actually does and doesn't do. People seem to have made up their minds that they're happy with existing init systems and then looked for reasons to dislike systemd rather than finding out in detail about the potential advantages and disadvantages.

Poettering: The Biggest Myths

Posted Feb 6, 2013 12:29 UTC (Wed) by nye (subscriber, #51576) [Link] (1 responses)

To begin with, personally I'm actually not one of those people who doesn't have those problems. I do on occasion want to be able to do things that systemd makes easier, and am in principle very much in favour of systemd, though in practice I'm not actually willing to try it for myself yet until it's had at least another couple of years to mature - let's say jessie (Debian 8.0).

However, I have a great sympathy for those who are in that position, because that's where I was (and actually still am) with PulseAudio (which, I'll note, still doesn't work properly with Wine on 64-bit systems, at least not on Debian, and still provides no benefits I actually care about).

This is roughly how the exchange sounds from that perspective:

Person A: Try this new snibulator; it's great!
Person B: I'm happy with my old snibulator, thanks.
A: But this one allows you to snibulate even when you're under water!
B: Huh, that sounds cool, but I've never needed to do that.
A: Well you're just too stupid to realise that you need this after all.
B: Please stop telling me what to do.
A: Stop being so hostile! You're clearly too lazy to spend the time to work out that you really need waterproof snibulation. If I had any employees like you, I'd fire them on the spot! I'm so important!

The last line is the one that really grates on me, because it's so arrogant and so aggressively obnoxious. And I haven't hardly even had to exaggerate in my paraphrasing; many of the responses really are that hyperbolic.

Poettering: The Biggest Myths

Posted Feb 6, 2013 19:00 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

> …with PulseAudio (which, I'll note, still doesn't work properly with Wine on 64-bit systems, at least not on Debian, and still provides no benefits I actually care about).

It Works For Me™ on Fedora. Rawhide no less (in December; not sure about this month yet). As for features, I keep an instance of MPlayer streaming some internet radio at work. It's nice to be able to mute it and still play some video with sound and then go back and unmute the stream seamlessly.

Poettering: The Biggest Myths

Posted Jan 27, 2013 14:51 UTC (Sun) by ovitters (guest, #27950) [Link] (7 responses)

You're suggesting that systemd requires loads of time to digress. I don't get why though. Could you please explain in practice?

I mean, I still can use chkconfig, service and dump a sysvinit script using systemd. Those are not the systemd ways, but it works fine anyway.

I also wonder why you think you represent an entire user-base or that you aren't listened to?

The most confusing thing about systemd for the distro I help out in is that we disabled the syslog daemon by default. That takes getting used to, no matter how easy it is to reenable it again.

Poettering: The Biggest Myths

Posted Jan 27, 2013 15:18 UTC (Sun) by hazard (guest, #3695) [Link] (6 responses)

Firstly I never wrote that I represent entire user-base. However, I do feel that many sysadmins share the same feelings about systemd as I do. At least I never heard praise of systemd from any of my peers.

systemd requires time to understand in its entirety. I'm not speaking about ability to start or stop particular service or init-script, I'm speaking about understanding the grand picture of how the new startup works in detail, what all those files in /lib/systemd mean and how they interact with each other.

Without having a good grasp on the startup process overall, one is unable to quickly troubleshoot issues. This causes a commercial risk of longer downtime, as sysadmins will take longer to bring the system up in case of a fault.

Poettering: The Biggest Myths

Posted Jan 27, 2013 18:04 UTC (Sun) by jwarnica (subscriber, #27492) [Link] (1 responses)

So don't upgrade to a system that uses systemd.

What is the problem?

Poettering: The Biggest Myths

Posted Jan 27, 2013 18:36 UTC (Sun) by larryr (guest, #4030) [Link]

So don't upgrade to a system that uses systemd.
What is the problem?

The problem for me is I want the other updates to the other 99% of the software on the system. I am not a sysadmin by profession or by choice, but I need to be for the things I want to do with the system. Understanding the holistic and specific impacts of using of systemd for everything it takes to do what I want has taken me many hours, and I suspect it will take many more. But I am not going to introduce the even greater risk of switching to a different system to avoid using systemd.

Poettering: The Biggest Myths

Posted Jan 27, 2013 18:49 UTC (Sun) by HelloWorld (guest, #56129) [Link]

> systemd requires time to understand in its entirety.
Sysvinit based init schemes take orders of magnitudes more time and they even work differently on different distros.

> Without having a good grasp on the startup process overall, one is unable to quickly troubleshoot issues. This causes a commercial risk of longer downtime, as sysadmins will take longer to bring the system up in case of a fault.
This is a short-term problem that is easily offset by systemd's relative simplicity in the long term.

Poettering: The Biggest Myths

Posted Jan 27, 2013 23:13 UTC (Sun) by ovitters (guest, #27950) [Link]

If the init system was intended to be changed every few years, then I'd understand your point. But it is one change in 10+ years (depending on distro, etc).

So 'change is bad': agree. But this is a one-time change. Obviously there is pain in relearning things. But there are also loads of benefits, e.g. systemctl status $SERVICE actually gives meaningful output. Furthermore, you can have systemd reliably detect crashed services and have them restart automatically (no need for some monitor script). Things like this I see as hugely beneficial. One one server amavis dies every few months for reasons unknown. Restart and it'll work for months again. Pretty nice if that is supported instead of yet another monitoring script.

Regarding representing the entire user-base, you said:

Sorry, real life doesn't work this way. What people know DOES matter, what your existing customer base wants also DOES matter.

I doubt anyone fully knows what (e.g. Fedora) users want. Maybe they want systemd, maybe not. You suggest that change is bad and to me you were trying to speak on behalf of everyone with above. Misread that.

Poettering: The Biggest Myths

Posted Jan 28, 2013 4:20 UTC (Mon) by raven667 (subscriber, #5198) [Link]

I am a sysadmin and have been for around 15 years and I think that systemd is awesome so now you have some praise from your peers 8-)

I find systemd to be thorough well thought out, and am excited after doing some troubleshooting that required interaction with systemctl, loginctl and journalctl by how simple it makes things. In a way it reminds me of OpenBSD, which I also find to be well organized. The problems it solves are not fundamentally difficult, starting processes in a controlled environment, and clears away a lot of confusion by just, doing, that.

Poettering: The Biggest Myths

Posted Feb 6, 2013 5:45 UTC (Wed) by vonbrand (subscriber, #4458) [Link]

But as the knots SysVinit would tie the system into (by willy-nilly starting stuff "in order" when some precondition was delayed for whatever reason, something didn't start because of a mistake and the whole line of dominoes came down leaving no clue) are next to impossible with systemd, this point is moot...

Poettering: The Biggest Myths

Posted Jan 27, 2013 17:26 UTC (Sun) by HelloWorld (guest, #56129) [Link] (2 responses)

> There are reasons why Cisco keeps IOS syntax unchanged, why GUI apps keep using "floppy disk" icon to save files and why Linus insists to keep kernel's user-level API unchanged.
Yeah, it's probably for the same reason that systemd still supports SysV init scripts, /dev/initctl and all that crap. Seriously, who are you trying to shit here?

Poettering: The Biggest Myths

Posted Jan 28, 2013 13:36 UTC (Mon) by pboddie (guest, #50784) [Link] (1 responses)

Perhaps you would make your point better by pointing to the documentation for those compatibility features instead of assuming that someone who is honestly reporting their concerns has a hidden agenda and has to be "called out" over it.

I know it's fashionable to tell existing users to "get with the program" and to assume that they have just as much time to read up on all the new stuff as the people who made that stuff had to develop it, but the recommended transition happens a lot easier if you don't ridicule those people as the first thing you do.

Poettering: The Biggest Myths

Posted Jan 28, 2013 13:52 UTC (Mon) by niner (subscriber, #26151) [Link]

Information about these compatibility features is contained in man systemd which at less than 400 lines of text should be short enough to warrant a reading by someone who obviously has enough time to post multiple messages about systemd. Wouldn't it be prudent to at least read a minimum of documentation before claiming missing compatibility?

Poettering: The Biggest Myths

Posted Jan 27, 2013 17:03 UTC (Sun) by smurf (subscriber, #17840) [Link] (1 responses)

>> Actually, this is the core issue: systemd, as a complete replacement,
>> makes all these years of experience obsolete and irrelevant.

No it does not. Your /etc/rcX.d/SYY.foobar script still works as-is. In fact, on Debian (if it uses the LSB functions) it transparently redirects itself to systemd, so your basic "/etc/init.d/foobar restart" invocation stanza does the right thing.

Your assertion that

>> most of the new functionality could be implemented without
>> a complete eradication of SysV concepts.

is misleading. Most, probably. All, no way. Just read the systemd.service manpage. Lots of what it can do require assistance from /sbin/init; when you're done implementing all of that, and then write another heap of fragile helper programs along the lines of Debian's(?) start-stop-daemon to do whatever can be done withoout /bin/init's help, you have basically created a wholly new startup system anyway. So, you can just as well drop that idea and use systemd. :-P

Just answering the question "is server FOO running" in traditional SysV is an exercise in fragility. PID file exists? Contents match a running process? With the same executable and/or process name? (Oops, does your SysV init script checks THAT? What happens if you rename inetd to inetd.OLD before restarting?)
If not, can we safely restart it – are all its child processes dead yet? What if two sysadmins restart a service at roughly the same time, would that cause a problem?

And so on.

Yeah, we old fogeys need to learn a few new magic incantations, but let's face it: they are a whole lot less fragile than the ones they replace, and they tell us a lot more about the state of the system. Given the choice between what "systemd status foo" does for me and what I had to do before to figure out what's going on (grep through ps, verify that the server I do see doesn't happen to run in some chroot-ish namespace, grep through a few syslog files), the choice is a no-brainer. Even for me.

Poettering: The Biggest Myths

Posted Jan 27, 2013 21:07 UTC (Sun) by pizza (subscriber, #46) [Link]

>Yeah, we old fogeys need to learn a few new magic incantations, but let's face it: they are a whole lot less fragile than the ones they replace, and they tell us a lot more about the state of the system. Given the choice between what "systemd status foo" does for me and what I had to do before to figure out what's going on (grep through ps, verify that the server I do see doesn't happen to run in some chroot-ish namespace, grep through a few syslog files), the choice is a no-brainer. Even for me.

This. A thousand times this.

Poettering: The Biggest Myths

Posted Jan 28, 2013 2:19 UTC (Mon) by mezcalero (subscriber, #45103) [Link] (2 responses)

You know, regarding your example with spawning something on a serial tty: if I assume right you'd have edited /etc/inittab and add a new specialized line there, for your purpose, right?

As it turns out this already didn't work anymore long before systemd came along. And that's because Fedora and RHEL 6 used Upstart for a while, and Upstart already dropped support for inittab entirely.

In systemd, we could actually relatively nicely have provided compatibility with most features of inittab with a "generator" tool, which can extend systemd's dependency tree nicely from external configuration in other formats, such as inittab. However, given that Upstart already dropped support for inittab we decided there was no point in reintroducing it again.

So anyway, for this specific issue you cannot actually blame us systemd folks, blame Upstart instead please. (Not saying you can't blame us for a lot of other changes, but hey, for this specific one we do not take any credit.)

Lennart

Poettering: The Biggest Myths

Posted Jan 28, 2013 2:33 UTC (Mon) by dlang (guest, #313) [Link] (1 responses)

doing a quick bit of googleing here, it looks like RHEL is shipping support for inittab in it's upstart driven versions.

This makes a lot of sense to me, the lack of inittab support in upstart is a mistake. It's too bad that systemd is copying this mistake. It will just require people to work around it.

Poettering: The Biggest Myths

Posted Jan 29, 2013 10:01 UTC (Tue) by michich (guest, #17902) [Link]

No, one cannot use inittab in RHEL 6 to spawn any custom services. The RHEL 6 Migration Planning Guide says:
The /etc/inittab file is deprecated, and is now used only for setting up the default runlevel via the initdefault line. Other configuration is done via upstart jobs in the /etc/init directory.
... and then goes on to describe how to setup a custom getty instance using an upstart job.

Poettering: The Biggest Myths

Posted Jan 28, 2013 10:49 UTC (Mon) by pbonzini (subscriber, #60935) [Link] (11 responses)

> Unfortunately this is not the case and systemd designers preferred to throw out anything that reminds of SysV boot process. People like me simply don't understand why they have to spend hours learning new system when the old one wasn't "broken enough" for them. Once RHEL7 comes out, they'd just plainly reject it and look for a CentOS fork that keeps SysV in.

That's called RHEL6. It will be supported for several years after RHEL7 comes out.

Poettering: The Biggest Myths

Posted Jan 28, 2013 10:58 UTC (Mon) by dlang (guest, #313) [Link] (3 responses)

so you want to encourage people to not upgrade to RHEL7??!?!

just being techincally better (assuming systemd is), isn't enough to drive people to switch to something different.

If it was there wouldn't be nearly as many peole using Windows :-)

but even in the *nix space, look at syslog. For many years syslog-ng was out there and nobody disputed that it was a better syslog daemon than what everyone was using, but for the majority of the people plain syslog wasn't 'broken' enough for them to be willing to take on the load of learning a completely different config syntax.

rsyslog displaced sysklog very rapidly in part because it didn't force people who didn't want to use the new features to learn new ways of doing the same thing (and those wanting to do something different were willing to learn how)

I know people keep repeating how systemd doesn't force you to change, you can still use init scripts, but that's not how it's being used and setup in the distros using it so far.

Poettering: The Biggest Myths

Posted Jan 28, 2013 11:15 UTC (Mon) by pbonzini (subscriber, #60935) [Link]

> so you want to encourage people to not upgrade to RHEL7??!?!

Absolutely not, just pointing out the obvious: it's not like RHEL7's release will represent a flag day for everyone using RHEL. Not for systemd, not for anything else.

Of course the RHEL7 documentation will cover systemd, but even the training material covering systemd for RHCSA and RHCE courses will likely come out a few months later (at least that was the case for RHEL6).

> I know people keep repeating how systemd doesn't force you to change, you can still use init scripts, but that's not how it's being used and setup in the distros using it so far.

I agree, and that's why a "RHEL7 without systemd" fork just cannot exist.

Poettering: The Biggest Myths

Posted Jan 28, 2013 17:56 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

I looked at syslog-ng a couple of times, and while it's better, it's not that MUCH better. There are no "killer features" for me there, so I just don't bother.

Poettering: The Biggest Myths

Posted Jan 28, 2013 21:27 UTC (Mon) by dlang (guest, #313) [Link]

> There are no "killer features" for me there, so I just don't bother.

That is my exact point.

'killer features' for me

with syslog-ng there were a LOT of features that were absolutly killer for people who needed them, but it turns out that the vast majority of people were happy with what they had, and so the pain of having to change how everything currently worked in order to get new features that they didn't care about was enough to prevent just about any distro from switching to syslog-ng as the default syslog

along comes rsyslog, and while it's config files have some 'new, strange' stuff up at the beginning, the main part of the config files looks just like classic syslog, and it's maintained. Within a year, just about every distro switched to using rsyslog as the default, and it didn't bother anyone because their configs either 'just worked' (if they were included), or were dead trivial to adapt.

Now, several years later, there is a significant uptick in the number of people doing things with rsyslog that could never have been done with sysklog (but that could have been done with syslog-ng), in part because the transition was evolutionary, with full support (not just in theory, but in practice by the distros) for using the old style configs.

Poettering: The Biggest Myths

Posted Jan 28, 2013 11:19 UTC (Mon) by davidstrauss (guest, #85867) [Link] (6 responses)

> That's called RHEL6. It will be supported for several years after RHEL7 comes out.

RHEL6 runs Upstart, not SysV init.

Poettering: The Biggest Myths

Posted Jan 28, 2013 11:21 UTC (Mon) by pbonzini (subscriber, #60935) [Link] (5 responses)

When people mean sysvinit usually they only care about /etc/rc.d.

Poettering: The Biggest Myths

Posted Jan 28, 2013 12:02 UTC (Mon) by davidstrauss (guest, #85867) [Link] (4 responses)

> When people mean sysvinit usually they only care about /etc/rc.d.

You'll have to clarify which people you're referring to. Without knowing, I have to assume you're just speaking for yourself.

In any case, let's review the rule you've proposed for for what qualifies as "SysV init" (or, really, what's more accommodating to long-time SysV init users).

== Upstart ==

Upstart is "SysV init" because it allows rc.d for traditional init scripts. Unfortunately, it lacks first-class support for the SysV init scripts themselves. By "first-class," I mean where administrators can control the SysV services using normal Upstart commands and have the services participate in dependencies trees with other services. This is not possible on an Upstart machine [1].

Upstart's rc.d support does not extend to native Upstart services. There's not even an equivalent of enabling and disabling configured services in Upstart. You could use symlinks from /etc/init, but it wouldn't be a standard approach supported by any CLI tools or configuration management utilities like Puppet or Chef.

Adapting a SysV init-based service to run as a first-class citizen on an Upstart machine requires complete conversion to a native Upstart service. Once that's done, the service is simply always enabled (because of the aforementioned lack of rc.d or equivalent for native services).

== systemd ==

systemd is apparently *not* "SysV init" because it lacks rc.d. However, is does have first-class support for /etc/init.d scripts, including support for native systemd service management commands, integration into dependency trees with other services, and automatic mapping of the LSB runlevels to the systemd equivalents [2].

While systemd lacks rc.d support, it has a superset of the rc.d/runlevel capability with named targets. And, as long as the administrator has been using chkconfig (or equivalent) to enable or disable services, there's no change in the commands to enable or disable services -- whether systemd native or SysV-based.

So, which one is really more of a departure for long-time SysV init administrators?

[1] http://askubuntu.com/a/14823
[2] http://docs.fedoraproject.org/en-US/Fedora/16/html/System...

Poettering: The Biggest Myths

Posted Jan 28, 2013 12:20 UTC (Mon) by davidstrauss (guest, #85867) [Link]

One correction: Upstart apparently now has support to disable native services. But, it's not via rc.d and it doesn't support disabling SysV init-style services the same way as the new method for native services.

Poettering: The Biggest Myths

Posted Jan 28, 2013 12:25 UTC (Mon) by pbonzini (subscriber, #60935) [Link] (2 responses)

> You'll have to clarify which people you're referring to. Without knowing, I have to assume you're just speaking for yourself.

For example the guy that started this thread (http://lwn.net/Articles/534260/). In general most of the practical complaints I heard about systemd (i.e. not "oh but the Unix way") are about /etc/rc.d.

For upstart, nobody did the work of converting most services to native, which is why you hear screams of horror for RHEL7's systemd but not for RHEL6's upstart. The transition was hardly visible.

FWIW, you're preaching to the choir. I like systemd.

Poettering: The Biggest Myths

Posted Jan 28, 2013 12:38 UTC (Mon) by davidstrauss (guest, #85867) [Link] (1 responses)

> For example the guy that started this thread (http://lwn.net/Articles/534260/). In general most of the practical complaints I heard about systemd (i.e. not "oh but the Unix way") are about /etc/rc.d.

It's pretty arcane to go rooting around in rc.d when there are tools like chkconfig that have handled such needs for years. systemd still tracks which services are enabled for a target using symlinks, so it's no more or less "the Unix way" than rc.d.

> For upstart, nobody did the work of converting most services to native, which is why you hear screams of horror for RHEL7's systemd but not for RHEL6's upstart. The transition was hardly visible.

Upstart pretty much leaves the SysV init side of things alone and does its own thing. That has a low impact on administrators when you don't convert any of the services over, but it creates a schizophrenic mess once you have a mix. (It's actually alright with *only* native Upstart services, too.)

That's still pretty different from Upstart being more like SysV init or more accommodating to experienced SysV init users.

Poettering: The Biggest Myths

Posted Jan 28, 2013 12:49 UTC (Mon) by davidstrauss (guest, #85867) [Link]

Looking back at the referenced post (regarding rc.d expectations) again, it seems that the author was actually looking at the init.d script rather than the enabled/disabled symlink structure, which is what I usually think of when someone mentions "rc.d." So, chkconfig obviously wouldn't handle what he was trying to do.

Oh well.

Poettering: The Biggest Myths

Posted Jan 27, 2013 14:16 UTC (Sun) by marduk (subscriber, #3831) [Link]

> 3) systemd makes the boot process non-linear and complex to comprehend and predict. I cannot easily see the boot process visually as it is happening and I have to rely on external utilities to understand the order in which everything will be started. With systemd, I get a feeling that I'm booting Windows - I don't have a feeling that I'm in control anymore.

Possibly true.. but I've never needed to know the exact order that things are started, just that they are started in the order they need to be. I used to use openrc, which has similar behavior). As long as the service dependencies are made explicit. The syntax of specifiying dependencies/orderings in systemd is similar to openrc, the main difference is that in systemd they are plain text files and in openrc they are shell scripts.

> 4) systemd is more difficult to troubleshoot. Starting a SysV script, I get output and errors straightaway on the screen. With systemd, at best this information is hidden somewhere in the logs which I have to hunt for.

Also coming from openrc, I have to disagree. A lot of times with openrc you don't see an error message on the console, and sometimes it doesn't even get logged to a log file. You have to retry the command with a special flag passed to the init script, or sometimes even edit the init script to get useful error messages. So for, my experience with systemd has been different. A simple "systemctl status" on a failed service shows the tailed messages from the daemon (stderr, syslog, etc). It's been really easy for me to see why a system failed.

> 5) systemd has a steep learning curve. There are lots of config files and interaction between them is not obvious. It'll be quite difficult to explain it to a junior system administrator.

If you want to learn *everything* about systemd then you're probably right.. then again I didn't know everything about openrc either. In my experience though, just doing 1:1 conversions form openrc to systemd has been relatively easy. But there's a lot more that you can do with systemd... and I'm learning it incrementally. But the easy stuff is easy.

I am comparing systemd to openrc, which is already probably considered "advanced" to some compared to other/older init systems. It's been a *long* time since I used "my father's" sysv init. But I don't miss it at all.

Poettering: The Biggest Myths

Posted Jan 27, 2013 16:43 UTC (Sun) by jch (guest, #51929) [Link]

> systemd is a solution for a problem that doesn't exist for me

While many of may not like Poettering's solution, it's difficult to argue that SV is well designed. Just look at a typical Debian setup -- init is parsing inittab, which launches /etc/init/rc, which launches a bunch of scripts in /etc/rc?.d, which in turn use the start-stop-daemon binary to launch your daemons, which still need to double fork in order to create a new session. We've gone through four layers of indirection, and I still need to duplicate the double fork, setsid, open(/var/run/deamon.pid) incantation in every single daemon that I write!

(Let alone the fact that /etc/rc?.d is just one of the many ways to start a service -- there's /etc/inittab, there's /etc/inetd.conf, there's crontab, there's at, there's /etc/acpi/event.d, and probably others I've never used.)

I think that DJB has expressed the issue fairly clearly:

http://cr.yp.to/daemontools/faq/create.html#why

--jch

Poettering: The Biggest Myths

Posted Jan 27, 2013 20:42 UTC (Sun) by lu_zero (guest, #72556) [Link] (6 responses)

https://plus.google.com/113237307210359236747/posts/PCfHd...

That pretty much says everything.

systemd has interesting ideas and wrong implementations for a number of usecases.

OpenRC covers better more usecases and is not so terrible for the rest w/out forcing dependencies nor requiring specific software (in fact you can use it pretty much everywhere, even as secondary rc-system)

Poettering: The Biggest Myths

Posted Jan 27, 2013 21:26 UTC (Sun) by HelloWorld (guest, #56129) [Link] (5 responses)

> That pretty much says everything.
Actually, it says nothing at all as the guy is confused. The session bus and the system bus run in different dbus-daemon instances, so unless both crash (unlikely), either the desktop or systemd is affected, but not both.

Also, I just tried "crashing" the system bus dbus-daemon (the one with the system bus) on my Fedora 18 system with a SIGKILL. Systemd respawned it, I can still log in on the console, the desktop (KDE Plasma 4.9.5) works fine. If dbus-daemon had actually crashed, I probably wouldn't have noticed at all. So sorry, but I'm not impressed. Try harder.

Poettering: The Biggest Myths

Posted Jan 28, 2013 3:38 UTC (Mon) by zlynx (guest, #2285) [Link] (2 responses)

I *have* seen a dbus daemon get wedged. It was a few years ago and there may have been a kernel bug involved.

It definitely had messages waiting to process but it was stuck in a poll somehow.

Poettering: The Biggest Myths

Posted Jan 29, 2013 9:05 UTC (Tue) by ovitters (guest, #27950) [Link]

In addition to what you said: Every so often dbus has security bugs. No software is without bugs, but at least the heavy usage of dbus ensures that at least a lot of bugs have been caught.

Alternative would be to make something like dbus yourself. That still will have bugs, could potentially have sscurity issues, etc. Maybe would be smaller than dbus, but then also would not have all the support that dbus has.

In short: No clue what is better (roll your own vs rely on dbus), but disagree with people who suggest dbus is "obviously wrong" (to be clear: do not mean you :)).

Poettering: The Biggest Myths

Posted Jan 29, 2013 9:25 UTC (Tue) by nix (subscriber, #2304) [Link]

There was another recent bug that might have had the same effect. Same symptoms, poll()ers and select()ers on localhost got stuck after a while.

TBH, if something like that breaks, you're lucky to have a system at all. Those syscalls are *so* widely used...

Poettering: The Biggest Myths

Posted Jan 28, 2013 21:53 UTC (Mon) by lu_zero (guest, #72556) [Link] (1 responses)

You didn't try the right way or Fedora found a way to avoid the situation, if it is the latter I'd love to know how.

dbus _does_ have this kind of issues and moving in the kernel the dispatcher/broker does help mitigating some of them (if the implementation doesn't share the same issues, but I hope it does not).

Poettering: The Biggest Myths

Posted Jan 29, 2013 14:36 UTC (Tue) by foom (subscriber, #14868) [Link]

So the argument is "we wrote some terrible software with crap performance and poor failure modes. Guess it needs to be in the kernel." What?? Surely these problems are fixable in user space!

Poettering: The Biggest Myths

Posted Jan 27, 2013 21:23 UTC (Sun) by theophrastus (guest, #80847) [Link]

Just a quick aside, from an -old- linux veteran, to thank LWN for hosting conversations *exactly* like this. Filter out just a bit of the name calling and I learn a great deal about the forces currently affecting linux in specific, and computer developers in general.

Thankee LWN! keep posting the controversial stuff (systemd, UEFI, and Wayland...oh my!) because that's where there is information.

Poettering: The Biggest Myths

Posted Jan 28, 2013 5:05 UTC (Mon) by mgb (guest, #3226) [Link] (23 responses)

The problem is not change per se. Us "old fogeys" have successfully administered and programmed numerous radically different operating systems. We love to learn new techniques.

But good change is good and bad change is bad. And purporting to refute a stack of straw myths doesn't make a bad change any better.

The problem with systemd is that it is politically monolithic and controlling and therefore harmful to FLOSS progress. Stick it in a DSL router and sell a billion units - no problem. Stick it in the main branch and throttle Linux progress - big problem.

Poettering: The Biggest Myths

Posted Jan 28, 2013 5:32 UTC (Mon) by raven667 (subscriber, #5198) [Link] (22 responses)

What harm?

You keep using that word but I can't for the life of me see how reliably launching your systems processes is harmful. What harm do you see to FLOSS progress by running daemons in a consistent environment and providing tools for managing them?

Poettering: The Biggest Myths

Posted Jan 28, 2013 5:40 UTC (Mon) by mgb (guest, #3226) [Link] (21 responses)

... politically monolithic and controlling and therefore harmful ...

Poettering: The Biggest Myths

Posted Jan 28, 2013 5:45 UTC (Mon) by raven667 (subscriber, #5198) [Link] (20 responses)

But there is no "therefore", the fact that systemd is in a single source tree which is maintained by an open community of developers from several different companies or individuals dosen't imply or logically lead to any "therefore harmful" that I can understand.

Again, what harm are you talking about. I really, really don't understand your point.

Poettering: The Biggest Myths

Posted Jan 28, 2013 10:07 UTC (Mon) by mgb (guest, #3226) [Link] (19 responses)

I am so glad to hear that systemd is no longer monolithic. But I'll believe it when they split their monolith into separate uncoupled GitHub projects so that FLOSS can easily use the best and replace the worst.

This will be easy as systemd is no longer monolithic: systemd has one small UNIXy library for parsing unit files, a small UNIXy binary whose sole function is to control a cgroup, a small Unixy udev, small UNIXy tools for serial and for parallel startup, a small UNIXy inetd, and a small UNIXy init. And they are all as elegantly factored, modularized, and uncoupled as any good software engineer would insist. Not.

Sadly that's not how Poettering works. He uses political control of critical elements to push his unUNIXy monolithic ideas. systemd is not UNIX. It is the opposite of UNIX.

Now systemd in Fedora and Redhat is not a problem - other distros will continue to progress. But if systemd were simultaneously blocking the progress of key distros such as Debian, Fedora, and Ubuntu that would seriously harm FLOSS.

Poettering: The Biggest Myths

Posted Jan 28, 2013 11:07 UTC (Mon) by pizza (subscriber, #46) [Link]

>Now systemd in Fedora and Redhat is not a problem - other distros will continue to progress. But if systemd were simultaneously blocking the progress of key distros such as Debian, Fedora, and Ubuntu that would seriously harm FLOSS.

You seem to have a very different definition of "progress" than everyone else. The way you're writing here, "progress" means "keeping things unchanged"

Taken that way, your argument at least makes sense, but while you're entitled to your opinions, you aren't entitled to your own facts/definitions.

Poettering: The Biggest Myths

Posted Jan 28, 2013 12:03 UTC (Mon) by heijo (guest, #88363) [Link] (10 responses)

The Linux kernel is also an "unUNIXy monolithic idea", "not UNIX" and "the opposite of UNIX" according to these criteria.

Poettering: The Biggest Myths

Posted Jan 28, 2013 12:28 UTC (Mon) by mgb (guest, #3226) [Link] (9 responses)

> The Linux kernel is also an "unUNIXy monolithic idea", "not UNIX" and "the opposite of UNIX" according to these criteria.

Which is why the idea of a micro-kernel never completely dies. A micro-kernel would allow faster progress but in the case of the Linux kernel the performance disadvantages are too severe.

While no-one has yet devised an efficient micro-kernel replacement for Linux, implementing UNIXy cgroup monitoring and parallel boot are no more difficult than creating a monolithic unengineered monster to provide the same functionality.

Poettering keeps borging key FLOSS components because it gives him more power, not because his monolith is better. Carefully factored software - the UNIX way - is far more conducive to progress.

Poettering: The Biggest Myths

Posted Jan 28, 2013 13:01 UTC (Mon) by smurf (subscriber, #17840) [Link] (1 responses)

> Carefully factored software - the UNIX way - is far more conducive to progress.

Unless the elements are so tightly coupled that to change one without the other would not make any sense.

Case in point: cgroup monitoring. init needs to do that by itself, in order to know when a daemon's processes have all died. Please demonstrate that compartmentalizing this feature to another process is an advantage in some way, because I can't think of one.

Maybe merging the dbus and systemd repositories was a mistake, maybe not. I don't know; absent any demonstrated technical reasons why it's a bad idea I continue to assume that there have been sound technical reasons to do it. "Avoid a monolith, designed by a power-hungry <whatever>" is not a technical reason.

But you can still build the dbus stuff without systemd, and you can check out the repository and basically ignore all the systemd stuff, so what's your problem?

Poettering: The Biggest Myths

Posted Jan 28, 2013 13:06 UTC (Mon) by davidstrauss (guest, #85867) [Link]

> Maybe merging the dbus and systemd repositories was a mistake, maybe not.

Presumably, you mean udev and systemd. D-Bus is still quite independent [1].

[1] http://cgit.freedesktop.org/dbus/dbus/

Poettering: The Biggest Myths

Posted Jan 28, 2013 16:58 UTC (Mon) by raven667 (subscriber, #5198) [Link] (5 responses)

> Poettering keeps borging key FLOSS components because it gives him more power, not because his monolith is [technically] better.

That is pretty certainly not true based on the available evidence. You don't have to guess, you can watch his presentations, read his writings and make your decisions based on the technical arguments that he makes. Believing that he is living under a secret base in a volcano cackling at his "power" like a fictional Bond villan, twirling his mustache is your right I guess but just because that's what you believe doesn't mean it has any relation to reality. Making decisions based on opinions which are not real will not have the desired effects.

systemd to absorb cron next

Posted Jan 28, 2013 21:14 UTC (Mon) by dlang (guest, #313) [Link] (4 responses)

at least according to an article on phronix, they are looking at having systemd do the job that cron has been doing for decades next

systemd to absorb cron next

Posted Jan 28, 2013 21:36 UTC (Mon) by raven667 (subscriber, #5198) [Link]

Well I take anything said on phoronix with a grain or two of salt but I don't see any reason that cron/at and functionality can't be part of systemd as long as it can parse the existing format files. systemd already has all the infrastructure for starting processes in a defined environment on-demand, the only difference is being time activated instead of socket activated. It seems like it would result in a net reduction of complexity of the OS while increasing capability which I think is a very good thing indeed.

I don't see how that provides any relevance to "political power play" rhetoric as there are clearly defined technical reasons that have nothing to do with politics or "power" (whatever "power" means in the context of open source projects that are governed openly).

systemd to absorb cron next

Posted Jan 28, 2013 21:38 UTC (Mon) by BlueLightning (subscriber, #38978) [Link]

Honestly I'm surprised they haven't already done this. If there's value in tracking processes and putting them into cgroups for daemons, surely the same thing applies to things started as scheduled tasks.

systemd to absorb cron next

Posted Jan 29, 2013 8:59 UTC (Tue) by ovitters (guest, #27950) [Link]

That would allow any package to provide the systemd service file as well as any time/timer file (=cron) which it needs as well. Makes packaging much easier, avoids that a distribution might forget about cron as well as making things more consistent across distributions. IMO the goal should be to do as little changes during packaging as possible (meaning: bugs should be fixed upstream).

systemd to absorb cron next

Posted Jan 29, 2013 15:14 UTC (Tue) by mmeehan (subscriber, #72524) [Link]

As it should. Cron doesn't meausure up well in 2013. No way to detect job failures, establish job dependencies, or allocate processes to cgroups. In any important environment it's already been replaced with full-scale enterprise batch managment suites because basic cron is a liability.

Think of every "batch" out there implmented in cron with overly long timings between jobs, or hacked together with cron, shells, and lockfiles. There's a need for a serious cron replacement, and for a per-user daemon managment system. It looks like systemd is going to fill those shoes, which means they'll have a consistent interface and tools.

Poettering: The Biggest Myths

Posted Jan 29, 2013 0:42 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link]

Carefully factored software - the UNIX way - is far more conducive to progress.

An excellent point. Now you just have to show that the current set of software actually is carefully and correctly factored and you'll have proven your point. As it is, you seem to be assuming this point and proceeding directly from that assumption to accusations of bad behavior on the part of Lennart Poettering. In fact the only detailed discussion of factoring that I've seen has come from systemd developers trying to justify their decisions. I think the discussion would be a lot more productive if you could focus on refuting those technical arguments rather than attacking developers' motives.

Poettering: The Biggest Myths

Posted Jan 28, 2013 16:14 UTC (Mon) by mezcalero (subscriber, #45103) [Link] (6 responses)

systemd never was monolithic. Hence we couldn't turn it into something that "is no longer monolithic".

See, there, you are creating a new myth. Do I now have to update the article with a 31st entry?

Poettering: The Biggest Myths

Posted Jan 28, 2013 17:18 UTC (Mon) by mgb (guest, #3226) [Link] (5 responses)

> Do I now have to update the article with a 31st entry?

It would be far better if you spent your time factoring systemd into properly engineered and decoupled projects.

I'm sure you can figure out how to manage a cgroup from other than pid 1.

Poettering: The Biggest Myths

Posted Jan 28, 2013 17:48 UTC (Mon) by HelloWorld (guest, #56129) [Link] (3 responses)

Dude, if you think it's so easy to provide the set of features that systemd does in a significantly less-coupled and modular way, go ahead and do it. But if you can't, then stop telling other people how to do their job. Because unlike you, they are actually doing something *useful* instead of bitching about other people's work.

Poettering: The Biggest Myths

Posted Jan 28, 2013 18:16 UTC (Mon) by mgb (guest, #3226) [Link] (1 responses)

We don't need to create a well-engineered decoupled solution because most distros still have it.

Udev modularity has gone from many distros but there is no engineering reason for that, just the usual Poettering Power Play.

Read up on cgroups if you want to see how easy it is to not include the cgroups functionality in init.

Poettering: The Biggest Myths

Posted Jan 28, 2013 21:22 UTC (Mon) by niner (subscriber, #26151) [Link]

I just can't tell you how glad I am that this "well-engineered decoupled" piece of crap that's SysV init is no longer in my distribution of choice. Because _finally_ we got rid of the serious flaws.

Thank you very much to all systemd developers (they are more than one you know).

As soon as we see any contribution by you at all, you may earn our gratitude as well.

Poettering: The Biggest Myths

Posted Jan 29, 2013 21:42 UTC (Tue) by aleXXX (subscriber, #2742) [Link]

Please stop calling people "Dude".

Alex

Poettering: The Biggest Myths

Posted Jan 29, 2013 8:54 UTC (Tue) by ovitters (guest, #27950) [Link]

I assume you never used sysvinit on a distribution where /bin/sh does not point to bash? Obviously if you use /bin/sh you should not use any bash features. However, real life is different. I prefer systemd over anything that you call well engineered :P

Multiseat

Posted Jan 28, 2013 14:06 UTC (Mon) by Yenya (subscriber, #52846) [Link] (4 responses)

Disclaimer: I am a long-term multiseat Linux user - my home workstation has been dual-seat for 10+ years.

For me, there are two stumbling blocks for a multiseat workstation in current Linux distros (e.g. Fedora 17): pulseaudio and gdm/systemd.

For pulseaudio, the situation is actually not so bad: PA has been originally designed as a flexible multi-purpose tool. It is possible to run PA in system mode, and use a single system-wide daemon to split a 6-channel on-board sound card to three independent dual-channel outputs. The problem is that system-wide mode is not supported (even discouraged in the PA documentation[1]), and for example Fedora does not even contain the init-script (or .service file) for system-wide PA, and it also does not contain suitable SELinux policy module for it.

The other problem is this shiny new systemd/gdm "autoconfiguration". It would be nice for those rare cases, where everything works as authors intended. My problem is that the Xorg.conf file is used only for the primary seat, and for other seats, a temporary minimal file is generated under /run, with no control over its contents and no control over Xorg(1) command-line arguments. For example, my ~3 years old home workstation with two Radeon HD 3450 boards needs the -isolateDevice command-line argument, and there is no way to set it with systemd/gdm. I have been using manual setup and xdm for the last several years, which works without problems.

So from my point of view, systemd definitely does not enable the multiseat capability. This capability has been there for at least ten years, and works without (if not despite) systemd.

FWIW, there is 5+ years old Fedora Bugzilla entry about gdm not being able to pass the arbitrary Xorg command-line options:
https://bugzilla.redhat.com/show_bug.cgi?id=451562

[1] http://www.freedesktop.org/wiki/Software/PulseAudio/Docum...

Multiseat

Posted Jan 28, 2013 14:24 UTC (Mon) by cortana (subscriber, #24596) [Link] (1 responses)

It's a sad bug, but I'm surprised no one in 5 years cooked up a simple patch for gdm-simple-slave to make it pull config options from a file in /etc.

Multiseat

Posted Jan 28, 2013 14:27 UTC (Mon) by ovitters (guest, #27950) [Link]

There is a patch, it was rejected as they wanted to wait on multiseat. There now is multiseat, but bug is still lost in the bugtracker. See my other reply on the recent thing on systemd-devel.

Multiseat

Posted Jan 28, 2013 14:26 UTC (Mon) by ovitters (guest, #27950) [Link] (1 responses)

For PA system-wide, suggest filing a bug. That something is not supported does not mean you cannot have an init script for it. Maybe it was done to discourage or to make clear that it is not the supported mode, no idea.

For multiseat, only this week there was a thread on systemd-devel that multiseat sometimes needs more configuration. I have 0 clue on multiseat, and don't understand what kind of configuration was needed, nor what you're doing with multiseat. Maybe they are targetting what you need, maybe not. Suggest writing to systemd-devel.

Multiseat

Posted Jan 28, 2013 15:27 UTC (Mon) by Yenya (subscriber, #52846) [Link]

As for the system-wide PA: read the bottom of the page mentioned in my previous post:

> But my distro is shipping an init script for PA!
>
> Ignore that. It's stupid. They shouldn't be doing this.

Given Lennart Poettering's involvement in Fedora, I don't expect Fedora to go directly against his suggestions in this case. But yes, I may give it a try nevertheless.

Poettering: The Biggest Myths

Posted Jan 28, 2013 15:50 UTC (Mon) by nhippi (subscriber, #34640) [Link]

I'm not convinced myth 25 is just a myth. Take the following post:

https://plus.google.com/106265217227408958782/posts/e2Y1a...

I use systemd on my debian system and I feel systemd is the future - I just hope more attention is put into making troubleshooting easier.

There are real reasons to question systemd

Posted Jan 28, 2013 16:16 UTC (Mon) by ebiederm (subscriber, #35028) [Link] (16 responses)

Since no one has mentioned this.

FACT: systemd has had at least one well publicized failure to do the job it is responsible for so severe the linux kernel had to be changed because the systemd maintainers could not get their act together.

There are real reasons to question systemd

Posted Jan 28, 2013 17:52 UTC (Mon) by mezcalero (subscriber, #45103) [Link] (11 responses)

Here's another FACT: the kernel broke API more than once so that systemd broke too, even though the kernel folks consider breaking userspace a total no-no. Also FACT: one thing that broke systemd was namespacing code, the removal of "nspid". Namespacing is what a certain "ebiederm" hacked on.

There are real reasons to question systemd

Posted Jan 29, 2013 11:27 UTC (Tue) by nix (subscriber, #2304) [Link] (8 responses)

You can't break userspace! for values of 'userspace' equal to 'the syscall API'. What, you're using something the kernel exports other than syscalls? Sucks to be you, expect repeated breakage :( (though the churn has slowed somewhat in the last few years, thank goodness.)

Breaking userspace

Posted Jan 29, 2013 13:12 UTC (Tue) by eru (subscriber, #2753) [Link] (7 responses)

I recently wrote some code that wants to know how many cores or processors there are in the Linux system, so it can estimate how many parallel threads it makes sense to set up. I did this by scanning "/proc/cpuinfo". Is this now in danger of breaking at any time, because it relies on "something the kernel exports other than syscalls?"

In that case I would like to know of some alternate, more stable way to get this information - and preferably without adding dependencies on some heavy-duty libraries or middleware, since the program currently depends on nothing but its own code, the standard C library, and pthreads.

Breaking userspace

Posted Jan 29, 2013 13:45 UTC (Tue) by cortana (subscriber, #24596) [Link]

Breaking userspace

Posted Jan 29, 2013 13:53 UTC (Tue) by khim (subscriber, #9252) [Link]

What's wrong with a standard C library?

Breaking userspace

Posted Jan 29, 2013 16:20 UTC (Tue) by nix (subscriber, #2304) [Link] (3 responses)

IIRC, that's actually never worked on all multi-CPU architectures: it just happens that x86 and x86_64 report one stanza per CPU.

Breaking userspace

Posted Jan 29, 2013 16:54 UTC (Tue) by cesarb (subscriber, #6266) [Link] (2 responses)

/sys/devices/system/cpu seems better, and looks the same at least on this desktop (x86-64) and my phone (arm).

Breaking userspace

Posted Jan 29, 2013 19:01 UTC (Tue) by nix (subscriber, #2304) [Link] (1 responses)

But that's /sys, which is also rather more mutable than the syscall interface.

getconf _NPROCESSORS_CONF (or _NPROCESSORS_ONLN) is the thing to use (or, from a program, sysconf(_SC_NPROCESSORS_CONF) et seq. This is in glibc, so has a stability guarantee that is worth something. Let it figure out where to go for the answer.

Breaking userspace

Posted Jan 29, 2013 20:10 UTC (Tue) by eru (subscriber, #2753) [Link]

Thanks nix, and khim, that looks like what I need.

Breaking userspace

Posted Jan 29, 2013 16:29 UTC (Tue) by cesarb (subscriber, #6266) [Link]

Copying an old Stack Overflow answer of mine:

The definitive resource for /sys is Documentation/sysfs-rules.txt. The definitive resource for /proc/sys is Documentation/sysctl/. The definitive resource for the rest of /proc appears to be Documentation/filesystems/proc.txt. The rest of the Documentation/ directory in the Linux kernel source has other interesting information. In particular, Documentation/ABI/ mentions the stability of each interface.

(Unfortunately for your use case, I found nothing about /proc/cpuinfo in these files, and I know that the output is very different depending on the architecture, so it is probably not stable. The /sys/devices/system/cpu directory has some of that information, however, including how many cores there are, and should be the same for all architectures, so I would recommend looking there first and using /proc/cpuinfo as a fallback.)

There are real reasons to question systemd

Posted Jan 29, 2013 12:16 UTC (Tue) by HelloWorld (guest, #56129) [Link] (1 responses)

Would you mind telling us what those alleged kernel ABI breakages were?

There are real reasons to question systemd

Posted Jan 29, 2013 13:30 UTC (Tue) by lsl (subscriber, #86508) [Link]

Presumably files exported in sysfs. The very nature of sysfs makes keeping it stable an awkward task. As the first few lines in Documentation/sysfs-rules.txt say:

> The kernel-exported sysfs exports internal kernel implementation details
> and depends on internal kernel structures and layout. It is agreed upon
> by the kernel developers that the Linux kernel does not provide a stable
> internal API. Therefore, there are aspects of the sysfs interface that
> may not be stable across kernel releases.

There are real reasons to question systemd

Posted Jan 28, 2013 19:07 UTC (Mon) by smurf (subscriber, #17840) [Link]

(a) assuming that you talk about what I think you talk about, exactly whose failure it was is open to debate.

(b) not everybody religiously reads LWN and/or can read minds, so kindly name names.

Thank you.

There are real reasons to question systemd

Posted Jan 29, 2013 0:40 UTC (Tue) by robert_s (subscriber, #42402) [Link] (2 responses)

I've become awfully suspicious of the word FACT in capital letters.

There are real reasons to question systemd

Posted Jan 30, 2013 19:12 UTC (Wed) by ttelford (guest, #44176) [Link] (1 responses)

Depends on your source. Given a choice between ebiederm and any other human on the planet, I'll take ebiederm's word.

There are real reasons to question systemd

Posted Jan 30, 2013 19:45 UTC (Wed) by raven667 (subscriber, #5198) [Link]

Facts are not a matter of taking anyones word about it, they are either true or not irrespective of human concerns or politics.

Poettering: The Biggest Myths

Posted Jan 28, 2013 20:08 UTC (Mon) by cb064 (guest, #88059) [Link] (6 responses)

I just can not express how much i am fed up with these Poettering bashing flame fests.

Would it be possible to return to technical and on-topic discussions about this and similar endeavours?

Poettering: The Biggest Myths

Posted Jan 29, 2013 14:10 UTC (Tue) by bluss (guest, #47454) [Link] (5 responses)

We need to generate a lot of random bytes. It is a good idea to perform some other action (post a news story about systemd, Lennart Poettering, or Ubuntu) during the prime generation; this gives the random number generator a better chance to gain enough entropy.

Poettering: The Biggest Myths

Posted Jan 29, 2013 20:00 UTC (Tue) by Eckhart (guest, #74500) [Link] (1 responses)

> We need to generate a lot of random bytes. It is a good idea to perform some other action (post a news story about systemd, Lennart Poettering, or Ubuntu) during the prime generation; this gives the random number generator a better chance to gain enough entropy.

You just provided another argument against systemd: it ships with its own version of a random number generator, tightly coupled into the init system. Myth 1 demythified!

(The implementation is also pretty bad, there are quite a lot of predictable comments here…)

Poettering: The Biggest Myths

Posted Jan 30, 2013 8:38 UTC (Wed) by micka (subscriber, #38720) [Link]

> (The implementation is also pretty bad, there are quite a lot of predictable comments here…)

... but the one you respond to was probably the least predictable !

Poettering: The Biggest Myths

Posted Jan 29, 2013 22:14 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link] (2 responses)

An amusing theory with just one small problem: LWN uses https by default. That means they consume a lot of entropy every time somebody connects to read the latest flames. All the SSL connections from a massive systemd flamewar will actually deplete their entropy pool far faster than the posts will refresh it. Not to mention the absolute foolishness of depending on user-provided information to replenish the entropy pool in the first place...

Poettering: The Biggest Myths

Posted Jan 30, 2013 1:13 UTC (Wed) by Trelane (subscriber, #56877) [Link] (1 responses)

It's not like systemd or gnome3 threads contain any entropy anyway.... ;)

Poettering: The Biggest Myths

Posted Jan 30, 2013 2:28 UTC (Wed) by cb064 (guest, #88059) [Link]

the timing of our network packets might just contain enough of it .. Though the time between requests might correlate with the current length of the thread (at least for individuals reading the comments) .

Poettering: The Biggest Myths

Posted Jan 29, 2013 3:15 UTC (Tue) by cmccabe (guest, #60281) [Link]

Having used systemd for a few months, I can say that it is a great tool. Thanks, Lennart.

Poettering: The Biggest Myths

Posted Jan 30, 2013 15:36 UTC (Wed) by pspinler (subscriber, #2922) [Link] (4 responses)

As an enterprise level sysadmin, there's a couple of things in systemd that I think are pretty darn cool, and there's a couple of things I'm not looking forward to.

On the cool side, I rather like that systemd can reliably detect daemon termination and restart it, as well as some of the cgroup daemon isolation stuff. This is quite welcome.

On the concern side, I worry about the extra time it's going to take to debug issues in large chunks of compiled code v. script, the extra dependencies (e.g. on dbus), and I am annoyed at having to learn yet one more subsystem.

The cool stuff is pretty self explanatory, but allow me to explain my concerns.

Extra debugging effort: I recently (in the last two weeks) had to debug a LDAP client performance issue related to over large LDAP queries.

In one case, I found the bug in a set of perl scripts used as service monitors for our veritas clusters. I was able to find the offending code, modify it in place and test the fix in about 2 hours.

In another case, the issue occurred in a bit of compiled code using LDAP (sudo in this case), I had to download a large bit of code, spend a good amount of time reading it, spin up a compiler / development environment, *then* make changes and test 'em. Note that this had the advantage, since it's a pure user space program, that I could test it without e.g. system reboots, and the codebase was smaller than systemd's. Total time, a day and a half.

It'd be even slower for systemd, 'cause the codebase is notably larger, and it's significantly more tied into the system; I'm not sure it could be tested without rebooting.

Extra dependencies: as a good sysadmin, I try to follow the principle of running only the minimum stuff required on a server. For instance, my web servers have only 26 processes on 'em. More stuff running == more stuff to go wrong, and more attack surfaces. In general, I prefer to not do so. From what I can see, systemd requires at least d-bus, and maybe more besides.

Yet one more subsystem: the thing here is simply that sysV init isn't going to go away. It's not like our many hundreds of customers running on rhel5 are going to magically migrate to rhel7 the moment it hits. So I'm going to have to continue to know sysV init, and at least a bit of upstart for personal use, then add systemd knowledge on top of that.

For instance, when my boss tells me to install a new monitoring agent, and I write a puppet recipe go install and configure the service, and it doesn't work, I now have to be concerned about double the domain knowledge as before (remember, heterogeneous deployment across older sysV and newer systemd systems).

I know systemd is fun and it does cool stuff, I'll likely be learning it and using it at home if only because my home distro will use it; it's just that I only have so many hours in the day and both my personal and professional time is valuable. In this case I'm not yet convinced the benefits of systemd outweigh the time and opportunity costs of it.

-- Pat

Poettering: The Biggest Myths

Posted Jan 30, 2013 17:12 UTC (Wed) by rgmoore (✭ supporter ✭, #75) [Link] (2 responses)

On the concern side, I worry about the extra time it's going to take to debug issues in large chunks of compiled code v. script

This seems like a misplaced worry to me. Consider your example of the perl scripts you debugged. You assumed correctly that any performance problems would be in the scripts rather than the compiled code (i.e. Perl itself). That's a reasonable assumption because your scripts were something you apparently put together in house, so it hadn't had much performance debugging effort. In contrast, Perl is used by millions of people all over the world, and any critical performance issues in it would have been noticed long before you got there. The same thing is true of the shell and standard shell functions when you're writing scripts in sh; you assume that the compiled code has been thoroughly debugged and had its performance problems worked out, so any residual problem must be in your script.

I would argue that systemd is going to be much closer to Perl, bash, etc. than it is to your hand-rolled monitoring scripts. Systemd is going to be doing performance critical tasks on tens or hundreds of millions of machines, so any critical problems are going to be found and fixed quickly. The equivalents of your scripts will be the systemd unit files, which can be much simpler and easier to debug than traditional scripts because they don't require a lot of implementation details.

Poettering: The Biggest Myths

Posted Jan 30, 2013 19:22 UTC (Wed) by pspinler (subscriber, #2922) [Link] (1 responses)

Well, actually, the perl scripts I'm referring to were themselves vendor written, and presumably in wide use (there's a pretty big customer base for HA clusters running Oracle, after all). Ergo, wide deployment is no guarantee of non-bugginess.

Also, the analogy you make isn't quite getting my point, which is: it's easier and faster to debug script code than compiled code. It's further easier to debug small amounts of code than large amounts of code.

Systemd is both large, and compiled, and thus harder to debug.

To your point: I'm skeptical that systemd will remain so bug free as you imply. Perhaps after 5-6 years of bake in; but now? Uh, sure. It's a large, complicated body of code, which has taken on and rewritten a lot of functionality (initd, inetd, logging, scheduling, timezone, yadda yadda yadda).

That amount of functionality is going to be quite quite hard to get right in any short amount of time, no matter how good your software engineering is. In fact, to quote Mr Poettering from this very discussion, earlier:

> Heck, we have so much more bad code in our stack, Upstart totally stands out in quality.

I'm going to refer to an interesting blog post on software engineering by Joel Spolsky. He's a windows dev, and not a big open source guy, but I think he has some pretty good insights, here:

http://www.joelonsoftware.com/articles/fog0000000069.html

Joel S. argues that rewriting from scratch is one of the worst software project choices that you can make. Why? Here's the crux of it:

> The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they've been fixed.
...snip...
> that two page function. Yes, I know, it's just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I'll tell you why: those are bug fixes.
...snip...
> When you throw away code and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work.

There's more, but I think the above is telling. We've spent huge amounts of time making sure that the stuff we have works -- it'll take years more to make sure that the new stuff works (and here's the key point) _no matter how much better an architecture it has_.

So yeah, when systemd actually reaches that level of stability, then at that point I'll unlikely have to debug it in an enterprise environment. Right now, I'm skeptical it's there yet.

-- Pat

Poettering: The Biggest Myths

Posted Jan 30, 2013 19:43 UTC (Wed) by raven667 (subscriber, #5198) [Link]

I think you are right to be skeptical but I think that the systemd team is using appropriate software engineering standards leading to decent initial code quality. While you say the code is complex it is no where near as complicated as, say, a filesystem, so will take less time to stabilize, it has been worked on for several years now and shipped on several systems for at least one release cycle. It represents a net reduction in code compared to the previous systems (sysvinit, startup scripts, shell functions commonly sourced by scripts, ancillary tools used by scripts to daemonize and track pidfiles, etc.) which is another factor which should lead to the core systemd stabilizing quickly.

There will be bugs and problems in the future but I don't expect them to be common or widespread and I'd be surprised if it were in the core functionality because the core has such a limited and well defined scope (starting and killing processes).

Poettering: The Biggest Myths

Posted Jan 30, 2013 17:43 UTC (Wed) by raven667 (subscriber, #5198) [Link]

I too don't expect to be reading the source code to systemd when troubleshooting problems, it's the unit files where the logic is, and they should be much easier to debug than shell scripts. systemd has such a limited scope, it just needs to start processes in a configured environment, and has a well defined interface to the kernel I expect the core of it to be pretty stable. There may be more churn in the ancillary tools but they should be fairly easy to debug by logging or watching their inputs/outputs as they are separate processes.

Poettering: The Biggest Myths

Posted Jan 30, 2013 18:54 UTC (Wed) by ttelford (guest, #44176) [Link] (1 responses)

I realize I'm days late to the party, but there is one thing that has always bothered me about systemd: boot up fsck.
Let's say, for example, we have the following scenario:
  • We have critical systems, including several large disk arrays, that have lost power with no chance of shutting down properly. (Let's say there are tightwads who feel hardware failure and data loss is cheaper than a UPS).
  • No problem, when power comes back, the system starts booting up.
  • Naturally, our arrays require FSCK'ing... and here we have a problem.
  • Systemd (reasonably) halts all boot progress until fsck is done.
  • Systemd does not, however, provide any sort of feedback as to how the FSCK is progressing.
    • Obviously, this is all console mode, as the boot splash hides everything anyway
  • Even worse, systemd doesn't provide any indication of a problem when an FSCK fails, requiring manual intervention.
    • A sysadmin has no clue whether the FSCK is simply taking a long time, or failed altogether.
  • Since there's no feedback, an Admin doesn't know if s/he needs to reboot in a recovery mode (to perform the manual FSCK actions), or if the fsck is simply taking its sweet time.
  • Even without a power failure, many FSes require an FSCk after so many mounts and/or days. All the admin knows is that the boot isn't completing.
It's been a few months since I've tried using systemd because of this... and even then, I was using Debian systems, which have an old version of systemd.

So has systemd actually fixed the fsck clusterfsck? Or is it still impossible to tell whether the kernel has panicked on boot or the system is just fsck'ing?

Poettering: The Biggest Myths

Posted Jan 30, 2013 19:56 UTC (Wed) by michich (guest, #17902) [Link]

Either your experience with invisible fsck is obsolete, or you saw a bug. systemd-fsck (a small wrapper for the real fsck) prints the progress to the console.

Poettering: The Biggest Myths

Posted Feb 1, 2013 14:47 UTC (Fri) by barryascott (subscriber, #80640) [Link]

We converted from sysV init to systemd with F15 and have never looked back.

Odd issues with statup of daemons are now gone.
Code inspections of start code is easier
Dealing with a hot plug world is graceful and we are in control

Is systemd perfect? No, it can be improved. But it so much better then what went before.
I found drop offs in the early docs and provided patches to improve the after helpful discussions on the systemd mailing list.

It took us a little tim to learn how to use the power of systemd. It was time well spent. I encourage anyone that admisters a linux systm to find the time to learn the systemd basics. You will not regret it.


Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds