LWN: Comments on "The road forward for systemd"

The road forward for systemd

sebikal — Thu, 21 Oct 2010 08:17:25 +0000

ssh per connection still not working
When trying to connect to the ssh server, systemd spits out a "function not implemented" message. Any thought on this?

thank you

The road forward for systemd

sebikal — Wed, 28 Jul 2010 07:12:37 +0000

Thanks for the suggestion. Has anybody had any success with on demand starting of sshd?

The road forward for systemd

rahulsundaram — Tue, 27 Jul 2010 21:33:00 +0000

This sounds like a bug report you should file in http://bugzilla.redhat.com against systemd. Please do that.

The road forward for systemd

sebikal — Tue, 27 Jul 2010 21:20:25 +0000

Hello systemd folks :)
I've been trying out systemd for a few days and seem to have a bit of an issue. The way I've installed it is using a Fedora core 13 installation, then "yum install fedora-release-rawhide" then installed fc-14 systemd with yum. I'm booting the system using kernel boot args "init=/bin/systemd systemd.default=multi-user.target". The first time I did this I was unable to log in the system (not even through virtual consoles). To get past this, I issued "chcon -t init_exec_t /bin/systemd" and that fixed the login issue. However, sshd starting per connection basis is not working. Systemd is not starting sshd when I try to connect through ssh. Initially, I did not have any of the sshd.service, sshd@.service nor sshd.socket files. Shouldn't they have been installed by the systemd-units rpm package? Then, I copied them over from http://0pointer.de/public/systemd-units/ but to no avail. I first need to manually start sshd in order to connect with ssh. Any ideas?

Better kernel support

paulj — Tue, 13 Jul 2010 09:09:12 +0000

Yes it did. IIRC it used an ethertap device to forward the packets up to itself in user-space. Then when the line was up it would repoint the default route to the ppp (or whichever) device and send out the packets. Very neat.

Better kernel support

dlang — Tue, 13 Jul 2010 09:01:02 +0000

hmm, diald was caching network packets while it opened a network connection a decade ago. why can't the same mechanism be used for the inbound packets?

no, I don't remember the mechanism, but in the dial-up days I used diald extensively

Better kernel support

smurf — Tue, 13 Jul 2010 08:40:37 +0000

Perhaps a user space program could register its interest in a certain events (e.g. connection to a certain port) and take appropriate actions (e.g. start a daemon).

That's exactly what systemd does right now.

The kernel doesn't provide a mechanism to queue TCP SYN packets until the corresponding port gets opened. I don't think there should be. Either some process is listening on a port (you send a SYN+ACK back), or not (you send a REJ back).

Better kernel support

nas — Wed, 07 Jul 2010 21:28:06 +0000

It seems to me that a better solution would be for the kernel to provide explicit support for this model. To be sure, it would require some serious work to get the interface correct. I'm thinking of something similar to udev. Perhaps a user space program could register its interest in a certain events (e.g. connection to a certain port) and take appropriate actions (e.g. start a daemon).

The road forward for systemd

geocar — Wed, 07 Jul 2010 10:15:25 +0000

The comment is rebutted in the article. systemd and inetd are as meaningfully similar as postfix and stunnel, and maybe less so.

The road forward for systemd

Triffid — Wed, 07 Jul 2010 06:06:20 +0000

PulseAudio is terrible. I always try to rip that out first, but the distro vendors really seem to love this it. If it was up to me, I'd at least give users a choice.

Between PA and Avahi, I have had quite a few problems with stuff this guy has written. Systemd looks like another attempt to change things for no good reason.

Annoying time based fsck

hackerb9 — Wed, 09 Jun 2010 12:37:28 +0000

This means you have 2 distinct time-jumps. The first one is to avoid the annoyingly bad fsck times when a filesystem is 480+ days out of fsck ( right.)

I realize I'm going far off-topic by not contributing to the init flame war and instead giving a small helpful hint. Please forgive me. Don't jump your clock just to avoid fsck. If you're going to skip regular fscks anyway, you can use tune2fs -i 0 /dev/sdaX to disable the time based fsck.

ObligatoryFlameContribution: "NO! If you had read my blog post you'd realize you are ALL wrong! Using Makefiles for RC dependency is the one true way!

The road forward for systemd

michich — Tue, 08 Jun 2010 07:12:58 +0000

I'm not seeing any perceptible latencies with PA.. Can you point me to a detailed bugreport?

The road forward for systemd

marcH — Mon, 07 Jun 2010 20:06:58 +0000

At this time, the blog includes a mostly political (and buried) FAQ that does not answer any technical question.

This blog is light years away from being a good systemd sales pitch. That's perfectly OK as long as you do not get angry and annoying when people fail to read between its (too numerous) lines.

Just try to compare your blog to any LWN article and you might understand better the misunderstandings happening here.

The road forward for systemd

nye — Mon, 07 Jun 2010 16:23:00 +0000

Not the only reason, but the triggering reason. The thing that made me finally decide 'the people who design current desktop Linux systems have goals which are wholly incompatible with mine'.

The road forward for systemd

anselm — Mon, 07 Jun 2010 13:15:42 +0000

The only reason? IMHO this amounts to cutting off one's nose to spite one's face. It is after all quite possible to run Linux on a workstation without PulseAudio. I do, for example (on Debian sid, too), and audio still seems to work for me even so.

The road forward for systemd

nye — Mon, 07 Jun 2010 11:58:44 +0000

>PulseAudio is the reason I run a Mac. Seriously.

And the reason I run Windows, in fact.

The road forward for systemd

segedunum — Sat, 05 Jun 2010 13:35:48 +0000

> Please stop the FUDding!

Heh. Did you think Linux was any kind of gaming platform before? It certainly isn't with the multi-second latencies we get with it, and any audio work is completely out of the window now. Don't give me any of that crap about that not being the right use cases either. The Mac manages it. Windows manages it. They're called desktops for a reason.

It's certainly been a nice way to finish desktop Linux off completely.

The road forward for systemd

segedunum — Sat, 05 Jun 2010 13:29:00 +0000

> P.S. Lennart thanks for Pulse, it's been an awesome improvement.

Pffffff. Lennart should fix PulseAudio properly first by solving its extreme latency issues, amongst other things, before faffing about with something else.

PulseAudio is the reason I run a Mac. Seriously.

Too novel

Wol — Sat, 05 Jun 2010 11:27:10 +0000

The trouble with djb (and I speak from personal experience) was that he was a d***head.

I know I made a conscious decision that I was not going to touch any of his software, not because it was bad or good, but because I didn't want anything to do with him the man.

Cheers,
Wol

Why it's so hard to understand?

Wol — Fri, 04 Jun 2010 23:37:06 +0000

"Either" (in normal English usage) does *not* imply the two sides of the "or" are exclusive.

"You can borrow my car if you do either A or B" - in other words I don't care which you do and you could do both if you wish. "either" is a filler word which implies "xor" but doesn't require it. If you *really* mean XOR, you have to say "either but not both".

Cheers,
Wol

The road forward for systemd

obi — Fri, 04 Jun 2010 14:18:32 +0000

Irrespective of whether SystemD or Upstart is the greatest thing since sliced bread - honestly, both would be an improvement, and I can't yet wrap my head around which one would be better - the mess with udev is hard to miss.

Just recently I got a message while upgrading saying udev >150 is incompatible with a kernel I was running, that had CONFIG_SYSFS_DEPRECATED compiled in. Now, it's not a huge deal, but udev and init are "special"; developers can collectively decide that using certain new features and conflicting with deprecated features is the right way forward, but I'd prefer people to thread carefully and would like this to happen rarely. With udev this happened a bit too often, and it has burned users in the past. The whole "upgrade udev in lock-step with your kernel" is a little... annoying.

That being said I do understand it's worth it sometimes.

The road forward for systemd

oak — Thu, 03 Jun 2010 21:37:41 +0000

I think the largest Upstart issue is that its dependencies don't solve the issue that just starting the service is not enough for the clients. The server's socket needs to be available too, otherwise the clients can abort.

With Upstart one needs to modify the servers and have some extra code to communicate with these mods from the Upstart or its scripts. Or to have some trivial utility waiting & polling the server's socket availability before Upstart can fire the "started" event. If you don't have these with Upstart, your bootup process is fragile to timings issues.

That's the problem that systemd seems to be nicely solving. Doing the dependencies _correctly_ (based on server's service, not execution) with minimal modifications & complications required.

The road forward for systemd

eduperez — Thu, 03 Jun 2010 07:33:43 +0000

Some RAID controllers are painfully slow, and I mean they take several minutes to initialize and start loading the boot manager.

The answers are there, you just don't want to admit they are right answers...

Darkmere — Wed, 02 Jun 2010 07:38:58 +0000

Case in hand, several of our machines run without bios batteries (concious decision) and are reset to "plate" state if they boot up. This means dates set at somewhere in 2002. Early part bootup scripts check the time of /etc/init.d, and sets local time to modification date of init.d + 15s. This is done wether or not we have a working clock for consistency. At this point, if time is right, we break it, if it's wrong, we break it. All to enforce consistency and make sure we recover from the error.

After network is established, time is then fast-forwarded to "real" time via ntpdate.

This means you have 2 distinct time-jumps. The first one is to avoid the annoyingly bad fsck times when a filesystem is 480+ days out of fsck ( right.) the second one _has_ to run before dovecot, which will detect time warp and decide "Life is bad, hardware broken, we die now" and block. ( doesn't properly shut down, just stops working properly )

So, yes. A few services, mostly mailservices, and some other ones do not like it when time changes too much. ntp itself _requires_ ntpdate early on, or it will simply decide that the time is too much out of sync to even bother adjusting it. And with large timedrifts, ntp isn't fun anyhow.

So, You may consider those services that require proper timekeeping to be broken ( perhaps they are ) but they are common and have to be managed with. And it's easier to deal with that than to deal with other situations.

The answers are there, you just don't want to admit they are right answers...

paulj — Wed, 02 Jun 2010 06:33:33 +0000

It's not a safe assumption unfortunately. There are distressingly many machines out there which deliberately are run without batteries (lower field maintenance) and which hence have the time set at boot. Note that "ntpdate -s" implies setting the time - not slowly adjusting it. Even if you discount this example, I have a strong sense that there are many other high-level events (e.g. the network and printer examples).

What exactly is the "native notification logic"? (Note that many events are application layer).

Also, I'm not saying systemd needs eventing logic. I'm asking whether it makes sense to try solve these problems in a init process, external to applications. (for value of apps that includes those that would be started by it).

In short I'm asking whether actually its user-space that needs fixing to cope with differences in and changes to environmental state? Because it seems that doing that correctly would allow a not-too-fancy init to fire off apps in parallel and not worry about dependencies, as you argue systemd should be able to do with good apps. It seems applications will have to be modified to do this anyway, to get best effect from systemd.

The road forward for systemd

liljencrantz — Tue, 01 Jun 2010 23:01:16 +0000

Thanks for clarifying.

The road forward for systemd

anselm — Tue, 01 Jun 2010 17:55:41 +0000

I don't worry about the boot times a great deal – even with the traditional init my machine boots a lot quicker these days than it used to.

So never mind the benchmarks; if systemd merely does the other things it is said to do, at upstart's or even Sys-V init's speed, that will be just fine by me, and I'd be happy to just skip over upstart, thank you very much.

The road forward for systemd

__alex — Tue, 01 Jun 2010 16:18:59 +0000

I don't have a problem with it. It just seems that discussing it's technical merits is a bit philosophical when there's no published data on exactly the sort of thing people will be expecting it to improve.

The road forward for systemd

SEMW — Tue, 01 Jun 2010 16:11:12 +0000

If you find the explanation in the article inadequate ("Sorry, no bootcharts or hard data on start-up times for the moment. We'll publish that as soon as we have fully parallelized all services from the default Fedora install. Then, we'll welcome you to benchmark the systemd approach, and provide our own benchmark data as well"), you might get farther by posting your actual problem with it. Just rhetorically asking "Where are the benchmarks?" might mislead people into wrongly thinking that you just hadn't read the article.

The answers are there, you just don't want to admit they are right answers...

dlang — Tue, 01 Jun 2010 13:33:18 +0000

I don't know what datacenter you are working in, but I sure don't trust the system time in my datacenter until NTP has started, and at initial startup I don't have it gradually adjust the time as it may be so far off that NTP will decide that it will never get it right and shut down. Instead it does a one-time large jump to get the system time correct.

on some systems I use the -G option, on others I use ntpdate.

The answers are there, you just don't want to admit they are right answers...

mezcalero — Tue, 01 Jun 2010 11:56:35 +0000

If your event "system time may now be considered stable" shall be about NTP then it makes little sense since NTP clients tend to slowly adjust the time instead of making it jump. That means that the time is adjusted fully only after quite a bit of time. Applications should not wait for that. They should just assume that time is correct, right from the beginning. And that is a safe assumption for all machines built in the last 25y. And again, I don't see what systemd has to do with eventing like that anyway. I see no need to multiplex events through an eventing system. Get your events from the respective subsystems directly. Don't add an indirection layer here.

People should handle "in-lifetime" events (as you call them) with the native notification logic available. no need to involve systemd, or dbus or anything.

The answers are there, you just don't want to admit they are right answers...

mezcalero — Tue, 01 Jun 2010 11:47:07 +0000

I don't think that "System time may now be considered stable" is a valid, or relevant or necessary event.

The answers are there, you just don't want to admit they are right answers...

mezcalero — Tue, 01 Jun 2010 11:43:51 +0000

I am not sure why you get the idea that you need to poll for anything if systemd is used. systemd itself does not do repetitive polling anywhere, although we now can watch quite a few different kinds of resources for you: devices, file systems, automounts, sockets, paths, timers, swap files and more.

I am not sure what you mean by "basic dependencies". systemd actually has a pretty elaborate dependency system, which distinguishes eight kinds of dependencies: Requires, RequiresOverridable, Requisite, RequisiteOverridable, Wants, Conflicts, Before, After. And you can use that to build dependencies between all ten kinds of units we have.

There doesn't need to be an event for "name resolution available", because in the socket-based actviation scheme it is always available. And most services which need a live network, such as IPv6 discovery daemons, or Avahi or suchlike hook into netlink anyway to get notifications for this -- and rightly so. I see little need for another generalized eventing system. And I don't think that "System time may now be considered stable" is a valid event. On all machines built in the last 25 years or so the RTC should be "stable" from the beginning. I mean, it would be nice to have a notification system where the kernel informs us about a jumping time (i.e. on timezone changes), but that has nothing to do with the monotonic clock or systemd.

So, I fail to see why you'd want any generalized eventing system beyond the dependency system that systemd offers to you and the various notification systems the kernel already provides, such as inotify, netlink, udev, poll() on /proc/mount, and similar. If apps want those events they should use those notification facilities natively, there is little need to involve systemd in that.

Does that answer your question? Because quite frankly, I am not sure I understood the question entirely...

The road forward for systemd

__alex — Tue, 01 Jun 2010 09:06:56 +0000

If the merit in systemd is indeed technical and it is already possible to boot systems with it, where are the benchmarks showing it beating upstart?

The answers are there, you just don't want to admit they are right answers...

paulj — Tue, 01 Jun 2010 01:21:07 +0000

If systemd does indeed have some simple dependencies, then "system time may now be considered stable" == having a dependency on a script that runs ntpdate -s. I'm more concerned that there may be events that do not fall easily into "write to an fd, at the boundary of process lifetimes (either before or after)" model. Such events are not visible to systemd. If there's a significant amount of those, then you need something higher-level.

E.g. the printer example Lennart gives, and says "printer plugged in" can be depended on. But what if I want to depend on the type of printer? That's treading into udev below and DBus services above, depending on exactly what I want to do. Or "network available" - but what if I want to depend on a certain kind of network interface? Or a certain location (e.g. "start the automounter, for corporate NFS if ...")? Lots of the information you might use there is being maintained by NetworkManager (using DBus to publish that info).

Basically, if we add systemd to the mix, we're going to have udev, then systemd, then DBus + {various DBus services: NetworkManager, ModemManager, gdm, polkitd, bluez, etc}. Do we really need that extra layer of management? And many services modified for systemd would have to bind into a DBus(-like)? layer anyway, to handle in-lifetime events.

The answers are there, you just don't want to admit they are right answers...

nix — Mon, 31 May 2010 23:34:21 +0000

'network available' and 'name resolution service available' are handled by blocking the network socket of discourse until the service *is* available, by opening the socket but not accept()ing on it until the child is started. (It remains slightly unclear to me how you can fd-pass the socket fd to the child without hacking the child to accept a passed-in fd, which most unmodified daemons cannot accept -- in fact I'm not sure I've ever seen one which can.)

'System time may now be considered stable' I have no idea how you could handle. Lennart?

The answers are there, you just don't want to admit they are right answers...

paulj — Mon, 31 May 2010 22:40:18 +0000

I'm not FUDing. I was replying to a specific example about openvpn - not systemd. And then generalising from it to *polling* - not to name resolution! I said *IF* systemd leads to more polling in apps, that would be bad.

I understand systemd does allow for basic dependencies (the fulfillment state transition of which is obv. an event). However, the stated philosophy in systemd is to eschew encoding dependencies in config files and instead have the apps just "Do The Right Thing" (whatever that may be), as much as possible.

My question then is: Doesn't this philosophy mean that, in addition to systemd managing process lifetimes, that you also need an a higher-layer event-posting system above systemd to allow apps to stay informed of event fullfilment and transitions, and handle them? Events such as "network available", "name resolution service available", "System time may now be considered stable", etc.

Basically, systemd deliberately does not answer these questions, other than that it can act as a proxy for certain services by handling their fd, correct? If so, do you agree that there would such a higher-level system? If so, why do you think it is worth having BOTH a fancy init AND a fancy higher-level event-handling system? Why not just do all the work in this fancier higher-level service, and migrate services to be started by this higher-level service, and stick with the dumb init?

You may say I'm FUDing, but none my assertions in the comment you replied were about systemd. Anything relating to systemd were in the form of questions (except perhaps my fears of instability), as with this comment, and all I'm interested in is to have my questions/concerns addressed.

The road forward for systemd

nix — Mon, 31 May 2010 18:21:47 +0000

Exactly. What would be worthy of disparagement was if you'd decided that we *could* no longer use shell script fragments to start services. But, really, the repetitive umpty-Kb-long shell scripts to start random daemons on current systems are just ridiculous. Most of that isn't needed to *start the daemon* at all (BSD-style boot scripts didn't need it), but to provide the other services (stop/restart, easy reconfiguration via stuff in /etc/default et al) that the distribution needs. And *that* doesn't really belong in *every single* startup script. Better to centralize it.

The road forward for systemd

mezcalero — Mon, 31 May 2010 17:40:41 +0000

Hmm, my comments there don't seem to show up on the comments section on your blog. So here's my reply to your blog story:

First of all, you create the impression that systemds core design was about dependencies. Well, it is not. Dependencies are supported by systemd only for the purpose of early boot. Normal services should not really use that for much. One of the nice things in systemd is that the kernel will get the dependencies and ordering right for you, out of the box.

Also, it is misleading that the hw hotplug logic would only really make sense in an event-based systemd. But well, thats not true. It fits really nicely in a system that does dependencies: for example, there is a dependency between bluetooth dongle is plugged in and bluetoothd must be running. Systems that support dependencies can express that dependency explicitly, while event-based systems cannot.

Did you never get the idea that maybe nobody else is using your event-based design, because it is simply broken? (For the reasons I pointed out in my original blog story.)

Also, claiming that launchds or systemds core design was around on-demand loading of services is misleading. While we support that too, we do socket-based activation mostly to parellelize boot-up and to get rid of explicitly configured dependencies. Only a few services will actually be started on-demand in a systemd system. Most will be started in any case, but fully in parallel due to the socket activation logic.

Yes, systemd does both socket-based activation and dependencies. Manually configured dependencies are however only really needed for early boot-up or late shut-down and a few other cases. launchd does not support dependencies at all, for the price that their early boot-up is completely seperate from the starting of daemons, while in systemd this can be interleaved quite a bit.

OpenSUSE has already given up all Upstart experiments btw. Right now only Ubuntu uses Upstart fully, and Fedora and RHEL use it in sysv compatbility mode only.

And generally I think it would be nice to talk more about stuff Upstart has actually already accomplished, instead of great plans. In my postings about systemd I kept the part about my plans very short. Almost all features I talked about in my various postings can be listed under DONE, not under WILL BE DOING.

Note that systemd supports quite a few triggers for starting a service. And they can be combined in various ways. For example, we can start CUPS as a dependency of printer plugged in, as well as a dependency of client requests CUPS services, as well as theres a file in the CUPS spool dir. Its all implemented, and works fine. CUPS fits perfectly in the systemd model, as all its triggers are available already. If one of the conditions holds, CUPS is started. I dont see how the current Upstart could do anything like this So, on this topic: systemd 1, upstart 0.

And finally, I dont think you have really understood launchd or systemd. The focus is on using socket-activation for parallelizing and getting rid of explicit dependencies. Its much less on on-demand loading.

The road forward for systemd

mezcalero — Mon, 31 May 2010 17:12:44 +0000

Well, first of all, it's Peter Zijlstra who called us "dumb-ass", I didn't come up with that.

And if cgroups are really so ugly, why didn't anybody point that out when it got merged? Seems you are criticising us for a failure of the kernel community in reviewing the cgroup code then. And that is unfair.

Well, and it is really a very kernel-developer-centric view of the world if you say that running kernels without sysfs or cgroups or stuff like that should work with every init system. But, first of all, kernel developers make up only a tiny fraction of our users, and they are the ones who should be able to pass "init=/bin/sh" to the kernel, to make the system boot up in any case. Also, nothing forces you to use systemd. You can always continue to run SysV init if that makes you happy.

And I fail to see that there was any mess with udev. Also, isn't it a bit unfair if you ask us to basically stop development of userspace where it was 10 years ago, just because you are unwilling to follow any new userspace development and find it boring and stupid in general?