LWN: Comments on "How Debian managed the systemd transition"

How Debian managed the systemd transition

nix — Tue, 13 Oct 2015 22:49:03 +0000

Oh, I agree it would be bad by modern standards -- however, it was quite clearly capable of scaling to the point of handling graphics data with no more kernel support than that. To get back to the original point: unless you think D-Bus is not just going to be asked to handle graphics data but the full graphics flow of a 3D game I think the volume of data involved in graphics should not serve as an argument for needing kernel support just to handle that.

How Debian managed the systemd transition

raven667 — Tue, 13 Oct 2015 15:09:00 +0000

That's a good point, but even if you don't consider allowing the userspace app to just bang away at /dev/mem "kernel support" because that really isn't a defined API, certainly we say that limiting to the performance and capabilities of the 1990's X stack would be considered "degraded" by modern standards and applications. Making this behave safely without degraded performance required the addition of dedicated APIs, to talk to the graphics co-processor, to share memory buffers, beyond the 1980's UNIX standard ones.

We've already gone down the route of adding dedicated IPC APIs for SysV, for Netlink, for X/Wayland and now for DBUS, which I see as following the evolution of OS design and the needs of the applications of the era when these interfaces were designed.

How Debian managed the systemd transition

nybble41 — Tue, 13 Oct 2015 14:45:04 +0000

> X ran undegraded without kernel support for literally a decade plus

I think one could argue that being given direct access to the graphics hardware, and thus effectively unlimited access to the entire system, should count as "kernel support". Sure, the driver code was inside the X server rather than compiled into the kernel or a loadable module, but it still required special interfaces used primarily by X, and it wasn't possible to run the X server as an ordinary, non-root user process.

How Debian managed the systemd transition

nix — Tue, 13 Oct 2015 13:50:21 +0000

What? X ran undegraded without kernel support for literally a decade plus, until hardware 3D stuff started turning up. MIT-SHM provided everything needed.

They did a good job

mathstuf — Sun, 11 Oct 2015 00:15:02 +0000

One of systemd's goals is to make a layer for Linux that can be depended upon. Minor differences like this are the "death by 1000 papercuts" that makes cross-distro deployment a pain. Breaking every other distro in the same way Debian has for years is certainly not the better solution here. More docs would have been better. Even better would be some check in the updater about possible semantic changes in /etc/fstab entries.

They did a good job

cebewee — Sat, 10 Oct 2015 09:49:20 +0000

A behavior does not need to be "intended" to be relied upon. It suffices if this behavior is consistently exposed. Then a change has a high chance of breaking user expectations. Quoting dlang's comment:

> you should read Ingo's comments from the kernel qotw http://lwn.net/Articles/657428/

Now, there are of course cases where breaking changes are justified (and I won't judge whether this was the case here), but you cannot blame people for being upset when said changes break their system.

They did a good job

zdzichu — Sat, 10 Oct 2015 09:24:28 +0000

If it's not documented, how can it be "intended"? How can you judge if the behavior is correct and it is not a bug? See also encrypted swap support in article – it was another bug exposed by systemd.

How Debian managed the systemd transition

raven667 — Sat, 10 Oct 2015 01:24:59 +0000

Shared memory is a kernel feature that gets you some of the way there but doesn't have the access control interface that these applications require and the DRI/DRM interfaces in the kernel were created for graphics applications like X, much like memfd which was created for kdbus, so I don't think its fair to say that X runs undegraded without special kernel support.

How Debian managed the systemd transition

nix — Fri, 09 Oct 2015 23:29:47 +0000

Experience with the X Window protocol is instructive here as it sits in a very similar place in the software stack and there was a desire for dbus to be able to scale to the point of handling graphics data, which has already been demonstrated with X that a userspace daemon cannot do this without kernel support.

X was doing just that without kernel support for nearly two decades. The MIT-SHM extension is worth noting.

You don't need to be the kernel to share memory... and with memfds, you don't even need to be the kernel to share memory with untrusted partners.

They did a good job

nix — Fri, 09 Oct 2015 22:49:50 +0000

You're saying that only documented behaviour needs to be preserved? That users are expected to read *all* the documentation that applies to every little part of their system, and if they don't, it's their own damn fault for being annoyed when something that turns out not to have been documented randomly changes behaviour and breaks things?

This is a very legalistic (read, unhelpful) way to make a system. It's a way to make a system that, to be blunt, doesn't work very well.

How Debian managed the systemd transition

Creideiki — Mon, 05 Oct 2015 14:22:39 +0000

Another problem is that while the interface names might (sometimes) be predictable across time on a single machine, they are emphatically not predictable across several machines at the same time. Especially with hardware manufacturers changing how they wire up their PCI buses without changing anything other than the revision number printed on the motherboard.

I've had a batch of computers, all the same nominal model from the same (big) manufacturer, all installed from the same netboot image, half calling their only Ethernet interface one thing and half calling it another. System administration scripting was so much easier when all machines had an eth0 connecting to the main network.

How Debian managed the systemd transition

bronson — Mon, 28 Sep 2015 18:44:01 +0000

Just another data point: I've only tried systemd on the server but I can't recommend it highly enough. I replaced a few different monit/god/custom setups started by all sorts of different distro-specific init.d files with a single systemd unit file. It worked first try and it's been stone stable ever since (so, no idea how helpful the support channels are since I haven't had a chance to use them).

Still waiting for the other shoe to drop.. but it's been excellent so far.

How Debian managed the systemd transition

flussence — Mon, 28 Sep 2015 17:30:27 +0000

That's an impressive number! I doubt systemd would do much for me though; my browser already takes longer from exec to window draw than the rest of the boot process...

How Debian managed the systemd transition

sbergman27 — Sun, 27 Sep 2015 21:50:54 +0000

Based upon all my bad experiences with PulseAudio... I admit I was skeptical of systemd. But I have to admit, it's working well on my Debian 8 desktop. The administration tools are well designed. And after years of hearing about other people's fast boot times... which I'd decided I wasn't sure I believed... I was shocked when after the upgrade I was able to go from Grub screen to a fully operational system, logged in, and with Chrome up and on the Google page, in (drum-roll please) 5.7 seconds. And speed was not even the main focus of the systemd design. Just a side effect. Regarding the socket-based dependencies, I think it's clear that Poettering was right and Remnant was wrong. They really are are good as they look on paper.

They did a good job

zdzichu — Sat, 26 Sep 2015 11:16:53 +0000

Could you show where it was documented to behave that way?

They did a good job

rleigh — Sat, 26 Sep 2015 10:22:26 +0000

It wasn't an "unusual default". It had behaved this way for ~20 years and was the intended and expected behaviour for a Debian system.

How Debian managed the systemd transition

oak — Fri, 25 Sep 2015 22:24:44 +0000

Yea, the buffering is much larger performance issue than context switching. All it takes is some message that is generated very frequently, and a client that has subscribed to the message, but isn't reading its messages (e.g. because it's suspended for few days while on background).

Result is that daemon message buffers grow until they take all your memory, your system message transport goes to swap (with everything else) and things become *really* slow until the problematic client is killed. If the client is woken up, daemon and client can spend many minutes (or hours depending on how much swap & buffering you have) during which bus isn't very responsive. If allocations were mixed well enough, emptying the message buffer on daemon doesn't actually free its dirtied memory because it's gotten fragmented.

This is D-BUS experience from 5-10 years ago on semi-embedded device. Even worse, the user-space daemon gets it's memory fragmented very easily and doesn't return to system memory it's once allocated. So, local DOS is trivial to do with any client that can connect to bus.

Some of the things where kernel *might* be able to improve on this are:
* Assigning message buffers memory cost to corresponding client, so that admin can identify who's the culprit
* Better allocator that guarantees that after processing the messages, the emptied buffer can actually be freed for other purposes (i.e. allocation blocks don't mix data with unrelated life-times, e.g. send and receive messages or messages from/to different clients)
* If message is status broadcast, maybe having some mechanism where only last status update is buffered
* Suspending message sending if receiver isn't processing the messages

How Debian managed the systemd transition

dlang — Fri, 25 Sep 2015 20:36:45 +0000

actually, if the firewall interfaces get swapped, it's not going to talk to anything, because it is going to send the reply packets out the wrong interface (unless you also have dynamic routing)

Yes, it is possible to configure a system so that it's IPs and routes will swap with the interface changes, but the firewall rules won't, but that's getting into rather contrived territory.

How Debian managed the systemd transition

Wol — Fri, 25 Sep 2015 20:16:09 +0000

> What I mean is that linking the ethernet name to the hardware is useful in some contexts

You mean, like in a firewall, for instance ...

I saw a comment somewhere where eth0 and eth1 got swapped. In other words, until someone noticed, the firewall's soft, unprotected, meant for the internal network, interface was the interface to the hostile outside world ...

Cue major panic, lan disconnected from the internet, rebuild the firewall from scratch, ...

At the end of the day, unpredictable behaviour is a security risk. What I think happened was that the system had always been booted from cold, and had always been predictable. Then for some reason, one day it did a warm-start, and came back with the interfaces swapped over ... oops ...

Cheers,
Wol

Sigh

bronson — Fri, 25 Sep 2015 04:08:23 +0000

I'm guessing it means any future trolling will be redacted or deleted? It's a symbolic bit?

Sigh

mathstuf — Fri, 25 Sep 2015 02:11:18 +0000

Umm…okay. So a comment *starting off* invoking Godwin's Law is more useful to you because you spent hours reading debian-devel. Great, maybe, just *maybe* there are some folks here don't follow debian-devel who are interested in how the transition was done (such as me who does some support for various Linux system issues at work, some of them being "how does systemd do this?"[1]).

I don't really see anything interesting in the original post other than a "see how much of am asshat I can be" link to a proposal for systemd.conf akin to asking to present a "90% of the world doesn't want to run Debian, nevermind know what it even is" poll done (based on results at a university computer lab) at a Debian conference asking them to give up and go back home.

The rest, I'm going to assume is cherry picking quotes because he has earned that reputation at least (and having read the source of one, yeah, there's no need to reconsider it right now).

[1]And the majority of the time it's "oh, that's much nicer", the rest being "hmm, that's different, but OK". But I'm going to assume you'll just deny their existence.

Sigh

mgb — Fri, 25 Sep 2015 01:12:31 +0000

I found what you have designated as a troll to be a tad more useful than the article itself which was old news to those of us who have following events as they transpired.

But you clearly disagree with his opinion while I share it.

Sigh

alvherre — Thu, 24 Sep 2015 22:42:10 +0000

What is the "troll" bit? I don't think I see it set for this guy.

Sigh

corbet — Thu, 24 Sep 2015 13:33:36 +0000

So I was really hoping we would get through this one without somebody trying to restart the whole systemd flamewar. No such luck. The last time you did this, just over a month ago, you were warned that your troll bit would be set the next time. This is the next time.

How Debian managed the systemd transition

jb.1234abcd — Thu, 24 Sep 2015 04:09:12 +0000

Yo, men !

http://ibin.co/2Ge1kuufRD1G

The plot:

Recently a poster presented a view of systemD(efeat) - mostly pro, with
the exception of its packaging.
http://lists.freedesktop.org/archives/systemd-devel/2015-...

His packaging proposal was brushed off as wrong.

http://lists.freedesktop.org/archives/systemd-devel/2015-...
"It's 119 executables now, btw."
"It's a set of basic building blocks distributions can build an OS from."
"We provide people with sources, with a git tree, and it's up to them how they
decide to package that."
"(...) we keep all components of our system together in one repo, under
a single release schedule and without stable, internal APIs."
"(...) We need to keep things maintainable. And you don't make things
maintainable (...) by forcing us to stabilize internal APIs (...)".
"(...) the only folks who should care about our updates are those who
are technically versed enough to not need version numbers, but who can
read our NEWS files."
"Well, it's supposed to be a steady stream of smaller additions instead
of major feature additions in long intervals."
"We can drop things from our git tree from time to time, (...)".

There is a darker side behind the facade.

http://ewontfix.com/14/
"... an aggressive, dictatorial marketing strategy including elements such as:
Engulfing other "essential" system components like udev and making them difficult or impossible to use without systemd (but see eudev).
Setting up for API lock-in (having the DBus interfaces provided by systemd become a necessary API that user-level programs depend on).
Dictating policy rather than being scoped such that the user, administrator, or systems integrator (distribution) has to provide glue. This eliminates bikesheds and thereby fast-tracks adoption at the expense of flexibility and diversity.
"

An0nym0usC0ward
"True, systemd consists of 69 [edit: 119] binaries. But they are so tightly coupled that to my knowledge nobody so far was able to provide a fully compatible alternative implementation for any of them. Besides, since the APIs internal to those systemd binaries are not only undocumented but also constantly changing, providing a compatible alternative implementation means aiming for a moving target.
Even if LP holds the 69 [edit: 119] binaries as proof that systemd isn't monolithic, to me that's BS. The strong coupling is what makes for a monolith."

The plot thickens:

http://lists.freedesktop.org/archives/systemd-devel/2015-...
"systemd is a core port of Linux ecosystems these days, just like
the kernel. It's a foundation layer that finally brings some much
needed order, coherency, and vertical integration to the mess that
is Linux userspace."

The plot thickens more:

http://lists.freedesktop.org/archives/systemd-devel/2015-...
"To further resonate that. Just like with kernel, every vendor make
their own longterm maintenance thing of systemd.
Look at Centos vs Debian kernel, they are widely different, even if
released from same series or at the same time.
Ditto systemd, integration done in Debian, Ubuntu, openSUSE, Fedora
are all different as well."

Ouch !

The plot thickens even more:

Reason got an upper hand over vanity and a proposal of a prospect of
systemD(efeat) becoming a "Linux-based Init and Service Management System
Standard":
"Yeah, but no. I doubt writing standards like that makes any sense at all...
Sorry,
...
Lennart Poettering, Red Hat"

I hope the above topic and the following one will be discussed during their
upcoming systemD(efeat) conference:
http://lists.freedesktop.org/archives/systemd-devel/2015-...

Btw, this is Pieter Bruegel's take on it:
https://en.wikipedia.org/wiki/Satire#/media/File:Pieter_B...

How Debian managed the systemd transition

foom — Wed, 23 Sep 2015 20:35:37 +0000

... Except, it failed to do so. (Man, what a *waste* of a third attempt to have functional fs watching functionality...)

How Debian managed the systemd transition

raven667 — Wed, 23 Sep 2015 16:29:53 +0000

> presumably without having thought too deeply about things

I've been on the sidelines, following development on LWN, but that doesn't seem representative of the people involved or the effort which has gone into this, so I wouldn't presume that at all.

> not having questioned the notion that the performance problems with dbus-daemon were to do with kernel-userspace transitions

I believe there was awareness that the existing dbus-daemon implementation was not performant but also awareness that even a perfectly implemented userspace daemon has an upper limit on what it can do because of serializing, memory copying and context switches. Experience with the X Window protocol is instructive here as it sits in a very similar place in the software stack and there was a desire for dbus to be able to scale to the point of handling graphics data, which has already been demonstrated with X that a userspace daemon cannot do this without kernel support. Less copying and less context switches are also a boon for power usage which is becoming more important every year, both for battery powered and datacenter devices.

> efficient user-space implementation + whatever generalised kernel services are needed for IPC problems in the abstract.

This was the original goal and implementation many years ago but was flatly rejected by the kernel developers who would have needed to approve it which is why we have the kdbus implementation we have today as opposed to some other design. The original thought would be for a multicast AF_UNIX type socket that a userspace daemon could control which would be capable of zero-copy message delivery but the network subsystem maintainers refused to entertain the changes required to make something like that work and be supportable, so a different design which is much more self-contained is being proposed instead.

How Debian managed the systemd transition

dlang — Wed, 23 Sep 2015 15:51:56 +0000

Linus has pointed out that the performance wins of kdbus have far more to do with horribly inefficient userspace dbus code than any advantage of being in the kernel (context switches or memory copies)

So the 'official' justification for kdbus is no longer performance, but rather security and/or reliability

How Debian managed the systemd transition

luto — Wed, 23 Sep 2015 15:06:12 +0000

Indeed, kdbus saves a memory copy in the common case if the receiver is able to consume data straight from the "pool" without copying the data itself.

For small messages, this barely matters, and for large messages, both kdbus and AF_UNIX users can use memfds, which does even less copying.

Actually, for small messages, I'll only believe that the kdbus approach is faster if someone benchmarks it cleanly. The saved copy is only possible because the kernel writes to the receiver's pool when the message is sent, and that means that the kernel has to map the receiver's pool, and that's not free. (In fact it can be very slow -- modern CPUs are very good at mapping things, but at least x86 makes *unmapping* extremely expensive.)

How Debian managed the systemd transition

lgeorget — Wed, 23 Sep 2015 09:47:44 +0000

> The problem is some people already went and implemented a kernel DBUS, presumably without having thought too deeply about things and not having questioned the notion that the performance problems with dbus-daemon were to do with kernel-userspace transitions.

Actually, if I recall correctly the discussions on that matter, the main advantage of the in-kernel implementation of dbus was not that it reduces the number of context switches but that it reduces the number of memory copies because for the kernel, unlike a user-space daemon, copying memory can be as simple as mapping the same pages in two processes.

> those people (like any others) aren't keen to have their work wasted, there will now be pressure to integrate it.

As far as I can tell from reading the mails on the Linux mailing list, Greg Kroah-Hartmann has shown to be very professional. He would surely be pleased to see his work in the mainline kernel, but not to the point to "pressure" anyone.

How Debian managed the systemd transition

paulj — Wed, 23 Sep 2015 09:33:29 +0000

The problem is some people already went and implemented a kernel DBUS, presumably without having thought too deeply about things and not having questioned the notion that the performance problems with dbus-daemon were to do with kernel-userspace transitions. So given it exists and does improve performance over the inefficient user-space implementation, and given those people (like any others) aren't keen to have their work wasted, there will now be pressure to integrate it.

That pressure will be hard to deflect by pointing out the correct solution to an inefficient user-space implementation is not a very $FAVOURED_IPC_OF_THIS_DECADE-specific kernel implementation, but instead to implement an efficient user-space implementation + whatever generalised kernel services are needed for IPC problems in the abstract. To deflect that pressure for good requires coming up with that efficient user-space implementation really.

How Debian managed the systemd transition

jejb — Tue, 22 Sep 2015 16:20:27 +0000

> If someone implements a reboot-less upgrade from x.y to x.(y+1), and it actually works, I will personally buy them the beer or tasty non-alcoholic beverage of their choice*. Snapshotting the world and restoring using CRIU or similar tools doesn't count.

Hey, that's not fair: to go from n to n+1 you know the only way is to save and restore the kernel state in a version independent manner, so you're trying to define the only possible method out of your challenge. The problem with the method is the time it takes, but there are people working on it

https://sslab.gtisc.gatech.edu/pages/kup.html

How Debian managed the systemd transition

cortana — Sun, 20 Sep 2015 13:14:14 +0000

It is a probabilistic problem; you might never see it in your whole career; or you might see it twice in a week. It is annoying when those who have never seen it happen to them (and I am not including you in this set) assume that it is not a problem, and rubbish the efforts of those who are trying to fix it properly by removing the underlying race condition.

How Debian managed the systemd transition

patrakov — Sun, 20 Sep 2015 12:09:42 +0000

Well, I could not say "eth0 became eth1", but I had a case when eth1 became eth2. On Debian.

How Debian managed the systemd transition

dlang — Sun, 20 Sep 2015 10:52:07 +0000

I agree, I always disable that on my systems. But I can see that if someone was dealing with multiple USB interfaces, or wants to do async module loading (with all the ordering race conditions it introduces) in search of faster boot times, I can see why it could be useful.

How Debian managed the systemd transition

kreijack — Sun, 20 Sep 2015 10:29:14 +0000

> > I think that a more reasonable solution is to still provide the ethN interface name as alias of the new name.
> Network code doesn't support it and network devices have always been a 'special case' in Linux.
> Notice how there are no '/dev/ethX' files.

May be that *now* doesn't exist a technical solution for doing that. But I don't see any reason which prevent us to implement it.
Let me to rephrase my sentence: my suggestion could be cause of some problem, or today it is impossible because nobody had coded that solution.
The former may/is a stop, the latter could be solved by some hours of coding (and several days of testing :-) )

How Debian managed the systemd transition

kreijack — Sun, 20 Sep 2015 10:22:30 +0000

> a swap can happen if you have cards..
What you are saying is correct.... But I repeat in my (very limited) experience I never seen that.
What I see is that the ethernet name change from eth0 to eth1 when I moved from a disk image from one host to another, and that cause me more headache then the fact that I have more ethernet device of different hardware...

What I mean is that linking the ethernet name to the hardware is useful in some contexts, but in others not; and I suspect that these "others" are more common than the former...

How Debian managed the systemd transition

Cyberax — Sun, 20 Sep 2015 07:37:51 +0000

IFNAMSIZ is equal to 16, so you only have 15 letters for the full path.

And the second problem is that you don't actually _need_ your devices to be mounted at '/dev'. A device node can be anywhere on the system.

It doesn't even require anything exotic - a host might simply with to get info about a TAP/TUN device in a container.

The correct way to fix it would be by creating a new API that uses file descriptors instead of interface names.

How Debian managed the systemd transition

dlang — Sun, 20 Sep 2015 00:04:50 +0000

Interesting, i never ran into that.

It may be that I got bit by the rename due to card changes and disabled it entirely before the odds caught up with me.

How Debian managed the systemd transition

dlang — Sun, 20 Sep 2015 00:02:32 +0000

if they can do eth<mac> then /dev/ethN isn't unreasonably long