LWN: Comments on "The "Devuan" Debian fork"

This whole debate saddens me

anselm — Sun, 21 Dec 2014 00:01:06 +0000

The execution environment for cron jobs is not the same as the one in force when you're typing commands at the terminal. This is just one thing that you need to take into account.

The “maintainability” issue comes in because you will need to come up with all the necessary stuff by yourself, and you will have to do it over again for each new cron job that needs customisation. Writing the commands may be OK if you're a seasoned administrator, but it is tedious to query a shell script for the customised settings it incorporates the way you can for systemd units – you need to inspect the actual shell code, which is a hassle if you're a human but almost impossible if you're another shell script. (Systemd even makes it convenient to change various customisation settings on the fly, through simple commands, which is again almost impossible if customisation happens via shell scripts like you're suggesting.) With the traditional setup, there is no documented method for carrying such customisations across distribution upgrades, much less one that is standardised between distributions in the same way that systemd affords. And of course customsing a cron job is quite different from doing the same thing for an at job, or an init script, or a background service started by inetd or xinetd. Finally, forcing people to write shell code invites errors and inefficiencies, may increase exposure to security issues, and there are situations (like with some embedded systems) where a shell is otherwise not required at all, so having to introduce one simply to be able to set resource limits or a chroot for a service process would mean a considerable inconvenience.

This whole debate saddens me

vonbrand — Sat, 20 Dec 2014 21:42:21 +0000

I'm sorry, but as sonn as your cron job has to do even a minimal setup for doing whatever task you want to run, you are in a world of hurt. Yes, you write a shell script; no, it isn't just "what you write on the CLI", you need to consider it is running in a weird, locked down environment.

This whole debate saddens me

flussence — Sat, 20 Dec 2014 21:14:48 +0000

> you're reduced to sticking any unusual stuff in shell scripts, which is a hassle and difficult to maintain.

No, it isn't a hassle or difficult to maintain a plain text file containing lines corresponding 1:1 to what you can type at any terminal. The only valid reason for saying such things is if you're physically or mentally handicapped.

This whole debate saddens me

johannbg — Sat, 20 Dec 2014 01:12:11 +0000

Actually timer units and cron job complement each other shortcomings and coexist quite nicely

This whole debate saddens me

anselm — Sat, 20 Dec 2014 00:35:15 +0000

Especially because systemd, when starting processes, can do all sorts of things that cron can't, and it works the same across all types of units (whether started on a timer or not). With cron, you're reduced to sticking any unusual stuff in shell scripts, which is a hassle and difficult to maintain.

This whole debate saddens me

raven667 — Fri, 19 Dec 2014 17:57:42 +0000

> why is cron required to be part of systemd?

Of course you aren't precluded from using a traditional cron daemon but fundamentally what cron does is start one-shot services on a timer, there is nothing magic that needs to be added to systemd, it has all the functionality to start services, and it has timers, so it can fundamentally do anything cron can do. If someone added timers to sysvinit instead of creating a whole nother utility then inittab would have been crontab too.

Why wouldn't you allow units to be started with timers? It seems like such a straightforward logical feature I can't imaging what justification could exist for forbidding it.

This whole debate saddens me

anselm — Fri, 19 Dec 2014 10:08:20 +0000

Here we go again …

The “logging requirement” has already been explained (once more). Name resolution, cron, and network configuration are not “required to be part of systemd” – nothing prevents you (or a distribution provider) from using non-systemd components to implement that functionality. Stuff like dnsmasq, Vixie cron, or NetworkManager works fine with systemd.

What systemd is intended do is to provide reasonable standardised baseline implementations for the functionality in question. The idea is that what comes with systemd should be adequate to cover many use cases to a point where the previous hodgepodge of tools is no longer required. The advantage of this is that with systemd there is a unified set of components with well-defined interfaces, a single mechanism to start service processes, a single configuration file format, convenient methods to change and adapt the configuration, and good documentation, none of which are provided by the traditional setup (or for that matter competing init systems). People who need to deal with more complex use cases are still free to slot in more complex components that do what they require.

Having said that, there are reasonable criticisms that can be made regarding some of the baseline components in systemd. It is however well to remember that systemd is still being developed further. Systemd's basic plumbing components should occupy a “sweet spot” that balances complexity and required functionality for a significant subset of use cases, and exactly where that sweet spot is located is presumably subject to well-argued debate (as opposed to repetitive uninformed argument based on gut feelings).

This whole debate saddens me

jgg — Fri, 19 Dec 2014 01:26:17 +0000

So, I can comment on the logging aspect (not sure about the others, they don't seem as integral to systemd anyhow)

As soon as you say you want to have good logging for your daemons, that is to say their stdout/err and syslog are all captured to the system log and not to some arbitrary tty, you need logging integrated, in some significant way, with PID 1.

Specifically, the very first thing PID 1 must do is start another process to be the other end of all those logging FDs init will create when it starts spawning daemons. During spawn init arranges to connect stdout/err to a socket where the read end resides in the logger process.

Of course, this now creates state in the logging process, it has all these FDs that cannot be recreated, so it can never die, and is essentially tightly coupled with init. You certainly don't want it to be something complex and network facing like rsyslog or otherwise.

Also since this PID 2 must be started before anything beyond tmpfs can be mounted it can't reasonably be any sort of already existing syslog daemon, none are suitable to run in that sort of environment.

So now we have a consolidation and storage requirement for our PID 2, which gets us pretty close to where systemd is. There are various small trade offs here and there, but the integration of the two is not one of them.

Give up the special logging arrangement and you have to give up universal stdout/stderr capture - which is a huge long standing defect in sysvinit.

As an aside, the scheme I talked about above actually does work like this, the log collector PID 2 is fairly simple and only writes logs to structured binary files. Other tasks like syslog delivery to a remote host are done by 'tail -f' on the binary log files and never interact with the logger process. Why binary? To do the async tail -f for syslog purposes we need to have a robust cursor mechanism to iterate asynchronously over the log data, the binary format is essentially the length, line number and then data. Having a unique line number allows us to construct a reliable cursor.

The "Devuan" Debian fork

mathstuf — Mon, 15 Dec 2014 20:28:35 +0000

FWIW, it also seems that ALSA isn't really the epitome[1] of design[2] either.

[1]https://github.com/mpv-player/mpv/commit/ae5fd4a809bcba8e...
[2]https://github.com/mpv-player/mpv/commit/49df01323e59526d...

The "Devuan" Debian fork

mathstuf — Mon, 15 Dec 2014 01:03:08 +0000

> Pulseaudio's injection between ALSA and userspace, was an unmitigated disaster

Having used Fedora and not had many issues personally (across at least 8 machines of varying provenance), the *vast* majority of issues I've heard about are from Debian-land (particularly Ubuntu) which always made me assume improper packaging and deployment. I've found that disabling the flat-volume option makes things much better. I enjoy being able to mute my music streaming to play a video or something instead of having to kill it to get any audio from anything else.

> Pöttering

Not his name; it's Poettering.

This whole debate saddens me

deepfire — Sun, 14 Dec 2014 21:05:36 +0000

It should be a FAQ entry somewhere. You've written it up quite well..

This whole debate saddens me

anselm — Sun, 14 Dec 2014 20:15:45 +0000

I don't know whether explaining this to you once more will make any difference or if you're deliberately trying to be obtuse, but here goes:

systemd as a whole consists of a number of modules that communicate via documented external interfaces. There is a stability promise that covers most of these interfaces. It is possible to replace most of those modules by independent implementations that are otherwise unrelated to the systemd code base as long as one sticks to the documented interfaces.

In the interest of saving work, various modules avail themselves of common subroutines in a separate directory in the source tree. Since all of systemd is released together and nothing outside of systemd uses these subroutines, there is little point in making these into a shared library (especially since then invariably somebody outside the project would want to to use that, and that would impact the systemd developers' freedom to change things around as needed to suit the implementation), and systemd uses linker features to ensure that every systemd modules contains only the code it actually uses.

Therefore systemd's "library API" is an internal implementation detail similar to the internal APIs of the Linux kernel. It only exists to make the systemd developers' life easier. Whether systemd consists of separate modules (it does) is of no consequence to this, because it is still possible to reimplement many of these separate systemd modules independently based on the documented external interfaces and the API stability promise.

The "Devuan" Debian fork

nix — Sun, 14 Dec 2014 17:27:39 +0000

Since then it's been worming its way in piece by piece through dependencies in other programs

Most of those other programs call a couple of functions to set up socket activation. That's all. These functions are stable and their interfaces do not change. This is not a high bar for reimplementation (indeed, they have been reimplemented). As signs of impending world domination by our lizard overlords go, this is not a particularly impressive one.

This whole debate saddens me

nix — Sun, 14 Dec 2014 17:24:51 +0000

It's not shoddy design unless you then turn around and say that things that use said functions in those compilation units are not really part of that software at all, they're just bundled into the same source tree so they can use that library interface! That position implies very strongly that that interface is actually an external interface, since things can call it that are not 'really' part of the project.

As far as can see, there are two possibilities here:

- Either those interfaces are external interfaces, and they should be a stable shared library API, and the systemd developers are doing shoddy work by not making them stable, or

- those interfaces are internal interfaces, subject to change as and when needed, and those things in the systemd source tree that use them are *part of systemd* and claims that oh no they just share a source tree and are not really part of it at all are, not to put too fine a point on it, lies.

I don't see a third option, and neither of these are particularly pleasant.

This whole debate saddens me

nix — Sun, 14 Dec 2014 17:21:55 +0000

Projects at the same layer: glibc. libgcc. libstdc++. Projects at a different layer: kdelibs (yes, really, right down to internal functions). libX11.

There is a certain degree of conservatism that should come with writing software at the kernel/userspace boundary, or at the 'everyone has lots of dependencies on me' level. glibc has it. systemd (like libpng) doesn't.

Devuan as Lefthanded Debian, a different view of the future.

anselm — Sun, 14 Dec 2014 10:31:13 +0000

Two points:

We will have to wait and see whether the Devuan project will ever release anything that involves actual work (a rebranded Debian jessie probably doesn't yet count).
There is nothing that would prevent anyone from developing a systemd alternative — or for that matter non-mainstream alternatives to individual systemd components — based on stock Debian.

The "Devuan" Debian fork

ksandstr — Sun, 14 Dec 2014 08:55:21 +0000

Thank you for posting this comment. It describes the case exactly. I'd like to add a piece to the puzzle so far.

For some the reason for separating from Debian is that the previous iteration of Lennartware, i.e. Pulseaudio's injection between ALSA and userspace, was an unmitigated disaster that (in gentle terms) brought no increase in performance or functionality to systems that were well-configured with ALSA. Instead it changed e.g. the meaning of the mixer volume level as applied by most applications, breaking UI behind the application's back.

I waited out the pulseaudio storm to no decrease in functionality. The way I achieved this is with about five stanzas in an apt-prefs file that's still necessary to keep pulseaudio the hell out. This is because even years after Pöttering discontinued his involvement, PA continues to hang on to the model of being the only network-capable sound service in town. (In the case of PA's design it makes the other servers its clients, which brings out various implementation discrepancies in the way PA reimplemented the ALSA APIs.)

Then along came that same guy's "init system, device manager, login manager, cron, DNS resolver, console terminal emulator, package manager, kitchen sink, slopbucket, API breaker" project's adoption as dependency in the Debian Gnome packages. Since then it's been worming its way in piece by piece through dependencies in other programs; most recently bsdutils pulls in libsystemd0 which combines the freshly de-modularized group of libsystemd-{journal,daemon,login,...} program-level APIs. (who knows, maybe it looks smaller that way.)

Opinions of course vary and the technical minutes are too involved to quote here, but in my view there is no problem that systemd solves that isn't one of those introduced by the systemd design itself. Out of the possible reasons to have such an architecture in GNU/Linux, systemd doesn't meet utility, necessity, or even practical desirability.

The inclusion of systemd's libraries (and PAM modules, and root-privileged daemons, and...) isn't the problem by itself, of course: a library that implements its APIs in terms of local services that don't exist can't cause much trouble (unless root). Rather these libraries are an indication, a first warning of some program being about to become feature-dependent on an API from an architecture controlled by people who're on public record wrt their intent to e.g. break interfaces and make lock-step upgrades the norm. Having something like systemd-shim replacing Pöttering's systemd doesn't help here: for a well-defined architecture it matters relatively little who completed the implementation (or even how).

There are many people who'd like to sit out systemd as could be done with Pulseaudio. Personally I'd prefer to accomplish this by some means that isn't eternal vigilance towards my preferred distribution's packagers and politicians. Such things weren't needed in the time when Debian was a distribution of GNU/Linux first and one of Gnome only a distant second alongside every other userspace program.

Pinning against the entirety of systemd and packages that depend on it would result in many programs staying behind from a time between Wheezy and Jessie while others moved ahead. This tends to expose unstated dependencies between userspace programs which hadn't been reported by the legion of people upgrading regularly from `testing'. The mutually exclusive options for systemd doubters are therefore 0) adopt systemd and damn the icebergs, 1) pin against systemd and become a dinosaur, and 2) pool efforts, fork Debian, and move on.

Hence Devuan. (which, for the record, I'm not actively involved in.)

If things don't work out for it, there'll be others. Haters won't slow this puppy down.

Devuan as Lefthanded Debian, a different view of the future.

BoSox — Sat, 13 Dec 2014 22:22:24 +0000

I think the "Devuan" fork has been misread. Certain dissatisfied sysadmins and developers are creating a traditional init Debian fork. Keeping the older strain of Debian alive is a good thing. Debian itself has chosen a system-d based path, like many other distros. Someone keeping the older gene pool alive is very good. It is not however a mainstream thing.

A .deb based distro which is easier to customize or shrink will be very useful. This left handed Debian will be where alternatives to systemd can be developed. It is where non-mainstream options to the suite of tools in the systemd universe can be grown. It is a new reference implementation of debian. It is unfortunate that so much friction has come from what may be a welcome offshoot of Debian.

Devuan's porters and supporters will be doing a lot of work. They are creating another option for those who choose it in the future. People who support their vision of things will package or repackage things their way. Nothing they do will reduce our options. They may provide a fallback position for the rest of us. Linux itself provides a fallback position from many ex XP users. Options are what linux is about.

This whole debate saddens me

cesarb — Sat, 13 Dec 2014 12:39:59 +0000

> > But that's your opinion. It's completely unreasonable to expect any software project to go to the trouble of maintaining bug-compatible stability of internal interfaces that are explicitly labeled as not for public use.
> >
> > Show me a free software project that has such a ridiculous policy.

> Does Wine count?

Wine's main internal, not-for-public-use interface is the server protocol, which is the RPC between the lower-level Wine DLLs and the shared wineserver process.

They don't maintain bug-compatible stability of that interface. Every time they feel the need, they add, remove, or change its methods and structures. The only concession to compatibility is a single version number, which is increased ever time they make a change, and used to detect version mismatches between the server and the DLL. That version number is already over 450, meaning there were over four hundred times that interface was changed in an incompatible manner.

So no, Wine doesn't count.

This whole debate saddens me

rahulsundaram — Sat, 13 Dec 2014 03:30:50 +0000

Not really. They had no choice given the purpose of their project.

This whole debate saddens me

flussence — Sat, 13 Dec 2014 02:49:12 +0000

Does Wine count?

The "Devuan" Debian fork

rodgerd — Sat, 13 Dec 2014 01:20:11 +0000

> It was *not* dropped because of inadequate maintenance, by the way.

Probably a bad choice of word - the email I saw said it was because they were down to one maintainer (an inadequate *number* of mainters, not that the maintainer is inadequate).

This whole debate saddens me

njs — Sat, 13 Dec 2014 00:29:56 +0000

I don't see the comparison to the cut-and-paste code or libpng/zlib issues. The problems there are that you end up with multiple copies of the same source code, which then diverge.

In the systemd case, there's exactly one copy of the source code, and it always gets updated in sync with its reverse-dependencies. You do end up with multiple copies of the *compiled* code, but I don't see how this can cause problems beyond a miniscule waste of space. It's not really any different from how macros or inline functions or C++ templates work. Plus it almost certainly increases reliability to avoid adding another .so dependency to your early boot stack, esp. once you take into the account the possibility of version skew etc.

The "Devuan" Debian fork

Zack — Fri, 12 Dec 2014 20:36:01 +0000

I think rodgerd was referring to GNU/kFreeBSD being dropped as a release architecture.

It was *not* dropped because of inadequate maintenance, by the way. With zero RC bugs and 90% of the archive built, it was in ship shape. The main release concern for the maintainers was in fact, that it was probably too late to ship with a FreeBSD 10 kernel.

Why it was dropped isn't 100% clear (https://release.debian.org/jessie/arch_qualify.html doesn't seem up to date), but the only reason I can surmise is that Steven Chamberlain in his role as super-human maintainer of all things kFreeBSD is a single point of failure, and as such, as a release architecture it would be prone to the malevolent whims of renegade buses.

Last thing I heard it was still going to be released, on schedule, alongside Jessie, but probably with a FreeBSD 10 kernel, since the freeze doesn't affect the architecture anymore.

The "Devuan" Debian fork

andreasb — Fri, 12 Dec 2014 19:13:21 +0000

>> This is *Debian*. The distribution which has support for more than one non-Linux kernel.
>
> Not any more. It just dropped one of the two because of inadequate maintenance.

That doesn't make any sense as you wrote it. If you only count release architectures as being supported, then Debian never had support for more than one non-Linux kernel. Hurd has not made it that far.

This whole debate saddens me

mchapman — Thu, 11 Dec 2014 23:49:33 +0000

> I happen to think this is shoddy design

This the first time I've heard of "calling a function in a different compilation unit" shoddy design. If that's what you really think, then all software is shoddy.

This whole debate saddens me

peter-b — Thu, 11 Dec 2014 19:30:50 +0000

Well, you can think it's shoddy design all you like. But that's your opinion. It's completely unreasonable to expect any software project to go to the trouble of maintaining bug-compatible stability of internal interfaces that are explicitly labeled as not for public use.

Show me a free software project that has such a ridiculous policy.

This whole debate saddens me

nix — Thu, 11 Dec 2014 17:22:48 +0000

Quite. I happen to think this is shoddy design, unless (like, say, gnulib) there are very good reasons for it (in gnulib's case, that a lot of the horrifically evil dark magic it does happens before compile time, at the configure stage, and that it's only trying to fix up holes in already-public APIs anyway). It was a bad idea when GNOME did it with its 'cut-n-paste-code' directories, it was a bad idea when people embedded dozens of copies of libpng and zlib, and it's a bad idea now. Yes, doing it right is more work. Doing it right often is.

And no, I'm not going to go away and fork or reimplement systemd or shut up as peter-b so charmingly suggested. I don't have infinite time. I am, in fact, permitted to say 'this is bad design, other projects don't do this, it has been done this way before and every time turned out to be a bad idea' without simultaneously being obligated to fix every single instance of bad design I see. Nobody could ever have enough time to do that.

This is a classic systemd proponents' attack: if you criticise the holy project in any way you should shut up unless you have personally implemented something better. Sorry, it doesn't work that way, and repeatedly attempting to shut down any and all criticism this way does *not* make me feel better about the project as a whole.

This whole debate saddens me

mchapman — Thu, 11 Dec 2014 03:11:00 +0000

> but the systemd folks keep claiming that systemd isn't monolithic, that anyone is free to replace any component, they just need to implement the same functions.

Oh criminy. Let's take this from the top again.

You asked why the different systemd components are in the one tree. I said that one likely reason is so that they can share code. nix wondered why they didn't just use a shared library for that. I said that it might be because the shared code isn't part of the *public* interface of systemd, so there is no compelling reason to make it a dynamically-linkable library.

Got it yet? We're talking about code that is statically linked into multiple, separate, independently-reimplementable binaries. This has *absolutely nothing* to do with their public APIs (i.e. a set of D-Bus interfaces) provided by those binaries.

This whole debate saddens me

pizza — Thu, 11 Dec 2014 00:49:46 +0000

> but if the API of those functions is "internal" then it is monolithic in design (even if implemented in multiple binaries)

The API *between* components is a published, stable DBUS (and/or command-line) interface.

The implementation details, ie the C functions, are internal, private. You know, not unlike Linux's published userspace API, vs the internal implementation details that are changing all the time?

This whole debate saddens me

dlang — Thu, 11 Dec 2014 00:41:57 +0000

but the systemd folks keep claiming that systemd isn't monolithic, that anyone is free to replace any component, they just need to implement the same functions.

but if the API of those functions is "internal" then it is monolithic in design (even if implemented in multiple binaries)

This whole debate saddens me

mchapman — Thu, 11 Dec 2014 00:02:19 +0000

> To me that's a sign of developer laziness: they can't be bothered to design APIs right the first time, and they're making everyone else pay for that.

Did you completely miss the word "internal"?

These APIs are *not* public APIs. They are *not* used by other software. They are merely the internal function calls from one part of a binary to another. They aren't even visible to or accessible by end-users.

The "Devuan" Debian fork and the fuss about systemd

mathstuf — Wed, 10 Dec 2014 21:03:21 +0000

Seeing as he pretty much evaporated since the discussion about him offering nothing new to the threads he comments in, the indication that he's basically got no standing from his declarations that he never has nor will try systemd personally, and the lack of a denial here, I wonder if a "trap" condition got triggered in his source code.

This whole debate saddens me

peter-b — Wed, 10 Dec 2014 18:41:47 +0000

>> Of course it has. But, like the Linux kernel internal API, it's fluid and can change from one commit to the other.

> To me that's a sign of developer laziness: they can't be bothered to design APIs right the first time, and they're making everyone else pay for that. (It *can* be done, though it is a lot harder than just guessing and constantly chopping and changing them.)

Designing good, future-proof APIs is extremely difficult, as you acknowledge. Maintaining APIs in a bug-compatible manner is much more difficult still. It's totally reasonable for developers to define a small, stable, external API, while maintaining flexibility within their software to modify internal implementation details. It enables improvements to the efficiency, reliability, or features of the implementation. In fact, this approach is evidence of a robust development process and a sustainable project.

By contrast, it is completely *unreasonable* to expect people to only ever design, write and release perfect, flawless, future-proof code, and accuse people who have realistic expectations of what is achievable of "laziness". It is, in fact, arrogant and insulting.

What's going on here is that the systemd developers defined some public APIs that they wanted to provide, and went ahead and implemented a system that provided those APIs. You're now nitpicking their implementation. systemd's public APIs are fully documented, with stability guarantees, and that makes the project a heck of a lot more professionally developed than 99.9% of FOSS software projects.

I invite you to: 1. "improve" systemd to match your expectations (or pay someone else to), 2. fork systemd and "fix" it to your satisfaction, or 3. produce a plumbing suite that solves the same key problems in a "better" way. Until then, perhaps you should consider *not* wantonly slinging dismissive insults around. They're not going to get you whatever it is that you think that you want.

The "Devuan" Debian fork

nix — Wed, 10 Dec 2014 17:56:28 +0000

Exactly so. This is particularly true for performance regressions, which are both prone to be overlooked when they happen, and hard to verify except on bare metal.

This whole debate saddens me

nix — Wed, 10 Dec 2014 17:55:00 +0000

Of course it has. But, like the Linux kernel internal API, it's fluid and can change from one commit to the other.

To me that's a sign of developer laziness: they can't be bothered to design APIs right the first time, and they're making everyone else pay for that. (It *can* be done, though it is a lot harder than just guessing and constantly chopping and changing them.)

The "Devuan" Debian fork and the fuss about systemd

nix — Wed, 10 Dec 2014 17:19:34 +0000

No, this is Serdar Argic, except someone's changed the trigger string to 'systemd' and altered the templates it sprays out when triggered.

Gentoo Resisting?

johannbg — Tue, 09 Dec 2014 20:08:46 +0000

Shortcomings of systemd setup in initramfs is irrelevant to systemd itself ( other than it being used ) and bugs against upstream implementing this ( dracut, mkinitcpio etc ) should be filed so they can address the issue.

Gentoo Resisting?

cesarb — Tue, 09 Dec 2014 10:14:52 +0000

> What initramfs has to do with systemd?

At least on some distributions, the initrd nowadays runs systemd. Run "journalctl -b" and look for a "systemd[1]: Running in initial RAM disk" line; if you see it, your distribution is running systemd within the initrd. You should also see a "systemd[1]: Switching root" line just before the switch to the real root.

Gentoo Resisting?

Cyberax — Tue, 09 Dec 2014 08:54:35 +0000

What initramfs has to do with systemd?