LWN: Comments on "Another daemon for managing control groups"

Another daemon for managing control groups

nix — Mon, 27 Jan 2014 22:40:56 +0000

Someone needs to implement an init system as an IOCCC entry, clearly.

Another daemon for managing control groups

khim — Mon, 27 Jan 2014 20:45:46 +0000

This disregards the obvious fact that while SysV init is surely doing something, there is no way that one could claim it is doing it well.

It does pretty decent work on scaring newbies away from Linux. Perhaps that is the thing is was designed for? Of course it's kinda overenginered for such a use case, you can create a much smaller and simpler mess which will be much scarier.

Another daemon for managing control groups

anselm — Mon, 27 Jan 2014 19:38:30 +0000

The current init/service activation infrastructure (System V init, inetd, cron, …) does not have anything worth calling »a design«. It is cobbled together out of random pieces from different sources and the amazing thing is that it works at all. The pieces don't talk to one another and are all configured differently. If it had in fact been »designed« you would expect at least a minimum of consistency and cooperation between the pieces.

If somebody proposed this setup today as a new system they would be laughed out of the lecture theatre. The only reason this is still in actual use at all is that for the longest time nobody has dared come up with an alternative. Most of the other Unix-like systems have replaced it long ago.

People always cite »the Unix philosophy« as »do one thing and do it well«, and point to System V init as a prime example of this approach. This disregards the obvious fact that while SysV init is surely doing something, there is no way that one could claim it is doing it well.

Another daemon for managing control groups

mathstuf — Mon, 27 Jan 2014 15:07:28 +0000

If you're so lazy, why are you even trying to follow everyone else? Is sticking with Slackware not the laziest thing one can do these days anymore?</snark>

Maybe others aren't so lazy to not see the benefits it brings to the table and that's why it has been adopted? Making my user setup use unit files took maybe a day or two (and that includes a *lot* of unit files; at least 30) and I just got hung up on needing suid to start X. Systems does bring a lot to the table. Have you even tried it or did "service" not working prevent you from digging deeper?

Anyways, an anecdote from dealing with FreeBSD this weekend: I tried getting a taskd server running, but unfortunately I get no indication that it has failed to start, logs as to why it might have happened, or what the init system even attempted to do. And BSD init scripts are easier to deal with than sysvinit. I really would rather have systemd with all the tools it gives me to deal with such problems (so I know whether I should go poke upstream).

Another daemon for managing control groups

HelloWorld — Sun, 26 Jan 2014 01:20:20 +0000

> Yes, the design of sysVinit is a *bit* rough. But it works,
It *doesn't work*. Stop denying it, sysvinit can't even stop a service reliably and correctly.

> and you don't have to read thousands of lines of source to figure out what it's doing.
Yeah, you just have to read thousands of lines of shell. With systemd otoh, you read *documentation*. You know, this stuff that is actually meant to be read by humans. Because unlike the sysvinit quagmire, systemd actually *has* documentation that's worth mentioning.

> And if one part of it falls over, ps will tell you, and you can kill just that one part.
Yeah, like systemd somehow magicall broke ps.

> And you can grep the damn logs.
You can do that with journalctl too. But actually most people don't do it, because it *sucks* and just using the domain-specific features of journalctl is easier, faster and more robust.

Another daemon for managing control groups

Baylink — Sat, 25 Jan 2014 20:50:00 +0000

No, the modularity comes from *the pieces not all being one big piece with no visibility or lines of separation*.

Yes, the design of sysVinit is a *bit* rough. But it works, and it's easy to understand if you know basic Unix, and you don't have to read thousands of lines of source to figure out what it's doing.

And if one part of it falls over, ps will tell you, and you can kill just that one part.

And you can grep the damn logs.

Another daemon for managing control groups

Baylink — Sat, 25 Jan 2014 20:47:25 +0000

That's the part that I really don't understand.

I can get that Lennart doesn't understand the Unix Philosophy, or why it's more than "just a philosophy"; I can get that he doesn't care about why binary log files are bad, or why not being able to see *which* component of the core system went runaway on ps and kill just it is bad, or why being unable to replace those components with the specific ones you need for your job is bad. Or why optimizing for something that the vast majority of non-developer users of Linux don't care about -- booting at ludicrous speed -- is bad.

Why, in short: *requiring me to UTSL* just to do admin work, is bad.

What I *cannot* understand is how *entire committees* of people who drive the designs of distributions drank the Flavor Aid.

It is effectively impossible now to run sysVinit, because the people who package things for dnew Tweets istributions *no longer include, much less support* the components necessary to do that. And unless you're google-scale (in which case you're probably building your distro from scratch anyway, with a dedicated department of staffers), you can't fix that.

And that's not to mention the *literally decades* of sysadmin reflexes that have been flushed down the drain by this decision on the part of distro managers -- that's the part that really frosts my ass. I have better things to do than to relearn the basics all over again, guys, really.

I got into Unix because I am *lazy*. I want to learn small things once, and leverage them into big things, to make my life easier.

No matter how much better it might be at the actual work, as a sysadmin, systemd continues to make my life harder and more frustrating, at every single turn.

My philosophy of life is that our morale is proportional not to the size of our wins and losses, but to their *number*; that's why I use Linux.

systemd just made it go the other way.

Thanks, guys.

Another daemon for managing control groups

Cyberax — Fri, 17 Jan 2014 10:36:17 +0000

Yes, it's absolutely possible to boot a computer without any shell using systemd.

Another daemon for managing control groups

etienne — Fri, 17 Jan 2014 10:27:03 +0000

> ... let PID 1 exit to have a secure router ...

I have a new question to an old answer, to secure a system after it has been correctly configured:
Can you get rid of /bin/sh (i.e. any sort of shell: bash, ksh, zsh...) and still have a booting computer if you use systemd?
That would be very difficult to try to compromise a computer if the shell has been removed...

Another daemon for managing control groups

raven667 — Sat, 04 Jan 2014 17:56:30 +0000

In some ways this is like Plan9 again, a re-thinking of the UNIX concepts with an eye toward completeness and consistency with a lot of operational experience behind it that didn't exist when many of these systems were first developed.

Another daemon for managing control groups

anselm — Sat, 04 Jan 2014 14:50:44 +0000

the truth is that systemd just isn't good enough to be worth the price of loss of modularity and replacability (nothing could be good enough to be worth that price)

Any »modularity« in System-V init is there purely by accident. Actual »modularity« would imply a certain amount of unity of design and basic concepts, and communication between the components by means of well-defined and documented interfaces. None of these apply to System-V init and its associated tools such as inetd, cron, etc.

It's not as if System-V init had been consciously designed the way it is. It is a hodgepodge of stuff from different sources that were developed separately from each other, don't interact well (if at all) and are each configured completely differently from the rest. If there is a long-overdue opportunity to replace this utter mess by something that has actually been properly thought out as a complete system, works a lot better in real life, and is easier to understand, develop for, maintain, teach, and learn, then I'm all for it.

Another daemon for managing control groups

neilbrown — Sat, 04 Jan 2014 03:19:26 +0000

> I really don't care who he is. what people say and do is more important than what their name is.

Well said!

As for the rest of your post, I guess we'll have to agree to disagree.

(and no, I didn't mean to exaggerate. I love Unix tools and use them all the time, but I also believe in choosing the right tool for the job)

Another daemon for managing control groups

raven667 — Sat, 04 Jan 2014 02:27:48 +0000

Well this helps standardize service startup and login session management, the CLI and GUI shell userspace is not affected in any way

Another daemon for managing control groups

Cyberax — Sat, 04 Jan 2014 02:12:30 +0000

>anyone who says that sort, grep, tr, comm etc 'don't combine together well' and '"do one thing and do it well" is good for prototypes, not for final products' does not understand unix. read his post - the entire thing was "waah! unix toosl are too hard and complicated, systemd saves us from that!".

Frankly, I'm sick of total ignoramuses who repeat this very argument.

Get the freaking systemd and see what it can do. Now try to replicate it with "do no thing well" tools like SysV Init.

Another daemon for managing control groups

dlang — Fri, 03 Jan 2014 23:47:39 +0000

ahh, but they aren't trying to just create a new init system. They are defining a new standard linux userspace. That is enough to justify anything.

I just wish they would do this on their own distro.

Another daemon for managing control groups

cas — Fri, 03 Jan 2014 23:02:36 +0000

I really don't care who he is. what people say and do is more important than what their name is.

anyone who says that sort, grep, tr, comm etc 'don't combine together well' and '"do one thing and do it well" is good for prototypes, not for final products' does not understand unix. read his post - the entire thing was "waah! unix toosl are too hard and complicated, systemd saves us from that!".

perhaps he was just indulging in some exaggeration - it's a popular systemd pusher's tactic to exaggerate the alleged 'difficulty' or 'inadeqacy'of all previous/competing methods so that systemd's meager benefits seem greater by comparison. in this particular case, he has to grossly exaggerate the difficulty and inadequacy and just plain awfulness of the small-tools approach.

the truth is that systemd just isn't good enough to be worth the price of loss of modularity and replacability (nothing could be good enough to be worth that price), especially when the minor benefits that systemd does provide could easily be provided without a monolithic mono-culture monstrosity absorbing every low-level userland function in sight. init systems really don't have to be that complicated, they don't have to absorb syslog and cron and automount and everything else.

Another daemon for managing control groups

raven667 — Fri, 03 Jan 2014 22:06:21 +0000

> journald works pretty well for a single machine personal system, but it was built in ignorance of how logging works in larger environments.

Maybe you are right but it seems that when the systemd developers do something they do a ton of research before hand before committing resources. In any event it is pretty plain and stated that the journal isn't even trying to deal with networked logging or larger environments, it's main use case is the single personal machine, and capturing logs from early-boot which are normally lost. You don't judge a fish by how well it can climb trees.

Another daemon for managing control groups

raven667 — Fri, 03 Jan 2014 22:02:19 +0000

But they haven't removed rsyslog, and in fact rsyslog is required if you want to do central logging using syslog, systemd and the journal will pass through log messages to it. What they aren't doing is installing both logging systems by default. There is also no default sendmail in Fedora either.

Other distros make different decisions but rsyslog integration is still an important part of the journal, heck its even an important part of rsyslog which added support for the journal on-disk data format as I recall.

Another daemon for managing control groups

mchapman — Fri, 03 Jan 2014 21:59:00 +0000

> they can campaign to have their system added, but what's the justification for removing the other option, especially when it can co-exist with theirs and provide capabilities that they don't offer?

For Rsyslog in Fedora specifically?

According to the Change page [1]:

1. Less disk and runtime footprint
2. Fewer packages in the default install
3. Faster booting

Now, you can argue whether those are *sufficient* justification for the removal of syslog from the default install. I'm sure people did.

At the end of the day, though, the vote was in that this Change should go ahead.

[1] https://fedoraproject.org/wiki/Changes/NoDefaultSyslog

Another daemon for managing control groups

mchapman — Fri, 03 Jan 2014 21:54:43 +0000

> This tells me that the people working on this are not actually using these tools enough.

That is as ludicrous as saying any long-standing bug in the kernel shows that the kernel developers don't use Linux enough.

Bugs happen. People fix them (or at least, *should* fix them) when they discover them, not before.

Another daemon for managing control groups

dlang — Fri, 03 Jan 2014 21:54:32 +0000

they can campaign to have their system added, but what's the justification for removing the other option, especially when it can co-exist with theirs and provide capabilities that they don't offer?

Another daemon for managing control groups

dlang — Fri, 03 Jan 2014 21:53:01 +0000

rsyslog can get the logs from journald very rapidly, pretty close to real-time.

the designers of journald didn't have any idea what rsyslog, syslog-ng and similar logging daemons were able to do at the time they started journald, their justification of journald is full of "syslog can't do this" arguments that were true of traditional syslog, but not true on any of the modern syslog daemons.

journald works pretty well for a single machine personal system, but it was built in ignorance of how logging works in larger environments.

Another daemon for managing control groups

dlang — Fri, 03 Jan 2014 21:50:06 +0000

I would believe this more if there hadn't been a bug that prevented you from accessing the journald binary logs that went undetected for several months until it made it into a Fedora release that included rsyslog reading the files and going into a endless loop doing so (the journald tools also went into an endless loop, so you can't just call this a rsyslog bug)

This tells me that the people working on this are not actually using these tools enough.

Another daemon for managing control groups

mchapman — Fri, 03 Jan 2014 21:48:34 +0000

> they also campaign to get the competition removed from the distro (or at least the default installation of the distro)

Is that so surprising? Clearly they think they have a better solution. They *should* be campaigning for it, if that's what they really think.

I'm sure distro maintainers are smart enough to judge the various options available on their merits.

Another daemon for managing control groups

dlang — Fri, 03 Jan 2014 21:46:42 +0000

> Really, how is this different from today where if you want to use this kernel feature you have to have your software talk to the kernel and twiddle it. How is it different than iproute2 or iptables?

iproute2 and iptables are happy to sit on a system and only be used for the parts that you want them to be used for. They don't try really hard to force you to do things a specific way and push you to replace a bunch of other tools if you use them.

Another daemon for managing control groups

dlang — Fri, 03 Jan 2014 21:44:40 +0000

> The only way they are "killing the competition" is by making a high-quality, feature-rich offering which people want to use, they are out-competing the competition.

not quite, they also campaign to get the competition removed from the distro (or at least the default installation of the distro), rsyslog vs journald in Fedora is an example of this in practice.

Another daemon for managing control groups

raven667 — Fri, 03 Jan 2014 20:50:16 +0000

This is a reasonable place to do a risk analysis, how difficult is it to run the tool that can handle the file format of journald on a system that may not be working 100%, how often are the local logs not going to be sent via syslog to a central location? I don't think the designers of this would have done it this way unless they were convinced this was still a win but it's a discussion that can be had.

Another daemon for managing control groups

rodgerd — Fri, 03 Jan 2014 20:43:28 +0000

There's a reasonable objection to be made about binary logging formats - system recovery and general analysis can be vastly more painful with such. It's a toss-up, because in many day-to-day cases journald's filtering is a win. But it's hardly an invalid complaint, unless you're considering "how we use logs to do our job" to be a non-technical complain, which would seem to be stretching.

Another daemon for managing control groups

rodgerd — Fri, 03 Jan 2014 20:41:03 +0000

I think before you go off on another sweary rant about the quality of discussion you might want to consider raising your own.

Another daemon for managing control groups

rodgerd — Fri, 03 Jan 2014 20:31:21 +0000

There is a lot more "vendor lock-in" in a situation where every Linux vendor implements their userland in a special snowflake way such that it requires significant effort to move from one distro to another.

systemd reduces vendor lock-in.

Another daemon for managing control groups

Cyberax — Fri, 03 Jan 2014 20:26:57 +0000

> only for the short term until they figure out how to allow for wider access.
I haven't seen any indications of that.

And that's actually my sticking point with cgroups - I think that cgroups nesting and delegation is a must.

>(and even then the old way will remain for compatibility for a while)
Yeah, sure. Without any new features.

Another daemon for managing control groups

raven667 — Fri, 03 Jan 2014 20:03:23 +0000

It's a tautology really, if you don't want to run systemd for whatever reason, then don't run systemd.

> ... features that systemd is trying to take over and kill off the competition in.

I'm not sure what "take over" means in this context if you aren't running systemd on a system because you don't want to than what systemd does is not of relevance to you except as an example. The only way they are "killing the competition" is by making a high-quality, feature-rich offering which people want to use, they are out-competing the competition. I hope we can all agree that they shouldn't hobble the implementation and make it less useful just to make the competition look better...

> how would you use cgroups on a wireless access point where you don't want to run systemd but still want to limit some set of processes?

You do whatever you want to do, within the confines of the kernel interface, which in the future will want a single-writer (although compatibility with the existing interface will be maintained for some time as I understand it). So you can make your own cgroup manager daemon, being advised that there are fundamental problems with restarts, race conditions, etc. and that this can only manage a subset of processes (ones that it starts), or you write your custom manager into pid 1 either from scratch or patching one of the existing inits to have the logic you want.

There are probably going to be a handful of small cgroup managers that you can start from which are dedicated to specific tasks like HPC or Google, although if systemd incorporates whatever configuration toggles these workloads need then many will probably just use that rather than re-implement something themselves, but there is always someone out there who wants to do it themselves and is distrustful of any third party code 8-)

Really, how is this different from today where if you want to use this kernel feature you have to have your software talk to the kernel and twiddle it. How is it different than iproute2 or iptables?

Another daemon for managing control groups

dlang — Fri, 03 Jan 2014 19:15:50 +0000

even the systemd developers admit that there are linux systems out there that should not run systemd (embedded type systems for example), but those systems may still want to use some of the features that systemd is trying to take over and kill off the competition in.

how would you use cgroups on a wireless access point where you don't want to run systemd but still want to limit some set of processes?

Another daemon for managing control groups

dlang — Fri, 03 Jan 2014 19:13:00 +0000

> In the new world order only ONE process will be able to create/modify cgroups. That's going to be a kernel requirement.

only for the short term until they figure out how to allow for wider access. (and even then the old way will remain for compatibility for a while)

It also would't the only time some kernel developers have said that there was going to be a kernel requirement that they later back away from before enforcing it.

so it's very possible that there will never be a kernel release that will require only having one process manage cgroups

Another daemon for managing control groups

raven667 — Fri, 03 Jan 2014 18:57:53 +0000

> The refusal of systemd developers to cooperate in a flexible solution (combined with the marketing drive to make systemdbe default in major distros) is the very thing that is poisonous to alternative cgroup management implementations.

No, what is poisonous to other competing implementations is that systemd is technically advanced and comprehensive so that there is little need for other re-implementations because it works so well. That is the reason it has gained so much uptake, do you really think that all the people who build distributions are technically incompetent and are switching to systemd components because of some "marketing drive" or because these components solve the problems they are trying to solve better than the systems which came before?

It's true that the systemd developers refuse to re-design their system to have cgroup management handled by an external process, the "flexability" you talk about, but there are very rational technical reason for that which you have not addressed. Do you understand the kernel cgroups api and your workload and the systemd configuration knobs enough to be able to tell us where the systemd cgroup managment fails in your workload that another cgroup manager would be able to do better that couldn't be implemented in systemd? Do you have any examples or is this an entirely philosophical conversation with no real backing in reality?

Another daemon for managing control groups

Cyberax — Fri, 03 Jan 2014 18:44:48 +0000

You don't get it.

In the new world order only ONE process will be able to create/modify cgroups. That's going to be a kernel requirement.

This in turn requires cgroups management daemon to be constantly active.

Another daemon for managing control groups

aigarius — Fri, 03 Jan 2014 18:23:49 +0000

Those are two competing points right there - currently systemd developers want to manage cgroups inside systemd and do not want to support delegating this management to any other outside daemon or library. For any other implementation of cgroups management to be able to work on a systemd system it would need systemd cooperation as above.

Therefore there are two options - either systemd must be relegated to be a tiny minority init version that can easily be replaced with something else on any distro, so that you can actually use a different cgroups manager or any other cgroups managers will be unusable without a very major hassle on many popular distros.

The refusal of systemd developers to cooperate in a flexible solution (combined with the marketing drive to make systemdbe default in major distros) is the very thing that is poisonous to alternative cgroup management implementations.

Another daemon for managing control groups

Cyberax — Fri, 03 Jan 2014 18:08:58 +0000

>Right, and car manufacturers in this analogy are distributions and system administrators of large installations. Systemd is not flexible enough as designed. I want to replace the cron-equivalent with another tool, I want to use a different logind, I want to use a different cgroups manager.

Nothing stops you from running your own cron implementation, syslog implementation or eschewing logind entirely. Systemd simply provides a lot of very convenient utilities but they are not mandatory.

The only problem is in the boneheaded single-writer cgroups hierarchy. But that's not entirely a systemd's failure.

Another daemon for managing control groups

aigarius — Fri, 03 Jan 2014 18:08:07 +0000

None of these require these things to be bundled together within a single binary or project.
Init systems have communicated with udev for startup for years before systemd came along. It is trivial and elegant - you start the udev and then start a service to mount/fsck all the filesystems. This service might do the work itself or just ask udev if its done the work - none of that matters to the init. Init only needs to know when the filesystems are ready for the start of launching of other services, which is communicated by the script exiting with exit code 0.
Init system does not have to include a cgroup manager to start services in their appropriate cgroups, you can trivially provide an equivalent of cgroups-launch wrapper that would communicate with a cgroups daemon and ensure that cgroup is switched before launching the secondary executable. Init system might provide a handy shorthand, for example that if a CGRoup option is specified for a service, then it will be launched via a cgroups-launch wrapper with the CGroup contents passed on to that launcher as a parameter. Trivial, decoupled, just as functional.
And cgroups daemon can trivially provide hooks for the udev to call when devices are added or removed in the usual way. There is no need to invent some tight coupling.

It seems that systemd developers not only "look at all that was before and found it lacking", but also at that point stopped looking and thinking outside the box that they put themselves into. Refusing cooperation with other projects and instead trying embrace-and-extend them is a huge red flag in free software communities. That never ends well.

Another daemon for managing control groups

andresfreund — Fri, 03 Jan 2014 17:43:37 +0000

Uh. And what exactly stops you from using vixie cron or your own logind implementation?

The case with cgroups is a bit different, but that's not really systemd's fault - the kernel/Tejun wants to move towards a single hierarchy, not systemd. Since systemd relies on controlling the cgroup hierarchy it cannot easily use a separate cgroup controlling daemon without having pid 1 rely on other processes without similar protections as pid 1 has from the kernel.

If there's things to critize about system's cgroup handling it's that they've built an API to control cgroups in a relatively short time which they are bound to continue to provide. Since it has to be generic enough to a) be implementable by others b) not block any future cgroup usage that makes me a bit wary.