Another daemon for managing control groups [LWN.net]

By Jake Edge
December 5, 2013

Control groups (cgroups) in the kernel are changing, in both their implementation and their interface. One of those changes is that systems that use the new cgroup interface will require a single management process to coordinate access. For many, that management daemon may well be the systemd init replacement, but there are distributions (notably Ubuntu) and users who will want a different choice. To that end, Serge E. Hallyn is working on an alternative cgroup management daemon that he calls "cgmanager".

Cgroups are a way to group processes and to apply various resource limits to the group. The current cgroup API in the kernel is a filesystem-based interface called cgroupfs. Groups are represented as directories and processes are placed into them by writing their process ID to a special file. Additional special files are used to set various limits and to control other aspects of the group, which depend on the specific controllers that are associated with the group (i.e. CPU, memory, block I/O, etc.).

Depending on permissions, any process may be able use that interface to set up its own hierarchy of groups, which is one of the problems with the existing implementation, according to cgroup co-maintainer Tejun Heo and others. So, in the future, there will be a single process that is responsible for managing the single hierarchy of groups that will be allowed under the new cgroup interface. While systemd already has code to perform that job, systemd has never been a requirement to use cgroups—but if it isn't used, something has to take its place.

So Hallyn put out a preliminary design for cgmanager on the lxc-devel mailing list at the end of November. It envisions a single daemon process with a D-Bus interface that will manage the hierarchy, which will mount cgroupfs inside a private namespace that is accessible only by cgmanager. Processes will make requests of the daemon to create cgroups, configure them, and to move processes into them. Obviously, some of those operations are privileged, and Hallyn has worked out a set of privileges required for each. Users own the cgroups they create and can place their own processes into those cgroups. More complicated arrangements are possible, of course, but generally either require real root privileges or at least root inside a user namespace owned by the user.

There is also the concept of handing the ownership of a cgroup (or hierarchy) to another user. Doing that using the standard filesystem permissions on the cgroupfs, as is done today, is a big security hole, which can lead to various kinds of denial of service. Mediating that access through cgmanager should alleviate that problem, however.

So far, Hallyn's design has been well-received; there are questions and suggestions, of course, but overall it would seem that Hallyn has taken several constituencies into account.

The first of those is the LXC containers project, but there are several other interested parties as well. Upstart-using distributions (mostly limited to Ubuntu and derivatives—though possibly Debian too, depending) will also need some kind of cgroup manager if they intend to use cgroups. Hallyn has clearly been talking to the Upstart folks, so it would seem that cgmanager should eventually fit the bill there.

Another constituent is Google, which uses cgroups extensively in its fairly sizable datacenter. Tim Hockin from the search giant had been rather critical of the single hierarchy cgroup plan. He has since largely made his peace with the plan, but an article from July outlines Google's use of cgroups as well as some of the problems with the "new cgroups" that Hockin had foreseen. Clearly Hallyn was also paying attention to Google's needs as Hockin seems relatively pleased with the direction of cgmanager's design. In particular, Google uses the "change ownership" feature today, but it trusts all of the users on its cluster. For others, who may not be able to have that trust level, Hallyn has provided a means to get that functionality, while reducing its security impact.

Where does systemd fit into this picture? The answer to that seems to essentially be: not at all. Systemd has its own control group interface that it intends to proceed with. Two messages from June make it pretty clear that Lennart Poettering, at least, is not particularly interested in having systemd work with another cgroup manager or to provide cgroup management in a library. Poettering's contention that init should not be dependent on some other daemon seems sensible, but forcing anyone who wants to use cgroups to use systemd clearly isn't.

It would be nice if some kind of common API could be worked out so that applications and users don't have to support two different ways of accessing cgroups. It would seem that Hockin approached the systemd folks about working together but was evidently rebuffed. That may make sense for the systemd project, but will unfortunately leave applications and users who need cgroup functionality in something of a bind. Supporting both systemd and cgmanager will likely be required.

While it will undoubtedly be annoying to support two (or more if another competitor shows up) interfaces to cgroups, having two separate implementations may actually be a good thing—at least in the short term. That wouldn't preclude a common interface, of course, but that doesn't seem to be in the cards. Cgroups have been a problematic feature in the kernel since they were merged for 2.6.24 back in 2007. One hopes we are on our way toward a better implementation and user-space interface for the feature—so the more code that exercises all of that, the better.

Index entries for this article
Kernel	Control groups

to post comments

Another daemon for managing control groups

Posted Dec 5, 2013 1:55 UTC (Thu) by Baylink (guest, #755) [Link] (60 responses)

Am I really the *only* person who hasn't drunk the koolaid?

Does not lumping *everything your Linux system does* inside the opaque ball of systemd violate nearly 35 years of Unix Philosophy pretty violently?

Do one thing. Do it well.

WWBKKRS&SD?

Another daemon for managing control groups

Posted Dec 5, 2013 3:46 UTC (Thu) by neilbrown (subscriber, #359) [Link] (42 responses)

Next you'll be telling us that we should re-write Linux as a micro-kernel with a herd of daemons, 'cause as it is Linux does *lots* of things.

You might have noticed that Unix has faded. Maybe its philosophy is wasn't sufficient.

The Linux philosophy (i.e. what Linus regularly rants of favour of) is pragmatism: Do what works.
Plus competitive evolution: Do several conflicting and incompatible things and may the best code win.

"Do one thing and do it well" certainly has its place, but that place isn't everywhere.

Another daemon for managing control groups

Posted Dec 5, 2013 5:02 UTC (Thu) by luto (subscriber, #39314) [Link] (25 responses)

Except that you won't even be able to try the competing solution on a Red Hat, Fedora, or other systemd-using distribution.

At the risk of harping on something I've said before: there needs to be a way to tell systemd to stay the $*!& away from the cgroup hierarchy.

If this can't be done today, let's fix it.

Another daemon for managing control groups

Posted Dec 5, 2013 8:07 UTC (Thu) by ovitters (guest, #27950) [Link]

Seems like you like to tell other people what to do. Maybe try convincing them. Because with telling, you're very likely going to be ignored. And no, you cannot tell people what to do and expect them to actually listen to you. "Fixing that" is not possible, nor wanted.

Another daemon for managing control groups

Posted Dec 5, 2013 13:29 UTC (Thu) by mezcalero (subscriber, #45103) [Link] (23 responses)

You can also turn this around: You won't be able to try the systemd solution on systems using that new Ubuntu cgroup manager either. There can only be one cgroup manager, and you have to choose between systemd or the Ubuntu one. You cannot use both. And that's not because systemd was evil, that's because the kernel iface is the way it is and the only secure way to allow multiple managers live side by side in userspace fiddling around in the same tree...

Another daemon for managing control groups

Posted Dec 7, 2013 0:43 UTC (Sat) by isilmendil (guest, #80522) [Link] (22 responses)

Actually, turning this around is a good excercise:
You can't use the systemd solution on a Ubuntu system obviously (since systemd would also conflict with upstart and maybe udev), but you can swap the cgroup management for a (not currently existing) different solution, and you won't be forced to remove upstart or any other program.

I'm not saying that systemd is evil -- it just forces an all or nothing approach. Essentially it's a new Linux stack: additionally to GNU/Linux and Android/Linux we now have GNU/systemd/Linux...

Another daemon for managing control groups

Posted Dec 8, 2013 3:55 UTC (Sun) by cas (guest, #52554) [Link] (21 responses)

this "new stack", GNU/systemd/Linux is not like unix, and not like linux - it is the complete opposite of the traditional small-tools, do one job and only one job but do it extremely well approach. it's a jack-of-all-trades master-of-none that tries to do too much, tries to do everything, in fact.

call it what it is, Lennartix, or Lennix. It's certainly not Linux. or Unix.

Another daemon for managing control groups

Posted Dec 8, 2013 5:19 UTC (Sun) by neilbrown (subscriber, #359) [Link] (19 responses)

"Linux" is just a kernel, so stuff in user-space certainly isn't "Linux".
It also certainly isn't "Unix" as that is fairly dead. It met needs in the 60s and 70s and even to some extent the 80s quite well. But the needs we have today are very different.
And it clearly isn't "GNU/Linux" as that was largely an attempt to re-implement UNIX is free software, and add more options. So it maybe carried us to the 90s.
I think calling it "Lennartix" gives way too much credit (or otherwise) to Lennart. He certainly instigated systemd and a few other bits, but the whole userspace that Fedora or openSUSE enjoy today is much more than any one person's innovation. Trying to name the whole after any one person is a mistake.

One of the big weaknesses of the "do one job and do it well" approach is that those individual tools didn't really combine very well. sort, join, cut, paste, cat, grep, comm etc make a nice set of tools for simple text-database work, but they all have slightly different ways of identifying and selecting fields and sort orders etc. You can sort-of stick them together with pipes and shell scripts, but it is rather messy and always error prone.

I remember being severly disillusioned by this in my early days. I read some article that explained how a "spell" program can be written to report the spelling errors in a file. It uses 'tr' to split into words, then "sort" and "uniq" to get a word list, then "comm" to find the differences. "cool" I thought. Then I looked at the actual "spell" program on my university's Unix installation. It used a special 'dcomm' (or something like that) which knew about "dictionary ordering" (Which ignores case - sometimes). Suddenly the whole illusion came shattering down. Lots of separate tools only do 90% of the work. To do really complete work, you need real purpose-built tools. "do one thing and do it well" is good for prototypes, not for final products.

One thing Unix never gave us was a clear big picture. It was always lots of bits that could mostly be stuck together to mostly work. I spent a good many years as a Unix sysadmin at a University and I got to see a lot of the rough edges and paper over some of them.

Systemd provides us with a clear big picture for one fairly important part of the sysadmin story: daemon process control. It may not be a perfect picture, but it is a whole lot better than no real picture at all. Back when I was being a sysadmin I would have loved something that would give me real control over all the various deamons: status, control, auto-restart, dependency etc. I even wrote something. It was nowhere near as ambitious as systemd but as far as it went, it was designed along very similar lines.

There certainly may be technical details about systemd that can be honestly debated, and there may be social/community issues that not everyone feels comfortable with (but then ambitious people often come across as intimidating), but to say that systemd is bad because it is not the "Unix way" is to simply have your head in the sand. The core idea behind systemd of unifying system service management into a single infrastructure is something that Unix has badly needed for years. It's about time we got it.
The thing that annoys me most about systemd is that I didn't write it first!

Another daemon for managing control groups

Posted Dec 8, 2013 5:59 UTC (Sun) by cas (guest, #52554) [Link] (18 responses)

> "Linux" is just a kernel, so stuff in user-space certainly isn't "Linux".
> It also certainly isn't "Unix" as that is fairly dead

wow, thanks for that startling revelation. Next you'll be telling me that water is wet, or that nobody ever actually drinks the kool-aid because that's a specific brand-name that's only available in certain countries(*)

Followed, no doubt, by the conclusion that nobody wants to (or should ever want to) combine water + flavouring agent because that's too messy and inconvenient and far too difficult so it's easier to just buy a can of some fizzy drink.

pure pedantic-literalist genius!

(*) US only? i dunno. i've never seen a kool-aid brand of anything, ever. i'm not even entirely sure of what it is - some kind of cordial or powdered flavour concentrate, i believe. but i know what someone means when they use a phrase like "drink the kool-aid". hint: it's not a literal statement about imbibing a flavoured liquid.

> One of the big weaknesses of the "do one job and do it well" approach
> is that those individual tools didn't really combine very well. sort,
> join, cut, paste, cat, grep, comm etc
> [ ... blah blah blah, i don't understand it, it's too hard ... ]

thank you for demonstrating that you don't understand unix.

that's the trouble with systemd advocates - they don't understand what it is that they're pushing to replace and, worse, universally think that their ignorance is a compelling argument for dismissing any relevant concerns by those who do know how the unix small-tools modular approach works.

Another daemon for managing control groups

Posted Dec 8, 2013 12:38 UTC (Sun) by pizza (subscriber, #46) [Link]

> that's the trouble with systemd advocates - they don't understand what it is that they're pushing to replace and, worse, universally think that their ignorance is a compelling argument for dismissing any relevant concerns by those who do know how the unix small-tools modular approach works.

BTW, That argument turns out to work quite well in reverse, as the systemd detractors (as a whole) spend very little time discussing the technical [de]merits of systemd (usually rather incorrectly) and prefer to just bash Lennart instead.

Another daemon for managing control groups

Posted Dec 8, 2013 23:19 UTC (Sun) by mathstuf (subscriber, #69389) [Link]

> hint: it's not a literal statement about imbibing a flavoured liquid.

While the phrase doesn't mean it literally, it is a reference to an actual event where Kool-Aid was used[1] for one of those murder-suicide cult things.

[1]https://en.wikipedia.org/wiki/Drinking_the_Kool-Aid#Jones...

Another daemon for managing control groups

Posted Dec 9, 2013 1:39 UTC (Mon) by nix (subscriber, #2304) [Link]

thank you for demonstrating that you don't understand unix.

Oh, thank you. Unintentional funny of the week. Neil Brown doesn't understand Unix?! Who are you going to accuse of not understanding Unix next, Roland McGrath? (GNU Make has far too many features and so does glibc, they are not in the Unix spirit!). Rob Pike? Ken Thompson?

Another daemon for managing control groups

Posted Jan 2, 2014 11:29 UTC (Thu) by Rudd-O (guest, #61155) [Link] (14 responses)

You are talking to Neil Brown. Where is your respect? You look like an idiot accusing him of "not understanding UNIX". Apologize.

Another daemon for managing control groups

Posted Jan 3, 2014 23:02 UTC (Fri) by cas (guest, #52554) [Link] (13 responses)

I really don't care who he is. what people say and do is more important than what their name is.

anyone who says that sort, grep, tr, comm etc 'don't combine together well' and '"do one thing and do it well" is good for prototypes, not for final products' does not understand unix. read his post - the entire thing was "waah! unix toosl are too hard and complicated, systemd saves us from that!".

perhaps he was just indulging in some exaggeration - it's a popular systemd pusher's tactic to exaggerate the alleged 'difficulty' or 'inadeqacy'of all previous/competing methods so that systemd's meager benefits seem greater by comparison. in this particular case, he has to grossly exaggerate the difficulty and inadequacy and just plain awfulness of the small-tools approach.

the truth is that systemd just isn't good enough to be worth the price of loss of modularity and replacability (nothing could be good enough to be worth that price), especially when the minor benefits that systemd does provide could easily be provided without a monolithic mono-culture monstrosity absorbing every low-level userland function in sight. init systems really don't have to be that complicated, they don't have to absorb syslog and cron and automount and everything else.

Another daemon for managing control groups

Posted Jan 3, 2014 23:47 UTC (Fri) by dlang (guest, #313) [Link] (3 responses)

ahh, but they aren't trying to just create a new init system. They are defining a new standard linux userspace. That is enough to justify anything.

I just wish they would do this on their own distro.

Another daemon for managing control groups

Posted Jan 4, 2014 2:27 UTC (Sat) by raven667 (subscriber, #5198) [Link]

Well this helps standardize service startup and login session management, the CLI and GUI shell userspace is not affected in any way

Another daemon for managing control groups

Posted Jan 25, 2014 20:47 UTC (Sat) by Baylink (guest, #755) [Link] (1 responses)

That's the part that I really don't understand.

I can get that Lennart doesn't understand the Unix Philosophy, or why it's more than "just a philosophy"; I can get that he doesn't care about why binary log files are bad, or why not being able to see *which* component of the core system went runaway on ps and kill just it is bad, or why being unable to replace those components with the specific ones you need for your job is bad. Or why optimizing for something that the vast majority of non-developer users of Linux don't care about -- booting at ludicrous speed -- is bad.

Why, in short: *requiring me to UTSL* just to do admin work, is bad.

What I *cannot* understand is how *entire committees* of people who drive the designs of distributions drank the Flavor Aid.

It is effectively impossible now to run sysVinit, because the people who package things for dnew Tweets istributions *no longer include, much less support* the components necessary to do that. And unless you're google-scale (in which case you're probably building your distro from scratch anyway, with a dedicated department of staffers), you can't fix that.

And that's not to mention the *literally decades* of sysadmin reflexes that have been flushed down the drain by this decision on the part of distro managers -- that's the part that really frosts my ass. I have better things to do than to relearn the basics all over again, guys, really.

I got into Unix because I am *lazy*. I want to learn small things once, and leverage them into big things, to make my life easier.

No matter how much better it might be at the actual work, as a sysadmin, systemd continues to make my life harder and more frustrating, at every single turn.

My philosophy of life is that our morale is proportional not to the size of our wins and losses, but to their *number*; that's why I use Linux.

systemd just made it go the other way.

Thanks, guys.

Another daemon for managing control groups

Posted Jan 27, 2014 15:07 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

If you're so lazy, why are you even trying to follow everyone else? Is sticking with Slackware not the laziest thing one can do these days anymore?</snark>

Maybe others aren't so lazy to not see the benefits it brings to the table and that's why it has been adopted? Making my user setup use unit files took maybe a day or two (and that includes a *lot* of unit files; at least 30) and I just got hung up on needing suid to start X. Systems does bring a lot to the table. Have you even tried it or did "service" not working prevent you from digging deeper?

Anyways, an anecdote from dealing with FreeBSD this weekend: I tried getting a taskd server running, but unfortunately I get no indication that it has failed to start, logs as to why it might have happened, or what the init system even attempted to do. And BSD init scripts are easier to deal with than sysvinit. I really would rather have systemd with all the tools it gives me to deal with such problems (so I know whether I should go poke upstream).

Another daemon for managing control groups

Posted Jan 4, 2014 2:12 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

>anyone who says that sort, grep, tr, comm etc 'don't combine together well' and '"do one thing and do it well" is good for prototypes, not for final products' does not understand unix. read his post - the entire thing was "waah! unix toosl are too hard and complicated, systemd saves us from that!".

Frankly, I'm sick of total ignoramuses who repeat this very argument.

Get the freaking systemd and see what it can do. Now try to replicate it with "do no thing well" tools like SysV Init.

Another daemon for managing control groups

Posted Jan 4, 2014 3:19 UTC (Sat) by neilbrown (subscriber, #359) [Link]

> I really don't care who he is. what people say and do is more important than what their name is.

Well said!

As for the rest of your post, I guess we'll have to agree to disagree.

(and no, I didn't mean to exaggerate. I love Unix tools and use them all the time, but I also believe in choosing the right tool for the job)

Another daemon for managing control groups

Posted Jan 4, 2014 14:50 UTC (Sat) by anselm (subscriber, #2796) [Link] (6 responses)

the truth is that systemd just isn't good enough to be worth the price of loss of modularity and replacability (nothing could be good enough to be worth that price)

Any »modularity« in System-V init is there purely by accident. Actual »modularity« would imply a certain amount of unity of design and basic concepts, and communication between the components by means of well-defined and documented interfaces. None of these apply to System-V init and its associated tools such as inetd, cron, etc.

It's not as if System-V init had been consciously designed the way it is. It is a hodgepodge of stuff from different sources that were developed separately from each other, don't interact well (if at all) and are each configured completely differently from the rest. If there is a long-overdue opportunity to replace this utter mess by something that has actually been properly thought out as a complete system, works a lot better in real life, and is easier to understand, develop for, maintain, teach, and learn, then I'm all for it.

Another daemon for managing control groups

Posted Jan 4, 2014 17:56 UTC (Sat) by raven667 (subscriber, #5198) [Link]

In some ways this is like Plan9 again, a re-thinking of the UNIX concepts with an eye toward completeness and consistency with a lot of operational experience behind it that didn't exist when many of these systems were first developed.

Another daemon for managing control groups

Posted Jan 25, 2014 20:50 UTC (Sat) by Baylink (guest, #755) [Link] (4 responses)

No, the modularity comes from *the pieces not all being one big piece with no visibility or lines of separation*.

Yes, the design of sysVinit is a *bit* rough. But it works, and it's easy to understand if you know basic Unix, and you don't have to read thousands of lines of source to figure out what it's doing.

And if one part of it falls over, ps will tell you, and you can kill just that one part.

And you can grep the damn logs.

Another daemon for managing control groups

Posted Jan 26, 2014 1:20 UTC (Sun) by HelloWorld (guest, #56129) [Link]

> Yes, the design of sysVinit is a *bit* rough. But it works,
It *doesn't work*. Stop denying it, sysvinit can't even stop a service reliably and correctly.

> and you don't have to read thousands of lines of source to figure out what it's doing.
Yeah, you just have to read thousands of lines of shell. With systemd otoh, you read *documentation*. You know, this stuff that is actually meant to be read by humans. Because unlike the sysvinit quagmire, systemd actually *has* documentation that's worth mentioning.

> And if one part of it falls over, ps will tell you, and you can kill just that one part.
Yeah, like systemd somehow magicall broke ps.

> And you can grep the damn logs.
You can do that with journalctl too. But actually most people don't do it, because it *sucks* and just using the domain-specific features of journalctl is easier, faster and more robust.

Another daemon for managing control groups

Posted Jan 27, 2014 19:38 UTC (Mon) by anselm (subscriber, #2796) [Link] (2 responses)

The current init/service activation infrastructure (System V init, inetd, cron, …) does not have anything worth calling »a design«. It is cobbled together out of random pieces from different sources and the amazing thing is that it works at all. The pieces don't talk to one another and are all configured differently. If it had in fact been »designed« you would expect at least a minimum of consistency and cooperation between the pieces.

If somebody proposed this setup today as a new system they would be laughed out of the lecture theatre. The only reason this is still in actual use at all is that for the longest time nobody has dared come up with an alternative. Most of the other Unix-like systems have replaced it long ago.

People always cite »the Unix philosophy« as »do one thing and do it well«, and point to System V init as a prime example of this approach. This disregards the obvious fact that while SysV init is surely doing something, there is no way that one could claim it is doing it well.

Another daemon for managing control groups

Posted Jan 27, 2014 20:45 UTC (Mon) by khim (subscriber, #9252) [Link] (1 responses)

This disregards the obvious fact that while SysV init is surely doing something, there is no way that one could claim it is doing it well.

It does pretty decent work on scaring newbies away from Linux. Perhaps that is the thing is was designed for? Of course it's kinda overenginered for such a use case, you can create a much smaller and simpler mess which will be much scarier.

Another daemon for managing control groups

Posted Jan 27, 2014 22:40 UTC (Mon) by nix (subscriber, #2304) [Link]

Someone needs to implement an init system as an IOCCC entry, clearly.

Another daemon for managing control groups

Posted Dec 8, 2013 13:00 UTC (Sun) by anselm (subscriber, #2796) [Link]

it's a jack-of-all-trades master-of-none that tries to do too much, tries to do everything, in fact.

Systemd is really a collection of co–operating tools, each of which tries to do one thing well (including but not limited to service process configuration, control, and management).

Another daemon for managing control groups

Posted Dec 6, 2013 13:13 UTC (Fri) by roblucid (guest, #48964) [Link] (15 responses)

Whilst appreciating Neil's point, it also seems to a more open philosophy to have program packages focussed on related tasks eg) setting up & managing raid whilst leaving FS layout, backup and so on to other tools. That makes it easier to evolve with time, because new features or projects improving on past work, only need to replicate a limited set of functionality.

The monolithic approach is disliked, because it leads to stagnation and lock in. Perhaps a new kernel feeature, would in practice require support by systemd, which in might not be of interest to that project; yet other projects would be faced with a daunting task to replicate a whole host of systemd compatible features, because of it's wide remit. This is why it could be so hard to replace MS Exchange servers in business; they weren't just doing email, but group meeting planning and so on.

systemd, merging init(8) and inetd(8) makes good sense, it's less clear what the benefit is of multiple incompatible implementations of cgroup managament to me. systemd's implementation is going to be rather experimental, which is very different from integrating in a better more modern implementation init & inetd; they may not have the best ideas but because of systemd becoming ubiquitous we could be lumbered with something satisfactory, rather than the best.

Another daemon for managing control groups

Posted Dec 6, 2013 15:00 UTC (Fri) by anselm (subscriber, #2796) [Link] (13 responses)

it also seems to a more open philosophy to have program packages focussed on related tasks eg) setting up & managing raid whilst leaving FS layout, backup and so on to other tools. That makes it easier to evolve with time, because new features or projects improving on past work, only need to replicate a limited set of functionality.

On the other hand, the latest fashion in file systems seems to be all-in-one approaches like ZFS and Btrfs, which combine file, volume, and RAID management, snapshots, etc. in one single implementation instead of relying on existing system features such as LVM and Linux kernel RAID.

It is interesting to note that the backlash against these new file systems is a lot less emotional than the one against systemd, even though the underlying idea of coming up with a radical new integrated design to replace a number of loosely-connected existing but sub-optimal infrastructure elements is quite similar. (But then again, Btrfs wasn't invented by Lennart Poettering.)

Another daemon for managing control groups

Posted Dec 6, 2013 16:27 UTC (Fri) by dlang (guest, #313) [Link] (10 responses)

> It is interesting to note that the backlash against these new file systems is a lot less emotional than the one against systemd

The reason for this is very easy to understand, nobody is forcing anyone to use the new filesystem.

Even if a distro were to change to use one of these new filesystems by default, the distro would not remove support for the existing filesystems, and would be extremely unlikely to make it so that if you selected one of the other filesystems you would have lots of other things you had to change in the distro as well.

It's really easy to have your system using different filesystems for different tasks.

It's really hard to have multiple init systems in your distro, even as options. And it's impossible to have a new init program on your system that you only use for some testing things.

Another daemon for managing control groups

Posted Dec 6, 2013 16:50 UTC (Fri) by anselm (subscriber, #2796) [Link] (6 responses)

nobody is forcing anyone to use the new filesystem.

Nobody is »forcing« anyone to use the new init system, either. It may be the case that a Linux distribution or a project like GNOME decides to take advantage of the new features and better integration that something like systemd has to offer, but if you don't want to come along for the ride there's always the likes of Slackware. The beauty of free software means that for as long as enough people want to stick with SysV init desperately enough, there will be a distribution (or distributions) around that is based on SysV init.

As far as file systems are concerned, the most recent SUSE distributions, for example, do »force« you onto Btrfs if you want to avail yourself of advanced tools like Snapper – and SUSE's enterprise-class distribution doesn't even support Ext4 (which arguably is the file system one would want to use if one didn't want to use Btrfs). Hence the situation isn't all that different, it's just that with file systems the »out with the old, in with the new« phenomenon isn't as widespread yet. This may also be to do with the fact that an init system is conceptually a lot simpler than a modern advanced file system, and that, for the time being, people (and in particular distribution makers) appear to be more inclined to trust systemd over SysV init than Btrfs over Ext4.

Another daemon for managing control groups

Posted Dec 6, 2013 18:09 UTC (Fri) by dlang (guest, #313) [Link]

sure, you can abandon the distro that you have been using when they change init systems.

you then have to listen to people harass other distros because they are being 'backwards' and not using the new init system.

people are far more careful about changing filesystems because a filesystem bug will eat their data, and distros that eat people's data tend to disappear :-)

Another daemon for managing control groups

Posted Dec 6, 2013 23:19 UTC (Fri) by nix (subscriber, #2304) [Link] (4 responses)

I note that people who write filesystems are paranoid bastards who know what backward compatibility is for. The history of udev and pulseaudio and all of Lennart's projects makes me think that he wouldn't know backward compatibility if it shook him very hard by the throat. PA only settled down and stopped breaking things for all its users on almost every upgrade when Lennart was no longer doing most of the maintenance. I see no sign that udev (which I long gave up trying to upgrade at despair at its constant pointless disruptive changes) and systemd are ceasing in their ongoing attempts to break the world.

After all, adapting to all that breakage is up to the *distribution*! And, oh yes, if you're not using Fedora and/or otherwise *precisely* the same platform Lennart is and experience problems you can bugger off. Or that's what I've seen Lennart saying over and over and over to quite tiresome lengths. (And, no, I'm *not* going to use a distro with a six-month lifespan for anything important. I need a greater degree of stability, without being stuck in the six-year-old software mire of the 'enterprise' distros, which apparently makes me a niche market nobody cares about.)

Another daemon for managing control groups

Posted Dec 7, 2013 0:20 UTC (Sat) by pizza (subscriber, #46) [Link] (3 responses)

Comments:

udev was primarily developed by GregKH. Though udev is now part of the systemd tree it is an independently-developed component that is one of the fundamental building blocks of a Linux system.

Meanwhile, Pulseaudio's user-facing problems were threefold -- Bugs in hardware drivers, bugs in ALSA's user plugin system, and horrendously buggy applications. Oh, and it was still a vast, vast improvement over what came before. (I say this as someone who has written a decent amount of audio code over the years..)

This is not unlike NetworkManager, which exposed the inconsistent behaviour of wifi drivers supposedly implementing the same Wireless Extensions interface (and the even more inconsistent event interface) and how every single WEXT client had to use device-specific hacks to ensure consistent behaviour.

I find it sad that the folks who put forth the effort to sanitize core infrastructure bits are blasted for all of the bugs they uncover (and fix!) in the process, while those same voices complain about how Linux isn't ready because things aren't transparently integrated and fully tested like $somecommercialos. This work doesn't just happen, and there are frightfully few people with the technical chops and skin thick enough to accomplish this.

Another daemon for managing control groups

Posted Dec 9, 2013 2:40 UTC (Mon) by ThinkRob (guest, #64513) [Link] (2 responses)

> Meanwhile, Pulseaudio's user-facing problems were threefold -- Bugs in hardware drivers, bugs in ALSA's user plugin system, and horrendously buggy applications. Oh, and it was still a vast, vast improvement over what came before. (I say this as someone who has written a decent amount of audio code over the years..)

Wasn't one of the somethings that came before something that had per-application mixing, a consistent cross-platform API (and cross-platform implementations), existing application support, and precise, lower-latency mixing? i.e. most of pulse's features, sans network streaming?

Another daemon for managing control groups

Posted Dec 9, 2013 13:44 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (1 responses)

If you're thinking of JACK, it is a little on the high end and complicated for just getting mplayer to play audio at the same time as notifications. Pulseaudio works with JACK for handoff IIRC, so they can cooperate. If you're thinking of OSS4 or aRTS, I think those are pretty much dead in Linux-land.

Another daemon for managing control groups

Posted Dec 10, 2013 5:56 UTC (Tue) by ThinkRob (guest, #64513) [Link]

I was thinking of OSS4, which is both still developed for, and still quite usable in Linux. It's just (sadly) no longer the default, in large part due to self-inflicted wounds.

Another daemon for managing control groups

Posted Dec 10, 2013 19:06 UTC (Tue) by BenHutchings (subscriber, #37955) [Link] (2 responses)

Even if a distro were to change to use one of these new filesystems by default, the distro would not remove support for the existing filesystems,

It certainly might remove support for existing filesystems as root. For example, the package manager might be changed to require transaction support. (I would expect to hear a similar outcry if this actually happens in a mainstream distro.)

Another daemon for managing control groups

Posted Dec 10, 2013 22:12 UTC (Tue) by anselm (subscriber, #2796) [Link] (1 responses)

For example, the package manager might be changed to require transaction support. (I would expect to hear a similar outcry if this actually happens in a mainstream distro.)

OpenSUSE and SLES are apparently moving in that direction. Recent incarnations of YaST use a tool called »Snapper« to make snapshots of the root file system before and after configuration changes; if you want to avail yourself of that functionality you are pretty much forced to use Btrfs. So far people don't seem to get as worked up about this as they do over systemd (which to be fair was introduced to openSUSE some time ago, seemingly without much of a fuss either).

Another daemon for managing control groups

Posted Jan 1, 2014 12:27 UTC (Wed) by jospoortvliet (guest, #33164) [Link]

Note that some work is ongoing to support this on non-btrfs filesystems...

Another daemon for managing control groups

Posted Dec 9, 2013 2:36 UTC (Mon) by ThinkRob (guest, #64513) [Link]

<quote> It is interesting to note that the backlash against these new file systems is a lot less emotional than the one against systemd, even though the underlying idea of coming up with a radical new integrated design to replace a number of loosely-connected existing but sub-optimal infrastructure elements is quite similar. (But then again, Btrfs wasn't invented by Lennart Poettering.) </quote>

AFAIK, no distros have plans to adopt a single filesystem as the One True filesystem and stop supporting all others. The same cannot be said for systemd.

The upshot of this is that while you're always free to run whatever FS you choose despite the default choice of your distro's installer, the same cannot be said for the ease with which one can switch init systems.

Another daemon for managing control groups

Posted Dec 10, 2013 8:29 UTC (Tue) by zdzichu (subscriber, #17118) [Link]

Memory fades fast. There was "blatant layering violation" as a nickname for ZFS for some time.

Another daemon for managing control groups

Posted Dec 7, 2013 0:26 UTC (Sat) by rgmoore (✭ supporter ✭, #75) [Link]

it's less clear what the benefit is of multiple incompatible implementations of cgroup managament to me.

The requirement of a single manager for cgroups is coming from the kernel side, not from systemd. Since systemd makes very heavy use of cgroups for process management, it makes sense for systemd to be that manager in distributions that use it. Otherwise, you have the init system depending on another process to do a truly essential part of its job, which doesn't seem any simpler than having an integrated system.

Another daemon for managing control groups

Posted Dec 5, 2013 4:03 UTC (Thu) by daniels (subscriber, #16193) [Link]

inb4 ... damn, too late.

Another daemon for managing control groups

Posted Dec 5, 2013 6:55 UTC (Thu) by zdzichu (subscriber, #17118) [Link] (7 responses)

systemd is becoming *the* Linux userspace.
Just like glibc is *the* Linux system libc. There are other contenders, but they occupy their small niches.

Another daemon for managing control groups

Posted Dec 5, 2013 11:03 UTC (Thu) by rwmj (subscriber, #5474) [Link] (6 responses)

It's not the same thing. glibc provides a broad API, but with definite limits, and you can easily replace bits of glibc. Need to use a GUI? Link with gtk. Need a different way of handling memory? Link with talloc. You can even replace (for example) all str* functions with stricter versions from an external library.

systemd is a fine way to manage daemons, but there's no reason why it couldn't use a library or separate process for cgroup management.

Breaking up monolithic daemons into separate processes and libraries with well-defined interfaces between them is both the Unix philosophy AND a better way to develop more reliable software.

Another daemon for managing control groups

Posted Dec 5, 2013 11:17 UTC (Thu) by hummassa (guest, #307) [Link]

Actually, the only pro-systemd argument I heard that makes a lot of sense seems to contradict you and goes in the "yo dawg, I'll make init, that is a daemon, to init and manage the deamon that will init and manage the other daemons so you can init and manage the deamons" direction. If systemd/sysv-init will manage one deamon-managing process, isn't it the same as managing all the deamons? I really don't have an answer.

Another daemon for managing control groups

Posted Dec 5, 2013 13:03 UTC (Thu) by mezcalero (subscriber, #45103) [Link] (4 responses)

Same thing with systemd. You don't like logind? Turn it off at compilation, use ConsoleKit instead. Don't like journald? Just run syslog, too. Don't like it's SELinux support, then turn it off. Don't like its bootchart implementation, then turn it off, maybe use another one. Don't like its readahead implementation? Then turn it off, use another one. There are 35 things you can turn/off with configure switches, to tailor it exactly to your needs. And distros can split up systemd into multiple RPMs/DEBs if they desire so and make use of them independently.

However, there are a few things you cannot turn off: That's PID1's process management of which cgroups is part of. That's udev device management and that's the journal (but you can turn off the journal's storage on disk, we however need it to pass stdout/stderr of all daemons through).

And yes, there are tons of reasons why you want cgroup management in PID1: because it's trivially easy there and simple, and it is not if you do it outside of PID1. If you do it out-of-process, you need to replicate pretty much the entire state of your service manager in your cgroup manager, since the entities that are resource managed are in 95% of the cases exactly the same entitites that are service managed. So splitting them up means you need some form of IPC that constantly replicates the entire tree of services from PID1 in that other cgroup daemon. That's fragile and messy, and a lot of unnecessary code. IPC always is. Then, there is the issue of cyclic deps: a good PID 1 knows at any time securely and reliably of each process to which service it belongs. That's easy to do with cgroups. However, if cgroup management is done outside of PID1 in a different process, then PID 1 suddenly becomes a client to that other process, while that other process is also a managed process PID 1 wants to use cgroups to manage for. To start and manage that other daemon that will allow you to deal with cgroups you need cgroups in the first place. And that's just broken.

Beyond that: When doing cgroups you need an execution queue of some kind and you need a dependency tree between your groups, so that you can always rebuild your tree from the root down, in the right order, with each property set before you go to the next child. You also need to dynamically react to devices coming and going and rebuild/refresh your tree since many cgroup props take device major/minor pairs which are pretty much dynamically assigned to devices these days. You also need to be able to deal with groups coming, and going, and possible propagating changes then to other groups on the same level. Soooo, an execution queue that is based on a dependency tree that is influenced by services/cgroups coming and going and devices coming and going, that's pretty much exactly what systemd *is*. If you avoid doing this in PID 1 then you are just recreating systemd a second time in a different process. A lot more code to write, to test, to spend resources on. I am sorry, but I am just not going to write or help building such a needlessly complex system. You can have it easy and small and today. I am really not interested in having it complicated, redundant, larger and one later day.

(And no, a library certainly doesn't work at all for this, you need to cross the privilege boundary and a single active queue dispatcher, which doesn't really make a library such a good choice...)

Another daemon for managing control groups

Posted Dec 5, 2013 19:03 UTC (Thu) by butcher (guest, #856) [Link] (3 responses)

I believe it's good to have both, systemd/cgroups and cgroupd, but I'll probably be buildroot-ing images with systemd/cgroups. It seems like a more appropriate factoring of functionality.

Thanks all involved for the ability to choose.

Another daemon for managing control groups

Posted Dec 5, 2013 20:18 UTC (Thu) by drag (guest, #31333) [Link] (2 responses)

> I believe it's good to have both, systemd/cgroups and cgroupd,

Why is it good to have both?

In what way is systemd's solution inadequate?

Another daemon for managing control groups

Posted Dec 5, 2013 20:55 UTC (Thu) by jake (editor, #205) [Link] (1 responses)

> In what way is systemd's solution inadequate?

I think primarily the problem that some people have with systemd's solution is that it requires running systemd as the system's init, which some do not want to do. According to Lennart in http://lwn.net/Articles/575847/ there are "lots of cases where you don't need systemd", but if you want to use cgroups you (will) need some kind of manager.

systemd's solution may well be perfectly adequate (or, in my mind, way better than 'adequate'), but it does require systemd.

jake

Another daemon for managing control groups

Posted Dec 6, 2013 18:06 UTC (Fri) by drag (guest, #31333) [Link]

> but if you want to use cgroups you (will) need some kind of manager.

Yes, but only one can be used at a time. So any solution that will provide cgroups management will have the same problem as systemd.

No really good way to fix this I expect.

Another daemon for managing control groups

Posted Dec 5, 2013 8:05 UTC (Thu) by ovitters (guest, #27950) [Link] (3 responses)

Being able to setup tcp connections from bash is probably terribly evil in your book? Let's replace bash then, because that's what bash provides (built in, no external tools required).

Anyway, a philosophy has a certain purpose. Blindly following some philosophy is not useful. Calling things unneeded names like "opaque ball" is vague. Further, "everything your Linux system does" is factually wrong.

Try being specific: why *must* *everything* be reimplemented multiple times?

Further, if you continue to twist words like you did before I'll call you out on it. Good discussion ok, but twisting of words is poor form.

Another daemon for managing control groups

Posted Dec 5, 2013 12:28 UTC (Thu) by NAR (subscriber, #1313) [Link] (1 responses)

Try being specific: why *must* *everything* be reimplemented multiple times?

Exactly: why is an operating system (Linux) implemented when there were (and are) multiple other operating systems?

More seriously: one answer is that specifications can be "verified" by implementing them multiple times by different people and see if the result interoperates. Of course, my gut feeling is that the systemd and cgroup developers are not interested in interoperability or portability or even stable APIs because they hinder progress. Whether one agrees with this position or not, that's an other question.

Another daemon for managing control groups

Posted Dec 5, 2013 12:41 UTC (Thu) by ovitters (guest, #27950) [Link]

You're not being specific, you're just answering one question. Where is the specification for every software available on Debian? Where are the multiple implementations for every software?

I was asking why *everything must* be implemented multiple times.

Also your answer implies that systemd has a specification which it is being written against. I'm not aware of such specification. Systemd did create various specifications.

Suggest thinking about my question a little bit more.

Another daemon for managing control groups

Posted Dec 6, 2013 2:38 UTC (Fri) by rgmoore (✭ supporter ✭, #75) [Link]

Try being specific: why *must* *everything* be reimplemented multiple times?

I'm not sure if everything must be reimplemented multiple times, but everything should be capable of being reimplemented multiple times so that components can be torn out and replaced with better ones. That has been a significant way that Unix/Linux has made progress. Of course it's helpful to have standardized interfaces so that new implementations can be drop-in compatible with old ones. It may even be helpful to have multiple implementations of critical infrastructure so that it can be replaced quickly if it's ever discovered to have critical flaws.

Another daemon for managing control groups

Posted Dec 5, 2013 9:53 UTC (Thu) by Karellen (subscriber, #67644) [Link] (1 responses)

"Do one thing. Do it well."

I don't use systemd (yet) but I do see it doing one thing well. It manages the other processes (or groups of processes) on your system.

That means starting them up, monitoring them, re-starting them if necessary, and shutting them down. Whether those processes start on system boot, on receipt of a network connection, on a timer, or on some other event, it makes sense to me that there's one process that knows the whole policy, and can make sure it's being followed as intended.

And that to me means that if you want to set up a special environment for a new process, that daemon is the place to do it. Want to specify which user/group the process runs as? Be able to configure that in the process-managing deamon. Want to specify resource limits for the new processes (rlimits/ulimits)? Configure it in the process-managing daemon. Want to specify an alternate rootfs - configure it in the process-managing daemon.

Want to manage the control groups that new processes are started in, or move processes between control groups? Why not do it in the process-management daemon.

Another daemon for managing control groups

Posted Dec 6, 2013 23:23 UTC (Fri) by nix (subscriber, #2304) [Link]

That means starting them up, monitoring them, re-starting them if necessary, and shutting them down.

Shame it's trying to do so many other things too.

Another daemon for managing control groups

Posted Dec 5, 2013 19:57 UTC (Thu) by drag (guest, #31333) [Link]

> Does not lumping *everything your Linux system does* inside the opaque ball of systemd violate nearly 35 years of Unix Philosophy pretty violently?

No. Because that is not what is happening.

> Do one thing. Do it well.

Cgroups is about managing resources of processes in a hierarchical/grouped manner from what I can tell. Linux/Unix processes are already arranged in a hierarchical manner due to the parent child process. The parent process is PID 1, which is the Init.

Makes sense to me that you'd want to be able to manage things from the top down instead of having a entire separate process you have reprogram your applications deal with via dbus.

Another daemon for managing control groups

Posted Dec 12, 2013 8:19 UTC (Thu) by oldtomas (guest, #72579) [Link]

> Am I really the *only* person who hasn't drunk the koolaid?

No, you aren't the only one. "Me Too".

I've managed to avoid systemd up to now, and hope I'll manage in the future.

It's just that the discourse with its very vocal proposers isn't fun anymore.

Another daemon for managing control groups

Posted Dec 5, 2013 2:26 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (10 responses)

> Depending on permissions, any process may be able use that interface to set up its own hierarchy of groups, which is one of the problems with the existing implementation, according to cgroup co-maintainer Tejun Heo and others.

Can ANYBODY describe the relevant problems???

I've been hearing this statement about problems with cgroup delegation about 27863 times but I haven't been able to find what the problems are. Apart from the utterly trivial "starve your siblings" one.

Another daemon for managing control groups

Posted Dec 5, 2013 5:31 UTC (Thu) by Fowl (subscriber, #65667) [Link] (8 responses)

Yes I'd very much like to see a comprehensive overview of what exactly is wrong with the status quo. It seems that the kernel is the perfect place to mediate between processes, delegate system resources and enforce security... I mean we don't have a user space "file manager".

Or is the current implementation unworkable somehow?

Another daemon for managing control groups

Posted Dec 5, 2013 7:59 UTC (Thu) by ovitters (guest, #27950) [Link] (6 responses)

It was explained at least a few times in previous LWN articles AFAIK. Think as recently as 2-4 weeks or so.

Another daemon for managing control groups

Posted Dec 7, 2013 4:26 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

NO IT WASN'T!

The only scenario explained was a brain-dead "starve your siblings" one. Which is extremely easy to avoid.

Another daemon for managing control groups

Posted Dec 7, 2013 4:41 UTC (Sat) by dlang (guest, #313) [Link] (2 responses)

> The only scenario explained was a brain-dead "starve your siblings" one. Which is extremely easy to avoid.

In the comments here I have heard one other issue, the fact that this turns kernel-internal APIs into something accessed by unprivileged users, and those interfaces may not have been hardened suitably.

Now, while I agree this is a valid concern, the 'solution' of cut off all possible access except through a single userspace daemon does not seem like the appropriate long-term answer.

Another daemon for managing control groups

Posted Dec 7, 2013 6:21 UTC (Sat) by raven667 (subscriber, #5198) [Link] (1 responses)

> Now, while I agree this is a valid concern, the 'solution' of cut off all possible access except through a single userspace daemon does not seem like the appropriate long-term answer.

I think you are exactly right, and I would guess that Tejun Heo would agree with you, but the time and effort it is going to take to build a cgroup API in kernel space which could be delegated to untrusted users and have some guarantees that they couldn't cause trouble will take many years of effort and probably a thorough rewrite of the cgroup implementation to put in appropriate policy and access control.

In the mean time the single-userspace-manager approach will both help solve the immediate issue and provide very useful operational experience which will inform the choices what that eventual kernel API should look like based on the real-world cases of how the userspace managers are actually used.

Another daemon for managing control groups

Posted Dec 7, 2013 8:44 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

What is so freakingly complicated in delegating?

Cgroups API is not something magical with tons of different settings! There are maybe around 50 user changeable settings in _total_ for all cgroups and most of them are uncomplicated.

Sure, there might be subtle race conditions or something, but solution for them is not to hide our head in sand, but to solve them.

Look, /proc is much more complicated than /sys/fs/cgroup and yet it has been secured adequately for non-privileged users to create their own namespaces. Of course, Lennart's way for this would be to create NamespaceD with DBUS-based interface and pull it into SystemD.

Another daemon for managing control groups

Posted Dec 7, 2013 6:25 UTC (Sat) by raven667 (subscriber, #5198) [Link] (1 responses)

Hey man, no point in arguing about what is or isn't here with us, you should just have this out with Tejun Heo, you can explain to him how his understanding of what the core cgroup failings are totally wrong and how any problems he believes he has uncovered are actually extremely easy to avoid. I'm sure he'll be grateful for saving him so much work once you explain that all the design flaws he uncovered are actually features.\

I hope you take my sarcasm in the gentile way it was intended 8-).

Another daemon for managing control groups

Posted Dec 7, 2013 8:51 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

I sent him a message some time ago. It has been ignored (as I suspected it would be). I'm preparing a long message to LKML, but I'm sure it's going to be ignored too.

My company provides a platform for hundreds of customers and we depend heavily on cgroups for our infrastructure. That's why it worries me that cgroups developers simply declare that cgroups ABI is somehow magically insecure and can NOT be fixed at all. And the only solution is to develop a One Daemon To Rule Them All with an API that almost, but not quite completely replicates the filesystem interface.

Another daemon for managing control groups

Posted Dec 10, 2013 22:24 UTC (Tue) by Fowl (subscriber, #65667) [Link]

(this was a request for more LWN content, not an attack on any of the players)

Another daemon for managing control groups

Posted Dec 7, 2013 21:07 UTC (Sat) by hallyn (subscriber, #22558) [Link]

I can't, and agree with your position. We have two practical needs for the manager for lxc. First is that lxc itself not have to worry about nesting for children - it just wants containers to be under itself regardless if what its current cgroup is. Don't want that code in lxc. Second is user namespaces - you cannot make changes to devices cgroup settings if you are root in a child userns (need sys_admin targeted at init_user_ns), and a patch to allow this (which should be safe due to in-kernel hierarchical constraints) was rejected. The manager handles that for us.

Well, one danger is simply too-deep nesting of cgroups by unpriv users which could exhaust kmem. That and yours are all I know of.

Another daemon for managing control groups

Posted Dec 5, 2013 10:23 UTC (Thu) by tdalman (guest, #41971) [Link] (113 responses)

I tend to agree with the author that exclusively binding the cgroups feature into systemd is more than problematic. I also agree that lumping many (independent) functionalities in a single component is against my understanding of high quality software. I fail to understand how an init process is connected to udev or cgroups. As long as systemd forces me to use the whole package or none, I will certainly choose the latter and stick to working software (like sysvinit) and new approaches (like cgmanager). For now, systemd is an absolute no-go for me.

Another daemon for managing control groups

Posted Dec 5, 2013 11:24 UTC (Thu) by anselm (subscriber, #2796) [Link] (103 responses)

I also agree that lumping many (independent) functionalities in a single component is against my understanding of high quality software. I fail to understand how an init process is connected to udev or cgroups.

Many of the so-called »independent« functionalities are only »independent« because they have been developed at different times by different people, not because of deliberate design. For example, in the traditional setup there are four or five different ways of starting background service processes, all configured differently, most not really working properly, and none aware of each other – mostly through a series of accidents of history. With respect, this is not how one constructs »high quality software«. I wonder how people can claim, with a straight face, that doing things this way is a Good Idea, and that the method of choice for improving the situation is by adding more complexity through yet another disjointed service.

The big advantage of systemd is that it unifies a lot of previously disjointed but similar functionality under a single interface. This means, among other things, that the configuration of the init process (together with other purportedly »independent« bits like inetd, cron, …) is that much easier to understand, to manage, to learn, and to teach. This will be an advantage in the long run.

It is also worth reiterating that Unix didn't start out with the SysV init system. When that was first proposed in the 1980s, the objections against it were pretty similar to the ones fielded against systemd these days – it was accused, by the same sort of people, of being overengineered, needlessly complex, and generally not needed in the first place because the existing approaches were considered perfectly adequate. It's funny how, nearly 30 years later, SysV init seems to have suddenly become the epitome of simplicity and sound Unix-style engineering, and how we must hang on to it at all costs. Incidentally, pretty much none of the other popular Unix implementations out there still use SysV init, and that should also tell us something.

Another daemon for managing control groups

Posted Dec 6, 2013 8:42 UTC (Fri) by jezuch (subscriber, #52988) [Link]

> For example, in the traditional setup there are four or five different ways of starting background service processes, all configured differently, most not really working properly, and none aware of each other – mostly through a series of accidents of history. With respect, this is not how one constructs »high quality software«. I wonder how people can claim, with a straight face, that doing things this way is a Good Idea

I remember reading about an experiment with monkeys in a cage. There was a banana hanging from the ceiling but when any of the monkeys tried to reach for it, all of them were sprinkled with cold water. Avoiding the cold shower was more important than getting a banana, so soon whenever one monkey tried to reach for the banana, other monkeys prevented it. Over time the monkeys were being replaced one by one and each time the newcomer would try to reach for the banana and would be violently suppressed by others. And this persisted even when there were no monkeys left that experienced the cold shower. The banana became "taboo". This is how tradition emerges.

I think of this story each time I read someone passionately defending the "Unix Way" and using it against systemd...

Another daemon for managing control groups

Posted Dec 6, 2013 21:39 UTC (Fri) by sorpigal (subscriber, #36106) [Link] (3 responses)

> The big advantage of systemd is that it unifies a lot of previously disjointed but similar functionality under a single interface. This means, among other things, that the configuration of the init process (together with other purportedly »independent« bits like inetd, cron, …) is that much easier to understand, to manage, to learn, and to teach. This will be an advantage in the long run.
While you are not wrong the advent of systemd seems to presume that we now know how to do this and don't need to experiment any more. Any replacement for systemd once it is entrenched would have to have at least identical features--the lowest common denominator is now not as low.

Anecdote time: I recently added a ./var/ directory in a little app that I'm building for my $job, because I didn't need to explain to anyone else on the team what var meant or why logs and caches would be written there. In the long run perhaps everyone will know what .unit means, but not today. This is the secondary value of Unix, after simplicity; over time conventions gain power without regard to their intrinsic values. Repeating those conventions makes them more valuable each time. Violating those conventions raises hackles even if the violation has a lot of intrinsic value.

Another daemon for managing control groups

Posted Dec 6, 2013 22:35 UTC (Fri) by anselm (subscriber, #2796) [Link] (2 responses)

While you are not wrong the advent of systemd seems to presume that we now know how to do this and don't need to experiment any more.

SysV init has been around for 30 years or so. It is reasonable to assume that any significant use cases that it and its friends don't cover in some (convoluted and suboptimal) way would have shown up by now.

The other point worth making is that systemd doesn't come to us out of thin air. There was considerable experimentation going on in init-like systems in the shape of things like daemontools, launchd, Upstart, and so on, all of which have influenced the design of systemd. Systemd builds on these other tools in various ways and takes many of their ideas even further.

Finally, in case some revolutionary way of starting a service comes up that systemd doesn't cover, surely – systemd being open-source – it would make sense to extend systemd to deal with it rather than replace systemd outright with something else?

Any replacement for systemd once it is entrenched would have to have at least identical features--the lowest common denominator is now not as low.

I don't see this as an argument in favour of SysV init. Surely we don't want to stick with SysV init for the indefinite future because it offers fewer features and is easier to replace with something else than systemd?

I recently added a ./var/ directory in a little app that I'm building for my $job, because I didn't need to explain to anyone else on the team what var meant or why logs and caches would be written there. In the long run perhaps everyone will know what .unit means, but not today.

The /var directory came along in the 1980s when Unix had already been around for almost 20 years. It is safe to assume that there must have been old Unix hands at the time who would fight this innovation tooth and nail because it violated the conventions they had been repeating for more than a decade. The same applies to many of the other innovations that happened in nearly 50 years of Unix history (including but not limited to SysV init, which people seem to like to pretend goes back at least to Noah's ark).

People point to the hallowed »Unix tradition« as if Unix had sprung from the Earth in 1969, fully formed, and then didn't change at all until Linux came along 20 years later. The truth is that Unix changed all the time, and many things that we now consider part of the original »traditional« feature set only appeared way later than people generally accept. For example, the Bourne shell was introduced in 1977 with Unix Version 7, and still we read in actual published books that »The Bourne shell was the original Unix shell«. Not. Even something as central to the »Unix philosophy« as pipes was only implemented after Unix had already been around for several years.

SysV init and friends, in 2013, are the equivalents of baling wire and twine. It is amazing that they can be made to work at all. We can consider ourselves fortunate that somebody of Lennart Poettering's calibre has decided to come up with an alternative that will hopefully be suitable for the next few decades. I'm frankly a bit sick of those people who, because they have no technical arguments to make, will come up with quaint ways of saying »This is not what I'm used to and therefore I don't like it«. They can stick with SysV init for the next century or so, but they don't get to dictate policy to everybody else. If systemd is really as bad as they say they can rest assured that they will have the last laugh.

Another daemon for managing control groups

Posted Dec 7, 2013 3:23 UTC (Sat) by neilbrown (subscriber, #359) [Link]

> The same applies to many of the other innovations that happened in nearly 50 years of Unix history (including but not limited to SysV init, which people seem to like to pretend goes back at least to Noah's ark).

"If it can't be done by adding a few lines to /etc/rc.local, then it isn't worth doing!" - Noah.

Another daemon for managing control groups

Posted Dec 7, 2013 12:35 UTC (Sat) by sorpigal (subscriber, #36106) [Link]

I think you misunderstand, I was attempting to explain the reaction against systemd and part of why it might exist. Clinging to tradition seems to me to be a major source of negative reaction against systemd and you don't have to be an unreasonable person to react that way. The punch line would have been that eventually systemd is likely to receive hallowed status itself and there is no down side unless it has design flaws that are not apparent now and not fixable without total replacement.

Traditions and conventions have nothing specifically to do with Unix or the Unix Way, except insofar as its longevity means it has time to develop them while other things may not. Making new conventions is socially hard no matter the merit. Try convincing everyone to stop saying "Hello, world" and use something else instead; even for an arbitrary string it would be a hard sell.

> For example, the Bourne shell was introduced in 1977 with Unix Version 7
Indeed, and Bourne apparently spent quite a lot of effort individually convincing each of his early users to switch to his shell even though it was "obviously" superior and largely a superset of functionality. It was years before one could presume a Bourne sh. Change is hard (and slow).

Another daemon for managing control groups

Posted Dec 7, 2013 8:29 UTC (Sat) by tdalman (guest, #41971) [Link] (97 responses)

> It is also worth reiterating that Unix didn't start out with the SysV init
> system. When that was first proposed in the 1980s, the objections against
> it were pretty similar to the ones fielded against systemd these days – it
> was accused, by the same sort of people, of being overengineered,
> needlessly complex, and generally not needed in the first place because
> the existing approaches were considered perfectly adequate. It's funny
> how, nearly 30 years later, SysV init seems to have suddenly become the
> epitome of simplicity and sound Unix-style engineering, and how we must
> hang on to it at all costs. Incidentally, pretty much none of the other
> popular Unix implementations out there still use SysV init, and that
> should also tell us something.

Perhaps I wasn't clear enough: I don't care about the UNIX-way and whether is it the right or wrong thing to do. For me, the path that systemd has chosen is definitely the wrong one. I understand that you say that it unifies different functionalities with a single interface. Yes, that's indeed sometimes a very good idea to reduce complexity, especially when there is some kind of standardization involved. However, I am not yet convinced that (a) the operations in question NEED to be unified; and (b) that this kind of (almost proprietary, non-extensible) interface which is offered by systemd is the approach I prefer.
Unless Poettering et al. stop being ignorant towards other software solutions, I don't see systemd being a part of the future of Linux.

Another daemon for managing control groups

Posted Dec 8, 2013 0:09 UTC (Sun) by anselm (subscriber, #2796) [Link] (96 responses)

However, I am not yet convinced that (a) the operations in question NEED to be unified; and (b) that this kind of (almost proprietary, non-extensible) interface which is offered by systemd is the approach I prefer.

The operations don't »need« to be unified – we've been limping along with the non-unified hack-type approach for almost 30 years or so. However, if you want to call something »almost proprietary, non-extensible« then that should surely apply to the old (SysV init + friends) setup rather than systemd. The way you »extend« SysV init is by coming up with an entirely new infrastructure to tack on the side. Consider, for example, inetd for socket activation, which like most of these add-ons is not integrated with the rest of the init system at all – to a point where the system administrator needs to personally make sure, by simultaneously tweaking two completely different configuration systems, that a service is either launched via inetd or else via the runlevel system (and of course inetd has no notion of runlevels). This is horrible software design. Systemd, on the other hand, seems to cover pretty much all the bases in one single infrastructure and configuration scheme already (so to change a service from running permanently to being socket-activated you only need to tweak one single configuration file), and there is no reason to assume that, being free software, it could not be extended to handle any additional service management tasks as they arise, with comparatively less effort and a more manageable result.

SysV init is also »proprietary« in the sense that every distribution essentially had to do their own thing when it came to things like init scripts. Systemd, on the other hand, makes it easier to provide unit files with upstream packages so there can be greater standardisation between distributions and less distribution-specific work during packaging. And it still supports SysV-style init scripts so it can do anything SysV init can do.

Unless Poettering et al. stop being ignorant towards other software solutions, I don't see systemd being a part of the future of Linux.

It turns out that Poettering et al. did look at the »other software solutions« in this space and found them deficient. Systemd does incorporate many of the features found in other init replacements and builds on them.

Finally, I don't think you get to decide what becomes »a part of the future of Linux« and what doesn't. Judging from what most of the high-profile Linux distributions are doing with systemd these days, I believe it is fairly safe to assume that in a few years from now we will look back and wonder what all the fuss around systemd was about, much like we look back today and wonder what all the fuss around X.org, or glibc2, or ELF binaries was about when those were new.

Another daemon for managing control groups

Posted Dec 8, 2013 4:30 UTC (Sun) by cas (guest, #52554) [Link] (89 responses)

> The way you »extend« SysV init is by coming up with an entirely new
> infrastructure to tack on the side. Consider, for example, inetd for
> socket activation, which like most of these add-ons is not integrated
> with the rest of the init system at all

that's called modularity. it means I can replace any individual component without affecting any of the others. I can have whatever inetd provider I like. I can have whatever cron daemon I like. I can have whatever system log daemon I like, and whatever cgroups manager I like, and so on and so on.

and each of those modules can evolve independently of each other, with different implementations of each all striving to be the best of their kind (or, at least, to provide their own unique trade-off of features vs benefits). remember: one size does not fit all.

Any one of these modular components can be replaced with different - generally improved - versions at any time, for any reason.

just as importantly, I can experiment with potentially-good replacements without committing to them forever...I can try them and if it turns out that they suck for some reason (i.e they don't meet my particular needs), then I can easily revert back to what I was using before (or trial some other substitute).

I have benefited from this kind of modularity many times over the years. I wish to continue doing so in future, so have no intention of switching to some mononlithic monstrosity like systemd.

systemd is the death of innovation. it has its tentacles in way too many pies, and once you switch to it you're stuck with it - any replacement would have to do *everything* that systemd does and (even if such a replacement ever existed) you'd have to replace it all at once rather than piecemeal/by-module. the result of that will be that there will *never* be a viable replacement for it, and the more functions that systemd assimilates over time, the less likely it will be that any replacement ever appears.

Another daemon for managing control groups

Posted Dec 8, 2013 12:56 UTC (Sun) by anselm (subscriber, #2796) [Link] (14 responses)

that's called modularity

That's called trying to defend an obsolete hodgepodge of barely cooperating pieces of software – which is limited in its potential, error-prone, and difficult to explain to newcomers.

»Modularity« might be an actual asset if the current system had been designed that way, which it hasn't. The current system has been pulled together from a variety of sources with little thought given to its interaction, maintainability, and teachability. It is easy to replace individual pieces of it only because the pieces don't really talk to one another, which means that a lot of useful functionality that would be both very convenient to have and reasonably straightforward to implement (as shown by systemd) simply does not get implemented at all.

Your »modularity« argument amounts to claiming that cars don't make sense because in your horse-and-buggy setup, which has several centuries of tradition behind it and allows you to travel the two miles from the farm to the village in a mere half hour, you can replace the engine and the passenger compartment independently of each other.

systemd is the death of innovation. it has its tentacles in way too many pies, and once you switch to it you're stuck with it - any replacement would have to do *everything* that systemd does

I presume this is why you use something like GNU Hurd rather than Linux. The Linux kernel, after all, has its tentacles in way too many pies, and once you switch to it you're stuck with it – any replacement would have to do everything that the Linux kernel does. Which is why no innovation ever takes place inside the Linux kernel.

Systemd is really quite modular once you look past the source tarball. Its PID 1 mostly does the things it does because it makes technical sense to put them into PID 1, not because it is supposed to do everything – lots of systemd's functionality resides in separate executables that can be dealt with individually, and their interfaces are, on the whole, documented way better than those of the traditional system. It is also quite possible to replace parts of systemd with your own code even now if you think you can do better than the systemd developers.

By contrast, with the traditional setup many distributions have been forced to come up with their own solutions for things that stock SysV init and friends simply do not do to begin with, especially when it comes to configuration and management, and that leads only to widespread incompatibility and the re-invention of the (square) wheel rather than (desirable) modularity.

Another daemon for managing control groups

Posted Dec 8, 2013 20:32 UTC (Sun) by cas (guest, #52554) [Link] (13 responses)

> »Modularity« might be an actual asset if the current system had been
> designed that way, which it hasn't.

except that it was - it's a large part of the point of the "small tools" approach.

but don't let facts get in the way of your dumb argument.

> I presume this is why you use something like GNU Hurd rather than Linux

ah, yes. the single most cogent and popular argument of systemd pushers - "you like modularity therefore you must hate linux and love microkernels".

as someone else in this thread so eloquently put it, the only answer to that is "fuck off". and take your moronic straw-man with you.

> Systemd is really quite modular once you [...]

systemd is really quite modular once you redefine "modular" enough so that a monolithic monstrosity qualifies. combine that with the systemd-favourity argument that "modularity is stupid and who wants it anyway?" and you get insanity.

Another daemon for managing control groups

Posted Dec 8, 2013 20:54 UTC (Sun) by anselm (subscriber, #2796) [Link] (10 responses)

except that it was - it's a large part of the point of the "small tools" approach.

Yeah right. You know inetd comes from 4.3BSD and SysV init from – wait for it – AT&T's Unix System V? No wonder they don't talk to each other and are configured completely differently from each other. Great design.

systemd is really quite modular once you redefine "modular" enough so that a monolithic monstrosity qualifies. combine that with the systemd-favourity argument that "modularity is stupid and who wants it anyway?" and you get insanity.

Check out myth #1 at The Biggest Myths, where Lennart Poettering says

1. Myth: systemd is monolithic.
If you build systemd with all configuration options enabled you will build 69 individual binaries. These binaries all serve different tasks, and are neatly separated for a number of reasons. For example, we designed systemd with security in mind, hence most daemons run at minimal privileges (using kernel capabilities, for example) and are responsible for very specific tasks only, to minimize their security surface and impact. Also, systemd parallelizes the boot more than any prior solution. This parallization happens by running more processes in parallel. Thus it is essential that systemd is nicely split up into many binaries and thus processes. In fact, many of these binaries are separated out so nicely that they are very useful outside of systemd, too.
A package involving 69 individual binaries can hardly be called monolithic. What is different from prior solutions however, is that we ship more components in a single tarball, and maintain them upstream in a single repository with a unified release cycle.

Who's putting up straw men now?

Another daemon for managing control groups

Posted Dec 10, 2013 12:34 UTC (Tue) by dgm (subscriber, #49227) [Link] (5 responses)

>> A package involving 69 individual binaries can hardly be called monolithic.

It can. Modularity is not a question of the number of pieces, but of the coupling between them.

AFAIK, systemd modules are not "replaceable" in any sensible way, because they are highly coupled, their interfaces undefined and continuously changing. But I may be wrong.

Another daemon for managing control groups

Posted Dec 10, 2013 13:01 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

There is a page on the systemd site which lists the interfaces systemd adheres to between its components. It has been linked from this thread at least twice (search for "Promise").

Another daemon for managing control groups

Posted Dec 10, 2013 13:21 UTC (Tue) by mchapman (subscriber, #66589) [Link]

> AFAIK, systemd modules are not "replaceable" in any sensible way, because they are highly coupled, their interfaces undefined and continuously changing. But I may be wrong.

I think the big problem here is that people are talking about (at least) two different kinds of "modularity".

There have been comments claiming that too much is being put into PID 1. The "69 individual binaries" comment should be a good counterclaim to that -- the majority of those binaries live in /usr/lib/systemd/ and are separate from PID 1 specifically because their functionality doesn't need to run in PID 1.

That's one kind of modularity. The other is "replaceability". Here I would have to agree with you: most of those /usr/lib/systemd/ binaries are not really that replaceable. (I'm not sure I'd say the interfaces are "undefined and continuously changing", though -- that makes no sense from a maintainability point of view, and at least the ones I poked directly on my system all responded nicely to --help). Perhaps this is a concern to some people... but I'm not sure what the alternative might be. They're only supposed to be used internally by systemd, so replacing them with functional equivalents wouldn't change anything.

Another daemon for managing control groups

Posted Dec 10, 2013 13:34 UTC (Tue) by HelloWorld (guest, #56129) [Link]

> AFAIK, systemd modules are not "replaceable" in any sensible way, because they are highly coupled, their interfaces undefined and continuously changing. But I may be wrong.
You are, there are lots of interfaces that can be reimplemented independently, and most of the systemd interfaces are fully documented.
http://www.freedesktop.org/wiki/Software/systemd/Interfac...

Another daemon for managing control groups

Posted Dec 10, 2013 16:55 UTC (Tue) by khim (subscriber, #9252) [Link] (1 responses)

AFAIK, systemd modules are not "replaceable" in any sensible way, because they are highly coupled, their interfaces undefined and continuously changing. But I may be wrong.

You are partially wrong (interfaces are defined and described here). But more important is somewhat relevant question: why do you care?

It's true that systemd components are separated into a bunch of small programs designed to work together… just like UNIX was always developed. It's true that systemd components are not designed to be easily replaceable… just like UNIX components were never designed to be easily replaceable. It's kept as one single (and relatively large) source codebase… just like which was delivered as large code dumps on a few tapes.

Basically the story with systemd always goes in the same circle again and again. Somewhat like this:

Detractor: Systemd developers totally don't understand “Unix Way” because they are doing A, B and C!
Supporter: Well, systemd does X, Y and Z—why it's not enough.
Detractor: If systemd does not do A, B and C then it's against “Unix Way”, it's an abomination to be stopped at any cost. Only if you do A, B and C you can call yourself “true Unix believer”
Old beard: I was there at the beginning and I can vouch that Unix was never about A, B or C.
Detractor: You, the doited idiot, you've made Unix, but you totally don't understand it. Your Unix never did A, B and C but of course it does not matter: if you don't do A, B or C then you are doing things wrong!

When you define something which Unix never did, something which it never planned to do, something which only some other systems ever did as as “true Unix Way” then it's kind of hard to discuss your arguments rationally.

Unix components were never intended to be used on other platforms. Many components were ported to other platforms or rewritten in compatible way (GNU project is most famous for that), but it was not something UNIX creators ever cared about and it was not something they planned for.

Yes, components had more-or-less clean interfaces, but UNIX man was designed to always usee UNIX troff and it was never designed to use groff, UNIX cc was designed to be used with UNIX as and UNIX ld, it was not compatible with other tools (indeed even GNU toolchain sometimes is forced to use native components on some systems because they are tied to tightly to other components). So why, just why systemd should be designed differently? Because some people who don't plan to use systemd anyway desire that? Makes no sense to me.

Another daemon for managing control groups

Posted Dec 11, 2013 16:59 UTC (Wed) by dgm (subscriber, #49227) [Link]

> You are partially wrong (interfaces are defined and described here).

It happens to me all the time. Oh, well.

Another daemon for managing control groups

Posted Dec 11, 2013 22:15 UTC (Wed) by nix (subscriber, #2304) [Link] (3 responses)

Yeah. systemd is not monolithic, except that if cooperation with someone else to use a common library is asked for, well, they can just sod off. Wonderful. (Mind you, given Lennart's ability to antagonize other people I can see why he might want to avoid cooperating with people in such areas. He'd never be able to keep it up.)

Another daemon for managing control groups

Posted Dec 12, 2013 14:02 UTC (Thu) by HelloWorld (guest, #56129) [Link] (2 responses)

> Yeah. systemd is not monolithic, except that if cooperation with someone else to use a common library is asked for, well, they can just sod off.
So systemd is "monolithic" because they refuse to use a library that a) doesn't exist as of yet and b) doesn't buy them anything?

Another daemon for managing control groups

Posted Dec 14, 2013 19:39 UTC (Sat) by nix (subscriber, #2304) [Link] (1 responses)

It's not that they refuse to use it. They refuse to even discuss design of such a library, and say that if one were to exist they would ignore it.

Another daemon for managing control groups

Posted Dec 14, 2013 21:18 UTC (Sat) by raven667 (subscriber, #5198) [Link]

What exactly would the abstraction layer be that such a library could provide, what code would be in it and what code would be needed to interface with it? This is pretty low level, if it can't effectively encapsulate complexity then there isn't any point. I'm not an expert but my guess is that for someone who is familiar with the systemd code base and the kernel interface they have an idea of what abstractions are possible and how useful they could be and decided that the cost/benefit isn't there to encapsulate this logic in a library.

How is this possible abstraction different than the dbus interface these tools are planning to provide?

Another daemon for managing control groups

Posted Dec 14, 2013 13:42 UTC (Sat) by kleptog (subscriber, #1183) [Link] (1 responses)

The thing is, there is no actual good reason why init, inetd, cron and at are separate processes with incompatable config files and features. They all start processes, just for slightly different triggers.

init doesn't let you specify a user to start as. cron can in /etc/crontab, but not in /etc/cron.d. cron and at capture output and send it via email, the others don't have a proviso for log output. cron allows configuration of environment variables, init doesn't.

None of them support ulimits, none of them support cgroups, There are no moves in that direction either.

And most importantly, none of them actually *manage* the processes they start. For example, you find the list of currently running cronjobs using ps, why? sysvint doesn't notice if a process dies or fails to start. For an actual production system you end up using something like daemontools, which despite having a lot less code than init actually ends up being more useful (especially the helper programs setuigid, envuidgid, setlock, envdir, softlimit which really should be standard stuff). ISTM people praising the Unix-way should be using daemontools as init, since it's modularity is much better than init's.

So in my opinion it's good that someone has picked up the ball and replaced all these mutually incompatible programs with a uniform system that allows me to configure everything about starting a process uniformly and manage them in a uniform way. Now if support for containers/umask/ulimits/chroots/etc is added it works for all kinds of processes, not just for the init/inetd/cron/at daemon that happens to have gotten support for it.

As for the journal, in daemontools you use multilog to replace syslog and logrotate in one go. All systemd has to do is something better than multilog (which isn't hard) and I think you have a winner.

So as far as I can see, systemd is a win for people who want to use their systems to do real work.

Another daemon for managing control groups

Posted Dec 14, 2013 21:02 UTC (Sat) by jubal (subscriber, #67202) [Link]

So as far as I can see, systemd is a win for people who want to use their systems to do real work.

That's a bold statement. (In both general and the Irish meaning.)

Another daemon for managing control groups

Posted Dec 8, 2013 13:13 UTC (Sun) by pizza (subscriber, #46) [Link] (7 responses)

> and each of those modules can evolve independently of each other, with different implementations of each all striving to be the best of their kind (or, at least, to provide their own unique trade-off of features vs benefits). remember: one size does not fit all.

The problem with your argument is that it presumes that any of the alternatives are worth using -- Especially when you factor in the very real opportunity/maintainence/support cost of continuing to support alternatives with drastically fewer capabilities. You're forced to write everything to the least common denominator because you can't assume anything about anything else, and you're strongly discouraged from trying anything new because, well, it's "different."

Meanwhile, in the real world, modularity has always fallen away in favor of tight integration. For example, cars no longer let you bolt in any engine the manufacturer makes. They now have a single integrated IVI system that encompasses the radio, HVAC, and telemetry information. They are less maintainable and modifiable than ever, but in exchange for that they are by far the safest, most reliable and most performant that they have ever been.

Closer to home, computers have shed most of their modular expandability with the mass shift to laptops, and even laptops with their modicum of modularity are rapidly falling to a totally integrated tablet. OSes too have shed most of their modularity in favor of integration.

Modularity used to be a necessity to get anything done, now it mostly just adds disproportionate amounts of complexity for its little (real-world) gain. This is the same reason we use pre-packaged Linux distros/installers rather than building everything completely from scratch each time we do an install on a new system...

Another daemon for managing control groups

Posted Dec 8, 2013 13:22 UTC (Sun) by Jandar (subscriber, #85683) [Link] (6 responses)

> For example, cars no longer let you bolt in any engine the manufacturer makes.

Car manufacturers are an example of increasing modularity. Where a few years ago every model had its own varying parts nowadays the same parts are used in numerous models.

Another daemon for managing control groups

Posted Dec 8, 2013 13:44 UTC (Sun) by pizza (subscriber, #46) [Link] (5 responses)

> Car manufacturers are an example of increasing modularity. Where a few years ago every model had its own varying parts nowadays the same parts are used in numerous models.

That's not *you* swapping parts willy-nilly, it's the manufacturer simplifying their designs to save themselves money at the time of design and manufacture.

For a typical modern car, you have no choice whatsoever when it comes to what core components you can use.

Try swapping out the fuel injection system. Or transmission. Or the engine. Or bolt on a supercharger/turbocharger. Or try to change the front-end suspension setup. Or raise the ground clearance. Or add a pickup bed.

Unless the manufacturer specifically sold that combination as an option, you're in for a serious world of hurt. Because modern cars are only modular when it comes to their manufacture, not their end-user.

Another daemon for managing control groups

Posted Dec 8, 2013 14:13 UTC (Sun) by Jandar (subscriber, #85683) [Link]

When was the last time where you could swap the engine between any chevrolet and any bmw?

Another daemon for managing control groups

Posted Dec 8, 2013 23:42 UTC (Sun) by dlang (guest, #313) [Link]

you must not be aware of the automotive aftermarket.

all the things that you talk about can be done, some brands are easier to work on than others, but there is a good supply of superchargers that you can add to your car, along with everything else.

Another daemon for managing control groups

Posted Jan 3, 2014 17:34 UTC (Fri) by aigarius (subscriber, #7329) [Link] (2 responses)

Right, and car manufacturers in this analogy are distributions and system administrators of large installations. Systemd is not flexible enough as designed. I want to replace the cron-equivalent with another tool, I want to use a different logind, I want to use a different cgroups manager.
Currently systemd attitude is that to get there I would need to recompile systemd with some custom options and possibly patch the code if I want different cgroups functionality.
That is not going to fly.
Systemd may expose some special functionality when run with its own cron, but to be a viable replacement it must be able to co-exist with the existing tools.
It could be as simple as wrapper-binaries - if you want proper permissions and monitoring for processes started from ye odle crond, you just add "systemd-start" to the beggining of the command line in your crontab.

The arrogance in refusing all the previous and competing solutions because "we've already considered it all and found it lacking" is a honking huge red flag for the health of the community.

Even if systemd was the greatest thing since sliced bread technically it would have to be forked and removed from the current development community before the rest of us could start becoming comfortable in depending on that.

Another daemon for managing control groups

Posted Jan 3, 2014 17:43 UTC (Fri) by andresfreund (subscriber, #69562) [Link]

Uh. And what exactly stops you from using vixie cron or your own logind implementation?

The case with cgroups is a bit different, but that's not really systemd's fault - the kernel/Tejun wants to move towards a single hierarchy, not systemd. Since systemd relies on controlling the cgroup hierarchy it cannot easily use a separate cgroup controlling daemon without having pid 1 rely on other processes without similar protections as pid 1 has from the kernel.

If there's things to critize about system's cgroup handling it's that they've built an API to control cgroups in a relatively short time which they are bound to continue to provide. Since it has to be generic enough to a) be implementable by others b) not block any future cgroup usage that makes me a bit wary.

Another daemon for managing control groups

Posted Jan 3, 2014 18:08 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

>Right, and car manufacturers in this analogy are distributions and system administrators of large installations. Systemd is not flexible enough as designed. I want to replace the cron-equivalent with another tool, I want to use a different logind, I want to use a different cgroups manager.

Nothing stops you from running your own cron implementation, syslog implementation or eschewing logind entirely. Systemd simply provides a lot of very convenient utilities but they are not mandatory.

The only problem is in the boneheaded single-writer cgroups hierarchy. But that's not entirely a systemd's failure.

Another daemon for managing control groups

Posted Dec 8, 2013 13:16 UTC (Sun) by Jandar (subscriber, #85683) [Link] (65 responses)

> the result of that will be that there will *never* be a viable replacement for it, and the more functions that systemd assimilates over time, the less likely it will be that any replacement ever appears.

That's called "vendor lock-in". It's almost everywhere undesired except by the systemd folks. Clean interfaces enabling change of components seem to be only wanted by backwoodsman who can't find the ONE TRUE way to enlightenment.

One of the greatest advantages of Linux is the freedom from vendor lock-in. I value my freedom to tinker far above a few seconds on every (seldom) boot, so unless systemd can limit itself to a replacement of init instead of being an all-embracing tentacle-monster it's banned from my systems.

If it's true that now gnome requires systemd it's also unfeasible.

Another daemon for managing control groups

Posted Dec 8, 2013 13:22 UTC (Sun) by pizza (subscriber, #46) [Link] (63 responses)

> That's called "vendor lock-in". It's almost everywhere undesired except by the systemd folks. Clean interfaces enabling change of components seem to be only wanted by backwoodsman who can't find the ONE TRUE way to enlightenment.

I'm sorry, no.

You cannot have "vendor lock-in" when you have the full source code to the *entire* system. You are just as free to tinker with systemd as you are with any other Free Software system.

Unless, of course, by "lock-in" you mean "the consequences of making a choice" which apply no matter if you chose A, B, or Q.

Another daemon for managing control groups

Posted Dec 8, 2013 13:40 UTC (Sun) by Jandar (subscriber, #85683) [Link] (62 responses)

> You cannot have "vendor lock-in" when you have the full source code to the *entire* system.

The problem is that systemd is the *entire system*. Systemd is an all or nothing approach.

I can't stand gnome but there are some programs from it I use with my kde desktop instead of the kde natives. This works fine because there is a standard for desktop communication.

There is an analogous standard for exchanging parts of the systemd *entire system*: fuck off.

Another daemon for managing control groups

Posted Dec 8, 2013 13:52 UTC (Sun) by anselm (subscriber, #2796) [Link]

There is an analogous standard for exchanging parts of the systemd *entire system*: fuck off.

I think this statement could use some fact-checking. See, for example, the systemd Interface Portability and Stability Chart, which documents various components of systemd (it's not one single big program) and their interfaces.

It would be quite possible to use some of systemd's components in different contexts without also making systemd PID 1, or to swap them out for different components while adhering to the published interfaces. This doesn't exactly look like an »all or nothing approach«, even though the systemd developers seem to attempt to offer a toolkit that covers most use cases by default.

Another daemon for managing control groups

Posted Dec 8, 2013 15:27 UTC (Sun) by mpr22 (subscriber, #60784) [Link] (60 responses)

The problem is that systemd is the *entire system*.

systemd is not an text editor, interactive MUA, C (or LISP / Fortran / Pascal / insert-your-compiled-language-here) compiler, interactive command-line interpeter, interactive IRC client, Perl (or Python / Ruby / Lua / insert-your-dynamic-language-here) interpreter, assembler, linker, window manager, media player, web browser, PDF viewer, or any of a myriad of other interactive or user-invoked-batch tools I would need or want my desktop/workstation Linux systems to have.

Neither is it a web server, MTA, news server, IRC daemon, or any of a myriad of other things I might need or want my Linux servers to have.

In short: I cannot do useful work with nothing but a Linux kernel and systemd, so in what sense, then, is it the "entire system"?

Another daemon for managing control groups

Posted Dec 8, 2013 20:15 UTC (Sun) by Jandar (subscriber, #85683) [Link] (39 responses)

Once upon a time systemd wasn't udevd or cgroupsd. Where is the boundary?

Another daemon for managing control groups

Posted Dec 8, 2013 20:41 UTC (Sun) by anselm (subscriber, #2796) [Link] (38 responses)

Systemd still isn't udevd. Udevd is a separate program which can be used without systemd – it is just part of the same source tarball.

It does make technical sense for systemd's PID 1 process to be in charge of cgroups. See, for example, this message where Lennart Poettering explains why this is a reasonable idea.

Another daemon for managing control groups

Posted Dec 9, 2013 11:39 UTC (Mon) by jubal (subscriber, #67202) [Link] (37 responses)

Are you sure we're reading the same text? Let me quote Lennart's explanation why this is a reasonable idea:

> a single-agent, we should make a kick-ass implementation that is
> flexible and scalable, and full-featured enough to not require
> divergence at the lowest layer of the stack.  Then build systemd on
> top of that. Let systemd offer more features and policies and
> "semantic" APIs.

Well, what if systemd is already kick-ass? I mean, if you have a problem 
with systemd, then that's your own problem, but I really don't think why 
I should bother?

I for sure am not going to make the PID 1 a client of another daemon. 
That's just wrong. If you have a daemon that is both conceptually the 
manager of another service and the client of that other service, then 
that's bad design and you will easily run into deadlocks and such. Just 
think about it: if you have some external daemon for managing cgroups, 
and you need cgroups for running external daemons, how are you going to 
start the external daemon for managing cgroups? Sure, you can hack 
around this, make that daemon special, and magic, and stuff -- or you 
can just not do such nonsense. There's no reason to repeat the fuckup 
that cgroup became in kernelspace a second time, but this time in 
userspace, with multiple manager daemons all with different and slightly 
incompatible definitions what a unit to manage actualy is...

We want to run fewer, simpler things on our systems, we want to reuse as 
much of the code as we can. You don't achieve that by running yet 
another daemon that does worse what systemd can anyway do simpler, 
easier and better.

The least you could grant us is to have a look at the final APIs we will 
have to offer before you already imply that systemd cannot be a valid 
implementation of any API people could ever agree on.

This is a reply to Tim Hockin's:

If systemd is the only upstream implementation of this single-agent
idea, we will have to invent our own, and continue to diverge rather
than converge.  I think that, if we are going to pursue this model of
a single-agent, we should make a kick-ass implementation that is
flexible and scalable, and full-featured enough to not require
divergence at the lowest layer of the stack.  Then build systemd on
top of that. Let systemd offer more features and policies and
"semantic" APIs.

We will build our own semantic APIs that are, necessarily, different
from systemd.  But we can all use the same low-level mechanism.

Note the usual non-confrontational Lennart's style of discussing obvious things and his typical respect to other participants in the discussion

Another daemon for managing control groups

Posted Dec 9, 2013 18:46 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link] (36 responses)

Note the usual non-confrontational Lennart's style of discussing obvious things and his typical respect to other participants in the discussion

If you want to dispute what he says, please address the technical point he goes into at some length. Either he has a valid point, in which case you need to address it technically, or he doesn't, in which case you can refute it. But as long as you spend your time ignoring his technical arguments and complaining that he's abrasive, people are left to conclude that you can't justify your objections to systemd and are just hating on it because of who designed it.

Another daemon for managing control groups

Posted Dec 9, 2013 21:49 UTC (Mon) by jspaleta (subscriber, #50639) [Link] (30 responses)

Its also instructive to go back and read the full cgroups list thread from the beginning to put the very short back and forth between Tim and Lennart into context. There is a ton of discussion well before Lennart pops in

Here's the deal... everybody except the systemd developers basically ignored the fact that the new "sane" cgroups interface roadmap for a year plus. systemd responded to the kernel dev side call and put effort into it. Everybody else basically just put their head in the sand for a year... and now that systemd has the first implementation meant to work with the new "sane" interface, the whole concept of having a single hierarchy with a dedicated manager is coming under fire.

I still don't see why the api systemd exposes is not adequate. I'm not saying that is is or is not. I'm saying I haven't seen anyone put effort into actually pointing out how its inadequate. I see a lot of discussion about why enforcing a single heirarchy is inadequate. I see a lot of discussion about how certain kernel-side changes enforced by "sane_behavior" cgroupfs mount option are going to require working userspace code. But an actual technical deficiency in the systemd abstraction to work with "sane_behavior" kernel development work..I haven't seen it.

And it still not clear to me that the new cgmanager implementation being spun up right not is going to meet the requirements imposed by the sane_behavior. I've taken a look at the cgmanager code, and I'm not convinced that its "sane_behavior" compatible at present. I think its relying on mechanisms marked as insane (for example it appears to me that its manipulating the tasks object, whcih I think was marked insane as of july in upstream patch commit)....if I'm reading the archived cgroups mailinglist discussion correctly.

I really need to stress this point. I do not think the alternative implementation consortium, that is pinning its hopes on cgmanager as a usable "sane_behavior" compatible manager implementation, is fully on the same page with where kernel devs are going with the sane_behavior work. There is a wealth of discussion archived in the cgroups mailinglist through all of 2013 concerning trying to mark functionality as sane or insane which anyone interested should go back and read up on. cgmanager may be a single hierarchy solution, but I'm not convinced that its going to meet the strictures of the "sane_behavior" mount options, of which a unified hierarchy is just one of the requirements.

I might be missing something significant, but it really seems like there continues to be a disconnect between what kernel side is doing to create the sane behavior and how cgmanager is being developed right now.

Another daemon for managing control groups

Posted Dec 9, 2013 22:14 UTC (Mon) by anselm (subscriber, #2796) [Link] (29 responses)

I still don't see why the api systemd exposes is not adequate. I'm not saying that is is or is not. I'm saying I haven't seen anyone put effort into actually pointing out how its inadequate.

That seems to be a motif in most discussions around systemd. Many people are quick to point out how (a) they think Lennart Poettering is a jerk, and (b) systemd is not what they're used to and they don't like it – but ask them to come up with something constructive and all you get is stunned silence.

Another daemon for managing control groups

Posted Dec 9, 2013 23:08 UTC (Mon) by khim (subscriber, #9252) [Link] (28 responses)

Note that here we have even worse situation.

Think about it. Kernel developers said that they could not offer API which is safe to use from untrusted code. Then they said that it's impossible to offer API which is safe to use from trusted code unless it's used by a single process. That is why they are planning to impose this change on a userspace.

Fine. Now cgmanager come along and basically proclaim: hey, kernel developers are lazy and stupid. They claim that it's impossible to offer sane cgroups API to userspace if it's supposed to be used by more then one process—but we'll just quickly go and cobble one together in hurry!

Now, I can easily imagine that they will solve their problem for some special case (note how Tim Hockin clearly says that we don't use udev (yet?) so we don't have these problems) and they will be able to cobble together some solution which will mostly work—if you are not doing “anything bad”. But if they will be able to invent something usable for general case? I highly doubt it. Why? It's easy: kernel developers can not do that. And kernel has access to all the knobs userspace can access and also to tons of other, userspace-invisible knobs and also can change kernel if it's really needed—and yet they could not do what cgmanager authors are planning to do.

Now, I don't fully understand why kernel developers insist that only one process must manage cgroups and why this work can not be delegated (even to other trusted processes, not to normal processes created by untrusted users) but if indeed cgmanager authors will be able to solve that issue sanely (which probably mostly means “safely in general case”) then indeed what James says will make perfect sense: If anyone, systemd included, wants to do a new API, it must support all use cases as well. Ideally, it should be agreed to and in the kernel as well rather than having some userspace filter. Yes, this is an ideal outcome and yes, if it's possible to do that then it's obviously must be done, but the whole story with a single controller in the userspace satrted with the supposition that such solution is impossible, remember?

Basically the whole discussion boils down one simple single point: systemd does not need any changes no matter what happens to cgmanager! If, indeed, cgmanager can offer sane, safe and flexible API to other daemons then kernel developers have made serious error in judgment and similar API can be offerend directly by kernel and can be used by systemd. If this endaveour will be found impossible and cgmanager will only manage to cover same corner cases (e.g. only cases where hardware is static and does not appear and disapper will-nilly) then, again, systemd should not be changed: it must work in general case, not just when cgmanager is safe to use!

Of course if cgmanager plans to offer somewhat restricted (and thus actually doable) API and not just a drop-in replacement for what we have now then said API can be added to systemd, but so far it does not look this way. On the contrary it looks like the plan is to offer the exact same API which kernel offers today (with all it's problems) just now as a separate daemon, not a kernel API. This is not a progress!

Another daemon for managing control groups

Posted Dec 10, 2013 2:50 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (27 responses)

>Why? It's easy: kernel developers can not do that
Wrong. They do not WANT to do that.

It's as simple as that. Right now delegation of cgroups to untrusted users _WORKS_ if one limits themselves to fully hierarchic groups like 'memory' or 'cpu'.

The proposed insane_behavior simply does the minimum amount required for SystemD to work and not an inch more.

Another daemon for managing control groups

Posted Dec 10, 2013 3:18 UTC (Tue) by pizza (subscriber, #46) [Link] (26 responses)

> Right now delegation of cgroups to untrusted users _WORKS_ if one limits themselves to fully hierarchic groups like 'memory' or 'cpu'.

Are you, as a kernel developer, willing to take the very massive chance that userspace will limit themselves to such an arrangement?

(If so, that is a very different stance than you typically take when it comes to such things...)

Another daemon for managing control groups

Posted Dec 10, 2013 3:28 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (25 responses)

> Are you, as a kernel developer, willing to take the very massive chance that userspace will limit themselves to such an arrangement?
Does kernel forbid to set SUID bit on /bin/bash? It's the same thing. If Lennart were designing Linux security then he'd rip out SUID bits and create a SuidD that would provide DBUS-based services to start SUID processes.

And trusting userspace to have interest in its own security is OK. For example, one can easily screw the kernel up by granting untrusted and malicious users excessive permissions on /sys. One can easily do "chmod -R a+w /sys", for FSM's sake!

Does it mean that kernel should forbid to change mode and ownership on /sys nodes and lock it down to be accessible only from SystemD? Oh wait, I don't want to give SystemD developers new ideas.

Another daemon for managing control groups

Posted Dec 10, 2013 3:53 UTC (Tue) by pizza (subscriber, #46) [Link] (7 responses)

Wow, way to throw out some strawmen arguments, peppered liberally with more uncalled-for Lennart-bashing. How was that in any way related to what I'd asked? How is *any* of this cgroup API nonsense Lennart's doing?

It's not like he wrote the original cgroup API, pointed out the problems with it, or wrote the new API. systemd was a user of the old API then adapted to kernel-imposed requirements with the new API, not the other way around.

I find it disappointing that you are trying to split hairs here, given how generously you tend to berate others for doing the same.

Another daemon for managing control groups

Posted Dec 10, 2013 4:07 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

> How is *any* of this cgroup API nonsense Lennart's doing?

Stop pretending. Systemd drives cgroup development (Tejun Heo works at RedHat, remember) and Lennart gleefully supported "single-writer" model.

> It's not like he wrote the original cgroup API, pointed out the problems with it, or wrote the new API. systemd was a user of the old API then adapted to kernel-imposed requirements with the new API, not the other way around.

Wrong. Tejun Heo basically redesigned cgroups to better fit SystemD model. Everything else (nesting, delegation) is brushed aside as "impossible" or "insecure" without explanations (really, try to find them - there are none).

And it shows, Lennart basically goes out and says: "if we allow delegation and nesting then we'd have to actually treat cgroups interface as a kernel ABI, and if we don't then we can pretend that systemd is the sole user of cgroups and do whatever we want with it in future".

Now, I really like SystemD and I'd like to see it in Debian. But I totally disdain the behavior of cgroups developers.

Another daemon for managing control groups

Posted Dec 10, 2013 5:47 UTC (Tue) by raven667 (subscriber, #5198) [Link] (1 responses)

Well you've laid out a clear worldview of you you see things, I don't think there is anything that can be added or taken away. I have to add though that I don't see the corruption and negativity that you see so I don't know if that means I'm blind or if you just need to relax. All the best.

Another daemon for managing control groups

Posted Dec 10, 2013 5:53 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

It's not a corruption in the sense of taking bribes or anything. It's just tunnel vision - cgroups and systemd developers simply do only what they need, and nothing more. All the requests to be more reasonable are simply ignored.

I'd very much prefer if systemd and cgroups were developed by two different competing companies. Absent that, I hope that someone with a large enough cluebat makes cgroup developers see the error of their way.

Another daemon for managing control groups

Posted Dec 10, 2013 13:50 UTC (Tue) by corbet (editor, #1) [Link] (3 responses)

Having watched this whole process, I have to disagree a bit. Tejun set in to the task of reworking cgroups to make them more maintainable and usable; the systemd folks found themselves having to react to that change. Systemd was using the multiple hierarchy feature, keeping its own special cgroup hierarchy off to the side. Remember the PaxControlGroups document? See this note from last June (and this one) where systemd's response to the cgroup changes was worked out.

Now, in the process of redesigning things, Tejun has certainly talked to the users of cgroups, including the systemd developers. Doing otherwise would not have been smart. But I do not think it's fair to say that systemd has driven these changes.

Another daemon for managing control groups

Posted Dec 10, 2013 16:12 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

I was referring to these messages as well. They read exactly as if Tejun Heo privately talked to SystemD developers and then systemd and cgroup folks presented the future changes as a fait-accompli with no public discussions.

Another daemon for managing control groups

Posted Dec 10, 2013 17:23 UTC (Tue) by rahulsundaram (subscriber, #21946) [Link] (1 responses)

In other words, your assertion is a guess and you don't really know but the language you use makes it sound almost like a conspiracy. It is misleading at best.

Another daemon for managing control groups

Posted Dec 10, 2013 18:06 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

Well, from my vantage point it definitely looks like cgroups developers are doing only what systemd developers want. It might be accidental, but I really don't care.

Another daemon for managing control groups

Posted Dec 10, 2013 10:49 UTC (Tue) by khim (subscriber, #9252) [Link] (15 responses)

If Lennart were designing Linux security then he'd rip out SUID bits and create a SuidD that would provide DBUS-based services to start SUID processes.

Sure. Setuid was an interesting hack, but in hindsight it's obvious that it created a lot of security problems and gave very few practical benefits. Windows uses central daemons with DBUS-services to impelement such functionality and it works just fine there.

The only big question is how to support backward-compatibility: it may be bigger hassle then keeping setuid bit around.

Another daemon for managing control groups

Posted Dec 10, 2013 16:13 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (13 responses)

Except that any DBUS-based service would get the same troubles, only more complicated. The need to create a privileged process remains, but it's obscured by complex interfaces.

Another daemon for managing control groups

Posted Dec 10, 2013 17:13 UTC (Tue) by khim (subscriber, #9252) [Link] (12 responses)

Really? What's so complicated in the interface which is supposed to just start the program? It just needs to check credentials and do that. It always start applications in pre-determined environent with known starting conditions.

Compare with today's approach where bazillion parts of kernel must know about suid bit (euid vs uid), many libraries need to know about suid bit (euid vs uid), glibc must specifically handle startup of setuid binaries (and there were many exploits around this process), binaries often need special handling if they are supposed to ever run as suid binaries. Sorry, but argument is nor convincing.

Note that even today when suid bit is actually available many programs are not using it and use cetralized-privileged-daemon scheme instead (things like apache, ftp, mysql and other countless daemons). Strange, isn't it?

Sorry, but setuid bit is obviously a mistake. It's not easy to replace setuid bit with a DBUS interface today and perhaps it's not even worth trying (transition pain can easily outweight and potential gain), but the design itself is obviously too complex and too fragile. That's not even worth discussing.

Another daemon for managing control groups

Posted Dec 10, 2013 18:10 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (11 responses)

>Really? What's so complicated in the interface which is supposed to just start the program? It just needs to check credentials and do that. It always start applications in pre-determined environent with known starting conditions.

Because it will have ALL the faults of suid and lots of additional faults of a half-baked userspace implementation. For example, think about signals (especially RT signals and SIGSTOP/SIGKILL). I can kill my SUID program using a straightforward "kill" utility, how would you do this with SuidD?

I'm actually speaking from experience - we have such a daemon in our system. It's simply not possible to replicate all the kernel-level functionality.

SystemD is repeating ALL the problems of this approach. For example, they have to cobble something together to handle delegation to containers while simple bind-mount is enough right now to nest cgroups.

Another daemon for managing control groups

Posted Dec 10, 2013 21:46 UTC (Tue) by khim (subscriber, #9252) [Link] (10 responses)

Because it will have ALL the faults of suid and lots of additional faults of a half-baked userspace implementation.

Really? Which ones?

For example, think about signals (especially RT signals and SIGSTOP/SIGKILL).

What about signals?

I can kill my SUID program using a straightforward "kill" utility

Yup. And it is a problem security-wise.

how would you do this with SuidD?

Most likely answer: you would not be able to do that. You will probably have some high-level knobs but you will not be able to just send random signals to random provileged programs. And this is “good thing”™.

I'm actually speaking from experience - we have such a daemon in our system. It's simply not possible to replicate all the kernel-level functionality.

Of course not! It'll be pointless excercise to just shuffle functionality around. It's the other way around: suid is a problem because it gives you huge amount of rope to tie itself. SuiD will give you much, much smaller amount of rope. Yes, this will also mean that some brain-dead designs will become impossible, but this will just mean that you will need to spend few more time thinking about design of your system upfront. What real-world task are you trying to solve with signals? Why do you think it can only be solved by giving the rights to affect priveleged process on your system from some random shell script?

SystemD is repeating ALL the problems of this approach.

Wow. Thanks for bringing that to my attention. Seriously, no sarcasm. This cgroups ≈ suid analogy really helps to show just why it's bad idea to give access to just some random user to the capabilities of cgroups… but it still does not explain why only one daemon can ever manipulate cgroups. Ok, It needs to be privileged daemon, but it's still not entirely clear to me just why it must be PID 1.

For example, they have to cobble something together to handle delegation to containers while simple bind-mount is enough right now to nest cgroups.

Well, it was always good idea to have daemon which does that thus I'm not sure why you are just now trying “to cobble something together”. The problematic fact is that all these solutions must be tied somehow to systemd, they can not just exist as yet-another-daemon on the side, but this is not systemd's fault, AFAICS it was imposed by kernel side changes.

Another daemon for managing control groups

Posted Dec 10, 2013 22:39 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (7 responses)

>Most likely answer: you would not be able to do that. You will probably have some high-level knobs but you will not be able to just send random signals to random provileged programs. And this is “good thing”™.

And here's the problem - it's "The my way or the highway".

Let's discuss a very simple SUID program - good old 'ping' utility. A user should be able to watch live its output, so some kind of shim utility should be used to transfer standard FDs to the DBUS service. This shim must also be running all the time while the DBUS ping service is running.

So far so good. But now I want to stop the service, so I press Ctrl-C. And nothing happens, unless the shim captures this signal and somehow communicates it to the DBUS service (oh, and don't forget to authenticate the transmission).

So far so good, until I press Ctrl-Z. Whoops. SIGSTOP can't be captured.

And that's without going into the gory details of controlling terminals, ptys and realtime signals (can you say 'priority inversion'?). It doesn't matter that YOU don't like signals - they are de-fact used in the world out there.

But let's go on. Suppose that we have a browser running in a sandbox. Should it be able to access DBUS? Likely. But I definitely don't want it to access the SUID-runner service, while my beloved Tilda should be able to start whatever processes I want. Can you tell me how DBUS services are secured? How can I audit this security? Can I write an AppArmor policy to restrict '/usr/bin/firefox' access to '*cgroup*'?

Oh, and we have this nice Criu project - but it won't be able to checkpoint the DBUS-based service (it can't checkpoint only one end of a Unix socket).

And we can leave out minor details like confusing 'ps' output.

In the end, the DBUS-based solution is going to be an inferior and unreliable construct. And that's exactly what is happening with SystemD and cgroups right now. They are building an inferior wrapper on top of a kernel interface, that's in itself WORSE than the status quo.

>Wow. Thanks for bringing that to my attention. Seriously, no sarcasm. This cgroups ≈ suid analogy really helps to show just why it's bad idea to give access to just some random user to the capabilities of cgroups…
Yes, probably there are several tight spots in the cgroups API that might give users too much capabilities to harm the system. But so does /sys, /proc and namespaces - yet all of them are accessible to users.

Another daemon for managing control groups

Posted Dec 11, 2013 0:35 UTC (Wed) by khim (subscriber, #9252) [Link] (3 responses)

And that's without going into the gory details of controlling terminals, ptys and realtime signals (can you say 'priority inversion'?). It doesn't matter that YOU don't like signals - they are de-fact used in the world out there.

It's not about signals. It's about system design. Any time indirection goes from unprivileged process to privileged one it must be accounted and cotrolled. It's really hard to do with setuid approach and most programs don't bother to do that.

Let's discuss a very simple SUID program - good old 'ping' utility.

Sure, let's do that. Consider the fact that said utility plays with very low-level stuff and can easily hurt not just your system but also neigbhoring systems. Let's see if we can actually do that:
$ ping -f www.google.com
PING www.google.com (74.125.143.106) 56(84) bytes of data.
ping: cannot flood; minimal interval, allowed for user, is 200ms

Wow! Lookie: there are a protection! But does it actually work? Of course not: you can still run 1000 ping's in parallel and this will have basically the same effect.

In most cases what you need it something similar to tcptraceroute -f 30 -q 10 (which works without any special permissions), anyway.

So far so good. But now I want to stop the service, so I press Ctrl-C. And nothing happens, unless the shim captures this signal and somehow communicates it to the DBUS service (oh, and don't forget to authenticate the transmission).

So far so good, until I press Ctrl-Z. Whoops. SIGSTOP can't be captured.

Which is good because it's NOT good idea to do something to highly privileged process behind it's back. Actual priveleged ping process may notice that shim is no longer responding and will probably stop doing it's work. That's fine, don't see anything wrong with that.

Basically you are explaining why current [broken] interface is hard to replicate with SuiD deamon. That's fine, I agree with you: it's really hard to replace it with anything sane and perhaps it's not ever a good idea to try to do that right now. It still does not mean that it was good idea to build it in this form initially.

But let's go on. Suppose that we have a browser running in a sandbox. Should it be able to access DBUS? Likely. But I definitely don't want it to access the SUID-runner service, while my beloved Tilda should be able to start whatever processes I want.

How is it different from /proc or /sys access?

Yes, probably there are several tight spots in the cgroups API that might give users too much capabilities to harm the system. But so does /sys, /proc and namespaces - yet all of them are accessible to users.

And that's why we must assume that any process started under any user yet with full access to all syscalls and /proc and /sys it having root access more or less automatically. It's basically impossible to make Linux kernel secure because it's attack surface is so wast.

Looks like people are really starting to think about it, but it's hard to change everything at once thus they are starting from most recent piece of the puzzle (which can be changed without affecting too many users yet). I'm just not sure what they are planning to do after that: sure, they will secure one tiny pice of the whole, but how exactly it'll help if everything else will remain in the same hodge-podge-with-bazillion-security-holes state?

Another daemon for managing control groups

Posted Dec 11, 2013 0:51 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

>It's not about signals. It's about system design. Any time indirection goes from unprivileged process to privileged one it must be accounted and cotrolled.
Welcome to Linux audit framework (that nobody uses, but it's there).

>Sure, let's do that. Consider the fact that said utility plays with very low-level stuff and can easily hurt not just your system but also neigbhoring systems.
Irrelevant. This particular warning is obsolete, anyway. I can just as well flood the network with UDP datagrams.

>Which is good because it's NOT good idea to do something to highly privileged process behind it's back.
Nope. It's a good idea, because SUID processes are specifically meant to interact with users. And SIGSTOP is one of the well-known ways to interact.

>Actual priveleged ping process may notice that shim is no longer responding and will probably stop doing it's work. That's fine, don't see anything wrong with that.
So there should be a heartbeat service? What about power consumption (all those spurious wakeups)?

You're digging hole even deeper.

>> But let's go on. Suppose that we have a browser running in a sandbox. Should it be able to access DBUS? Likely. But I definitely don't want it to access the SUID-runner service, while my beloved Tilda should be able to start whatever processes I want.
>How is it different from /proc or /sys access?
That's it - it's not different at all. Except that I have easy to use tools to restrict access to /sys and /proc - AppArmor or SELinux (for masochists). I'm not aware of similar infrastructure for DBUS.

>And that's why we must assume that any process started under any user yet with full access to all syscalls and /proc and /sys it having root access more or less automatically. It's basically impossible to make Linux kernel secure because it's attack surface is so wast.
Container people managed to fix this. It's possible to start a namespaced container with its own view of /proc and /sys with full root access in it and it will be reasonably secure.

And puzzle comparison is apt - for many years full container support was known as 'containers puzzle' (just search LWN). Many people diligently chipped away all the pieces to make full isolation possible. And it's finally there.

Except now cgroups developers say: "It's too complicated for us, we'll just throw in the towel and make it impossible even if it works right now for many users. For their own good."

Another daemon for managing control groups

Posted Dec 11, 2013 10:33 UTC (Wed) by zdzichu (subscriber, #17118) [Link] (1 responses)

It's built-in in D-Bus. See http://dbus.freedesktop.org/doc/dbus-daemon.1.html (search for policy) or content of /etc/dbus-1/ directory.

BTW. the proper spelling is "systemd" (no arbitrary uppercase letters).

Another daemon for managing control groups

Posted Dec 11, 2013 19:38 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

Nope. Can't use this policy to limit access to certain _processes_.

So we have two "solutions" already: polkit and DBUS policies. Which one is it?

Another daemon for managing control groups

Posted Dec 11, 2013 3:41 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

> Can I write an AppArmor policy to restrict '/usr/bin/firefox' access to '*cgroup*'?

There's polkit (which is used to restrict access to APIs such as sleep/hibernate/shutdown) which can be used. I don't know where it acts though (whether at the dbus-daemon level or the receiver making *another* call out to polkit asking for permission.

Another daemon for managing control groups

Posted Dec 14, 2013 12:48 UTC (Sat) by kleptog (subscriber, #1183) [Link] (1 responses)

Is it just me, or is this example flawed:

Let's discuss a very simple SUID program - good old 'ping' utility. A user should be able to watch live its output, so some kind of shim utility should be used to transfer standard FDs to the DBUS service. This shim must also be running all the time while the DBUS ping service is running.
So far so good. But now I want to stop the service, so I press Ctrl-C. And nothing happens, unless the shim captures this signal and somehow communicates it to the DBUS service (oh, and don't forget to authenticate the transmission).

Pressing Ctrl-C is different from sending a SIGINT. Namely, when you press Ctrl-C, the kernel sends a SIGINT to anything using that terminal. I imagine the shim would pass through all necessary file descriptors and hence CTRL-C will work fine.

Would it be weird if you were allowed to Ctrl-C a process, but not be allowed to send it a signal from another terminal?

(Hmm, ping drops back to the normal user after opening the socket, does that mean another process could ptrace it and get access to the socket that way? ptrace block it now, but it is something to consider)

Another daemon for managing control groups

Posted Dec 14, 2013 18:47 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

The shim is using the controlling terminal so it'll get a signal. Not the privileged binary. I checked.

Anyway, you'll still have the problem with SIGSTOP.

Another daemon for managing control groups

Posted Dec 11, 2013 3:36 UTC (Wed) by mathstuf (subscriber, #69389) [Link] (1 responses)

> but it's still not entirely clear to me just why it must be PID 1.

It doesn't *have* to be, just that with systemd, PID 1 is the simplest place to put it since it already dances around with cgroups pretty heavily and IPC to another service to manage cgroups makes that service a Special Snowflake that can't be set up using cgroups (since it would have to answer questions about how to start it before it starts).

Another daemon for managing control groups

Posted Dec 11, 2013 16:15 UTC (Wed) by khim (subscriber, #9252) [Link]

Well, that's the problem: PID 1 needs cgroups to manage services and container manager (presumably well-tested and quite privileged) needs them too. Why could not they both use cgroups? Why one must be client of the other one? That requirement was never properly explained.

Another daemon for managing control groups

Posted Dec 11, 2013 22:08 UTC (Wed) by nix (subscriber, #2304) [Link]

See Ian Jackson's 'userv'. I wish it had got more traction than the "none" it did...

Another daemon for managing control groups

Posted Dec 13, 2013 10:07 UTC (Fri) by gioele (subscriber, #61675) [Link]

> If Lennart were designing Linux security then he'd rip out SUID bits and create a SuidD that would provide DBUS-based services to start SUID processes.

That already exists and is widely used in RedHat and Debian systems: Polkit. https://en.wikipedia.org/wiki/Polkit

Another daemon for managing control groups

Posted Dec 10, 2013 17:59 UTC (Tue) by jubal (subscriber, #67202) [Link] (4 responses)

You might want to read the whole of my comment (yes, it contains two rather long quotes) and judge for yourself.

The italicised note was just a commentary, not a point in itself. The quotes were the point.

Another daemon for managing control groups

Posted Dec 10, 2013 20:58 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link] (3 responses)

I did read your comment, including the long quotes. I think Lennart is making a cogent point about why it makes sense to roll cgroup control into systemd. He might have been more polite about it, but his technical point seems sound. You have not yet addressed that technical point.

Another daemon for managing control groups

Posted Dec 11, 2013 13:27 UTC (Wed) by jubal (subscriber, #67202) [Link] (2 responses)

I don't see any technical point in telling other customers of the cgroups subsystem (and Google is probably the largest one at all, and will be in the foreseeable future) that their only options are to start using systemd or match its behaviour to the iota, just because it's more convenient for the authors of the systemd environment.

Another daemon for managing control groups

Posted Dec 11, 2013 15:23 UTC (Wed) by raven667 (subscriber, #5198) [Link] (1 responses)

What do you mean "match its behavior"? Do you mean having a system with a single writer or do you mean it's DBUS API? If you aren't using systemd as PID 1 and you don't need your userspace cgroup management client portable for other peoples who do run systemd then what systemd does or doesn't do is of no consequence to you, right? If anything you need to work with the kernel developers to make sure you understand their concerns and they understand your use cases. As has been pointed out many times the idea of a single userspace cgroups manager was an idea the kernel team had to remove cgroupfs as an attack surface for untrusted customer containers, they only wanted cgroupfs to provide the mechanism for changing settings and not also complicating the internals by encoding the security policy in the kernel.

Another daemon for managing control groups

Posted Dec 11, 2013 16:10 UTC (Wed) by jubal (subscriber, #67202) [Link]

That's grand and I don't think we're really in disagreement.

You may note I was referring to an old e-mail exchange (June, I believe), where Tim Hockin proposed to use a low-level library to provide basic cgroup management functions, a library that would be then used by both systemd and any non-systemd cgroup manager.

This approach was, as you may see, rejected by Mr. Poettering et co. as not viable and possibly non-constructive.

Another daemon for managing control groups

Posted Dec 8, 2013 20:36 UTC (Sun) by cas (guest, #52554) [Link] (19 responses)

> systemd is not an text editor, interactive MUA [...]
> Neither is it a web server, MTA, news server, IRC daemon, or any of
> a myriad of other things

not yet, give it time, be patient. you can't expect it to have *everything* in only a few years.

but it's well on the way, it's already assimilated logging, and there is no credible argument at all for that to be part of init or PID1.

Another daemon for managing control groups

Posted Dec 8, 2013 20:50 UTC (Sun) by mpr22 (subscriber, #60784) [Link]

And funnily enough, the systemd-journald executable does not run as PID 1 and is not a symlink to the systemd executable.

Another daemon for managing control groups

Posted Dec 8, 2013 21:25 UTC (Sun) by anselm (subscriber, #2796) [Link] (17 responses)

Systemd's logging functionality is neither part of init nor PID1. The journald process is separate from the init process.

What systemd's init process does as far as logging is concerned is arrange for output from a service process's stderr channel to go to journald (if the service process doesn't talk to journald directly either via the syslogd interface or else journald's native C library interface). This is (a) very convenient for the authors of service processes, and (b) hardly a difficult or problematic thing to do in principle. Thus it is probably worth the (little) additional complexity in PID 1.

Another daemon for managing control groups

Posted Dec 10, 2013 2:13 UTC (Tue) by cas (guest, #52554) [Link] (16 responses)

you say that as if it makes any practical difference at all. it doesn't.

can i run systemd as an init system *without* journald? no. systemd requires journald. it does not matter whether it forks a separate process or not.

and it does not help at all that systemd pushers say "just run *another* log daemon *as well as* (NOT instead of) journald if you don't like it". here's a free clue: someone saying "i don't want to run journald" is NOT saying "i want to run journald PLUS something else".

at best, if i play with the config, i can tell it not to store any log data on disk - but it's still running.

ditto with logind - WTF is the init system messing with logins? this is not something that an init system needs to do. it's just more systemd borg assimilation of everything.

and it's already got a half-arsed systemd/cron - systemd is the borg, it will assimilate everything.

so yeah, i fully expect systemd to have a mandatory web server within a few years (and if you're some kind of idiot that doesn't like it, just run something else on a high port and configure systemd-web to port-forward). and an irc server or text editor and whatever else lennartix feels like borging.

systemd probably wont have a C compiler because Pottering will no doubt invent his own super special language that is so much better than every other language ever that he'll just have to force everyone else to use it by making it mandatory for systemd. after all, he knows better than everyone else, so it's only fair that he gets to make that choice.

Another daemon for managing control groups

Posted Dec 10, 2013 2:33 UTC (Tue) by HelloWorld (guest, #56129) [Link]

> and it does not help at all that systemd pushers say "just run *another* log daemon *as well as* (NOT instead of) journald if you don't like it". here's a free clue: someone saying "i don't want to run journald" is NOT saying "i want to run journald PLUS something else".
And here's another free clue: journald can be considered an implementation detail as long as /var/log/journal doesn't exist. And complaining about implementation details usually doesn't make you look smart.

> ditto with logind - WTF is the init system messing with logins? this is not something that an init system needs to do. it's just more systemd borg assimilation of everything.
logind is completely optional.

The rest of your comment is so utterly stupid trolling that it doesn't deserve an answer.

Another daemon for managing control groups

Posted Dec 10, 2013 2:34 UTC (Tue) by dashesy (guest, #74652) [Link] (11 responses)

someone saying "i don't want to run journald"

Is analogous to "i don't want to eat my apple". It is like whining about kernel threads being running without being manually spawned, even though they do not hurt the performance or anything. Only if one has OCD, should care to control every single process running on a system.
Try it, it is one of the great things that happened to Linux and it is good for you.

Another daemon for managing control groups

Posted Dec 10, 2013 2:47 UTC (Tue) by cas (guest, #52554) [Link] (2 responses)

and here, along with the "response" by HelloWorld is a major part of the problem with systemd advocates.

*EVERY* *SINGLE* *OBJECTION* *TO* *ANYTHING* *ABOUT* *SYSTEMD* is dismissed with exactly the same unjustifiably arrogant 'we know better than you so just shut the fuck up and get with the program' response.

oddly enough, this is not likely to inspire any kind of trust or confidence.

worse, you lie. you say "oh, that's optional". except that it isn't optional. you can't disable journald. and if you try to disable logind or any of the other parts of systemd, you will break something - with a good chance of fucking up systemd so completely that it will fail to boot.

that does not meet the definition of "optional".

Another daemon for managing control groups

Posted Dec 10, 2013 3:10 UTC (Tue) by pizza (subscriber, #46) [Link]

> oddly enough, this is not likely to inspire any kind of trust or confidence.

Oddly enough, neither are raising objections that are plainly not supported by facts and have been debunked ad-nauseum.

Another daemon for managing control groups

Posted Jan 3, 2014 20:41 UTC (Fri) by rodgerd (guest, #58896) [Link]

I think before you go off on another sweary rant about the quality of discussion you might want to consider raising your own.

Another daemon for managing control groups

Posted Dec 10, 2013 7:33 UTC (Tue) by anselm (subscriber, #2796) [Link] (7 responses)

Try it, it is one of the great things that happened to Linux and it is good for you.

Doesn't matter. What matters is that it (a) was developed by Lennart Poettering, and (b) is not like syslogd, so must be bad – who cares if it comes with all sorts of compatibility features?

It's not as if these people could come up with a technical argument why systemd's journal (or indeed systemd in general) isn't a reasonable idea overall if their life depended on it. Bashing the very concept on general principles is their thing, and it is probably best to ignore them until they manage to find something to say that has actual substance.

Another daemon for managing control groups

Posted Dec 10, 2013 7:44 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

There is an argument to be made about too tight dependency on journald.

Systemd overall is a great idea, devil is in the details. As usual. And one of these small details is boneheaded attitude towards cgroups sharing with other actors.

Another daemon for managing control groups

Posted Jan 3, 2014 20:43 UTC (Fri) by rodgerd (guest, #58896) [Link] (5 responses)

There's a reasonable objection to be made about binary logging formats - system recovery and general analysis can be vastly more painful with such. It's a toss-up, because in many day-to-day cases journald's filtering is a win. But it's hardly an invalid complaint, unless you're considering "how we use logs to do our job" to be a non-technical complain, which would seem to be stretching.

Another daemon for managing control groups

Posted Jan 3, 2014 20:50 UTC (Fri) by raven667 (subscriber, #5198) [Link] (2 responses)

This is a reasonable place to do a risk analysis, how difficult is it to run the tool that can handle the file format of journald on a system that may not be working 100%, how often are the local logs not going to be sent via syslog to a central location? I don't think the designers of this would have done it this way unless they were convinced this was still a win but it's a discussion that can be had.

Another daemon for managing control groups

Posted Jan 3, 2014 21:53 UTC (Fri) by dlang (guest, #313) [Link] (1 responses)

rsyslog can get the logs from journald very rapidly, pretty close to real-time.

the designers of journald didn't have any idea what rsyslog, syslog-ng and similar logging daemons were able to do at the time they started journald, their justification of journald is full of "syslog can't do this" arguments that were true of traditional syslog, but not true on any of the modern syslog daemons.

journald works pretty well for a single machine personal system, but it was built in ignorance of how logging works in larger environments.

Another daemon for managing control groups

Posted Jan 3, 2014 22:06 UTC (Fri) by raven667 (subscriber, #5198) [Link]

> journald works pretty well for a single machine personal system, but it was built in ignorance of how logging works in larger environments.

Maybe you are right but it seems that when the systemd developers do something they do a ton of research before hand before committing resources. In any event it is pretty plain and stated that the journal isn't even trying to deal with networked logging or larger environments, it's main use case is the single personal machine, and capturing logs from early-boot which are normally lost. You don't judge a fish by how well it can climb trees.

Another daemon for managing control groups

Posted Jan 3, 2014 21:50 UTC (Fri) by dlang (guest, #313) [Link] (1 responses)

I would believe this more if there hadn't been a bug that prevented you from accessing the journald binary logs that went undetected for several months until it made it into a Fedora release that included rsyslog reading the files and going into a endless loop doing so (the journald tools also went into an endless loop, so you can't just call this a rsyslog bug)

This tells me that the people working on this are not actually using these tools enough.

Another daemon for managing control groups

Posted Jan 3, 2014 21:54 UTC (Fri) by mchapman (subscriber, #66589) [Link]

> This tells me that the people working on this are not actually using these tools enough.

That is as ludicrous as saying any long-standing bug in the kernel shows that the kernel developers don't use Linux enough.

Bugs happen. People fix them (or at least, *should* fix them) when they discover them, not before.

Another daemon for managing control groups

Posted Dec 10, 2013 13:35 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (2 responses)

I think you're missing some things. Whether willfully or because you inspire knee-jerk reactions which snowball away from the original assertions or whatever, I'll avoid passing judgement for now, but we'll see how your reply goes.

First, yes, systemd provides a PID 1 binary which acts as init. It's not like there aren't other alternatives to sysv init out there (OpenRC, upstart, SMF, and more), so this is nothing particularly new.

Second, systemd has a goal of bringing a single *interface* (if you want to replace components, feel free[1]) to managing a Linux *system*. They also bring an implementation of the interface, but that's easily understood as using an interface before freezing it is rarely a bad thing. Managing a system includes a lot of things these days, like it or not. From hardware (udev) to users (multiseat is supported only in logind/systemd AFAIK and requires hardware awareness), these things all need to be coordinated by *something*. So where do you propose that be done? Systemd developers have laid out interfaces for you to adhere to if you want to replace individual components at will. They choose to do it in (or near, more often) PID 1. You would probably want it to be some service, but then you have to tie it with PID 1 anyways for dealing with services which trigger on hardware or network events anyways. Not to mention some FD passing to PID 1 to pass to the service. That's a lot of code that can be avoided if your aim is to minimize what PID 1 does (and still meet the use cases).

Third, for the (limited) cron features, PID 1 knows these things already (when a service started, when it last finished, etc.) and best. You're introducing more code by writing a daemon which does the same thing because now you have to reach from to watch services dying and showing up as well. Either way is more code, but now I can say "check email every 5 minutes" (as a user-level .timer unit) based on when I log in, not the clock values. What would a crontab entry which does that look like?

Last, you do like strawmen (e.g., I see no indication of systemd gaining an IRC server, but a link could help here) and name calling (Borg). If you could keep your arguments technical in nature, maybe you'd have a clearer discussion with others about your issues.

What it sounds like you want to do is remove components entirely, ignoring the use cases they cover because you don't care for them or can't see why they might be important. Try reading the rationales behind the components. Lennart is very clear when it comes to that. As an example, journald is there because syslog *is not available* at early boot. Would you rather PID 1 to start caching log messages (meaning more code), then swamp syslog as soon as it gets to that point? Or stick with the "no logs before syslog" behavior of today (or maybe that should be yesterday; systemd solved this over a year ago)? I guess you could delay everything until after syslog starts, but then you are sticking a delay in where nothing can be parallelized.

[1]Go ahead and implement the journald interface which just forwards stuff to syslog if you want. The interface has to be there, not systemd's implementation. But of course, journald already does that and by default usually I might add (journald persistence wasn't default in Fedora for a while, but probably will if/when rsyslog is kicked out of @base).

Another daemon for managing control groups

Posted Dec 11, 2013 22:18 UTC (Wed) by nix (subscriber, #2304) [Link] (1 responses)

Would you rather PID 1 to start caching log messages (meaning more code), then swamp syslog as soon as it gets to that point?

If your syslogd cannot cope with a splurge of on-boot-time log messages, your syslogd is unbelievably awful (and would never be used).

Straw man.

Another daemon for managing control groups

Posted Dec 11, 2013 22:59 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

I'll grant that a decent syslog should be able handle the initial logs, but that still means PID 1 is caching things.

Another daemon for managing control groups

Posted Jan 3, 2014 20:31 UTC (Fri) by rodgerd (guest, #58896) [Link]

There is a lot more "vendor lock-in" in a situation where every Linux vendor implements their userland in a special snowflake way such that it requires significant effort to move from one distro to another.

systemd reduces vendor lock-in.

Another daemon for managing control groups

Posted Dec 8, 2013 10:55 UTC (Sun) by tdalman (guest, #41971) [Link] (5 responses)

I agree that we need a modern replacement for SysV init. However, I strongly disagree about systemd being that replacement. That said, I (and probably others as well) will stick to SysV init until an appropriate init solution is available.

Another daemon for managing control groups

Posted Dec 8, 2013 13:05 UTC (Sun) by anselm (subscriber, #2796) [Link]

I (and probably others as well) will stick to SysV init until an appropriate init solution is available.

That's your undisputed privilege (but don't hold your breath).

In the meantime the rest of the world – including leading Linux distributions and other software projects – can at their own peril use, improve, and build on systemd. Sounds fair to me.

Another daemon for managing control groups

Posted Dec 8, 2013 13:16 UTC (Sun) by pizza (subscriber, #46) [Link] (3 responses)

> I agree that we need a modern replacement for SysV init. However, I strongly disagree about systemd being that replacement. That said, I (and probably others as well) will stick to SysV init until an appropriate init solution is available.

I have a (completely serious) question.

What would a "modern replacement" for SysV init look like? How would it be any different than existing SysV init?

Another daemon for managing control groups

Posted Dec 9, 2013 1:51 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (2 responses)

I guess a related question would be: why do AIX/Solaris/FreeBSD/OS X/etc. all use different systems other than SysV init (or if they do, why are none of them using the same configuration or even runlevel meanings)? What use cases were they looking for that SysV init couldn't satisfy?

Another daemon for managing control groups

Posted Dec 10, 2013 17:06 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

FreeBSD's system is not wildly more capable than SysV. It just is different, mostly because of historical reasons. And there've been no compelling reasons to switch to SysV, so the split persists.

OS X initially used FreeBSD's init, but then developed launchd with additional functionality (like socket activation and declarative configuration).

Ditto for Solaris - it had a usual runlevel-based init that was replaced by SMF, which also replaced inittab and inetd.

Another daemon for managing control groups

Posted Dec 10, 2013 17:18 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

I wasn't focusing on the capabilities of those systems, more on that SysV is not the end-all-be-all of init systems to begin with. Basically, if SysV is so good that even systemd and all it brings isn't enough to provide enough "compelling reasons to switch", what would? I was more aiming at answering the "modern SysV" feature set by suggesting to look around to see what else is offered by other (non-obsolete and in-use) systems for a guide.

Different visions

Posted Dec 5, 2013 11:54 UTC (Thu) by cesarb (subscriber, #6266) [Link] (1 responses)

Perhaps the problem is that there seems to be two different visions as to PID 1's role.

For some people, PID 1 is the "initialization daemon", which loads the rest of the system and then stays out of the way.

For other people, PID 1 is the "system manager", which is responsible for managing the whole system. To these people, PID 1 being on the udev system bus and managing control groups for other processes makes complete sense.

Different visions

Posted Dec 5, 2013 14:09 UTC (Thu) by khim (subscriber, #9252) [Link]

Perhaps the problem is that there seems to be two different visions as to PID 1's role.

Indeed. Except it's not two “visions”. It's “wishfull thinking” vs “reality”.

For some people, PID 1 is the "initialization daemon", which loads the rest of the system and then stays out of the way.

Yup. Simple. Easy to understand. Impossible to implement on UNIX-like system including Linux.

For other people, PID 1 is the "system manager", which is responsible for managing the whole system.

In UNIX, for better or for worse, PID 1 is “system manager” because it's “parent of all orphans”. That's why initial vision of init included such management. Remember inittab? Unfortunately it was found limiting and instead of fining it people started introducing kludges upon kludges upon kludges.

To these people, PID 1 being on the udev system bus and managing control groups for other processes makes complete sense.

Yup. These are modern day replacements to venerable /etc/inittab and HUP signal. Nothing more, nothing less.

Systemd includes other components, too, but usually when these componets are simpler than interface needed to put them in separate programs.

Another daemon for managing control groups

Posted Dec 5, 2013 12:37 UTC (Thu) by ovitters (guest, #27950) [Link] (1 responses)

Systemd is meant to do one thing and do it well: be the basic building block for Linux. A lot of people have other incomplete components which lack certain features which they want to mix and match with systemd. So one has as scope "basic building block" while the other has "only do X and take various other bits from Y, Z, etc".

If the thing someones building partly wants to replace systemd while not replacing systemd but wanting to reuse bits of systemd, IMO you're not doing it well. pushing complexity on another project while claiming vague things like "Unix Philosophy".

Systemd usually provides d-bus interfaces as well as various other APIs. Just implement that! Complaining that you cannot take random bits or calling this "problematic" is not helpful.

Anyway, working software: systemd has been used by many distribution for various years. Please either explain why systemd is not working or keep it to yourself.

Another daemon for managing control groups

Posted Dec 6, 2013 23:29 UTC (Fri) by nix (subscriber, #2304) [Link]

Systemd is meant to do one thing and do it well: be the basic building block for Linux.

You complain about excessive vagueness right after saying that?!

OK, I'll pile the whole OS into one process then. It does one thing and does it well: lets me use my computer!

I think it uncontroversial that that statement is an example of bad reasoning through excessive vagueness. So too is yours. It does not provide clear limits on the functionality of the program: you could pile anything in there and say, oh, it's part of being a basic building block! Why not throw out glibc and replace it with IPC calls to systemd? glibc is just a basic building block, after all!

This is clearly ludicrous -- but your description of systemd's alleged purpose would seem to encourage it. If this really is the best idea systemd's authors have of its purpose, no wonder the thing has experienced scope creep like I've never seen outside government IT projects.

Another daemon for managing control groups

Posted Dec 5, 2013 13:26 UTC (Thu) by mezcalero (subscriber, #45103) [Link] (4 responses)

How service management is connected to device and cgroup management? That's pretty easy:

When you start services at boot-up, then you need to start them in the right order, possibly waiting for devices to show up first. Here's an example: for each block device that needs to be mounted, we first need to wait until it shows up (this is because device management is entirely hotpluggy and dynamic these days), then we need to run the "fsck" service on it. Then we need to mount it. Finally, after we did that for all file systems listed in /etc/fstab we can go on to the next step, and start the next service, for example apache, which expects all those file systems to have shown up, fscked, and mounted. So there you have the connection between device and service management.

Now, if you start services like apache and mysql, then in a large part of cases you want to resource manage them. Actually, without even configuring something explicitly it is highly preferable if apache and mysql get roughly the same amount of resources, even though apache runs tons of worker and CGI processes, and mysql only very few worker threads. Because if you don't even that out then apache might starve mysql to death even though it needs its service. Sooo. there you have the connection between service and cgroup management: for the services you start you need to set up cgroups and contain the service processes in them, racefreely, and unconditionally. So there you have the connection between cgroup and service management.

In addition there's also the third connection: between device and cgroup management. That's because for a number of cgroup properties you need to configure major/minor device ids. For example the "blkio" and "devices" controllers rely on that to set resource or access constraints per device. Now, major/minor assignment by the kernel is pretty much dynamic these days. Basically, before a device has shown up you cannot know which major/minor it will get. A cgroup manager which gets told to configure a 5MB/s bandwidth limit for a specific cgroup for a specific device hence needs to refresh the list of limits in the cgroup each time any device in that list pops up or is removed. Since admins probably prefer specifiying their devices with the stable per-uuid or per-path symlinks that udev creates the cgroup manager hence needs to be integrated with udev.

So there you have it: there's a connection between all three areas, in all directions.

Another daemon for managing control groups

Posted Jan 3, 2014 18:08 UTC (Fri) by aigarius (subscriber, #7329) [Link] (3 responses)

None of these require these things to be bundled together within a single binary or project.
Init systems have communicated with udev for startup for years before systemd came along. It is trivial and elegant - you start the udev and then start a service to mount/fsck all the filesystems. This service might do the work itself or just ask udev if its done the work - none of that matters to the init. Init only needs to know when the filesystems are ready for the start of launching of other services, which is communicated by the script exiting with exit code 0.
Init system does not have to include a cgroup manager to start services in their appropriate cgroups, you can trivially provide an equivalent of cgroups-launch wrapper that would communicate with a cgroups daemon and ensure that cgroup is switched before launching the secondary executable. Init system might provide a handy shorthand, for example that if a CGRoup option is specified for a service, then it will be launched via a cgroups-launch wrapper with the CGroup contents passed on to that launcher as a parameter. Trivial, decoupled, just as functional.
And cgroups daemon can trivially provide hooks for the udev to call when devices are added or removed in the usual way. There is no need to invent some tight coupling.

It seems that systemd developers not only "look at all that was before and found it lacking", but also at that point stopped looking and thinking outside the box that they put themselves into. Refusing cooperation with other projects and instead trying embrace-and-extend them is a huge red flag in free software communities. That never ends well.

Another daemon for managing control groups

Posted Jan 3, 2014 18:44 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

You don't get it.

In the new world order only ONE process will be able to create/modify cgroups. That's going to be a kernel requirement.

This in turn requires cgroups management daemon to be constantly active.

Another daemon for managing control groups

Posted Jan 3, 2014 19:13 UTC (Fri) by dlang (guest, #313) [Link] (1 responses)

> In the new world order only ONE process will be able to create/modify cgroups. That's going to be a kernel requirement.

only for the short term until they figure out how to allow for wider access. (and even then the old way will remain for compatibility for a while)

It also would't the only time some kernel developers have said that there was going to be a kernel requirement that they later back away from before enforcing it.

so it's very possible that there will never be a kernel release that will require only having one process manage cgroups

Another daemon for managing control groups

Posted Jan 3, 2014 20:26 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

> only for the short term until they figure out how to allow for wider access.
I haven't seen any indications of that.

And that's actually my sticking point with cgroups - I think that cgroups nesting and delegation is a must.

>(and even then the old way will remain for compatibility for a while)
Yeah, sure. Without any new features.

Another daemon for managing control groups

Posted Dec 5, 2013 13:11 UTC (Thu) by mezcalero (subscriber, #45103) [Link] (5 responses)

Jake, "forcing anyone who wants to use cgroups to use systemd clearly isn't"? Really? You think we are "forcing" anything? That's not a very smart thing to say, you know.

How are we forcing anything? People can write as many cgroup controller daemons as they wish. All we do is provide a sane API for cgroups, that you have if you use systemd. That's all. People can use a different init system, and can do whatever they want.

All we do is say that on systemd systems PID 1 is the cgroup manager. And that's because there can only be one, because we need it heavily in PID 1 and since resource management and service management is to 95% the same thing also the lightweightest option.

You know, complaining that PID 1 does cgroup management, and not allowing multiple cgroup managers run side by side is like complaining that you cannot have two libs provide /usr/lib64/libc.so, or that there can only be one PID 1 which gets SIGCHLD for orphan kids, or only one process that gets C-A-Del, and so on...

Another daemon for managing control groups

Posted Dec 5, 2013 15:09 UTC (Thu) by jake (editor, #205) [Link] (4 responses)

> Jake, "forcing anyone who wants to use cgroups to use systemd
> clearly isn't"? Really?

Well, Lennart, you provide no alternative and choose to ignore (and not work with) any alternatives. If you can say with a straight face that you do not want *everyone* to use systemd, I would be shocked. And you are doing everything in your power to make that happen. Perhaps that's not force, but it sure is pretty far down that path.

Sorry you didn't like that statement.

jake

Another daemon for managing control groups

Posted Dec 5, 2013 16:24 UTC (Thu) by mezcalero (subscriber, #45103) [Link] (3 responses)

I certainly don't want "everyone" to use systemd. All I am trying to do is cover the major usecases from embedded, to mobile, to desktop, to servers, with a general purpose system manager. If people find our offering compelling, then that's great. If they don't that's fine too by me and I don't care much.

There are lots of cases where you don't need systemd. For example, Google runs on its servers very static images, that know no hotplugging, that have a relatively static cgroup tree and so on. For them, having something that deals with all the dynamic bits like systemd is far less interesting than for the general audience. And that's totally fine.

Why for heaven's sake should I "provide" a second cgroup manager implementation? I only care for systemd. If people don't want systemd, they can use something else, but why for heaven's sake should *I* provide them the cgroup manager for that? My interest is making systemd good, and minimal, and integrated. I am certainly not compromising that in major ways, just to solve problems for people who hate anyway what we are working on? That's pointless, we cannot win that anyway.

If you say we don't work with any alternatives, then a) there acurrently are no alternatives, your story is about code that doesn't exist, about nothing but plans, that haven't even been thought to the end. And b) the work to make this all function in systemd was relatively simple and small, and just makes use of concepts that exist anyway in systemd, like the execution queue, like service management, the dependency tree or device management. We hence got it all done in little time, in in a very nicely integrated uniform way. You can list your cgroups ("scopes" and "slices") next to your services, you can influence them with the same tools, introspect them, and everything. If instead you rip all that out, do it in some separate project, that would be totally new and cross-init system, just to "work with alternatives", then we'd be nowhere, and our code would be much worse and way more complex. So yeah, my priority is quality and simplicity of code and the uniformity and usefulness of the user interface. That's what matters most for me. I will not compromise on that just so that you don't write about us that we "don't work with alternatives".

What you are writing here is not very smart. You are suggesting that I want to make life hard of people who don't want to use systemd. But you are confusing something here: I just *don't care* about non-systemd cases, and I don't have to. I don't care, not because I was evil, just because it's not interesting for my usecase. And I do not actively do anything against alternative impelemtnations of anything (how can I), I am just interested in my own code.

If others don't use systemd and use something else, then that's their choice and *they* have to come up with an alternative implementation, that's certainly not my duty, even if you claim it was.

Over the history of systemd we designed some things which needed little integration with PID 1 in a way that you can split them out and use them outside of systemd. For example, udev, tmpfiles, and hostnamed are components like that. Others OTOH need more integration with the rest of the system and you cannot just isolate them out. Which ones those are are explicitly documented in much detail here: http://www.freedesktop.org/wiki/Software/systemd/Interfac... -- we did that as a service to the people who don't use systemd as PID1. How nice of us! You should appreciate that. It's a question of honesty to list explicitly what is separable and what is not, and so we did that. If something is not separable, then that's due to technical reasons, that's all. And you should grant us that. SO much respect is necessary.

Anyway, Jake, I really don't appreciate what you are writing here. You shouldn't imply that we had ill intentions or when we don't. We don't force anyone to anything. We make an offer, we even make it easy to take only parts of the offer. It's all free. We aren't the only coders around, so if you don't like something, pick something else.

Jake, not cool. Not cool at all.

CGroups manager API

Posted Dec 5, 2013 17:27 UTC (Thu) by rvfh (guest, #31018) [Link] (2 responses)

Lennart,

On a slightly different note, what is your take on trying to have the same API as cgroupsmanager? Did they indeed contact you? Would you be happy to find a common ground?

CGroups manager API

Posted Dec 5, 2013 23:02 UTC (Thu) by drag (guest, #31333) [Link] (1 responses)

> On a slightly different note, what is your take on trying to have the same API as cgroupsmanager

It seems that systemd and cgroupsmanager have distinctly different goals in managing cgroups.

Systemd is looking to provide a high level API that piggy backs on the process management features that already exist and are being used by systemd.

It appears from the posts that I read that cgroupsmanager is looking to expose most of the the functionality of cgroupfs via dbus because the kernel developers are going to depreciate the current cgroupfs API and are discouraging having multiple processes from being able to manage hierarchical cgroups in the future.

It's the difference between trying to make something secure and easy to manage versus something that wants to allow as much flexibility as possible.

CGroups manager API

Posted Dec 6, 2013 1:46 UTC (Fri) by raven667 (subscriber, #5198) [Link]

I think thats a good summary, the question is whether there is value in systemd also offering the low level interface like cgroupsmanager intends, in addition to the high level interface, or would that break systemd service management if it exposed all the low level details? What kind of clients are expected to be using either of these interfaces, is this for virtualization management agents to setup containers across a shared cluster for customer jobs? Can this use case be accomplished with the kind of interface that systemd is planning to export or does it require the more low level interface that cgroupsmanager intends to export, or can the intended use cases for the dbus interface be accomplished with either system?

It would probably be suboptimal if an application need would create hard dependancies up the stack all the way to the init system such that certain workloads fundamentally can't be run on systems with incompatible init systems. I would prefer to pick the best init system (which I think is systemd) and not lose the ability to run whatever services I want.

Another daemon for managing control groups

Posted Dec 5, 2013 16:31 UTC (Thu) by raven667 (subscriber, #5198) [Link] (10 responses)

While this seems to have turned into the systemd thread with mezcalero trying to answer for every misconception people have, the article is about cgmanager and I think we've lost sight of the most important conclusion from the article which I agree with, it would be a better world if there was some level of agreement and standardization on the Control Group manager DBUS interface. I think that at the very least the core functions should use the same API even if the more advanced features are in systemd and cgmanager vendor specific interfaces. A userspace application that needs to interface with the Control Group manager shouldn't have to have two completely different implementations of the DBUS client depending on what OS its running on. All of the other systemd features are pretty agnostic and don't require such specific application support, this one feature is different than logind vs. ConsoleKit or other choices because Control Groups are so core to the modern Linux system, so I think that there is reason to take a different approach than "love it or leave it" for this one feature.

I mean, how different could the interfaces of systemd vs. cgmanager possibly be as they ultimately need the same information and are manipulating the same cgroupfs.

Another daemon for managing control groups

Posted Dec 5, 2013 16:43 UTC (Thu) by mezcalero (subscriber, #45103) [Link] (9 responses)

The cgroup APIs systemd exposes are described here:

http://www.freedesktop.org/wiki/Software/systemd/ControlG...

We introduced higher-level concepts of "slices" and "scopes" which will map to cgroup sin the end, but in more complex ways. For example, there's propagation of features between siblings and from child to parent, there are high-level properties which have an effect on much more than just one property of one cgroup. Not even the low-level naming of things is exposed, we have to escape some names in order not to clash with the names of the kernel-provided cgroup attributes. The APIs we expose hence are very high-level and they make use of a lot of concepts that systemd already knows like the execution queue, units or dependencies between units. It ultimately hides a lot of complexity with a simple interface (heck, we only added a *single* new bus call to systemd to cover the cgroup usescases!). OTOH the stuff the Ubuntu folks are working on goes in a completely different direction: they are much lower-level and closer to what the kernel provides itself. Trying to find an abstraction that works for both is not realistic. Sorry.

Another daemon for managing control groups

Posted Dec 5, 2013 17:28 UTC (Thu) by rvfh (guest, #31018) [Link] (8 responses)

Ah this is the answer to my question, sorry. Sad that you could not find a common ground though, but I understand.

Another daemon for managing control groups

Posted Dec 5, 2013 18:28 UTC (Thu) by jspaleta (subscriber, #50639) [Link] (7 responses)

I'm actually having a hard time finding a publicly archived discussion over a deficiency in the systemd API. I've even gone back to the lkml cgroup status quo thread starting Mayish 2013.

I'm not seeing an on-the-record deficiency or summary analysis of why the systemd cgroup management API is unusable.

It'll be interesting to see how different the alternative spec ends up being. But so far its just a draft spec its its literally half a year of inaction on even starting to draft an alternative spec.

It's still not clear to me that the lxc/ubuntu/google contigency have an understanding of the security issues manifest in multiple manager approach. Tejun has tried to explain the problem several times now in the archived discussion channels. First in the lkml and in the lxc-devel discussion..kickstarted by the vUDS session. It's just not clear to me that the leads in the alternative management effort understand why a single manager is needed at all. Nor do I think perhaps everyone in that camp gets how deeply dynamic device management can be now in very real world situations.

It sort of feels like they've been blindsided a bit...hit by the accelerating bus of kernel development which, last time they looked up to watch its progress, appeared to be miles away and moving slow. They put their head back down and..blam...runover as its picked up speed while they were not paying attention.

Don't fall into the trap on focusing on Lennart's contributions to the discussion. The real meat of the discussion is the discussion between Tejun, Tim and Serge as it spans the workman-devel list lkml in May/June and now in the lxc-devel list. The real discussion is the back and forth there as Tejun tries to explain why the new kernel API is being stood up. I don't see how in the world systemd and the yet to be built alternative are going to agree on an API, when the team building the alternative can't seem to understand why the kernel side needs to rely on a userspace with single manager (regardless of what that is). If they don't get the reasons, they won't built a solution that solves the security problems and the API will look crazy different as a result.

Honestly, Tejun is bending over backwards to be diplomatic and to try to explain things..repeatedly. It just doesn't seem to have sunk in. Oh well.

-jef

Another daemon for managing control groups

Posted Dec 5, 2013 21:42 UTC (Thu) by jubal (subscriber, #67202) [Link] (6 responses)

Are you trying to say that Tim Hockin does not have any idea what cgroups are and how they should be managed, Jef?

(BTW, Lennart, nice hissy fit both here and on your g+ account, additional ten points for class.)

Another daemon for managing control groups

Posted Dec 5, 2013 22:35 UTC (Thu) by jspaleta (subscriber, #50639) [Link] (2 responses)

No I'm saying he may not fully understand why the cgroup interface is being reworked because Google's internal use case don't need to consider the security issues Tejun is looking to solve by very deliberately cutting out functionality Tim is currently relying on.

The workman-devel list archived communication in June and July between Tejun and Tim really captures the meat of the dispute. Tim was blindsided by the speed at which systemd's implementation of a manager matured to meet the kernel side development of the new interface. And even though Tejun tries to explain why the kernel interface changes are happening, I never get the feeling Tim understands what Tejun is attempting to communicate. Tim's unhappy that the single heiarchy change is going to land on the kernel side. Tim is unhappy that the new kernel interface is taking away flexibility his use case currently relies on. The kernel side change is coming, regardless of systemd's high level implementation decision on how to deal with that inbound kernel change. It's coming. Tim admits Google already has an in-production in-house developed high level API for cgroup management in some form (oh the irony of that!)

The fact that systemd has a mature implementation of a cgroup manager API for the new kernel interface with kernel dev side buy-in is germane to Tim's core concern only because the maturity of that implementation means he potentially has less time to ignore the new kernel interface and to deal with adapting Google's in-house uncollaboratively developed tooling, by spinning up an alternative implementation to meet the requirements of the new cgroups interface. Even though Tejun diffuses this concern to some degree by stating the older, less secure interface will still be there, and Google should be able to continue running as is.

Thought out the entire discussion Tim's concern is still primarily about the collapse of the multi-hierarchy which Google's use case depends on and the requirement of having a central manager. Tim really doesn't care what systemd is doing as a high level API to wrap the kernel interface. Google isn't going to use it. Google is happy with their in-house tools.

There is a lot to chew on in that workman-devel thread... just in the discussion between Tejun and Tim and ignoring all other discussion. I'm left seriously wondering if Tim really understands and appreciates the security issues Tejun is attempting to solve with the new cgroups interface.

Tejun is serious about cleaning up the cgroup interface. Seriously serious.
The more people start pushing back about the proposed kernel side adjustments with details of how it breaks their use case, the more Tejun's concerns make sense to me from a security persepctive. People are doing some crazy crazy things with cgroups.

Another daemon for managing control groups

Posted Dec 5, 2013 22:53 UTC (Thu) by drag (guest, #31333) [Link] (1 responses)

I tried to read the thread and it seems like Tim never explained exactly why they can't do what they want with a single hierarchical structure and Tejun never explained in a understandable way the exact reasons why they cannot maintain a multi-hierarchical interface.

Although you are right that systemd has absolutely nothing to do with it. They'd have the same problem regardless of what method they use to manage cgroups.

Another daemon for managing control groups

Posted Dec 5, 2013 23:23 UTC (Thu) by jspaleta (subscriber, #50639) [Link]

Indeed,

I actually think it was surprising at how quickly systemd spun up a working manager interface. Tejun started this particular ball rolling in Feb/March 2012. LWN covered it. I think a lot of people, who should have been aware of Tejun's plans, just assumed it would take more time for anyone to spin up a working manager implementation for the saner cgroups interface he's spinning up.

The fact systemd that did it this quickly, I think speaks volumes.

The fact that everyone else sat on their hands for a year+ after Tejun's decision also speaks volumes.

Tejun is serious about cleaning up the lawless wild west. I'm not sure enough people took him seriously until systemd landed its public implementation. But really, a year later after Tejun stated his firm intention to move forward and you want to finally start to engage and push back? Meh.

-jef

Another daemon for managing control groups

Posted Dec 6, 2013 1:03 UTC (Fri) by ovitters (guest, #27950) [Link] (2 responses)

(BTW, Lennart, nice hissy fit both here and on your g+ account, additional ten points for class.)

If you don't like someone responding in a certain way, don't be an ass yourself ("nice hissy fit, additional ten points for class").

Another daemon for managing control groups

Posted Dec 6, 2013 14:21 UTC (Fri) by jubal (subscriber, #67202) [Link] (1 responses)

There's a difference between sarcasm and hissy fits.

Another daemon for managing control groups

Posted Dec 10, 2013 9:03 UTC (Tue) by mp (subscriber, #5615) [Link]

Pretty insignificant compared to the similarity in what they do to advance the discussion.

X doesn't have to worry about...

Posted Dec 6, 2013 4:02 UTC (Fri) by smoogen (subscriber, #97) [Link] (1 responses)

Does anyone else get the feeling that anytime a Google/Facebook/Amazon/etc engineer says something that boils down to "We trust everything in our cluster as we control it" someone in the NSA goes "Oh reallly.... " [You can trade out NSA with FSB, GCHQ, etc etc as needed.]

I have the feeling that as more data ends up being encrypted site to site and server to server, interserver exploits will be the area of highest concern and focus for these agencies.

X doesn't have to worry about...

Posted Dec 6, 2013 8:45 UTC (Fri) by khim (subscriber, #9252) [Link]

I'm not sure if you are underestimating or overestimating Google/Facebook/Amazon/etc engineer capabilities.

The primary maxima here is very simple: “if and when one can call arbitrary unrestricted Linux syscalls one can own the system”. Security track record of Linux kernel certainly support this POV (yes, even “hardened” ones had plethora of vulnerabilities exposed and undoubtedly have many more currently uncovered ones).

But if you filter Linux ABI access and not trust Linux kernel (without heavy-duty protections like seccomp-bpf) then why would you care about all these security implications Tejun is talking about?

Another daemon for managing control groups

Posted Dec 6, 2013 18:29 UTC (Fri) by jubal (subscriber, #67202) [Link] (1 responses)

BTW, I think it's worthy to paste here the Tejun Heo's comment from Lennart's exquisite google+ complaint note:

Ooh, some clarifications.

* Yes, I think cgroup should be managed by some entity which is trusted and privileged. There are two big reasons for this.

One is the fact that the whole thing is neither designed or implemented for delegation to untrusted domain. As I've now mentioned multiple times, security is mostly about logistics and details and implementing something with huge and complex interface surface without meticulously auditing every detail and expecting it to be secure is outright stupid. I can assure everyone that delegating cgroup to !priv has always been insecure and AFAICS it will continue to be - you can create vmalloc allocations of unlimited size, you can put processes into undefined hung state which will also drag your debugger into the same state if it tries to attach to it, and so on. Just imagine implementing this same feature with system calls and how much effort we'd have to be putting in in secure design and implementation. cgroup has not done any of it. Expecting it to be securely delegatable is fairy-tale naive.

The second is that delegating to !priv users often leads to each binary growing awareness of cgroup interface and manipulating it directly. This immediately promotes cgroup interface to the status of system calls, which is completely messed up. It becomes a side channel to breach the kernel API design and review process. I don't even understand how something like this has ever been allowed to happen but this has served as an active shortcut for both kernel and userland devs and is disastrous for both in the long term.

I don't know how designed this delegation to untrusted domain “feature” is. I suspect it came about as an accident just because the interface was filesystem based. Anyways, kernel devs have been working assuming the interface is a privileged admin thing and some userland has been happily delegating them out to untrusted domains.

* Serge seems to be mostly on the same page. I don't think there were any fundamental misunderstandings. It was just me being stupid about how chown was being used in its scheme which seems fine now.

* Google folks can speak for themselves but I asked around at the recent conferences and they don't seem to have any fundamental disagreements at this point. All the contention points were quite minor, or they at least seemed that way to me.

Hope this clears up at least some of the misunderstandings. Thanks.

Another daemon for managing control groups

Posted Dec 7, 2013 8:53 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

And let me post my rants from the comments:

I've been using delegation to !priv users with the current cgroup design just fine. It works as expected - I simply give permissions to a cgroup subtree to a user and they can do whatever they want.

I have heard the claim that cgroup delegation is uber-insecure and will eat your pets for lunch about 100^500 times by now. But somehow people always omit the exact details of the horrible gaping security holes. Except for the brain-damaged "starve your siblings" scenario and non-hierarchical cgroups (which are a mis-feature in themselves).

Let's see your technical objections:

1) "Putting processes into an undefined state and fouling up the debugger". Boohoo. So a !priv user can mess with their debugger. Horrible. And totally unprecedented, since ptrace is a nice rock-solid interface that always works flawlessly.

2) Unlimited vmalloc - totally a kernel issue that MUST be fixed anyway.

And your non-technical objections: cgroup ABI must be treated as an ABI. That ALREADY is true, systemd is NOT going to be the single user of the cgroups ABI and so it must be stable.

Next, let's see the requirements for the central cgroups daemon. It must support delegation to other users. At minimum, to support containers running old versions of systemd or other cgroup users.

How can this be done? I envision a way to delegate a subtree of the whole cgroups tree and allow the delegated software to be limited to using it.

So you need to have a way to isolate a part of DBUS tree and make sure only authorized users can access it.

Then there's a question of security - how do I audit cgroups delegation tree? Is there an audit API?

Can I use my favorite AppArmor (systemd is going to fully support AppArmor, right?) policies to make sure that only one process in my cgroups+namespaces based container is allowed to tinker with cgroups? How do I make sure that my browser can't ask for a delegated cgroups tree using this nice systemd-based DBUS interface?

Basically, ALL these issues are solved nicely with the classic Linux filesystem-based interfaces. And trying to solve them another way would just generate a solution isomorphic to a filesystem tree. Except it'll be stupid and buggy.

Another daemon for managing control groups

Posted Dec 16, 2013 23:44 UTC (Mon) by skissane (subscriber, #38675) [Link] (29 responses)

If there are to be two cgroup APIs, systemd and cgmanager, I wonder how hard it would be to create a translation layer between them. For instance, someone could write a patch to systemd to make it speak the cgmanager API too. (Then again, the systemd maintainers might not accept that patch, if they see no value in supporting another API.) Or, someone could write a translation daemon, which exports the cgmanager API, but then calls the systemd API. (Conversely, one might provide a translation layer in the opposite direction, from systemd API to cgmanager API, although an application using the control group parts of the systemd API might also expect to use other systemd functionality that cgmanager will not offer, which might make going in the other direction harder.)

Another daemon for managing control groups

Posted Dec 17, 2013 1:52 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (28 responses)

AIUI, systemd's API is higher level (it works on the service/scope/user level) while cgmanager is closer to a 1:1 of the kernel API. Any translation layer is going to need to implement systemd's logic to DTRT. It's not just a matter of calling a different DBus method.

Another daemon for managing control groups

Posted Dec 17, 2013 4:49 UTC (Tue) by dlang (guest, #313) [Link] (27 responses)

if your description is correct, it should be simple to make systemd use cgmanager, at which point we have one master daemon that manages userspace, and it isn't systemd, that should keep everyone (except the people who think systemd should be in control of everything) happy.

Another daemon for managing control groups

Posted Dec 17, 2013 4:59 UTC (Tue) by rahulsundaram (subscriber, #21946) [Link] (16 responses)

an init system shouldn't depend on another daemon. that doesn't make anybody happy.

Another daemon for managing control groups

Posted Dec 17, 2013 5:20 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (13 responses)

SHOULD is not MUST. An optional ability to use an external manager would do systemd only good.

Another daemon for managing control groups

Posted Dec 17, 2013 5:27 UTC (Tue) by rahulsundaram (subscriber, #21946) [Link] (12 responses)

It is only optional if systemd has its own implementation of similar functionality and at that point, there is even less of a reason to depend on another daemon. external dependencies for a init system must be as minimal as possible. standardize on external interfaces if needed but not a specific implementation itself.

Another daemon for managing control groups

Posted Dec 17, 2013 6:17 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (11 responses)

Right now the whole systemd and single-tree cgroups API is an abomination that should be burned alive and then have its ashes scattered over an active volcano.

So another API is sorely needed. And if another library forces systemd to adopt something sane then it might be the only solution.

Another daemon for managing control groups

Posted Dec 17, 2013 6:57 UTC (Tue) by mchapman (subscriber, #66589) [Link] (10 responses)

> So another API is sorely needed.

Why? How is that going to help anything?

Maybe I've missed something in this whole argument, but I fail to see why there must be only one implementation of a cgroups manager. Yes, you may only be able to *run* one at any time due to kernel limitations, but that doesn't mean multiple projects can't choose to implement the functionality in different ways if they have different goals.

Isn't that the whole point of FOSS?

Another thing I don't really understand is why systemd should be "forced" to adapt to use some other manager. systemd doesn't need another manager; it does enough management of cgroups for itself already. What compelling reason is there for the systemd developers to throw all that out?

Another daemon for managing control groups

Posted Jan 3, 2014 18:23 UTC (Fri) by aigarius (subscriber, #7329) [Link] (9 responses)

Those are two competing points right there - currently systemd developers want to manage cgroups inside systemd and do not want to support delegating this management to any other outside daemon or library. For any other implementation of cgroups management to be able to work on a systemd system it would need systemd cooperation as above.

Therefore there are two options - either systemd must be relegated to be a tiny minority init version that can easily be replaced with something else on any distro, so that you can actually use a different cgroups manager or any other cgroups managers will be unusable without a very major hassle on many popular distros.

The refusal of systemd developers to cooperate in a flexible solution (combined with the marketing drive to make systemdbe default in major distros) is the very thing that is poisonous to alternative cgroup management implementations.

Another daemon for managing control groups

Posted Jan 3, 2014 18:57 UTC (Fri) by raven667 (subscriber, #5198) [Link] (8 responses)

> The refusal of systemd developers to cooperate in a flexible solution (combined with the marketing drive to make systemdbe default in major distros) is the very thing that is poisonous to alternative cgroup management implementations.

No, what is poisonous to other competing implementations is that systemd is technically advanced and comprehensive so that there is little need for other re-implementations because it works so well. That is the reason it has gained so much uptake, do you really think that all the people who build distributions are technically incompetent and are switching to systemd components because of some "marketing drive" or because these components solve the problems they are trying to solve better than the systems which came before?

It's true that the systemd developers refuse to re-design their system to have cgroup management handled by an external process, the "flexability" you talk about, but there are very rational technical reason for that which you have not addressed. Do you understand the kernel cgroups api and your workload and the systemd configuration knobs enough to be able to tell us where the systemd cgroup managment fails in your workload that another cgroup manager would be able to do better that couldn't be implemented in systemd? Do you have any examples or is this an entirely philosophical conversation with no real backing in reality?

Another daemon for managing control groups

Posted Jan 3, 2014 19:15 UTC (Fri) by dlang (guest, #313) [Link] (7 responses)

even the systemd developers admit that there are linux systems out there that should not run systemd (embedded type systems for example), but those systems may still want to use some of the features that systemd is trying to take over and kill off the competition in.

how would you use cgroups on a wireless access point where you don't want to run systemd but still want to limit some set of processes?

Another daemon for managing control groups

Posted Jan 3, 2014 20:03 UTC (Fri) by raven667 (subscriber, #5198) [Link] (6 responses)

It's a tautology really, if you don't want to run systemd for whatever reason, then don't run systemd.

> ... features that systemd is trying to take over and kill off the competition in.

I'm not sure what "take over" means in this context if you aren't running systemd on a system because you don't want to than what systemd does is not of relevance to you except as an example. The only way they are "killing the competition" is by making a high-quality, feature-rich offering which people want to use, they are out-competing the competition. I hope we can all agree that they shouldn't hobble the implementation and make it less useful just to make the competition look better...

> how would you use cgroups on a wireless access point where you don't want to run systemd but still want to limit some set of processes?

You do whatever you want to do, within the confines of the kernel interface, which in the future will want a single-writer (although compatibility with the existing interface will be maintained for some time as I understand it). So you can make your own cgroup manager daemon, being advised that there are fundamental problems with restarts, race conditions, etc. and that this can only manage a subset of processes (ones that it starts), or you write your custom manager into pid 1 either from scratch or patching one of the existing inits to have the logic you want.

There are probably going to be a handful of small cgroup managers that you can start from which are dedicated to specific tasks like HPC or Google, although if systemd incorporates whatever configuration toggles these workloads need then many will probably just use that rather than re-implement something themselves, but there is always someone out there who wants to do it themselves and is distrustful of any third party code 8-)

Really, how is this different from today where if you want to use this kernel feature you have to have your software talk to the kernel and twiddle it. How is it different than iproute2 or iptables?

Another daemon for managing control groups

Posted Jan 3, 2014 21:44 UTC (Fri) by dlang (guest, #313) [Link] (4 responses)

> The only way they are "killing the competition" is by making a high-quality, feature-rich offering which people want to use, they are out-competing the competition.

not quite, they also campaign to get the competition removed from the distro (or at least the default installation of the distro), rsyslog vs journald in Fedora is an example of this in practice.

Another daemon for managing control groups

Posted Jan 3, 2014 21:48 UTC (Fri) by mchapman (subscriber, #66589) [Link] (3 responses)

> they also campaign to get the competition removed from the distro (or at least the default installation of the distro)

Is that so surprising? Clearly they think they have a better solution. They *should* be campaigning for it, if that's what they really think.

I'm sure distro maintainers are smart enough to judge the various options available on their merits.

Another daemon for managing control groups

Posted Jan 3, 2014 21:54 UTC (Fri) by dlang (guest, #313) [Link] (2 responses)

they can campaign to have their system added, but what's the justification for removing the other option, especially when it can co-exist with theirs and provide capabilities that they don't offer?

Another daemon for managing control groups

Posted Jan 3, 2014 21:59 UTC (Fri) by mchapman (subscriber, #66589) [Link]

> they can campaign to have their system added, but what's the justification for removing the other option, especially when it can co-exist with theirs and provide capabilities that they don't offer?

For Rsyslog in Fedora specifically?

According to the Change page [1]:

1. Less disk and runtime footprint
2. Fewer packages in the default install
3. Faster booting

Now, you can argue whether those are *sufficient* justification for the removal of syslog from the default install. I'm sure people did.

At the end of the day, though, the vote was in that this Change should go ahead.

[1] https://fedoraproject.org/wiki/Changes/NoDefaultSyslog

Another daemon for managing control groups

Posted Jan 3, 2014 22:02 UTC (Fri) by raven667 (subscriber, #5198) [Link]

But they haven't removed rsyslog, and in fact rsyslog is required if you want to do central logging using syslog, systemd and the journal will pass through log messages to it. What they aren't doing is installing both logging systems by default. There is also no default sendmail in Fedora either.

Other distros make different decisions but rsyslog integration is still an important part of the journal, heck its even an important part of rsyslog which added support for the journal on-disk data format as I recall.

Another daemon for managing control groups

Posted Jan 3, 2014 21:46 UTC (Fri) by dlang (guest, #313) [Link]

> Really, how is this different from today where if you want to use this kernel feature you have to have your software talk to the kernel and twiddle it. How is it different than iproute2 or iptables?

iproute2 and iptables are happy to sit on a system and only be used for the parts that you want them to be used for. They don't try really hard to force you to do things a specific way and push you to replace a bunch of other tools if you use them.

Another daemon for managing control groups

Posted Dec 17, 2013 5:36 UTC (Tue) by dlang (guest, #313) [Link] (1 responses)

so systemd really isn't modular because if you replaced any of the binaries, then it would be depending on another daemon.

you can't have it both ways

(yes I'm done trolling now :-)

Another daemon for managing control groups

Posted Dec 17, 2013 5:48 UTC (Tue) by rahulsundaram (subscriber, #21946) [Link]

I was referring only to the init system part. systemd itself is a collection of programs, many of them optional and some of them not. If you introduce extra dependencies on the non-init system parts, then it is not much of a problem.

Another daemon for managing control groups

Posted Dec 17, 2013 11:33 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (9 responses)

OK, so in this case, how does systemd manage cgmanager? Cgroups is out since cgmanager hasn't started. Or systemd has special logic for cgmanager. But then why have the logic in two places? Plus, cgmanager would probably end up as PID 2(ish)[1]. Sorry, I don't see how making systemd ask anything else about cgroups when it is such a core mechanism makes the overall system simpler and cleaner.

[1]Or 3; journald is pretty early too...but what is more important: logs or management? If logs, there's another thing that needs special group logic. Wait, cgmanager is over DBus, so add dbus-daemon to the list. What's the saying? "Do something once, hack it out; twice, use history; thrice, write a goddamn script" (or something). So if systems has special code to put the journal, DBus (modulo kdbus), and cgmanager, why the hell would you not just put all the logic in systemd to start with?

Another daemon for managing control groups

Posted Dec 17, 2013 13:23 UTC (Tue) by Jonno (guest, #49613) [Link] (8 responses)

Without having written any prototype code, but only looked at the problem architecturally, I agree that pushing cgroup management out of PID 1 will likely require *more* code in PID 1 than simply doing the cgroups management directly in systemd in the first place. But even so you don't need to drag dbus-daemon into the picture. cgmanager could use the same trick as systemd does, and initially only listen on a private socket for root-only direct libdbus communication, and then register itself on the dbus-daemon system bus whenever it becomes available.

That way cgmanager can start as PID 3 (after only systemd and journald), and the only early boot special handling in PID 1 would be to first start journald and cgmanager and then add them to their respective cgroups after the fact, but before continuing with any other units. Obviously that adds a bit of serialisation before going of and doing everything else in parallel, so will likely increase boot time by a second or two, but it's not like getting it to work is rocket science.

The trickier question is how to deal with cgmanager dying unexpectedly. PID 1 enjoys a privileged position and the kernel makes sure it never ever dies for any reason whatsoever, but PID 3 has no such protection. If systemd depends on it to do regular service management, it can't just restart cgmanager like it does any other service. Instead it would need special recovery code for that rare-but-inevitable case, code that likely won't get much testing, a perfect recipe for eventual disaster...

Another daemon for managing control groups

Posted Dec 17, 2013 18:07 UTC (Tue) by hummassa (guest, #307) [Link] (7 responses)

> PID 1 enjoys a privileged position and the kernel makes sure it never ever dies for any reason whatsoever

AFAICT the kernel dies and locks the machine out if PID 1 dies.

Another daemon for managing control groups

Posted Dec 17, 2013 18:14 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

I think it "just" panics. At that point, PID numbers are pretty useless I'd think :) .

Systemd's behavior in the face of a fatal error is to enter an infinite loop rather than dying[2].

[1]http://cgit.freedesktop.org/systemd/systemd/tree/src/core...
[2]http://cgit.freedesktop.org/systemd/systemd/tree/src/shar...

Another daemon for managing control groups

Posted Dec 17, 2013 19:23 UTC (Tue) by Jonno (guest, #49613) [Link]

> AFAICT the kernel dies and locks the machine out if PID 1 dies.
The kernel will panic if PID 1 exits on its own, but it will protect it from all outside interference (including blocking the normally unblockable SIGKILL and SIGSTOP), so as long as you handle all normal signals and never call exit() or return from main() you are golden.

As noted above systemd will enter an infinite loop (always yielding it's time slot back to the scheduler as to not monopolize the CPU) in the signal handler for any error signal it can't handle more gracefully, but that is only to protect from internal bugs, and not a normal operational condition, which a dying cgmanager would be.

Another daemon for managing control groups

Posted Dec 18, 2013 9:45 UTC (Wed) by anselm (subscriber, #2796) [Link] (4 responses)

The kernel won't lock the machine if PID 1 dies; it will complain but keep running by itself.

The legend goes that there used to be people whose PID 1 would set up a bunch of iptables rules and associated config and then exit – which would give you a firewall/router that was absolutely free from outside interference if a bit inconvenient to manage.

Another daemon for managing control groups

Posted Dec 18, 2013 15:14 UTC (Wed) by raven667 (subscriber, #5198) [Link]

I've had that happen, but not intentionally 8-) I've had devices too where the block device died but everything that was commonly used an in RAM/cache was still accessible, that machine actually ran for a year or two in that state doing its job before it could be decommissioned.

Another daemon for managing control groups

Posted Dec 18, 2013 18:33 UTC (Wed) by dlang (guest, #313) [Link]

I did that once, as much to show I could as anything else.

Another daemon for managing control groups

Posted Jan 17, 2014 10:27 UTC (Fri) by etienne (guest, #25256) [Link] (1 responses)

> ... let PID 1 exit to have a secure router ...

I have a new question to an old answer, to secure a system after it has been correctly configured:
Can you get rid of /bin/sh (i.e. any sort of shell: bash, ksh, zsh...) and still have a booting computer if you use systemd?
That would be very difficult to try to compromise a computer if the shell has been removed...

Another daemon for managing control groups

Posted Jan 17, 2014 10:36 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

Yes, it's absolutely possible to boot a computer without any shell using systemd.