Shuttleworth: Losing graciously

> This is Linux. This is not Plan 9 where you can open a TCP connection with mkdir.
Uh, I know this is Linux and not Plan 9. How is that supposed to be an argument? We should learn from Plan 9 instead of taking that kind of “us vs. them” stance.

> cgroupfs is fine for introspection, but control? that always seemed a bit unnatural to me.
And to me it seems unnatural that access control for cgroups is supposed to be done through a completely different mechanism than access control to files or devices. Though I agree with you that the current cgroups API isn't ideal. For one thing, I think the natural thing is to use
ln /proc/42 /sys/fs/cgroup/yaddah/cgroup.procs
and not
echo 42 > /sys/fs/cgroup/yaddah/cgroup.procs
to add processes to a cgroup.

* Only allow PID X to manage cgroup subtree /A/B/C.
* Processes in subtree /A/B/C/ may move their processes in /D/E/F but should not have any further control.

Traditional file permissions and even ACLs are incapable of dealing with either situation. In the first, case there's no way to grant that access without allowing all processing owned by that same user/group from controlling /A/B/C. In the second situation, there's no way to allow a one-way move.

Yeah, we can start adding more files to the cgroups to attempt finer control, but now things get messy, and it's likely the who permission issue goes out the window. That's just two random policies that a system may want to enforce. To do #1 you would likely have to give one of U, G, or O +w to the subtree for that user, but that's a broken definition because hidden kernel policy will revoke writes by other processes, even though they have the proper U or G or fall into O. In the end, a cgroup would spout many files to cover all sorts of policy combinations, and the kernel developers will be left with a monstrosity that can't be fixed because next time the same cries about API will arise.

A filesystem hierarchy is not suited to this complexity. It will have to be contorted into all kinds ways where the permissions scheme (even with ACLs) doesn't match traditional behavior.

It's far preferable to take the LSM approach. The kernel provides the primitives throughout the kernel subsystems and talks to another module to establish and enforce policy. It's true that LSM modules are kernel modules, but they remain separate from the LSM code. (I wouldn't have a problem with a cgroup manager living as a kernel module, but I don't see any inherent benefit either.)

Right now... its a mess. He even comments on the fact that the gentlemen's agreement in the form of PaxControlGroup shows exactly how problematic trying to support multiple writers actually is right now in the multi-heiarchy cgroups because they aren't all multi-heiarchy aware. You can't actually use all the controllers in a multiple writer fashion without them step on each other. Caveats abound. He even correctly notes that the way it works quite well right now for hand crafted setups... where one human has crafted all the cgroup interactions via scripting and its effectively "single writer" in a sense. But once you tack on automation or applications which want to make use of cgroups side-by-side with other applications or other automation or even hand-craft scripts... you run into problems. PAXControlGroups details the in and outs of those problems.

So as part of making room to clean up all the problems the single userspace writer will be mandated in the middle term of the kernel side work to make a flat hierarchy api. I'm not even sure the plan is to require that single writer model forever. I believe the plan right now is to stop pretending that cgroups and all the controllers work with the multiple writer model while that flat hierarchy is being developed and controllers are all reworked to correctly support that new model.

I also think you need to in mind that Tejun also mentions a pie-in-the-sky goal of merging cgroups into the process hierarchy.

Look I think the real issue here is that until the new work (both kernel and userspace) has progressed further along, there are going to be existing use cases that only served by the deprecated API. The kernel developers have always recognized this, from the start of the discussion.
I think the discussion here is only filling in the details of what Tejun knew were local admin policy scripting centric use cases that would be impacted in the short to mid term. Tejun clearly states the choices on how to proceed involved a trade-off that would impact some existing use cases.

The real question is, will the linux distribution vendors continue to provide userspace solution which support the old API as an option? I have no idea on that. I know that libvirt for example has a transition plan in place to support the older cgroupfs if the systemd D-Bus API is not available on the host. But I have no idea if any distribution vendors are going to expose a configuration option to pick which cgroups API to use when mounting cgroups. So Cyberax should probably start making the case to his distribution vendors about supporting his use case by making it possible to choose to run the deprecated cgroupfs API in the future, so long as the API is available and not pulled from the kernel the vendor ships.

Surprise: You are exactly right. The kernel people do not WANT to set policy there because the requirements are unknown / too diverse / we don't have much experience what we actually need in complex real-world scenarios / take your pick.

We do know that a user/group scheme will not work: you can nest namespaces, and the process which sets up the outer namespace's access rights does not know which user IDs will eventually end up being mapped to the processes inside these (sub)containers. This is (probably) why cgroupfs does not have ACLs: they'd be insufficient anyway.

It's not the kernel's job to set policy. It's the kernel's job to facilitate a stable ABI, and leave policy to user space where it belongs.

Besides, access rights are insufficient for another reason. Suppose you want to limit users' memory usage to 100MB each; if they want to have 1GB of main memory they can -- but only two at the same time, and only for 10 minutes.

This is not a particularly exotic requirement for a multiuser system. If somebody adds code for that kind of thing to their favorite cgroupsmanager program, no problem whatsoever. In the kernel? don't even think of doing that.

I'm pretty familiar with the current cgroups interface and it seems that an untrusted process _at_ _most_ can cause high load on the kernel and perhaps significantly slow down other processes.

There's also a small problem with several controllers which use weights to distribute resources, so it's possible for a cgroup to affect its siblings. But again, that's trivially worked around by using an intermediary tree level if one wants to delegate a subtree to untrusted processes.