LWN: Comments on "Control groups, part 5: The cgroup hierarchy"

Control groups, part 5: The cgroup hierarchy

ras — Thu, 22 Jan 2015 01:46:31 +0000

These is (or was) such a thing. It's called tcng. Sadly it's been unmaintained for a over a decade now.

A for the earlier comment "we should just use iptables", the reason hard core tc user doesn't want that is speed. Yes, iptables is easy to use, but it's naturally linear. A packet going through 20 or 30 rules is not at all uncommon. Yes, you can code your rules so that doesn't happen, but then it's not so easy to use. With tc's u32 on the other hand you don't have a choice. Every rule is just a 32 bit load, mask and compare, and possibly a jump via a hash table. You tend to use hash tables, so the number of rules evaluated is stuff all. As a consequence it runs like a scalded cat. There is some syntatic sugar, but it has the effect of turning what is essentially a machine language into assembler language.

But yes, speed concerns aside using one filtering language every place a filter is used seems like it would be a utopia compared to what we have now. And there are a lot of such places - ip route, ip rule, iptables, and u32 spring to mind. But if you are going to do that speed is of the absolute essence. A declarative language that is easily compiled and optimised into something like a u32 virtual machine seems like it could work.

I had hoped nftables give us an interface like that, but as far as I can tell it doesn't quite get there. It's compiled and all, but it is still encourages the user write a set of rules that are executed linearly. That makes it hard to optimise it to the same extent a hand u32 filter would be.

Control groups, part 5: The cgroup hierarchy

neilbrown — Sat, 09 Aug 2014 05:33:01 +0000

Thanks for asking - I did wonder if I had explained that well enough.

If we wanted to set the network class for sockets created by one set of processes that "joe" creates to "3" and for sockets created by some other set to "4", then in each session owned by "joe" we need two cgroups. Maybe call them "netcl_3" and "netcl_4".
If we have some other users logged in they will also each need "netcl_3" and "netcl_4" in their "session scope" cgroup.

These are the "several sub-cgroups in each session". We can assign lots of different cgroups the network class number of '3' and lots of others the network class number of '4'.

Suppose that we also want to restrict different process to different CPU sets, but this division of processes is quite independent of the divisiion by network class. Suppose there are two interesting sets of CPUs called "A" and "B".

Then each different session needs 4 cgroups: "netcl_3_cpu_A", "netcl_3_cpu_B" "netcl_4_cpu_A" "net_cl_cpu_B" (though other arrangements are possible).

If you wanted a different breakdown for imposing device access restrictions (for example) you would need many more groups, in each session. This is the "combinatorial explosion".

This only really works for the non-accounting cgroups. For CPU and memory you cannot achieve a useful effect by multiplying cgroups like this (at least I, like you, cannot see how).

Control groups, part 5: The cgroup hierarchy

IkeTo — Fri, 08 Aug 2014 01:52:16 +0000

I find the following difficult to follow:

If we included the net_cl subsystem in the systemd hierarchy described earlier, we would potentially need to create several sub-cgroups in each session for the different network classes that might be required. If other subsystems (e.g. cpu or blkio) wanted different classifications within each session, a combinatorial explosion of cgroups could result.

It isn't immediately clear to me how "create several sub-cgroups in each session" would work at all.

The Documentation/cgroups/cgroups.txt file is of a little help, but not sufficient for me to fully understand the alternative. It describes a scenario that the admin can raise the capability of a task temporarily in only one of the subsystems but not others. I can imagine that, instead of using a student's own quota in CPU when working for a professor, he can be configured to use the quota of the professor just for the CPU when running the simulator program.

What I don't see is, how that can be achieved without multiple hierarchies, even if creation of many sub-cgroups is allowed? Sure we can create a sub-cgroup of "student" as "student with professor cpu quota", but that cgroup cannot simultaneously be the child of the student cgroup and the professor cgroup with a single hierarchy. Perhaps it will all become very clear once part 7 is read.

Control groups, part 5: The cgroup hierarchy

kleptog — Wed, 06 Aug 2014 07:05:49 +0000

As an interface to the kernel integer handles are fine, see file descriptors for example.

But as a method of control by humans they are horrid. What was (is?) really needed is some kind of declarative language where you could describe what you want to achieve and then some compiler would compile that into a tc commands for you.

Control groups, part 5: The cgroup hierarchy

Cyberax — Thu, 31 Jul 2014 22:56:17 +0000

Probably a bit of both. An interface with integer handles is not easy to use - nice textual names instead of major:minor pairs would have helped a lot. Then there's the problem with error messages (they are opaque) and discoverability of settings.

And conceptually I never understood why qdisc is separate from iptables.

Control groups, part 5: The cgroup hierarchy

neilbrown — Thu, 31 Jul 2014 22:23:17 +0000

Thanks for your insights.

With qdisc, is it just the interface that is "hell to use", or might there be something deeper?
i.e. if someone created a FUSE filesystem interface, or a nice GUI which made the current structures easy to visualize and manipulate, would you expect that to make it heaven to use, or is there some issue that cannot be automated away?

Control groups, part 5: The cgroup hierarchy

Cyberax — Thu, 31 Jul 2014 12:03:42 +0000

I feel that NTC solves a different problem. It has to deal with network packets that have no relation to running processes, so its design is perforce quite different.

And from a personal point of view - qdisc interface is hell to use. Cgroups filesystem-based interface is an order of magnitude simpler.

Control groups, part 5: The cgroup hierarchy

riking — Thu, 31 Jul 2014 08:38:50 +0000

Thanks for making this series -- really enjoying it!