LWN: Comments on "Process containers"

Process containers

rijrunner — Fri, 25 Jan 2008 17:35:11 +0000

Well, my read of this is a bit different. This looks to me like a side-effect of the
virtualization changes to the kernel and how Oracle works. Basically, virtualization requires
carving out a set of system resources (memory, cpu, disk, network, etc, etc) and assigning
them to a virtual machine to manage. The key is that the kernel has to be able to define
parameters that can be isolated and restricted in their size and scope. What the container
concept seems to be - which I could be misunderstanding based on only a cursory reading - is
extending that ability to isolate resources to processes running within the base OS. 

ie, if you are putting hooks into the kernel to be able to define and limit system resources
for virtual machines, why not extend it to processes and resources at the OS level?

Process containers

Stephen_Beynon — Thu, 12 Jul 2007 10:19:05 +0000

It is possible to shape incoming traffic for tcp streams. Just drop any
packets that would cause the required bandwidth to be exceeded. TCP is
designed to assume packet loss means a saturated link, and backoff. While
it is not possible to get the bandwidth exact it is good enough to be
usefull.

When it comes to bittorrent I tend to find the problem is the upstream
bandwidth use, and that is much more controlable :)

Stephen

Process containers

muwlgr — Sat, 09 Jun 2007 10:57:40 +0000

And don't forget, there is no meaningful way to shape incoming traffic,
so the dream about BitTorrent&browser is just that, a dream :>

Process containers

vMeson — Tue, 05 Jun 2007 14:40:03 +0000

an industrial use case:
let's say you are a network infrastructure vendor,
you'd like to allocate 60% of cpu to processing packets for existing work, 10 % for handling new work, 10% for system maintenance, 10% for I/O, and 10% for spying^Hlawful intercept. ;-) The missing bit is how these containers or classes interact. Is system maintenance more important than new work or do you have a policy of fairness?

Containers coupled to the new scheduler: CFS seem like a powerful combination.

Process containers

IkeTo — Fri, 01 Jun 2007 16:40:21 +0000

Thanks. I understand your posts now. But I don't think I like the idea. At the very least, I don't think it reasonable to arbitrarily allocate user ID space to something completely unrelated to users this way. And of course it provides a horrible interface to users.

Process containers

utoddl — Fri, 01 Jun 2007 15:06:19 +0000

I was talking about supplementary group IDs as set by setgroups().

In the particular AFS context, when the older libafs kernel module loaded, it would swipe the setgroups entry in the sys_call_table (?sp) so it could handle the necessary details of associating an AFS PAG, token, and process. It was an admitted hack, but one that has worked in various forms for over a decade in a half dozen major flavors of UNIX. Other methods were invented for Linux when the kernel police make the sys_call_table read-only.

BTW, this was/is another reason to dislike what AFS does with the supplementary group list. It's rather disconcerting to do "id -a" and see groups with no associated names, but that's common if your shell is in a PAG. Behold:

$ id -a
uid=12428(utoddl) gid=12428(utoddl) \
groups=10(wheel),1511(atnid),12428(utoddl),1094942735

Process containers

IkeTo — Fri, 01 Jun 2007 14:33:56 +0000

> Process group lists have always been a light-weight set of properties that
> processes carry around and pass on through fork().

Can you clarify a little bit? AFAIK, there are two concepts of "groups" in the current kernel. One is called the "process group", as is set by setpgid(). Each process belong to one group (rather than many). That group is used for signal deliveries, allowing users to send signals to all processes of a group, either by explicit "kill" command/system call, or by using a special terminal character. The other is the "supplementary group IDs", as is set by setgroups(). Each process has a small number of those. It is used by system administrators to control the files or other resources that each user can access. The numeric values are meaningful not only to the kernel, but to the admin as well. They assign each user a list of such group IDs in /etc/group, and the login procedure will assign the login shell (or X session) process to use that list. There is also the session ID, but that doesn't seem like being what you mean.

So by "process group" do you mean one of these existing concepts, or is there yet another group concept in the process carried by the process that either is hidden in the kernel or that I forgot?

Process containers

utoddl — Fri, 01 Jun 2007 14:02:27 +0000

Fair enough. Let's see if I can connect the dots.

Ignore for the moment the implementation of either groups or process containers, and just look at the semantics. A given process can be in multiple groups; child processes inherit groups from their parents; special circumstances can alter which groups are added or dropped from a process' group list. Likewise for processes in containers. If you were to replace the labels in the diagram from the article with numbers, you could implement the processes "in-container-x" property with the existing group mechanism.

Process group lists have always been a light-weight set of properties that processes carry around and pass on through fork(). The fact that (almost) nothing except file systems uses them not withstanding, it seems somebody finally noticed that the semantics of passing around properties in this way is useful for other things like processor affinity, throttling, and other things the article mentions.

AFS (and later OpenAFS) piggy-backed process authentication group membership on the group mechanism. The AFS kernel module would add a group (actually a pair of group numbers) to a processes group list to create a new PAG. Child processes would inherit these just like any other groups through fork(), but no file system -- including AFS -- used these group numbers to check file access. Instead, AFS would use these numbers to associate a process with a specific PAG, which is just a set of processes which share a cached token. The token *is* used for access control, but membership in a PAG is just a property like any other group membership. The semantics for group membership and inheritance just happens to be exactly what you want for an authenticated file system like AFS.

Besides that, though, these semantics happen to be exactly what you want for processor affinity, bandwidth throttling, CPU limits, etc. But rather than piggy-backing these capabilities onto the existing group mechanism as AFS did, they've invented another parallel mechanism for passing process properties around. Group membership and process container "in-ness" are just properties after all.

To be fair, the time tested group mechanism has its limits. Group lists are rather short (or thay were last time I ran into that issue). They also aren't explicitly hierarchical like process containers (though what that buys us wasn't immediately obvious to me upon reading the article). It wouldn't surprise me if the old UNIX groups weren't eventually reimplemented as containers. Then you could eventually have hierarchical UNIX groups!

The point of my "camel in the tent" comment was that the way AFS piggy-backed the process properties it was interested in on top of groups was met with skepticism and sometimes out-right contempt by some kernel developers. The reasons include NIH (Not Invented Here -- AFS predates linux by a fair few years), the kernel module itself is maintained out-of-tree (it builds for several OSes other than Linux and not just on the current versions, so it contains a lot of "cruft", at least in the eyes of the kernel hard-core), and it's hobbled by being under the IPL license (basically IBM's GPL with a "we can take it proprietary later if we want" clause). AFS on recent kernels has switched to using keyrings -- yet another special purpose property propagation mechanism -- to implement PAGs, but the other factors still keep AFS/OpenAFS on the outside looking in.

The kernel goes through this periodic process where some new functionality is added, then somebody points out that this new thing and this other old thing have similar operations, then some common code is developed that they can both use or one gets folded into the other. We've seen it over and over, and I wouldn't be surprised to see it happen with groups and process properties.

Process containers

i3839 — Thu, 31 May 2007 23:34:05 +0000

They key point is:

> Other (not yet existing) subsystems could use containers to enforce
> limits on CPU time, I/O bandwidth usage, memory usage, filesystem
> visibility, and so on. Containers are hierarchical, in that one
> container can hold others.

Right now all resource management is done globally or per process/thread, but not much else. Process containers make it possible to group a bunch of processes and do resource allocation for them as a group (think ulimit, but more). What resource that is doesn't matter right now, as this article is about the basic infrastructure which is put into place to make everything possible.

This is useful for multi-purpose and multi-user machines. E.g. if you want your server to spend 50% of its CPU time, disk IO and/or memory on the webserver and a database, 25% on finding aliens, and the rest for reading LWn, it can be done.

It seems it can also function as a sort of jail, limiting the fs and process namespace view/access processes have.

(I might be mixing multiple things though.)

Process containers

riddochc — Thu, 31 May 2007 22:24:41 +0000

I must admit, this is more abstract than usual. I think the other two comments suggest that it's not really clear what exactly these are containers are *for*. Can someone give me an example of how such containers could be used? I'm confused.

Process containers

IkeTo — Thu, 31 May 2007 21:51:43 +0000

I have some difficulties understanding your comment. I've looked at OpenAFS for a tiny bit of time, my impression is exactly what you say: it is a filesystem, and PAG is a system for you to tell the filesystem who you are. How is this is anything to do with process container, which seems to be mainly a tool for system administrators or service startup scripts to limit the amount (rather than identities) of system resources like CPU and network bandwidth (rather than files) that the process can use, based on the "echo" commands executed by administrator manually or via scripts (rather than via the user creation and login procedure)?

Process containers

utoddl — Thu, 31 May 2007 18:20:39 +0000

This is a reimplementation of groups, but with more features attached than simply "you may or may not access this local file or directory". It looks like an extention of what OpenAFS's PAGs (process authentication groups) give you -- and what has kept their camel out of the kernel tent for years.