Another runc container breakout

If you want true privilege separation where violations can cost you a lot of money (banking, HIPAA, etc.) then you go with dedicated hosts. So each workload gets its own physical host, possibly running multiple VMs/containers but at the same privilege level.

If you're slightly less paranoid, then VMs that don't share resources (including CPUs in the same socket) are fine. One more step, and you can use VMs on oversubscribed hardware or VMs with dynamic resources. All these VMs can also run multiple containers, typically for the same application.

I don't think containers are good enough for _any_ privilege separation.

In general, you still use containers, but use some layer in between to provide each customer with a separate kernel. There's a large variety of ways to do this. On the simpler end, you can just statically provision normal dedicated hardware or VMs per customer that you operate with a shared control plane. This is kind of wasteful because you need to run a whole intermediate OS and size it for worst case memory use. You can be a bit smarter than this though, so there's a number of microvm based solution like Kata Containers that run each container as a lightweight KVM-based virtual machine instead, with impressively small overhead. But if even that is too much for you, you can also use something like gVisor, which implements the kernel interface entirely in userspace. Kind of like wine, but for running Linux applications on Linux.

There's a huge variety of solutions here, but what they have in common is that nobody trusts multiple customers with one Kernel.

There's a couple of options. At my day job we've been running a traditional UNIX environment for about 20 years for our HPC use case, and so we have relied on the normal UNIX privilege separation mechanism, namely UIDs. We mostly kept this when we started moving to containers: user-provided code only runs as their own user even inside the container. Users have historically had non-root shell access to our systems via plain SSH, and so as long as they remain non-root, a containerization breakout like this doesn't impact our security model. (Of course, keeping a Linux environment secure against local privilege escalation is challenging on its own, and if our untrusted users were anonymous people on the internet instead of employees who have gone through background checks, we would likely be more paranoid.)

We benefited from the fact that our workloads were already designed for non-containerized environments (and so we already had non-root ways of installing software etc.) and we could just containerize that model. If you're running container-native workloads that expect to have root, or even multi-tenant workloads that all use the same non-root UID (e.g. everyone runs as UID 1000), user namespaces are a great option: the code in the container feels like it's running as whatever UID, and if it's root it can do things like mount a tmpfs or reconfigure its own networking. But from outside the container, these are separate, unprivileged UIDs. So the attack in the article about reading /etc/passwd is still possible, but a container running as "root" cannot write to it. Kubernetes has user namespace support in alpha.

And then yes, you can use virtualization (of any level of hog). If you're on a public cloud platform and you're on VMs anyway, setting up your VMs to be per workload is pretty easy. There's also stuff like Kata Containers or gVisor (commercially offered by Google as GKE Sandbox) for having a "container" runtime that actually uses hardware virtualization.

I think about it like this: containers are a technology for distributing software artefacts, packaged up and ready to run on a host. Think shipping containers. They aren't designed to confine an executing workload within its container's namespaces; you still need other technologies, which can be used in conjunction with containers, for that.

So, for instance, looking at a platform like Red Hat OpenShift, which runs on top of RHEL CoreOS, a typical container:

* Must run with a UID, GID and supplemental groups from the range assigned to its project
* Must run within a restrictive domain (container_t) and a unique MCS category (e.g., c5,c67)
* Must run with the runtime/default seccomp profile applied

So for one workload to read the files of another, it would have to:

* Use an exploit to escape out of its namespaces
* Use an exploit to override DAC (i.e., read a file owned by another user; made more difficult by the seccomp profile which prevents the use of less common system calls)
* Use an exploit to override the SELinux policy which prevents, for instance, a process running as container_t:c5,c67 from accessing objects labelled with container_file_t:c78,c200

It's interesting you mention qemu, because the idea of using a restrictive label & MCS category to protect container workloads from one another came from the very same setup that RHEL uses to protect VMs from one other: The container_t label is (or was at some point) an alias to svirt_lxc_net_t, the label that libvirt applies to all the qemu processes that it launches (again, with each one being given a unique pair of MCS categories so that VMs can't attack each other).