LWN: Comments on "Updates in container isolation"

Updates in container isolation

ZhuYanhai — Fri, 01 Jun 2018 08:47:51 +0000

Sentry runs in guest ring0 for kvm platform. So the guest code doesn't trigger a vmexit for its syscall, unless the sentry itself needs some additional syscalls.

And sentry runs in ring3 for ptrace platform, which is designed for development and debug purpose only.

Updates in container isolation

bergwolf — Mon, 21 May 2018 03:26:40 +0000

> He cited the CVE-2017-1002101 vulnerability, which allows hostile container images to take over a host through specially crafted symbolic links. [Zhengyu He] Like native containers, hypervisors like Kata Containers also allow the guest to mount filesystems across the container boundary, so they are vulnerable to such an attack.

CVE-2017-1002101 is playing with symlinks and Kata Containers are vulnerable because of 9p, which is also true for gVisor. A simply show case:
[macbeth@/]$ls -l /tmp/foo/bar
lrwxrwxrwx 1 bergwolf bergwolf 9 May 21 11:15 /tmp/foo/bar -> ../../../
[macbeth@/]$docker run -it --runtime gvisor -v /tmp/foo/bar/etc:/vol busybox
/ # ls -l /vol |head
total 258
drwxr-xr-x 4 root root 35 Nov 6 2017 X11
drwxr-xr-x 3 root root 20 Nov 6 2017 acpi
-rw-r--r-- 1 root root 3028 Oct 17 2017 adduser.conf
drwxr-xr-x 2 root root 4096 May 2 11:00 alternatives
drwxr-xr-x 3 root root 21 Nov 6 2017 apm
drwxr-xr-x 3 root root 59 Feb 22 09:32 apparmor
drwxr-xr-x 9 root root 272 Feb 22 09:32 apparmor.d
drwxr-xr-x 3 root root 45 Feb 22 09:32 apport
drwxr-xr-x 6 root root 197 Mar 31 18:01 apt

To fix CVE-2017-1002101, both Kata Containers and gVisor should be more careful about what to map in the 9pfs share to the guest.

Updates in container isolation

bergwolf — Mon, 21 May 2018 01:39:24 +0000

> If it's ring 3 somehow, then that would be an additional security layer for gVisor, but worse for performance.

It's ring 3 and each syscall has to vmexit. Bad news for syscall intensive applications.

Updates in container isolation

bergwolf — Mon, 21 May 2018 01:35:43 +0000

Things like procfs, sysfs and various ioctls are the tricky part of Linux kernel APIs and hard to be compatible with.

Updates in container isolation

prattmic — Sun, 20 May 2018 06:31:53 +0000

The sandboxed application processes run in guest ring 3. The gVisor kernel runs in guest ring 0 (and host ring 3).

Updates in container isolation

roc — Thu, 17 May 2018 23:34:39 +0000

Because of this, I think the article obscures the security advantages of gVisor compared to something like Kata Containers. I don't understand the comparison between the number of lines of code in the Linux kernel and gVisor. Are they talking about the guest kernel size? But the guest kernel size in a VM container isn't an issue for security because it's inside the sandbox. Are they talking about the host kernel size making it hard to audit the security of KVM itself? But that would also affect gVisor-KVM, which is going to be the preferred deployment configuration ... assuming your host is running on bare metal or nested virtualization is available.

I think the security argument for gVisor-KVM is that if you have a KVM escape that escapes into the hypervisor's user-level, not the host kernel itself, then you're still in a very restrictive sandbox around the gVisor kernel. Whereas with Kata you'd be in QEMU which probably needs a much less restricted sandbox.

Although one interesting question is, do gVisor-KVM guest processes run at ring 0 or ring 3? If it's ring 3 somehow, then that would be an additional security layer for gVisor, but worse for performance.

I can see advantages for gVisor in terms of memory and storage usage, because the guest can share the host file system rather than mounting its own on a virtual block device.

Updates in container isolation

prattmic — Thu, 17 May 2018 16:38:39 +0000

Regarding /proc, gVisor does have a procfs implementation. Like system calls, procfs files are handled internally and we don't cover the entire surface, but do have the core functionality required by most applications (/proc/PID/maps, /proc/cpuinfo, parts of /proc/PID/status, etc).

This doc (https://github.com/google/gvisor/blob/master/pkg/sentry/f...) tries to explain which parts of procfs are implemented, though it is not 100% complete.

Updates in container isolation

anarcat — Thu, 17 May 2018 14:00:17 +0000

Right. I probably got this one backwards, apologies.

Still, the way Xen is designed just feels a little backwards to me as the first layer is not actually the hypervisor itself, but a (compatible) kernel that talks with the hypervisor. And yes, that *does* provide an *extra* layer of security at the cost of performance. But Xen's design also means you need a privileged supervisor domain (the dom0 in the case of Xen) is also part of the attack domain now, and I seem to recall that being used as an attack vector in the past, but I could be mistaken there. I think this is where my analogy came from, but I must admit I cannot substantiate this any further and I am forced to recognize that the attack surfaces are comparable with other hypervisor like gVisor, at least conceptually.

Updates in container isolation

thinxer — Thu, 17 May 2018 02:23:14 +0000

You don't actually worry about the kernel running inside the sandbox. You worry about the sandbox only, which usually has simpler interfaces than the kernel and thus a reduced attack surface.

Updates in container isolation

anarcat — Thu, 17 May 2018 01:36:35 +0000

The point is that instead of having just the kernel to worry about (which is still running inside the container), you now also have the hypervisor (in this case Xen) to worry about as well.

Updates in container isolation

roc — Thu, 17 May 2018 01:22:33 +0000

> Kata Containers relies on a kernel running inside the container, which actually expands the attack surface instead of reducing it.

This doesn't sound right. The kernel running inside the container is inside the sandbox (the virtual machine interface implemented by the hypervisor), therefore it cannot add to the attack surface.