Measuring container security

As a representative of the hosting community, I want to point out that isolation is not a real reason why hosters prefer VMs over containers. Security is important, but 'security' for freshly patched system is a gamble on 'what's next'. In this aspect I completely agree that containers may be broken at the same rate as VMs.

The Tenant Isolation is the keystone issue for hosters.

Fist, is a resource isolation. VM have four visible: CPU, net, memory and block IO, and three hidden: host services memory, cpu overhead and cache depletion. Is really easy to have two heavyweight VMs running on the same host without affecting each other. With a little tinkering I can push it to real-time level of isolation (pci pass through, CPU pinning, etc).

For containers it's not possible. There are few dozens of shared kernel resources, some if which aren't even counted against container cgroup limit. Any attempt to put limits on them is a double problem. First, is a vast amount of those resources. Second, the way those limits are translated to customers. No one want a list of 4k+ arbitrary restrictions.

Second, is a crash isolation. It somehow related to 'security', but not exactly. The Kernel sometimes can put processes into TASK_UNINTERRUPTABLE state. Any ideas how to terminate a container with such nastiness? Hint: 'virsh destroy' for VMs. There are many such corner cases in kernel, when bugs in one workload can affect another. If it happened in VM environment, all rouge guest can do is to hog resources up to the quota (see above on resources). In a container crazy guest can badly and undebuggingly (for container-level operator) affect innocent (other) containers.

Up to the point I thought that VMs is a way. Run VM for isolation purposes, put containers inside for management purposes.

Unfortunately, Spectre has been casting shadow long enough to kill all perspectives for VMs for isolation purposes. Baremetal for isolation, containers for management.

We should thank vms and clouds for help for building an infrastructure for image management and deployment patterns, so they could be adopted for baremetal deployment, but they (vms) are now limited to staging, or development purposes. One shall not trust isolation nor to containers nor to VMs.