LWN.net Logo

LinuxCon: Secure virtualization with sVirt

By Jake Edge
September 23, 2009

"I'm the rain in the cloud" is how Red Hat's Dan Walsh described himself at the beginning of his LinuxCon talk. There is much talk of "cloud computing" these days, but there has not been too much attention paid to the security aspects. Running multiple guest operating systems on the same hardware is "one of the scariest things you can do" from a security point of view, he said. sVirt was developed to combat the problem by applying SELinux mandatory access controls to restrict what guests can do—even if they break out of their containment and can access the Linux host OS.

Before virtualization, servers were separated by network connections, so a misbehaving server would have to launch a network-based attack to break into another server. There are lots of tools available to administrators that will alert or thwart network attacks, but when the servers are running on the same hardware, there is another line of attack: the hypervisor itself. Guests that can perform unauthorized actions on the host OS or hypervisor may be able to access information that is only supposed to be available to a different guest.

These are not theoretical attacks, Walsh said, as there have been successful attacks against Xen and others. Hypervisor vulnerabilities are the "number one goal" of the attacker community right now. The attack against Xen was able to subvert the SELinux policies that were in place on Red Hat Enterprise Linux (RHEL) specifically to stop that kind of attack. Those policies failed because the SELinux labeling of Xen processes and data were left up to administrators—something that sVirt is meant to fix.

Walsh pointed out that all guest OSes typically run as the same user in the Linux host. So, any exploit means that guests can access any other guest on that host. In the cloud computing scenario, users have no idea who else is sharing their machine, so it could easily be a competitor or someone with a malicious intent. But, enforcing separation between processes is a job that SELinux is good at.

In an SELinux-enabled system, processes and data both get labeled based on how they are allowed to be used. Since virtual machines are processes and their filesystem images are files on the host, proper application of SELinux labels—along with rules to govern the label interactions—will effectively disallow guests from unauthorized access to other guests. The host kernel enforces those rules so, as long as the kernel itself is uncompromised, rogue guests are confined.

As they learned from the Xen compromise, leaving the labeling up to administrators does not work, Walsh said, so they added dynamic labeling into libvirt. sVirt uses a largely unused field—for multi-category security (MCS)—in the SELinux label and generates a random unused value for that field. It labels the image file, then launches the virtual machine using that same label.

Using the MCS field allows the same SELinux rules to be used for all of the guests, but still restrict guests such that each guest can only access its process and data. When the guest exits, the guest image is then relabeled back to its original value. Different labels are used for shared images, depending on whether they are shared as read-only or read-write, which will allow administrators some flexibility while still restricting access to unrelated guest images.

Starting with Fedora 11, virt-manager will, by default, handle the automatic relabeling of virtual machines and data, Walsh said. One would guess that RHEL 6 will have that capability as well.

While it is certainly not a panacea for security in a virtualized environment, sVirt does provide some useful separation between guests. There is still cause to be concerned about potential kernel vulnerabilities that would allow end runs around SELinux, but sVirt reduces the exposure surface. As part of a multi-layered defense, sVirt effectively narrows the cracks that attackers can slip through.


(Log in to post comments)

LinuxCon: Secure virtualization with sVirt

Posted Sep 24, 2009 8:42 UTC (Thu) by djm (subscriber, #11651) [Link]

Let me get this straight: an attacker has gained access to a VM, escalated
privilege, escaped the (probably-hardware assisted) VM containment and they
think that more OS-level controls will prevent the same thing happening in
the host OS? IMO it is more likely that they escaped the VM by exploiting
bugs in the host OS kernel to begin with so sVirt couldn't help anyway...

LinuxCon: Secure virtualization with sVirt

Posted Sep 24, 2009 9:41 UTC (Thu) by danpb (subscriber, #4831) [Link]

If you look at the Xen vulnerabilities there has been a good split between flaws in the hypervisor/host kernel, and flaws in the QEMU device model. sVirt doesn't claim to protect the kernel, but it does offer valuable protection against QEMU device model flaws.

LinuxCon: Secure virtualization with sVirt

Posted Sep 24, 2009 9:28 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

"As they learned from the Xen compromise, leaving the labeling up to administrators does not work"

Yeah, have they _seen_ the complexity of SELinux policies? It's no wonder that most administrators dare not touch SELinux. Personally, I usually just pray that it works.

On the other hand, path-based approaches like AppArmor are very easy to use. But they had not gained any traction within the security community. Probably, because it's too easy to use.

LinuxCon: Secure virtualization with sVirt

Posted Oct 12, 2009 2:04 UTC (Mon) by vonbrand (subscriber, #4458) [Link]

On the other hand, path-based approaches like AppArmor are very easy to use. But they had not gained any traction within the security community. Probably, because it's too easy to use.

In Unix, the same object can be accessed by wildly different paths (think links) or can move around, so this won't give much security. That it is easy to use makes no difference if it is easy to bypass.

You also misrepresent the security community: A mechanism that is hard to understand and use won't be secure in practice, and they do know that very well; so they are looking for simple to use mechanisms.

All VMs run as the same user...

Posted Sep 24, 2009 11:00 UTC (Thu) by epa (subscriber, #39769) [Link]

If, as the article notes, all virtual machines run as the same user on the host system, wouldn't it make more sense to fix that by running each VM as a separate user and using classical Unix permissions?

All VMs run as the same user...

Posted Sep 24, 2009 17:44 UTC (Thu) by smoogen (subscriber, #97) [Link]

I believe (and this is a weak belief from too little research) that most VM's have to run with root priveledges at some place in their structure (this is to get use of the hypervisor CPU accelerations). Most of the people I know who are researching 'escapes' usually find the way out of the VM cage is in those areas.. thus the breakout has root access already.

The aim is with any of the security mechanisms is to limit what that root can do.

All VMs run as the same user...

Posted Sep 25, 2009 11:29 UTC (Fri) by rwmj (subscriber, #5474) [Link]

You can chmod 0666 /dev/kvm if you want to run KVM as non-root with hardware acceleration. (I
run it like this all the time).

The ability to run KVM processes as non-root is something that is to be added to libvirt in the near
future.

All VMs run as the same user...

Posted Sep 25, 2009 20:10 UTC (Fri) by lutchann (subscriber, #8872) [Link]

In addition to using one UID per KVM instance, use the new native container features in Linux to put each KVM into its own container. With an extremely limited view of the filesystem, namespaced process tables and IPC, an empty capabilities bounding set and appropriate iptables OUTPUT rules, breaking out of the VM into the KVM process does an attacker no good. No SELinux necessary.

With such a setup, the only thing you have to pray for is that there are no vulnerabilities that allow a guest VM to break into the host's ring 0. Unfortunately, such bugs have already been discovered in Xen.

(I can share my C wrapper for containerizing KVM if anybody's interested. Post a followup to this comment and I'll tar it up and post it somewhere.)

All VMs run as the same user...

Posted Sep 27, 2009 10:22 UTC (Sun) by nix (subscriber, #2304) [Link]

That would be extremely interesting, thanks. (I didn't realise the
namespaces stuff was at a usable state yet, but I haven't been paying much
attention to it.)

All VMs run as the same user...

Posted Sep 25, 2009 20:19 UTC (Fri) by smoogen (subscriber, #97) [Link]

Would not those permissions still allow for any process to look at another one through the device?

All VMs run as the same user...

Posted Sep 26, 2009 6:22 UTC (Sat) by Cato (subscriber, #7643) [Link]

Worse than that, presumably anyone wanting to write to kernel data structures can just write to /dev/kvm.

All VMs run as the same user...

Posted Sep 26, 2009 7:19 UTC (Sat) by rwmj (subscriber, #5474) [Link]

It's a good question. I talked to Gleb and Avi about this a few months back, and I came away with
the impression that it was safe. _However_ rereading their responses this morning, I'm now not so
sure it provides isolation between users who have VMs on the same system, so I guess I'm going to
have to dig into the code and check it myself.

Rich.

All VMs run as the same user...

Posted Sep 26, 2009 9:04 UTC (Sat) by avik (guest, #704) [Link]

It's safe. Access to /dev/kvm doesn't give any access to other virtual machines.

Of course, if a process has access to another process (via kill(2) or ptrace(2)) it can affect or access data belonging to that process. So if you run all virtual machines as the same user, you need to further isolate them. I believe sVirt does that with its random selinux contexts. but I'm no selinux expert.

All VMs run as the same user...

Posted Sep 26, 2009 10:17 UTC (Sat) by rwmj (subscriber, #5474) [Link]

There you have it. Thanks Avi :-)

All VMs run as the same user...

Posted Sep 28, 2009 17:35 UTC (Mon) by danpb (subscriber, #4831) [Link]

The ability to run KVM as non-root is already in libvirt. In Fedora 12 all 'qemu:///system' connections run VMs under a dedicated 'qemu' user account, while 'qemu:///session' connections run VMs under the UID of the user using that connection.

Also in Fedora 12, /dev/kvm has mode 0666 out of the box, allowing qemu:///session uses to use KVM acceleration.

The libvirt security architecture that deals with sVirt is modular allowing arbitrary security plugins. The Ubuntu devs have got an impl using AppArmour. It would also be possible to write an impl that ran each VM as a unique user ID.

Copyright © 2009, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds