Walsh: Secure Virtualization Using SELinux (sVirt) [LWN.net]

Red Hat SELinux hacker Dan Walsh writes about Secure Virtualization (sVirt) on his web log. The basic idea is to leverage SELinux to isolate virtual machines from each other and from the host. "After virtualization, we have multiple services running on the same host. If a virtual machine is broken into, the cracker just needs to break though the hypervisor. If a hypervisor vulnerability exists, the cracker can take over all of the virtual machines on the host. He can even write into any virtual host images that are accessible from the host machine. [...] This is very scary stuff. The question is not 'if', but 'when'. Hacker/cracker conventions are already examining hypervisor vulnerabilities. Crackers have already broken though the xen hypervisor, as I documented in one of my previous blogs."

to post comments

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 24, 2009 21:29 UTC (Mon) by caitlinbestler (guest, #32532) [Link] (17 responses)

Using SELinux to protect the Hypervisor is decidedly the wrong approach
unless you are using a large bloated hypervisor (such as Windows or Linux).

A more productive approach to security would be to have a small, stable
kernel(Hypervisor) and disaggregate the backend device drivers into
separate domains.

Yes, the single hypervisor is still an avenue of attack, but so is
the BIOS. Both achieve security by being small and stable rather than
jury rigging trip-wires all over.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 24, 2009 22:16 UTC (Mon) by foom (subscriber, #14868) [Link]

Great! So all you have to do is make a high-performance microkernel OS. Should be trivial. :)

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 24, 2009 22:21 UTC (Mon) by mmcgrath (guest, #44906) [Link] (2 responses)

> A more productive approach to security would be to have a small, stable
kernel(Hypervisor) and disaggregate the backend device drivers into
separate domains.

Why can't you have both? SELinux isn't that much bloat. And even the smallest of kernels will have a vulnerability at some point. SELinux can help mitigate that risk.

Even the smallest ...

Posted Aug 27, 2009 21:19 UTC (Thu) by AnswerGuy (guest, #1256) [Link] (1 responses)

... microkernel will have some vulnerabily.

And yet you assert that SELinux which it's "not that much bloat" will mitigate such risks (without introducing additional vulnerabilities).

On that last (implicit) point you are demonstrably wrong. There have been vulnerabilities introduced by the presence of SELinux.

What is more disturbing, however, is the overall blind assertion. "even the smallest" (any) X will Y but *my* Z will NOT Y" --- that you can make such an assertion without knowing anything about the hypothetical X suggests that you're advocacy of SELinux (Z) is in fact a matter of faith rather than objective analysis. For an issue such as this (a matter of science and pragmatic application -- engineering) this completely undermines your credibility.

Even the smallest ...

Posted Aug 27, 2009 22:03 UTC (Thu) by mmcgrath (guest, #44906) [Link]

> On that last (implicit) point you are demonstrably wrong. There have been vulnerabilities introduced by the presence of SELinux.

Since security is the overall goal here, has SELinux introduced more vulnerabilities then it's prevented? I think not.

> this completely undermines your credibility.

No need to undermine me :) Stick with what arguing what I said. Unless you have something to gain by my turning out to be stupid or otherwise incompetent.

/me breaks out the link so grown ups on the Internet can be grown up again - http://en.wikipedia.org/wiki/Ad_hominem

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 24, 2009 22:22 UTC (Mon) by jamesmrh (guest, #31622) [Link]

Indeed, Linux is the hypervisor in this case (kvm/qemu), although I'm personally more comfortable with a large well-tested codebase than a small, obscure one.

There is a project to separate Xen domains with MAC: XSM/Flask, which applies the Flask architecture to the Xen hypervisor and separates drivers into their own domains and also allows for runtime verification of guest kernels.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 1:05 UTC (Tue) by ebiederm (subscriber, #35028) [Link] (10 responses)

My definition of small < 10K lines of code. Something a single person can wrap their head around and reasonably maintain, something about the size of the original unix core. Has anyone ever seen a hypervisor that is less than 10K lines of code?

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 9:02 UTC (Tue) by johill (subscriber, #25196) [Link]

Most drivers people will want/require in the HV are larger than that.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 9:08 UTC (Tue) by gebi (guest, #59940) [Link] (8 responses)

sel4 (secure embedded L4) has about 7k5 lines of code.

And it's the one of the few hypervisors with a proof of correctness http://ertos.nicta.com.au/research/sel4/

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 10:22 UTC (Tue) by drag (guest, #31333) [Link] (1 responses)

And.. it does not actually do anything.

It's the perfect kernel from a security standpoint. Something with nearly
zero functionality has about zero chance of ever having a security
problem!!

To do anything with it you have to run Linux on top of it... which gets you
what, exactly? A system with nearly zero hardware support running a
application environment that is less secure and has less performance then
if you just ran it on bare hardware.

-------------------

And that goes pretty much the same for "lightweight" hypervisor like Xen or
ESX.

Lets look at Xen.(I am much more familiar with Xen, but I am sure that ESX
is in a similar situation)

In order to run anything you have to take a Linux kernel, shove it up into
Ring 1 and then have your hypervisor run in Ring0.

So.. theoretically you might have some security improvements, but in
practice the Linux environment needed for you to do anything and it has
pretty much full access to your hardware and all your guests. (anybody ever
heard of a term called "DMA"?)

Sooo.. in fact your _worse_ off. Because now your not just dealing with the
Xen hypervisor your dealing with all the code that is in Linux kernel and
then all the code that is required for the hardware emulation and I/O
virtualization and all that stuff.

And Xen is no featherweither either. 300k LoC I think.

So your "lightweight" hypervisor solution weighs in at about 2-3 times
amount of code running on your system to get the same work done as just
running Linux on the bare metal.

-------

To get rid of the dependence on Linux your going to have to write hardware
drivers, create a sort of userland interface, and then all the code
neccassry for management and I/O redirection/emulation and all that...

So now your hypervisor is essentially a Linux competitor and is huge, or
you have vastly reduced feature set.

You can't get away from the fact that if you want to support hardware and
have similar functionality that you get from a kernel your essentially just
going to have to write a kernel eventually. You can call it a hypervisor,
but a rose by any other name will require about the same amount of
functionality.

Meanwhile the "heavyweight" KVM code that turns Linux into a hypervisor is
what? 30k? 50k? If you want to go by LoC that is actually quite a bit less
then is required by Xen.

---------------

People need to get the idea out of their heads that virtualization can
increase security. This is a flawed way of looking at it.

The ideal solution is to have separate machines for each service you want
to deploy. But that is expensive. Virtualization can reduce costs.. and
that is the best way to look at it, I figure. VM's are a cost cutting
thing, not a security thing. When you can not afford to do the security
right (or at least the best way) then you use virtualization.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 11:17 UTC (Tue) by hppnq (guest, #14462) [Link]

People need to get the idea out of their heads that virtualization can increase security. This is a flawed way of looking at it.

Indeed, the article is about securing virtualized environments, not about using virtualization to secure systems.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 18:59 UTC (Tue) by mheily (guest, #27123) [Link] (5 responses)

seL4 sounded great until I read this part:

> The evolution of seL4 is continuing within NICTA, and it is the basis of ongoing research activities here at NICTA. However, it is also the subject of commercialisation activities, and thus is currently proprietary. We are endeavouring to release seL4 in the future, but the time-line and exact nature of the release is yet to be decided.

Thanks, but no thanks. There are already plenty of other free microkernels out there (L4, Mach, Coyotos, HURD, Minix, etc.)

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 26, 2009 9:36 UTC (Wed) by renox (guest, #23785) [Link] (4 responses)

> seL4 sounded great until I read this part:
>
>> The evolution of seL4 is continuing within NICTA, and it is the basis of ongoing research activities here at NICTA. However, it is also the subject of commercialisation activities, and thus is currently proprietary. We are endeavouring to release seL4 in the future, but the time-line and exact nature of the release is yet to be decided.

> Thanks, but no thanks. There are already plenty of other free microkernels out there (L4, Mach, Coyotos, HURD, Minix, etc.)

Sure, but those don't have the 'proof of correctness' feature so you shouldn't say that they're plenty of free alternative: if you really need this feature then AFAIK only proprietary OS provide it currently, otherwise sure there are plenty of free alternative.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 26, 2009 16:40 UTC (Wed) by Wol (subscriber, #4433) [Link] (2 responses)

Unfortunately, a "proof of correctness" isn't worth the paper it's written on.

It's maths, therefor, as per Godel, it's either trivial, wrongly spec'd, or wrongly proved. At least ONE (and probably two) of those conditions MUST be true.

Poof! there's your proof gone up in smoke...

Cheers,
Wol

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 26, 2009 20:34 UTC (Wed) by jordanb (guest, #45668) [Link]

Godel says no such thing.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 27, 2009 3:12 UTC (Thu) by njs (subscriber, #40338) [Link]

> It's maths, therefor, as per Godel, it's either trivial, wrongly spec'd, or wrongly proved.

Out of curiosity, which of those categories do you think Goedel's proof falls into?

Open Proofs

Posted Aug 27, 2009 14:05 UTC (Thu) by dwheeler (guest, #1216) [Link]

There's a big need for software that is proven correct and for which the implementation, proofs, and required tools are all open source software. I call such software "open proofs". If you're interested in helping, please visit the Open proofs web site; we need people who can package existing OSS tools, as well as people who can develop and prove programs.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 18:36 UTC (Tue) by aliguori (guest, #30636) [Link]

The BIOS isn't secure at all. It's only accessible by ring 0 and ring 0 is completely capable of taking over portions of the BIOS.

In fact, this is exactly how option roms work (hijacking BIOS vectors to extend BIOS funtionality).

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 24, 2009 22:24 UTC (Mon) by spender (guest, #23067) [Link] (3 responses)

(CLOUDBURST + kernel exploit) || (SMM bugs) = virtual wooden bridge over your virtual air gap

There is simply no replacement for physical separation, especially when we're talking about the kind of information this technology claims to protect and the kinds of people/organizations who want to access it.

-Brad

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 24, 2009 22:31 UTC (Mon) by jamesmrh (guest, #31622) [Link]

Yes, I made it clear in my sVirt talk at LCA that we're trying to restore some of the security benefits of physical separation, but that it's not quite the same.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 24, 2009 23:39 UTC (Mon) by eparis123 (guest, #59739) [Link] (1 responses)

Really, by now I believe if a security mechanism (e.g LSM modules) can't protect itself (as far as possible) from its own (or other kernel level) security bugs, it's close to useless.

Obviously recent and older history shows the validity of you comment ;-)

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 1:56 UTC (Tue) by jamesmrh (guest, #31622) [Link]

Security mechanisms need to be robust, but generally you can't expect them to protect themselves.

Grsec/pax folk are working on kernel heap hardening (KERNHEAP), which I hope we can get upstream in some form.

Protecting the kernel really needs hardware support, and features such as VT-d and TXT look promising here (although hardware can and does have bugs, too).

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 24, 2009 22:43 UTC (Mon) by jspaleta (subscriber, #50639) [Link] (14 responses)

So... obligatory Canonical question.

Is Canonical still positioning AppArmor as the default MAC option in its Eucalyptus based Ubuntu Entreprise Cloud Architecture product? Is there an equivalent effort underway to use AppArmor to provide a similar level of protection to what SVirt offers? Or has Canonical essentially punted on the issue of security in the cloud while pushing its server product.

-jef

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 0:14 UTC (Tue) by dlang (guest, #313) [Link] (8 responses)

app armor suffers from not being in the upstream kernel.

now that path based LSM hooks are available upstream, I would love to see AA updated to use them and resubmitted for inclusion.

personally I think that in practical terms AA is more useful (it's easier for an admin to understand, and therefor configure properly for their system)

I don't consider _any_ generic, out-of-the-box distro config to be reasonable (for AA or for SELinux). the specifics of what a particular service needs to do can change so drasticly from one application to another that there is no chance that a generic config can both protect the system and be broad enough to allow everything to work.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 0:31 UTC (Tue) by jspaleta (subscriber, #50639) [Link] (2 responses)

Is anyone...anywhere...working on AA policy that will provide the same protections as SVirt for any virtual host workload configuration which would help mitigate an attack vector from one virtual client to another like SVirt does with its security labels approach? And more specifically one that works with the Eucalyptus managed cloud architecture.

It's nice to say AA is easier to configure for site specific policy..great..yippie. Is anyone actually doing that with AA and configuring site specific policy for any virtualization workload that SVirt provides coverage for. If Canonical's UEC is the next big thing...there's been a distinct lack of discussion about security measures to protect attacks from other clients. I'm not seeing any discussion anywhere about how to use AA to mitigate the sort of things the SVirt developers are working on. Hell, Eucalyptus requires you to put Selinux into permissive mode last I checked so using the Selinux support in Ubuntu is probably not an option at all..even if they did support SVirt functionality in the SELinux implementation they offer.

If you can point me to any discussion concerning how to build AA policy to address the issues that SVirt addressess...please...point me to it.

-jef

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 2:23 UTC (Tue) by dlang (guest, #313) [Link] (1 responses)

while I admit to taking things on a tangent away from the original article, I will say that I wasn't the one who first introduced AA.

no, I would not expect to see someone trying to use AA to create generic seperation between virtual machines.

I could very easily see someone configuring AA to lock down a specific virtual machine to keep it from accessing things that another virtual machine is allowed to access. it would be based on the knowledge of what that particular virtual machine is supposed to access.

but since it's possible to run the virtual machines as seperate users, it would also be very easy to use the standard unix permissions to isolate them from each other, with probably 90% of the security for <<10% of the effort of using something like SELinux (and probably <50% of the effort of using AA)

with AA, if you have well defined applications (which I freely admit many/most people don't) it's relatively easy to create an AA profile for it. each AA profile is pretty much independent from other AA profiles, so if you just check them for overlapping you can combine them easily.

SELinux doesn't have tools (at least not well publicized tools) that let you do this. instead you have to understand the entire system (all local uses as well as all remotely accessible applications) and then make one big policy that defines all files correctly for all the applications (and you still have the problem of creating the appropriate tags for new files that are created), it's not something that your normal sysadmin can do. it's very hard for even extremely experienced security people to do (and it doesn't help that the common response to problems is "you don't know enough to be tinkering there, just use what your distro provides you with")

SELinux is more powerful than AA, no question, but it's complexity (and interactions in it's config) make it hard enough to use that most people don't bother (and if you think that the distro default configs are appropriate, I disagree with you, they are far too open because the sitros can't possibly know what's appropriate for a particular installation)

AA is far less powerful, but (if it was available in the default kernel) far more likely to be used.

I am not an AA user, because (having been burned a few times), I make it a policy to not use features that aren't in the kernel.org kernel unless I _absolutely_ have to, and so far my need for additional local security have not been that high yet.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 2:28 UTC (Tue) by dlang (guest, #313) [Link]

by the way, I don't choose to use SELinux due to the complexity, but I do use commercial firewalls that use similar security systems, and my opinion of them is poor, due to the demand of users the real security of the system has been scaled back to the point that there is really almost nothing that can be done without being the equivalent of 'root', so the first command anyone ever types is makes them this root equivalent. this just about destroys the value of having separate users, let alone the MAC tagging and controls. the firewall vendor used to have several different roles, each with different permissions, but users of these expensive commercial firewalls were unhappy with that so the vendor scaled back the evvective security.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 9:31 UTC (Tue) by danpb (subscriber, #4831) [Link] (4 responses)

> I don't consider _any_ generic, out-of-the-box distro config to be
> reasonable (for AA or for SELinux). the specifics of what a particular
> service needs to do can change so drasticly from one application to
> another that there is no chance that a generic config can both protect the
> system and be broad enough to allow everything to work.

This may have been accurate in previous version of Fedora <= 10), but this is a problem we explicitly tried to address in the sVirt project for Fedora >= 11.

For example, in old way of doing things in Fedora 10, the policy required that all QEMU/KVM disk images lived under /var/lib/libvirt/images. To deviate from this, the admin would need to manually change the policy or relabel files.

In Fedora 11 with the introduction of sVirt, libvirtd itself takes care of all disk labelling. So no matter where you, as an admin, decide to place files for virtual disks, libvirtd will set the correct labelling. It will also set correct labelling if you give a guest a physical disk/partition from the host. So the admin does not have to do any manual labelling of disks, nor are they restricted in where they can store them.

In Fedora 12, the automatic labelling is being taken further, so that libvirt will set correct labelling for any host USB or PCI devices that are assigned to the guest OS, removing another configuration from the admin.

This is the benefit of integrating SELinux support directly into applications, instead of having it applied as a separate layer - you are no longer restricted to a static policy.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 19:24 UTC (Tue) by SEJeff (guest, #51588) [Link] (3 responses)

And what if you store your images on NFSv3 using something like a NetAPP filer? WAFL doesn't support anything fancy like SELinux xattrs last I checked. What would you suggest be done in this situation?

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 20:02 UTC (Tue) by mmcgrath (guest, #44906) [Link]

> And what if you store your images on NFSv3 using something like a NetAPP filer?

Most netapps fully support iscsi. That's how we run images in Fedora's Infrastructure, iscsi + lvm on netapps.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 22:18 UTC (Tue) by jamesmrh (guest, #31622) [Link]

We're working on getting labeling support into NFS -- it's not easy.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 22:55 UTC (Tue) by danpb (subscriber, #4831) [Link]

NFS really sucks in this respect and there's no real satisfactory answer, since SELinux requires xattr labelling support on FS. So you won't get the sVirt per-guest disk image isolation on NFS at this time, only block device, local filesystems, or more advanced network FS' like GFS which can do xattr.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 23:10 UTC (Tue) by jdstrand (guest, #53302) [Link] (4 responses)

AppArmor is the default MAC option in Ubuntu, and will continue to be for the foreseeable future. Our kernel team has done a lot of work this cycle to get AppArmor to use the new hooks that (the already accepted) TOMOYO uses, with the goal of submitting AppArmor upstream. Progress by our kernel team has been great, and we hope to get it submitted this development cycle.

Regarding sVirt and AppArmor, a patch has been developed to use the sVirt framework provided by libvirt. It works and is currently undergoing testing. Karmic will have libvirt 0.7.0 with an AppArmor security plugin, which should be available in Ubuntu within a few days. I'll be submitting the patch to upstream libvirt shortly thereafter.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 23:15 UTC (Tue) by jdstrand (guest, #53302) [Link] (3 responses)

I forgot to mention that the specification for this is https://wiki.ubuntu.com/SecurityTeam/Specifications/AppAr....

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 25, 2009 23:54 UTC (Tue) by jspaleta (subscriber, #50639) [Link] (2 responses)

Thanks, good to see. Beyond the specification is there an archived discussion regarding the AA plugin and default policy configurations?

-jef

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 28, 2009 17:21 UTC (Fri) by jdstrand (guest, #53302) [Link] (1 responses)

I'm not sure why this didn't go through when I responded the day you asked, but since it seems it won't on its own, I'll respond again...

There hasn't been public discussion yet, beyond the specification and occasional references on IRC. I look forward to discussions with upstream when I submit.

Walsh: Secure Virtualization Using SELinux (sVirt)

Posted Aug 28, 2009 18:03 UTC (Fri) by jspaleta (subscriber, #50639) [Link]

Just a word of advice...when the AA plugin does become available for testing..make an effort to publicly broadcast an explanation of an example AA policy working in the context of the SVirt plugin. An in depth explanation about how the AA policy works similar to what Walsh has done in his blog post to how the SELinux labelling works with SVirt.

-jef