KVM, QEMU, and kernel project management [LWN.net]

KVM, QEMU, and kernel project management

Posted Mar 23, 2010 19:58 UTC (Tue) by mingo (subscriber, #31122)
Parent article: KVM, QEMU, and kernel project management

I still hope "perf kvm" will happen eventually: the tool is already useful even in its current prototype form and allows the analysis of guest overhead on the host side - and allows it to be compared to the host overhead.

Here's an example of the output:

[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms
--guestmodules=/home/ymzhang/guest/modules top

---------------------------------------------------------------------------
   PerfTop:   16010 irqs/sec  kernel:59.1% us: 1.5% guest kernel:31.9% 
              guest us: 7.5% exact:  0.0% [1000Hz cycles],  (all, 16 CPUs)
---------------------------------------------------------------------------

             samples  pcnt function                  DSO
             _______ _____ _________________________ _______________________

            38770.00 20.4% __ticket_spin_lock        [guest.kernel.kallsyms]
            22560.00 11.9% ftrace_likely_update      [kernel.kallsyms]
             9208.00  4.8% __lock_acquire            [kernel.kallsyms]
             5473.00  2.9% trace_hardirqs_off_caller [kernel.kallsyms]
             5222.00  2.7% copy_user_generic_string  [guest.kernel.kallsyms]
             4450.00  2.3% validate_chain            [kernel.kallsyms]
             4262.00  2.2% trace_hardirqs_on_caller  [kernel.kallsyms]
             4239.00  2.2% do_raw_spin_lock          [kernel.kallsyms]
             3548.00  1.9% do_raw_spin_unlock        [kernel.kallsyms]
             2487.00  1.3% lock_release              [kernel.kallsyms]
             2165.00  1.1% __local_bh_disable        [kernel.kallsyms]
             1905.00  1.0% check_chain_key           [kernel.kallsyms]
             1737.00  0.9% lock_acquire              [kernel.kallsyms]
             1604.00  0.8% tcp_recvmsg               [kernel.kallsyms]
             1524.00  0.8% mark_lock                 [kernel.kallsyms]
             1464.00  0.8% schedule                  [kernel.kallsyms]
             1423.00  0.7% __d_lookup                [guest.kernel.kallsyms]

(This profile shows an interesting phenomenon: ticket spin locks have way more overhead on the guest side than on the host side.)

To the observant reader the essence of the disagreement can be seen from that example already. The problem is that the following command:

perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms
--guestmodules=/home/ymzhang/guest/modules top

is fine for a prototype but is way too long and cumbersome for developers to type and use. It's a usability non-starter.

So i asked the KVM folks (this started the discussion) whether they'd accept two features that would enable perf to automate and shorten this into:

perf kvm top

That is where i ran into opposition (which was unexpected to me - these requests seemed rather natural to me) and that is when the flamewar erupted - and we need those features from KVM to support easier use and to support more advanced perf kvm features, and regarding those there indeed we are at an impasse.

I hope that it will happen eventually: multiple people expressed it in the thread that they find integration features like transparent, host-visible guest filesystems useful for many other purposes as well - not just for instrumentation.

Also, KVM will eventually have to face the guest enumeration problem as well - just like we enumerate network cards into eth0, eth1, eth2, etc. do we need a way to enumerate KVM guests programmatically and allow tools to connect to those guests.

Both got labeled as some sort of unreasonable, fringe requests in the heat of the discussion, but once the dust settles i hope someone will think it through and extend KVM with such kinds of features.

Thanks, Ingo

(Log in to post comments)

formatting issues

Posted Mar 23, 2010 20:12 UTC (Tue) by mingo (subscriber, #31122) [Link]

I cannot edit the text anymore (and it looked OK here in the preview and still looks readable on my browser) - but if anyone with access to the comment text wants to do it then please by all means feel free to break those lines (or trim/remove them).

Thanks, Ingo

formatting issues

Posted Mar 23, 2010 20:28 UTC (Tue) by corbet (editor, #1) [Link]

I took the liberty of breaking the heading line. You can always tell who prefers narrow browser windows and who doesn't....:)

formatting issues

Posted Mar 23, 2010 20:40 UTC (Tue) by martinfick (subscriber, #4455) [Link]

Perhaps there is a case for having a "no comments" link? Forgive me if this already exists and I missed it.

formatting issues

Posted Mar 23, 2010 21:54 UTC (Tue) by Frej (subscriber, #4165) [Link]

It's more usefull than you think. At least one browser relies on the layout of the page to
determine minimum window size! (Instead of switching beetween maximize<->custom size).
It is the stuff you don't see on screenshots - and a reason I miss safari :)....

KVM, QEMU, and kernel project management

Posted Mar 26, 2010 8:53 UTC (Fri) by ortalo (subscriber, #4654) [Link]

I propose the above comment (last sentence) for the next "LWN quote of the week".

KVM, QEMU, and kernel project management

Posted Mar 26, 2010 9:37 UTC (Fri) by mingo (subscriber, #31122) [Link]

When those "multiple people" will submit patches, said patches will be reviewed. Use distinctive subjects and do that in a separate thread.
Sorry, but what is your point?
And talk like a human - anything containing gems like "to jump to the next level of technological quality" will get a chance to find out which level of technological quality /dev/null has.

LOL. Nice soundbite, but is it fair for you to say that?

You characterise our efforts with a condescending tone but you don't actually make a point so it's hard for me to reply with exact details ...

Regarding "the next technological level" - that just matches our experience when we moved to tools/perf/. It's truly better compared to hacking on a separated package: it's more efficient, it leads to more and better changes, all in one it results in fundamentally better technology.

I shortened that into the "next level" because i think it is exactly that and i think you should try those waters before you form an opinion. (You, despite flaming it strongly, have still not tried to contribute a single line to that effort. You are certainly welcome!)

Al, a year ago you opposed the tools/perf/ move with very strong language and you predicted dire consequences before Linus said that he thinks you are wrong and overruled your opinion.

In hindsight (we are one year into tools/perf/) did your negative predictions about tools/perf/ materialize?

Thanks, Ingo

KVM, QEMU, and kernel project management

Posted Mar 26, 2010 21:43 UTC (Fri) by rahvin (subscriber, #16953) [Link]

I believe the response is (also as a non-hacker) that KVM is operated in a couple key situation that would either make that impossible or highly difficult. Elaborating, the data center is where most virtualization is being done, though it's moving into the local server cabinet. As a result you run into a couple situations, the first is that the people maintaining the server and the host have no data access to the guests for security reasons.

The second is that accessing the guest to run perf won't provide the complete picture and could mislead you. You really need perf on the host to see how the guest is interacting with the host and other guests. IIRC Vmware provides tools which allow this kind of performance analysis with their Sphere products and they don't require access to the hosts to do it.

KVM, QEMU, and kernel project management

Posted Mar 26, 2010 21:47 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]

in many cases you don't trust the people who have admin access inside the guest. As such you don't want this to involve anything that they could tinker with. This rules out any guest assisted schemes.

KVM, QEMU, and kernel project management

Posted Mar 27, 2010 1:57 UTC (Sat) by rahvin (subscriber, #16953) [Link]

That's what I meant to say with the security reasons comment. If that's not what I said your comment is an adequate clarification.

Basically you can't trust the guests and the guests can't trust the host as well. Contracted hosting is an interesting situation because neither group can realistically trust the other with their data because they don't screen each other's employees. Maybe there are hosting companies that offer to assign management screened by the contractee but it would seem easier to just put your own server in their rack and manage the whole thing yourself, otherwise you have to deal with a situation where no one can trust each other. So Emperor Mingo's comment that perf needs to be runnable without guest client access (or guest host access) is the reality of data center VM.

KVM, QEMU, and kernel project management

Posted Mar 27, 2010 2:12 UTC (Sat) by dlang (✭ supporter ✭, #313) [Link]

I disagree with you a bit.

I see it that the host does not want to trust the guest, but to a large extent the guest has no choice but to trust the host. There are just too many things that the host can do to the guest.

Remember the host has control of your cpu, your ram, and all your I/O

If the host really wants to it can search through the physical ram looking for 'interesting' data in the guest. the guest can try to keep things encrypted, but it has to store the encryption key somewhere, and wherever it is stored the host can get at it.

you can try to lock down the host to prevent it from getting at part of it's own resources (this is the type of thing that SELinux is designed for), but doing this completely is a very hard task, if it's even possible.

KVM, QEMU, and kernel project management

Posted Mar 29, 2010 19:07 UTC (Mon) by jeremiah (subscriber, #1221) [Link]

We run into this problem as well. We'd really like to offload some of our credit card processing but
we can't trust the hosts, much less the some of the other guests on the same host. My
understanding, is that guest to guest security has gotten better due to 'on chip' virtualization techs,
but I think the next place I'd like to see the KVM folks go is protecting the guest from the host as
much as possible.

As far as perf is concerned though, could this just be a parameter in the guest os that could disable
the feature. That way the guest could trust that it has disabled the host from spying or whatever on
it.

KVM, QEMU, and kernel project management

Posted Mar 29, 2010 21:03 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]

in many ways this is the same problem as a multi-user machine.

In theory SELinux can protect you, but you really have to trust both it's implementation and it's configuration. This requires placing a large amount of trust the the System Administrator team that you are trying to be protected from.

To some extent you either trust your system administrators or you don't.

If you don't how can you trust that they properly configured SELinux?

If you do, do you really need SELinux to be configured?

things do get a bit messier when you talk about multiple guests on one box and you want to make sure that you don't get attacked from the other guests, but there you can go a long way by simply having each guest run as a different user that has no permissions to anything else on the system (which does take careful auditing of the system, modern linux systems are not put together with multi-user security in mind)

but in my opinion, right now the real answer is that you really don't want to use virtualization as a security critical barrier between hostile parties and their targets.

KVM, QEMU, and kernel project management

Posted Mar 30, 2010 1:30 UTC (Tue) by jeremiah (subscriber, #1221) [Link]

And this is why we don't currently do it, or recommend it to others. I just think it would be nice for
a guest to be able to insure that the host couldn't access it in anyway. I don't think you could do
this in a non linux environment, but maybe though the sys api and have the guest kernel enforce it.
Who knows, but it sure would be nice.

KVM, QEMU, and kernel project management

Posted Mar 30, 2010 3:28 UTC (Tue) by dlang (✭ supporter ✭, #313) [Link]

given that the guest doesn't really control it's own ram, but the host OS does, there is no way that the guest can prevent the host OS from examining or changing the ram in the guest, there is no way for the guest to protect itself from the host if the host is malicious.

what is possible in theory is that the host could prevent one guest from escaping then using the host privileges to attack another guest. However this is the same theory that says that one user on a system can be prevented from attacking another user on the same system. That hasn't worked in real life, and I doubt if the protecting one guest form another will work much better.

KVM, QEMU, and kernel project management

Posted Mar 30, 2010 11:21 UTC (Tue) by jeremiah (subscriber, #1221) [Link]

I was thinking the the guest could encrypt or remap it's ram in a fashion that was known only to it.

KVM, QEMU, and kernel project management

Posted Mar 30, 2010 19:57 UTC (Tue) by nix (subscriber, #2304) [Link]

Sure it can. But the host can observe the guest's RAM, so can easily
acquire any necessary encryption keys and do the decryption itself. Even
if it got the key off the network, the host could spy on the network and
capture the key, or spy on the guest and watch the key come in, and then
capture it.

It is simply not possible to protect a VM guest from root on its host. The
host controls *everything*.