Using the KVM API [LWN.net]

Using the KVM API

Posted Sep 29, 2015 21:26 UTC (Tue) by josh (subscriber, #17465)
In reply to: Using the KVM API by pbonzini
Parent article: Using the KVM API

> Note that KVM used QEMU to handle the boot phase only in the very beginning; nowadays it uses vm86 mode or an in-kernel interpreter if unrestricted guest mode is unavailable. This avoids duplicating the emulation code between the kernel and userspace.

True. Out of curiosity, does any means exist to turn that *off*? I have some interest in compiling out most or all of the in-kernel instruction emulation, to reduce attack surface area.

> A KVM_EXIT_IO exit is very slow, on the order of a few thousand cycles on the newest processors. By comparison, QEMU can dispatch a single memory-mapped I/O operation in about 100 clock cycles, so 60-150 times faster than KVM.

What about with coalesced or fd-ed I/O?

to post comments

Using the KVM API

Posted Sep 30, 2015 13:45 UTC (Wed) by pbonzini (subscriber, #60935) [Link] (4 responses)

> True. Out of curiosity, does any means exist to turn that *off*? I have some interest
> in compiling out most or all of the in-kernel instruction emulation, to reduce attack
> surface area.

With unrestricted_guest=1 you only exit to the emulator for a few privileged instructions (where for simplicity KVM emulates them instead of having a mini-interpreter in vmx.c/svm.c) and for I/O. But unfortunately, thanks to the x86 ISA's read-modify-write instructions that's still a _lot_ of different instructions that you can emulate.

So there's not much that you can compile out. You could simply modify KVM to refuse loading if unrestricted_guest=0, but you can still trigger any bit of emulator code by setting up a race between two VCPUs. One triggers I/O continuously, the other races against the emulator changing the opcodes of the I/O instruction into something else. This actually used to be a vulnerability, but it's been patched for several years and the emulator is now considered a security sensitive component.

> > A KVM_EXIT_IO exit is very slow, on the order of a few thousand cycles on the
> > newest processors. By comparison, QEMU can dispatch a single memory-mapped I/O
> > operation in about 100 clock cycles, so 60-150 times faster than KVM.
>
> What about with coalesced or fd-ed I/O?

Still around 1500-2000 cycles. For ioeventfd you have to add the latency of waking up the I/O thread if it's sleeping (but if the fd is really busy, e.g. running fio in the guest, it won't have time to go to sleep).

Using the KVM API

Posted Sep 30, 2015 17:53 UTC (Wed) by josh (subscriber, #17465) [Link] (3 responses)

> For ioeventfd you have to add the latency of waking up the I/O thread if it's sleeping (but if the fd is really busy, e.g. running fio in the guest, it won't have time to go to sleep).

How much *minimum* latency comes from the vmexit, and how much gets added by the path from the in-kernel vmexit handling and whatever mechanism it uses to contact the I/O thread? If much of it comes from the latter, perhaps we could find a way to accelerate that via another (latency-optimized) interface.

Using the KVM API

Posted Oct 1, 2015 7:17 UTC (Thu) by pbonzini (subscriber, #60935) [Link] (2 responses)

It's about 50-50.

Using the KVM API

Posted Oct 1, 2015 15:50 UTC (Thu) by josh (subscriber, #17465) [Link] (1 responses)

Ouch. That really limits how much faster it can get, then, short of implementing the backend in the kernel.

Using the KVM API

Posted Oct 1, 2015 16:02 UTC (Thu) by pbonzini (subscriber, #60935) [Link]

Even if you implement in the kernel, it's hard to bypass all the stuff that KVM does at vmexit. The processor doesn't restore the whole host state at vmexit time, but KVM must restore at least whatever is needed to run Linux (e.g. the %fs base to access per_cpu variables).