LWN: Comments on "Using the KVM API"

Using the KVM API

josh — Tue, 18 Feb 2020 16:31:37 +0000

Yes, rust-vmm is extremely satisfying, and I highly recommend it. I've experimented with it, and it makes simple VMs quite straightforward to write.

I gave a talk on the /dev/kvm API, Rust, and rust-vmm, at BangBangCon: https://www.youtube.com/watch?v=A_diEEpAfpM

Using the KVM API

dizz — Tue, 19 Nov 2019 11:00:36 +0000

This is a very helpful article (including the comments). Thanks for the time in putting it together. If folks are interested a recent rust-related development in coding against KVM is the rust-vmm project [1]. It aims to create a common base line of libraries (crates) to ease the development of VMMs/Hypervisors. One of the related projects to rust-vmm is Intel's cloud-hypervisor [2] and while looking through its code base I found the article's sample code implemented [3] in rust using parts of rust-vmm.

[1] https://github.com/rust-vmm
[2] https://github.com/intel/cloud-hypervisor
[3] https://github.com/intel/cloud-hypervisor/blob/master/vmm...

Using the KVM API

sahil — Tue, 04 Oct 2016 16:51:44 +0000

I initially thought that this number '0x3f8' has a special significance for kvm api, as in, kvm would recognize this address as an memory mapped I/O or some other special address. The truth is that KVM exits due to 'out' opcode, and not because of the address. In the host, we then check the 1st operand of this 'out' opcode matches the value we gave, '0x3f8' in this case. We can have any other number (as long as it is 1 byte long) as operand, thus even 'out 0x4000, al', or 'out 0x0000, al' (intel syntax) would work just fine.

Using the KVM API

jem — Tue, 04 Oct 2016 07:01:00 +0000

> I searched the Intel software developer manual, didn't find any reference to it. Seems, to have a purely historical meaning.

You should search the IBM PC reference manual instead.

Using the KVM API

sahil — Tue, 04 Oct 2016 06:28:15 +0000

I searched the Intel software developer manual, didn't find any reference to it. Seems, to have a purely historical meaning.

Using the KVM API

zlynx — Tue, 04 Oct 2016 04:15:28 +0000

As far back as I can remember that was the serial port address. It's historical.

See COM on Wikipedia.

Using the KVM API

sahil — Tue, 04 Oct 2016 02:24:17 +0000

Why is 0x3f8 chosen for serial port?
Is this related to the guest physical address range, in which we have mapped the memory?

Using the KVM API with ARM?

marcH — Thu, 19 Nov 2015 23:00:24 +0000

> http://systems.cs.columbia.edu/files/wpid-asplos2014-kvm.pdf

1972, 2005, 2014,...

In some respects computers can be quite depressing: mostly re-inventing the old https://en.wikipedia.org/wiki/Hardware-assisted_virtualiz...

Sure, they're now much faster... to crash.

(on a more positive note the article is really good)

Using the KVM API with ARM?

yehuday — Thu, 29 Oct 2015 02:52:03 +0000

> Is anyone among the readers familiar enough with ARM to comment on how similar an implementation of a simple VM there would be? Obviously an ARM version would not employ Intel VT,
> and ARM ISA has its own notion of privilege levels, to begin with.
ARM added support for virtualization as an extension to ARMv7 architecture which later become part of ARMv8.
Hardware support for virtualization on ARM is a achieved by:
1. introducing a new CPU privellege level for hypervisors called Hyp or Excpetion Level 2 (EL2).
2. Adding 2nd stage translation to the CPUs MMU
3. Adding support for virtual interrupts to the generic interrupt controller (GIC)
4, Adding support for virtual timer in the ARM architected timer

there is a great article from Christopher Dall et all explaining how KVM was ported to ARM
http://systems.cs.columbia.edu/files/wpid-asplos2014-kvm.pdf

Using the KVM API with ARM?

alison — Sat, 10 Oct 2015 21:00:06 +0000

> we'll set up an x86 virtual machine using Intel VT. For another platform, you'd need to handle
> different registers, different virtual hardware, and different expectations about memory layout
> and initial state.

Is anyone among the readers familiar enough with ARM to comment on how similar an implementation of a simple VM there would be? Obviously an ARM version would not employ Intel VT, and ARM ISA has its own notion of privilege levels, to begin with.

> Other than learning, debugging a virtual machine implementation, or as a party trick, why use
> /dev/kvm directly?

Please invite me to your parties.

Using the KVM API

LightDot — Thu, 08 Oct 2015 16:48:00 +0000

You can pass raw qemu command line in the libvirt XML. This is an equivalent of '-vnc :30,tls -k fr -no-fd-bootchk':

The VNC option doesn't need to be presented as a command line, I just left it as an example.

novm / kvmtool still alive?

sjj — Sun, 04 Oct 2015 04:10:12 +0000

Thanks, I'll play with kvmtool a bit. I guess https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmt... is the current canonical source?

Using the KVM API

josh — Fri, 02 Oct 2015 18:54:01 +0000

> Is there any way to tell if a given process has created a KVM/VCPU?

Nothing better than lsof or equivalent.

> Is there a limit as to how many can be created on the system total,

As far as I know, no limit on the number of VMs other than available memory, but at the moment some limits exist on the number of VCPUs per VM. Those limits seem fixable, not inherent, though.

> and how to enumerate all of the existing ones?

Not as far as I know.

Using the KVM API

kjp — Fri, 02 Oct 2015 14:36:29 +0000

This is pretty cool. Is there any way to tell if a given process has created a KVM/VCPU? Is there a limit as to how many can be created on the system total, and how to enumerate all of the existing ones?

Using the KVM API

lkundrak — Fri, 02 Oct 2015 10:13:30 +0000

An excellent article. Thank you!

Using the KVM API

pbonzini — Thu, 01 Oct 2015 16:02:42 +0000

Even if you implement in the kernel, it's hard to bypass all the stuff that KVM does at vmexit. The processor doesn't restore the whole host state at vmexit time, but KVM must restore at least whatever is needed to run Linux (e.g. the %fs base to access per_cpu variables).

novm / kvmtool still alive?

josh — Thu, 01 Oct 2015 15:58:32 +0000

> Last novm commit on github is 8 months old.

True, novm appears somewhat stale, and I don't know if it sees active use. I mentioned it only as an example of an interesting project using KVM directly.

> kvmtool that I found doesn't seem all that active either.

It does most of what people using it want; it tend to grow new features as the KVM API does, grow new architectures when KVM supports them and someone wants to run kvmtool on them, and occasionally gain new drivers. I'd recommend trying kvmtool in its current state; it works quite well.

Using the KVM API

josh — Thu, 01 Oct 2015 15:54:37 +0000

> They are needed for unrestricted_guest=0. Remember that, in that case, the VM always runs in protected mode and with paging enabled. In order to emulate real mode you put the guest in a vm86 task, so you need some place for a TSS and for a page table, and they must be in guest RAM because the guest's TR and CR3 points to it.

Ah, I see.

> If you don't call KVM_SET_TSS_ADDR you actually get a complaint in dmesg, and the TR stays at 0.

While I saw the mention of that message in a few places, I don't actually get that message at any point. Presumably that only happens with unrestricted_guest=0?

Please consider documenting the use of these two ioctls and the data they point to, as well as what circumstances require them; the current KVM documentation doesn't mention any of that.

Using the KVM API

josh — Thu, 01 Oct 2015 15:50:32 +0000

Ouch. That really limits how much faster it can get, then, short of implementing the backend in the kernel.

novm / kvmtool still alive?

sjj — Thu, 01 Oct 2015 15:39:29 +0000

Last novm commit on github is 8 months old. kvmtool that I found doesn't seem all that active either. Is there something shinier out there that is actively being developed?

I generally don't run non-linux VMs, so these would be nice. Everytime I see a cloud image pull in some firmware packages I cry a little inside.

Using the KVM API

pbonzini — Thu, 01 Oct 2015 08:21:35 +0000

They are needed for unrestricted_guest=0. Remember that, in that case, the VM always runs in protected mode and with paging enabled. In order to emulate real mode you put the guest in a vm86 task, so you need some place for a TSS and for a page table, and they must be in guest RAM because the guest's TR and CR3 points to it. They are invisible to the guest, because the STR and MOV-from-CR instructions are invalid in vm86 mode, but it must be there.

If you don't call KVM_SET_TSS_ADDR you actually get a complaint in dmesg, and the TR stays at 0. I am not really sure what kind of bad things can happen with unrestricted_guest=0, probably you just get a VM Entry failure. The TSS takes 3 pages of memory. An interesting point is that you actually don't need to set the TR selector to a valid value (as you would do when running in "normal" vm86 mode), you can simply set the base and limit registers that are hidden in the processor, and generally inaccessible except through VMREAD/VMWRITE or system management mode. So KVM needs to set up a TSS but not a GDT.

For paging, instead, 1 page is enough because we have only 4GB of memory to address. KVM disables CR4.PAE (page address extensions, aka 8-byte entries in each page directory or page table) and enables CR4.PSE (page size extensions, aka 4MB huge pages support with 4-byte page directory entries). One page then fits 1024 4-byte page directory entries, each for a 4MB huge pages, totaling exactly 4GB. Here if you don't set it the page table is at address 0xFFFBC000. QEMU changes it to 0xFEFFC000 so that the BIOS can be up to 16MB in size (the default only allows 256k between 0xFFFC0000 and 0xFFFFFFFF).

The different handling, where only the page table has a default, is unfortunate, but so goes life...

Using the KVM API

pbonzini — Thu, 01 Oct 2015 07:21:13 +0000

Yes, it's still VMX, but the guest has CR0.PE=1 and EFLAGS.VM=1.

> Also, why does it care how the guest->host physical mappings are set up?

Can you explain your question better? Who is the subject?

Using the KVM API

pbonzini — Thu, 01 Oct 2015 07:17:15 +0000

It's about 50-50.

Using the KVM API

kleptog — Thu, 01 Oct 2015 06:29:43 +0000

I set up a system locally to run programs under a VM in qemu-kvm with shared folders and it was insanely slow, especially git and find. Stracing on the host showed that for each stat() in the guest, the host was doing a stat() on each directory component. So a stat() of a complete directory tree went O(N^2) on the directory depth.

Now, I get that it's probably a configuration issue since it clearly wasn't caching anything, but I found it really hard to find documentation about qemu that explained this behaviour. On top of that I'm managing them via libvirt, so even if I find a command-line option to deal with something, if libvirt doesn't support it I'm still SOL.

Overall, it hasn't been a great experience, next time I'll probably do what other people do, use VirtualBox or VMWare.

But back to the article, it's a pretty nice interface actually. Hopefully I'll find some reason to use it sometime :)

Using the KVM API

rahvin — Thu, 01 Oct 2015 00:22:15 +0000

If you think about it, this is probably the very reason most of the KVM internals were already in the Linux Kernel when Xen and VmWare were arguing over the "standard" for the changes to the Linux Kernel they needed. It pretty obvious that this is the reason why to me. The changes are straightforward and simple.

Using the KVM API

josh — Wed, 30 Sep 2015 23:04:35 +0000

Incidentally, with your background in qemu and kvm, perhaps you could explain (or point to documentation explaining) the purpose of KVM_SET_TSS_ADDR and KVM_SET_IDENTITY_MAP_ADDR, and what cases need them? A simple VM works without them, but the documentation lists them both as required for x86 VM setup.

Using the KVM API

luto — Wed, 30 Sep 2015 21:16:50 +0000

> Note that KVM used QEMU to handle the boot phase only in the very beginning; nowadays it uses vm86 mode or an in-kernel interpreter if unrestricted guest mode is unavailable. This avoids duplicating the emulation code between the kernel and userspace. Userspace doesn't know if you're running on a >=Westmere processor (or AMD, which has always had the equivalent of unrestricted mode--they called it "paged real mode"), or if you need to work around the limitations of the processor.

Please tell me that this is at least *guest* vm86 mode and not host vm86 mode.

Also, why does it care how the guest->host physical mappings are set up?

Using the KVM API

josh — Wed, 30 Sep 2015 17:53:26 +0000

> For ioeventfd you have to add the latency of waking up the I/O thread if it's sleeping (but if the fd is really busy, e.g. running fio in the guest, it won't have time to go to sleep).

How much *minimum* latency comes from the vmexit, and how much gets added by the path from the in-kernel vmexit handling and whatever mechanism it uses to contact the I/O thread? If much of it comes from the latter, perhaps we could find a way to accelerate that via another (latency-optimized) interface.

Using the KVM API

kvaneesh — Wed, 30 Sep 2015 17:49:17 +0000

With cache=loose, we should used the cached value in the inode in getattr (v9fs_vfs_getattr_dotl). So stat should be fast.

Using the KVM API

josh — Wed, 30 Sep 2015 17:34:45 +0000

I've been thinking about using the KVM API to prototype a new virtio filesystem, both for simplicity (this shouldn't need a full network filesystem) and to improve on 9p's very high access latency. (Try building a kernel under virtio-9p, and compare to host performance building the same kernel.) 9p provides reasonable throughput, but not reasonable latency. Given the ability to map guest memory, file data access could occur by mmap()ing the file on the host and then handing that memory to the guest; however, that doesn't address the performance of calls like stat(), which need to run absurdly fast and absurdly often.

Using the KVM API

felix.s — Wed, 30 Sep 2015 17:15:43 +0000

There's vvfat, but QEMU people say it's deprecated, and writing (which has to be manually enabled) currently seems broken. I found a virtio-9p driver for DOS: <http://www.zytor.com/pub/git/dos/virtio9p.git/>, but it's half-done, seems abandoned, and doesn't even compile properly as of now (binutils' fault though; bug 18963). On the other hand, the missing part is mostly the "boring" stuff, which is actually translating DOS file system accesses into 9P protocol exchanges (including file name translation). Maybe you can persuade hpa to get back to it :)

Also, funnily enough, some time ago I've been working on a DOS/BIOS ABI layer based on KVM (I tried to make backends interchangeable, but I'm not sure how well I've succeeded), and I think it would be ideal for the use case you describe. I even managed to include a simplistic packet driver, so I can use the FDNPKG package manager to download programs to test. However, the code is currently such a mess that I'm too embarrassed to publish it. Maybe some day...

Using the KVM API

eru — Wed, 30 Sep 2015 15:28:47 +0000

I probably should look into qemu again some day. One problem is file system access. As noted, I want the MS-DOS compilers to transparently compile sources in the Linux file system and write the objects there, and preferably without having to install any network support in the emulated MS-DOS or FreeDOS, so to leave maximum "real" memory for the compilers. Both dosemu and dosbox handle this requirement.

Using the KVM API

pbonzini — Wed, 30 Sep 2015 13:46:54 +0000

What makes you use dosemu or dosbox instead of QEMU (with or without KVM)?

Using the KVM API

pbonzini — Wed, 30 Sep 2015 13:46:03 +0000

I don't know, I play the shareware version. :)

Using the KVM API

pbonzini — Wed, 30 Sep 2015 13:45:05 +0000

> True. Out of curiosity, does any means exist to turn that *off*? I have some interest
> in compiling out most or all of the in-kernel instruction emulation, to reduce attack
> surface area.

With unrestricted_guest=1 you only exit to the emulator for a few privileged instructions (where for simplicity KVM emulates them instead of having a mini-interpreter in vmx.c/svm.c) and for I/O. But unfortunately, thanks to the x86 ISA's read-modify-write instructions that's still a _lot_ of different instructions that you can emulate.

So there's not much that you can compile out. You could simply modify KVM to refuse loading if unrestricted_guest=0, but you can still trigger any bit of emulator code by setting up a race between two VCPUs. One triggers I/O continuously, the other races against the emulator changing the opcodes of the I/O instruction into something else. This actually used to be a vulnerability, but it's been patched for several years and the emulator is now considered a security sensitive component.

> > A KVM_EXIT_IO exit is very slow, on the order of a few thousand cycles on the
> > newest processors. By comparison, QEMU can dispatch a single memory-mapped I/O
> > operation in about 100 clock cycles, so 60-150 times faster than KVM.
>
> What about with coalesced or fd-ed I/O?

Still around 1500-2000 cycles. For ioeventfd you have to add the latency of waking up the I/O thread if it's sleeping (but if the fd is really busy, e.g. running fio in the guest, it won't have time to go to sleep).

Using the KVM API

eru — Wed, 30 Sep 2015 06:17:51 +0000

on one hand you don't need near bare-metal performance that KVM provides, because dosemu/dosbox only need to emulate a 100 MHz machine or so, and a simple interpreter or a JIT compiler like QEMU's can handle it

Probably true for old games, but the situation I am thinking of involves using ancient cross-compilers to compiler large masses of legacy code for a weird environment that still has to be maintained. One would think (and I did think) this is an I/O-bound operation, but it turned out the speed difference between dosemu with VM86 and dosemu on x86_86 with emulation is very noticeable (order of magnitude for large inputs). On the other hand, dosemu also has advantages, because it can run "headless", can easily access native files, and starts up quickly. These are important features, because the ancient compilers are wrapped in layers that hide their MS-DOS internals, so from the Linux user's point of view they act like normal command-line tools.

Using the KVM API

voltagex — Wed, 30 Sep 2015 03:42:40 +0000

> (man, I should send those Jazz Jackrabbit patches upstream...)

Yes, yes you should. By the way, can you still buy that game? That and OMF 2097 are my favourites of all time.

Using the KVM API

josh — Tue, 29 Sep 2015 21:26:24 +0000

True. Out of curiosity, does any means exist to turn that *off*? I have some interest in compiling out most or all of the in-kernel instruction emulation, to reduce attack surface area.

> A KVM_EXIT_IO exit is very slow, on the order of a few thousand cycles on the newest processors. By comparison, QEMU can dispatch a single memory-mapped I/O operation in about 100 clock cycles, so 60-150 times faster than KVM.

What about with coalesced or fd-ed I/O?

Using the KVM API

pbonzini — Tue, 29 Sep 2015 21:18:45 +0000

Note that KVM used QEMU to handle the boot phase only in the very beginning; nowadays it uses vm86 mode or an in-kernel interpreter if unrestricted guest mode is unavailable. This avoids duplicating the emulation code between the kernel and userspace. Userspace doesn't know if you're running on a >=Westmere processor (or AMD, which has always had the equivalent of unrestricted mode--they called it "paged real mode"), or if you need to work around the limitations of the processor.

But KVM is not really needed in this case: on one hand you don't need near bare-metal performance that KVM provides, because dosemu/dosbox only need to emulate a 100 MHz machine or so, and a simple interpreter or a JIT compiler like QEMU's can handle it (QEMU is known as slowish for a JIT translator, but there's some work being done on that side as well). On the other hand KVM's performance comes with some fine print, which you cannot really afford in the case of dosemu/dosbox. A KVM_EXIT_IO exit is very slow, on the order of a few thousand cycles on the newest processors. By comparison, QEMU can dispatch a single memory-mapped I/O operation in about 100 clock cycles, so 60-150 times faster than KVM. Hence running demos like Unreal (https://www.youtube.com/watch?v=VjYQeMExIwk#t=7m) doesn't work too well on QEMU with KVM because they do an insane number of such exits.

To play old games (man, I should send those Jazz Jackrabbit patches upstream...) I typically use QEMU without KVM.

Using the KVM API

drag — Tue, 29 Sep 2015 20:17:28 +0000

Well if you look at what KVM does... all it ultimately does is allow you to execute ring0 code in ring2 on a x86 cpu. That way you don't have to modify the OS kernel to be able to execute it as a application. The alternative is to do what Xen did with Linux and patch it so that you can run the Linux kernel more like a regular application.

Previously you had kernel mode and then you had usermode ways to execute code. Now you have a third way, the kvm way. There isn't much to emulate because it's not using emulation at all. It's just executing the code directly on your processor. The only really magic/complicated stuff that happens is to deal with memory addressing for the guest OS.

Qemu does all the heavy lifting as far as 'system emulator' goes.