Announcing qboot, a minimal x86 firmware for QEMU

[Posted May 21, 2015 by jake]

From:		Paolo Bonzini <pbonzini-AT-redhat.com>
To:		qemu-devel <qemu-devel-AT-nongnu.org>, KVM list <kvm-AT-vger.kernel.org>
Subject:		Announcing qboot, a minimal x86 firmware for QEMU
Date:		Thu, 21 May 2015 15:51:43 +0200
Message-ID:		<555DE2EF.1000509@redhat.com>
Archive‑link:		Article

Some of you may have heard about the "Clear Containers" initiative from
Intel, which couple KVM with various kernel tricks to create extremely
lightweight virtual machines.  The experimental Clear Containers setup
requires only 18-20 MB to launch a virtual machine, and needs about 60
ms to boot.

Now, as all of you probably know, "QEMU is great for running Windows or
legacy Linux guests, but that flexibility comes at a hefty price. Not
only does all of the emulation consume memory, it also requires some
form of low-level firmware in the guest as well. All of this adds quite
a bit to virtual-machine startup times (500 to 700 milliseconds is not
unusual)".

Right?  In fact, it's for this reason that Clear Containers uses kvmtool
instead of QEMU.

No, wrong!  In fact, reporting bad performance is pretty much the same
as throwing down the gauntlet.

Enter qboot, a minimal x86 firmware that runs on QEMU and, together with
a slimmed-down QEMU configuration, boots a virtual machine in 40
milliseconds[2] on an Ivy Bridge Core i7 processor.

qboot is available at git://github.com/bonzini/qboot.git.  In all the
glory of its 8KB of code, it brings together various existing open
source components:

* a minimal (really minimal) 16-bit BIOS runtime based on kvmtool's own BIOS

* a couple hardware initialization routines written mostly from scratch
but with good help from SeaBIOS source code

* a minimal 32-bit libc based on kvm-unit-tests

* the Linux loader from QEMU itself

The repository has more information on how to achieve fast boot times,
and examples of using qboot.  Right now there is a limit of 8 MB for
vmlinuz+initrd+cmdline, which however should be enough for initrd-less
containers.

The first commit to qboot is more or less 24 hours old, so there is
definitely more work to do, in particular to extract ACPI tables from
QEMU and present them to the guest.  This is probably another day of
work or so, and it will enable multiprocessor guests with little or no
impact on the boot times.  SMBIOS information is also available from QEMU.

On the QEMU side, there is no support yet for persistent memory and the
NFIT tables from ACPI 6.0.  Once that (and ACPI support) is added, qboot
will automatically start using it.

Happy hacking!

Paolo

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 21, 2015 17:27 UTC (Thu) by arjan (subscriber, #36785) [Link] (4 responses)

I for one like this... and if this becomes good enough for what we need for clear containers, we have no qualms switching over...

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 21, 2015 18:27 UTC (Thu) by SEJeff (guest, #51588) [Link] (1 responses)

bonus if it lacks support for legacy baggage like oh maybe floppy disks.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 21, 2015 19:18 UTC (Thu) by pbonzini (subscriber, #60935) [Link]

Nope, floppy disk support is in QEMU. This is firmware, so it's on the other side.

But really, the VENOM thing was massively overhyped.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 21, 2015 19:31 UTC (Thu) by pbonzini (subscriber, #60935) [Link] (1 responses)

Thanks Arjan! Your setup is really clever (in the good sense), and we look forward to implementing persistent memory in QEMU!

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 22, 2015 4:50 UTC (Fri) by edomaur (subscriber, #14520) [Link]

I like that spirit :-)

Will this work on Xen ?

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 21, 2015 19:39 UTC (Thu) by luto (guest, #39314) [Link] (17 responses)

Is this clever enough to cause the guest kernel not to bother with expensive legacy probes like PS/2?

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 21, 2015 19:41 UTC (Thu) by pbonzini (subscriber, #60935) [Link] (4 responses)

No, that's something that QEMU does. If the hardware is there, it cannot be hidden by firmware.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 21, 2015 19:48 UTC (Thu) by luto (guest, #39314) [Link] (3 responses)

Maybe what we need is a modern, minimal x86 QEMU machine type. I think I asked about that once.

At the very least, I want all the legacy storage stuff gone. Even AHCI is kind of slow to probe.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 21, 2015 21:13 UTC (Thu) by arnd (subscriber, #8866) [Link] (1 responses)

Or maybe even CPU-architecture independent minimal machine for qemu, similar to what ARM kvmtool provides. We need a clocksource and irqchip that are likely to remain architecture specific (we have the infrastructure to do this architecture-independent too, but that's probably not worth it for x86). Beyond that, basically everything can be done using virtio-pci.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 23, 2015 7:29 UTC (Sat) by drag (guest, #31333) [Link]

Or maybe even a idealized computer architecture that will do away with x86 baggage completely and allow you to create images that will run anywhere once programmed. We can then name it after a slang term for coffee.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 22, 2015 6:29 UTC (Fri) by pbonzini (subscriber, #60935) [Link]

Yes, definitely. You just need a PCI host bridge (and possibly a few more PCI bridges if the machine is PCIe). Hopefully the kernel can boot well with no ISA bus.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 21, 2015 20:38 UTC (Thu) by mjg59 (subscriber, #23239) [Link] (11 responses)

Part of this is fixing the kernel so it pays attention when the firmware tells us that there's no PS/2…

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 21, 2015 22:19 UTC (Thu) by jlargentaye (subscriber, #75206) [Link]

And part of that is trusting the firmware.

But you'd know something about that wouldn't you :)

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 22, 2015 18:56 UTC (Fri) by dlang (guest, #313) [Link] (8 responses)

don't compile ps/2 support into the kernel, then it doesn't even look for it.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 22, 2015 19:01 UTC (Fri) by mjg59 (subscriber, #23239) [Link] (7 responses)

And now you're maintaining your own kernel builds and you get to look after security updates yourself and no thanks.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 22, 2015 19:15 UTC (Fri) by dlang (guest, #313) [Link] (2 responses)

well, unless your distro provides you with a virtualized kernel, yes.

If you are after the highest performance (and when we are quibbling about 10s of ms of boot time, you are), then you don't want to use a stock distro kernel that probes for 1000 different things that you don't have, and enables all sort of features that each have a small performance impact because "someone" will want them.

As the earlier article said, Intel got the super-fast startup of their VMs by stripping out all the 'real hardware' drivers and only including the drivers for the virtual devices provided by the hypervisor that are effectively instant initialization.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 22, 2015 19:20 UTC (Fri) by mjg59 (subscriber, #23239) [Link] (1 responses)

We only probe for hardware at all because we're ignoring information the platform gives us telling us not to probe for legacy hardware. So sure, you can just remove the feature and increase your workload, or we could just fix the kernel so it behaves in a sensible way by default.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 22, 2015 23:18 UTC (Fri) by josh (subscriber, #17465) [Link]

What information does the platform have to offer to explicitly say "no PS/2 keyboard or mouse here, move along"? Just the absence of such devices in ACPI?

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 23, 2015 2:44 UTC (Sat) by ploxiln (subscriber, #58395) [Link] (3 responses)

I'm already maintaining my own kernel config (just a diff from the distro), mostly to remove autofs and some cgroups, which systemd uses unconditionally.

Oh, the days when my total list of services fit in one screen, and reading the list of mounted filesystems an easy glance at 4 or 5 lines...

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 23, 2015 11:51 UTC (Sat) by mathstuf (subscriber, #69389) [Link]

> Oh, the days when my total list of services fit in one screen, and reading the list of mounted filesystems an easy glance at 4 or 5 lines...

Heh. My service list doesn't fit, but all non-kernel processes fits on half a screen now ;) .

Plus, autofs is useful outside of systemd anyways. I use it for on-demand mounting and unmounting of devices via cd rather than udisks.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 24, 2015 11:55 UTC (Sun) by ibukanov (subscriber, #3942) [Link]

> mostly to remove autofs

What is wrong with autofs? I use it to mount and *unmount* sshfs shares on demand.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 25, 2015 9:44 UTC (Mon) by intgr (subscriber, #39733) [Link]

> Oh, the days when my total list of services fit in one screen

Oh, that's easy to fix, clearly you need to upgrade to a 4k-5k display!

Seems like a good idea until you hit problems like this: https://i.imgur.com/6swhGrr.jpg

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 22, 2015 23:19 UTC (Fri) by josh (subscriber, #17465) [Link]

At least in the short-term, it's possible to pass command-line options to disable such probing.

Ideally, there'd be a single command-line option (*possibly* available as a firmware option) to say "no, really, if you don't see it explicitly listed as a device then it isn't there, don't guess at any legacy PC hardware".

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 21, 2015 20:14 UTC (Thu) by geertj (guest, #4116) [Link] (4 responses)

This is cool in and by itself. But I don't see this replacing containers.

The way I think of containers is that they are a separate layer in the stack, not a replacement for virtualiziation. Containers can be used with virtualization, or without. The big benefit of containers is that they improve application manageability by making apps "transparent". For example:

- Running a command line with an attached console in the container.
- Getting log information from the container to the host.
- Passing file descriptors into the container (for socket activated apps)
- Injecting environment variables into the app (e.g. communicating which network port a daemon should use).
- Monitoring processes inside a container.
- Debugging an application running in a container.

True, there's also the efficiency benefits of containers like lower memory usage and shorter boot times. But in my view those are secondary.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 21, 2015 22:25 UTC (Thu) by sodabrew (guest, #95737) [Link] (1 responses)

> True, there's also the efficiency benefits of containers like lower memory usage and shorter boot times. But in my view those are secondary.

Those concerns are primary, along with reduced CPU overhead, and are top reasons why large deployments are looking into containers in lieu of virtualization.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 22, 2015 17:44 UTC (Fri) by winden (subscriber, #60389) [Link]

Some large deployments are using containers for some time already...

http://research.google.com/pubs/pub43438.html

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 22, 2015 0:25 UTC (Fri) by flewellyn (subscriber, #5047) [Link]

This isn't about replacing containers. This is about a new way to do containers, using virtualization to make them better isolated while enjoying the performance benefits of not doing full virtualization.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 22, 2015 13:38 UTC (Fri) by bradfitz (subscriber, #4378) [Link]

See http://lwn.net/Articles/644675/ on Clear Containers for some background.

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 22, 2015 4:21 UTC (Fri) by saipursnal (guest, #102702) [Link] (2 responses)

"boots a virtual machine in 40 milliseconds[2] on an Ivy Bridge Core i7 processor"
40 milliseconds is the time to load the kernel and initrd into guest memory or to the guest login?

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 22, 2015 6:28 UTC (Fri) by pbonzini (subscriber, #60935) [Link] (1 responses)

It's the time to execute the first instruction of vmlinuz. After that there's no difference between kvmtool and QEMU's operation, basically, and also no difference between QEMU+SeaBIOS and QEMU+qboot.

Announcing qboot, a minimal x86 firmware for QEMU

Posted Jun 5, 2015 6:56 UTC (Fri) by robbe (guest, #16131) [Link]

But the guest OS could act quite differently, be simpler, etc.

Do we need devicetrees for x86?

Announcing qboot, a minimal x86 firmware for QEMU

Posted May 24, 2015 12:02 UTC (Sun) by ibukanov (subscriber, #3942) [Link] (1 responses)

Presumably this is to enable massive amount of VMs. But why not simply clone one that already initialized the kernel before it executes the init process?

Announcing qboot, a minimal x86 firmware for QEMU

Posted Jun 5, 2015 6:52 UTC (Fri) by robbe (guest, #16131) [Link]

This could definitely be a win. Is this realised in any existing virtualisation/container technology?

Would also help with the RAM sharing as discussed under https://lwn.net/Articles/646761/