KQEMU 1.3.0pre10 released - under the GPL

[Posted February 6, 2007 by corbet]

KQEMU is the kernel acceleration module for the QEMU processor emulation system. As can be seen in the KQEMU changelog, the 1.3.0pre10 release quietly added a couple of interesting features: full x86_64 virtualization support and a license change to version 2 of the GPL. Since KQEMU had previously been released under a proprietary license, this is a welcome change. (Thanks to Tzvetan Mikov).

to post comments

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 15:23 UTC (Tue) by ajross (guest, #4563) [Link] (17 responses)

Can anyone knowledgable comment on how this differs from the kvm stuff? The kvm environment is already a kernel-side virtualization framework that uses QEMU as its userspace hardware implementation.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 15:27 UTC (Tue) by alexl (guest, #19068) [Link]

kvm requires hardware virtualization support, kqemu doesn't (its similar to vmware in that sense).

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 15:29 UTC (Tue) by glikely (subscriber, #39601) [Link]

IIRC, kvm requires hardware virtualization support from the CPU. kqemu does not.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 15:32 UTC (Tue) by jdahlin (subscriber, #14990) [Link] (14 responses)

kvm depends on new hardware support (Intel VT/AMD-V), which has only been available a year or two.
kqemu is similar to vmware, it uses segmentation tricks to execute user mode code natively and translates the rest on the fly [which requires kernel support, thus requiring a kernel module].

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 16:56 UTC (Tue) by nix (subscriber, #2304) [Link] (12 responses)

Nope, it's smarter than that :) only code running with interrupts off needs complete interpreting. In user mode, it runs at near-native speed; in kernel mode, all segment loads must be intercepted so it's slower.

See the rather good technical docs. (Compare this to the way many companies in particular free code: it's the opposite of the usual 'tossed over the wall without documentation' rubbish.)

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 18:59 UTC (Tue) by mikov (guest, #33179) [Link] (11 responses)

To me the really impressive part of QEMU is not the virtualization using the kernel module, but rather the JIT-based emulation (which AFAIK has always been GPL).

When not using the kernel module, QEMU dynamically recompiles all guest instructions to the host instruction set, which makes it much faster than traditional emuilators like Bochs. It is actually possible to run Windows in QEMU in such emulated mode, and it is completely usable. Additionally, in this mode QEMU can emulate x86 with good speed running on any host (e.g. PowerPC). On top of that it is a 100% user node app, with no privileges (doesn't even have to run as root) so it is completely safe and secure.

AFAIK, QEMU uses GCC to generate binary code for the emulated instructions and then simply chains together these binary blobs. This allows it to be (almost) trivially ported to any architecture supported by GCC.

I think there is an opportunity for somebody to improve QEMU by replacing the code generation engine with a hand-tuned one (sacrificing the easy portability to any host). Perhaps LLVM could be used. I think this can bring the emulation speed from its current 10x slowdown to about 3-5x (without any kernel code!). This is a very exciting area.

QEMU slowdown and portability

Posted Feb 6, 2007 19:37 UTC (Tue) by jreiser (subscriber, #11027) [Link] (4 responses)

It is actually possible to run Windows in QEMU ... and it is completely usable.

QEMU and KQEMU are impressively good, but not that good. Installing TurboTax onto WindowsXP Pro running under Win4Lin Pro 3.0-0.7 [uses KQEMU] took around two hours, versus just a few minutes under native WindowsXP Pro running on the same hardware: a Pentium4 with 1GB RAM and "nothing" else running. Also, CPU utilization for (K)QEMU usually exceeds 90% even for simple things such as actually running TurboTax. WinXP graphics such as progress bars frequently do not get updated at all under Win4Lin. This shows that there is very little CPU power available for background tasks. TurboTax downloading updates thinks that something has gone wrong because the slowdown is excessive.

QEMU uses GCC ... (almost) trivially ported to any architecture supported by GCC

Except that QEMU's strategy requires a GCC version less than 4.0.

QEMU slowdown and portability

Posted Feb 6, 2007 23:54 UTC (Tue) by Richard_J_Neill (subscriber, #23093) [Link]

I think you've been unlucky. At any rate, I find Windows XP fairly usable(*) when running under Qemu/KQemu on a fast Pentium4. The slowdown is about 2-fold. This could be a Win4Lin-ism; alternatively, it could be a known bug (sorry, can't recall the reference) that is fixed by installing WinXP SP2.

(*)Disclaimer: I am not implying that Windows is ever truely "usable"!

QEMU slowdown and portability

Posted Feb 7, 2007 0:19 UTC (Wed) by mikov (guest, #33179) [Link]

I am not closely familiar Win4Lin, but I am pretty sure that you cannot draw conclusions about QEMU from Win4Lin. They are completely different products, so I don't really see the relevance. AFAIK, Win4Lin does not use QEMU's user mode component, which is quite important.

I suggest you test with QEMU and install Win2K instead of WinXP before you form an opinion. I have never experienced such terrible performance as you are describing even without the accelerator. When running with full kernel virtualization (which was added recently,AFAIK), I found the performance comparable to VMWare. Plus, it certainly did not take me many hours to install Windows itself, and it is much larger than TurboTax.

I fail to see your point about GCC 4: this is not a fundamental technological limitation. Plus, there are no significant host architectures supported only by GCC4 and not by GCC3 (are there any at all?).

BTW, I am not affiliated with QEMU in any way, except by being a happy user. It has literally enabled me to do things that I didn't think possible, like running Windows-only applications as software appliances on a headless Linux box, which I find to be incredibly cool.

I will grant you that QEMU is not perfect though - for example saving and loading the virtual machine state doesn't work well yet, as of version 0.9.

QEMU slowdown and portability

Posted Feb 7, 2007 1:18 UTC (Wed) by mgalgoci (guest, #24168) [Link] (1 responses)

I believe you are mistaken about Win4Lin. Win4Lin is a linux port of ancient SCO software called "Merge". It even goes so far as to replace windows DLLs with special versions that attach to hooks inside linux, so it's not really virtualization as much as it is running two OS at the same time.

http://www.sco.com/products/merge/

Of course that was last time I looked, which was like three years ago. Maybe Win4Lin swapped out their core technology for something more sane.

QEMU slowdown and portability

Posted Feb 7, 2007 5:59 UTC (Wed) by rloomans (subscriber, #759) [Link]

That, indeed, was the previous version. It only supported up to Win98/Me. The current version
appears to be based on QEMU and supports Win2k and up.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 7, 2007 22:08 UTC (Wed) by danshearer (guest, #18686) [Link] (5 responses)

> AFAIK, QEMU uses GCC to generate binary code for the emulated instructions > and then simply chains together these binary blobs. This allows it to be
> (almost) trivially ported to any architecture supported by GCC.

That was the idea, but it is quite fragile and tied to specific GCC versions.

> I think there is an opportunity for somebody to improve QEMU by replacing
> the code generation engine with a hand-tuned one (sacrificing the easy
> portability to any host). Perhaps LLVM could be used. I think this can
> bring the emulation speed from its current 10x slowdown to about 3-5x
> (without any kernel code!). This is a very exciting area.

See https://nowt.dyndns.org/ where Paul Brook has done exactly this, and I think Fabrice intends to merge Paul's code.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 8, 2007 0:16 UTC (Thu) by mikov (guest, #33179) [Link] (2 responses)

See https://nowt.dyndns.org/ where Paul Brook has done exactly this, and I think Fabrice intends to merge Paul's code.

Very interesting. Are there any details available on the implementation of the generator ? Intermediate representation, what optimizations it performs, etc ? Or any performance benchmarks ?

I couldn't find anything relevant with Google. All I was able to find was one sentence claiming 30% improvement, which is somewhat disappointing.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 8, 2007 11:09 UTC (Thu) by danshearer (guest, #18686) [Link] (1 responses)

> Very interesting. Are there any details available on the implementation of
> the generator ?

Not that I know of, apart from the code.

If you have the skills, why not document what you see as you read the code? That would be helpful to people like me. I am playing with Paul's work because it addresses a problem I care about (portability, preventing me from creating internal test suites for QEMU) but so long as it works my interests lie elsewhere inside QEMU. But if there was a guided tour I'd have a look for sure.

Dan

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 8, 2007 20:12 UTC (Thu) by mikov (guest, #33179) [Link]

I think the original author is in the best position to write something like that. It shouldn't take long to describe the basic idea of what he has done - just an outline. Anyway, I think writing a guide is not so much an issue of skill, but of time ... :-(

HP's Dynamo, Intel's IA32-EL, Transmeta and others have shown that there is definite potential in re-optimizing binary code. It is possible that an advanced optimizing code generator for QEMU could even outperform native code! Of course it would have to go significantly beyond the current QEMU infrastructure and rely on dynamic recompilation, trace optimization, etc. Some more info on this subject is available here: http://www.cag.csail.mit.edu/rio/#pubs/

In any case I will definitely take a closer look at Paul Brook's work. BTW, how does it address portability ?

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 8, 2007 11:12 UTC (Thu) by jfj (guest, #37917) [Link] (1 responses)

> That was the idea, but it is quite fragile and tied to specific GCC versions.

Some people bring this up as if it's a bad thing.

Why exactly it is do difficult to download gcc-core-4.0 and install it to /opt/gcc4.0 and use it for qemu?

The installation of gcc is very clean and easy, with modern computers it won't take more than 10 minutes, there are lots of mirrors of GNU software and you will always be able to find gcc-4.0.

And we are talking about developers. Distributions can arrange binaries that won't need gcc 4.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 8, 2007 11:29 UTC (Thu) by danshearer (guest, #18686) [Link]

> > That was the idea, but it is quite fragile and tied to specific GCC
> > versions.
> Some people bring this up as if it's a bad thing.
:
> And we are talking about developers. Distributions can arrange binaries
> that won't need gcc 4.

1. gcc 4 doesn't neccesarily work at all on a given architecture, and this will always be the case in some circumstances as versions shift under our feet. I was in this situation with QEMU on a pure AMD64 setup last year.

2. Most QEMU use is not production. For experimenting and testing people often want to compile QEMU, even if just to incorporate a patch someone posted. Such QEMU users usually have some other primary goal -- testing networking maybe, or some funny OS -- and building GCC isn't necessarily something they will think of or even be able to easily do.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 17:00 UTC (Tue) by mces (subscriber, #27668) [Link]

Looking at the "Future Developments" section in the "Documentation" page, this may sooner or later change:

"Support for hardware virtualization. The performance gains, if any, will be small but there would be no limitations regarding what the guest OS can do."

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 15:28 UTC (Tue) by orospakr (guest, #40684) [Link] (9 responses)

GPL, huh?

No doubt Mr. Bellard was starting to feel the squeeze from all the other virtualisation 'solutions'.

It was getting pretty obvious that no one was going to license the kqemu technology from him. There are already many other proprietary programs that do exactly the same thing (not to mention all the other Free programs like kvm and Xen, even if they do require the VT/Pacifica extensions to run unmodified operating systems).

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 15:34 UTC (Tue) by jdahlin (subscriber, #14990) [Link]

Win4Lin already licensed (K)QEMU.

I bet this is related to do with the recent GPL:ing of VirtualBox.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 16:43 UTC (Tue) by amando (guest, #36597) [Link] (3 responses)

``GPL, huh?
No doubt Mr. Bellard was starting to feel the squeeze from all the other virtualisation 'solutions'.''
This is not very nice.

I think it is great that qemu have become fully GPL.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 16:55 UTC (Tue) by rvfh (guest, #31018) [Link] (2 responses)

This was indeed not a nice thing to say. What I would rather say is:
Fabrice, un grand merci.

Note that Fabrice has been quite a genius so far, so I expect a number of
other great stuff to come from him. Maybe he GPL'd kqemu so others can
take the project further while he is busy on his next idea(s)?

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 17:57 UTC (Tue) by sjj (guest, #2020) [Link] (1 responses)

Absolutely.

Bellard is a hacker of the first order (look at qemu, ffmpeg, tcc). I hope the explosion of virtualization products that his work has helped bring about (I think at least virtualbox and kvm borrow from qemu) will bring him some financial rewards as well.

I sure hope he keeps on doing what he has been doing!

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 7, 2007 0:12 UTC (Wed) by drag (guest, #31333) [Link]

KVM is all Qemu userland.

KVM itself does not do anything that is Qemu-specific.. it can be used with any sort of VM environment you would like to port to use it...... but so far a hacked Qemu is all there is.

Xen uses Qemu stuff somewhat also for their stuff.

It's too bad that Kqemu has taken so long to become GPL'd. It's been around for a long time now, but it's better late then never and it is quite usefull for non-VT/SVM enabled machines.

Also hopefully the kernel hackers can look at the Kqemu code and figure out ways to fix some of KVM's existing limitations.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 16:44 UTC (Tue) by nix (subscriber, #2304) [Link] (2 responses)

Also, Fabrice is maintainer of a number of other GPLed programs so was probably waiting until (he could afford it | he'd obviously make no money off it) when it was proprietary. i.e., it was probably not a matter of `if' but `when'.

Welcome news, definitely!

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 17:31 UTC (Tue) by nix (subscriber, #2304) [Link] (1 responses)

Plus of course kqemu or no, qemu on its own is astonishingly cool and has added vastly to what my machine can do (and to debugging as well, of course, in a sense it's like being able to freeze time for tricky bugs, always hard when running on a physical CPU ;)) )

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 7, 2007 4:25 UTC (Wed) by robert_s (subscriber, #42402) [Link]

Yes - QEmu ( in all its variants - cross-arch JIT , kqemu , kvm etc. ) is a very important hacking tool. It's one of the areas I believe should get more attention due to its potential importance to Free software.

I also hope in the near future all QEmu's variants will undergo some kind of unification.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 7, 2007 1:45 UTC (Wed) by rganesan (guest, #1182) [Link]

You know, that's quite uncharitable. QEMU is used in a lot of projects including VirtualBox and KVM. If the author has chosen to release a key piece of work under GPL, whatever the reasons, I for one welcome it and applaud it.

KQEMU in Linux?

Posted Feb 6, 2007 17:23 UTC (Tue) by rvfh (guest, #31018) [Link]

Now that it's GPLv2, it would be nice for it to be merged into Linux...
Or maybe turn the KVM stuff compatible with older processors?

works as usual! :)

Posted Feb 6, 2007 17:55 UTC (Tue) by gvy (guest, #11981) [Link]

Runs like charm along with QEMU 0.9.0 on ALT Linux.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 18:11 UTC (Tue) by 01101010 (guest, #43187) [Link]

Virtualbox does the same thing as QEMU+KQEMU does, and I suppose both kernel modules have the same purpose. Are the modules supposed to be added to mainline Linux? Could both virtualization systems use a common kernel module?

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 6, 2007 18:25 UTC (Tue) by drosser (guest, #29597) [Link]

Does Fabrice have a PayPal or other sort of commerce account? I looked at qemu.org and didn't find anything. I'd like to at least buy him lunch since that's cheaper than upgrading my CPU to something that supports KVM.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 7, 2007 10:38 UTC (Wed) by irios (guest, #19838) [Link] (4 responses)

Am I the only one that is totally confused with all the options there are for virtualization?

I can recall kvm, xen, qemu, kqemu, user mode linux, virtual box, vmware (player, server and whatnot), Microsoft's virtualization solution, Win4Lin ... Parts of some are included in others, some seem to be compatible with others, some are totally free, some partially free, some only free as in beer, some totally unfree.

Is one solution going to become the winner, or maybe a couple, or is the panorama going to get muddier and muddier? Anybody knows of a good reference comparing the abilities of the different virtualizers?

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 7, 2007 12:26 UTC (Wed) by 01101010 (guest, #43187) [Link]

There will not be a winner, different virtualizers exist for different needs. Do you need complete emulation of an x86-system on any architecture? Only Bochs provides that, because it simulates a PC using C. But it is slow. If your CPU is supported by QEMU, which recompiles some code on the fly, it will not be that slow. Qemu can emulate different CPUs, not only x86. The common case of emulating x86 on x86 can be faster using native code execution when possible. KQemu and virtualbox do that (both extend qemu). Does your CPU support VT? Use KVM, then Qemu just has to emulate the hardware around the CPU. Dont want a full OS on top of the emulated machines? Xen provides that, also using VT and Qemu. Can still be faster if your guest os can be ported to the virtualizer, so you don't have to emulate complete hardware. Xen can do that. lguest uses a usual linux kernel as the hypervisor instead of Xen, but is slower and does not have all the interesting features Xen has (SMP, live migration). User mode linux is a port running as a userspace process, but cannot reach the speed of Xen. Sometimes you don't even want seperate kernels, just isolated userspace environments. That saves ram and is easier to manage. OpenVZ provides that.

So I think Qemu, either Qemu+KQemu or Virtualbox, Qemu+KVM, Xen, Xen+VT+Qemu, OpenVZ and something like lguest will stay alive for a long time, because each has a different approach. But that seems to be a full panorama.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 8, 2007 13:27 UTC (Thu) by drag (guest, #31333) [Link] (2 responses)

I wrote something a while ago. Keep in mind that I am very amature at this sort of thing. http://aplawrence.com/Linux/kvm_virtualization.html

Pretty innaccurate, but I suppose you can get my meanings by my descriptions. I know more about this stuff now.

Basicly you have hypervisors, VM-as-a-application, containers, and emulation. All these things overlap to one degree or another and it's very difficult to put some things into nice catagories.

But generally speaking you have hypervisors. Examples are going to be like Xen or Vmware ESX. These things are specialized kernel-like things that are specially designed to create VMs. Then in addition you'll have a userland based on a OS to manage and provide I/O emulation. Both of these things use Linux for that to one extent to another.

These provide the highest level of sophistication, isolation, and management options and generally have the best performance of a VM capable of running different sorts of operating systems. The most 'Enterprise'ish.

Then you have VM-as-a-application stuff. Like Qemu, or Vmware's Server/Workstation/player stuff. These things are more convient, but are subject heavily to the performance limitations of the host OS combined with the performance limiations of the hosted OS along with some emulation overhead. Not the fastest, but not the slowest either. Offer good-enough performance for the majority of tasks you'd want to do in a VM.

Something like KVM actually turns the Linux kernel into a hypervisor, but in practice it seems more like the second catagory.

Then you have container-type things. Like Vserver, or OpenVZ. Think about those things as 'chroot on steroids'. They share the same kernel and work about the same as a chroot, but are designed for much higher security and isolation then what is normally possible. Since there is no software vm or emulation of anything going on these provide the highest performance and most efficient use of resources. Much better then anything else. The downside is that a kernel exploit can open up all your VM's.. and you can't run anything that is not the same as the host OS.

Then you have something like Usermode Linux. It's sort of like Vserver, but is more like a real VM. You can't run anything like Windows in it, but it does provide higher level of isolation and such.

Then you have emulation. Bochs is the best example I can think of. It's a full-on computer-expressed-in-software. Everything is in software. Software sound card, software video card, software cpu, etc etc. The whole ten yards. With this you can run x86 code on Sparc or PowerPC or anything else, it's full software. It's also very slow.

For a more expert overview check out:
http://virt.kernelnewbies.org/TechComparison

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 8, 2007 20:16 UTC (Thu) by nix (subscriber, #2304) [Link]

The same techniques are used elsewhere, of course: for instance, valgrind
is a single-process caching JITting dynamic recompiler for x86-and-PPC
assembler with instrumentation support; as such, it's fairly close to
QEMU, but with all the complexity hidden from the user.

KQEMU 1.3.0pre10 released - under the GPL

Posted Feb 9, 2007 11:46 UTC (Fri) by irios (guest, #19838) [Link]

Thanks, I've got a much clearer picture now.