Who Is Developing KVM Linux Virtualization? (ServerWatch) [LWN.net]

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 12, 2010 23:29 UTC (Fri) by rahvin (subscriber, #16953) [Link]

Xen is an interesting example of creating a product to compete on the terms of the company you are competing against. Xen was a created as a competitor to VMWare, it's design and operation is almost identical to the VMWare method of doing things.

Well along come Qumarent, who realizes that the real answer to visualization is to build it into the OS (the strategy MS took to compete against VMWare BTW). The philosophy was why build a full hyper-visor to emulate a machine when you already have a kernel loaded with thousands of hardware drivers (many of which that can already be loaded without the hardware present). Because everyone had been trying to compete with VMWare they hadn't stepped back and asked themselves if the VMWare way was the right way. So Qumarent comes in and proposes and submits a method to do visualization using all the Kernel infrastructure that's already there. Qumarent's method differs greatly from the VMWare way of doing things, but IMO is a better way to do it (at least on Linux). Because their method doesn't require massive kernel changes and uses infrastructure in the Kernel that already exists Linus prefers it. Xen could never get upstream because the changes they needed were massive and I doubt they could have scaled it back enough to get fully integrated.

As Qumarent proved the solution wasn't to try to duplicate the VMWare model but to find the model that works the best for Linux. In the long run I expect that KVM will outpace and surpass Xen in every way. Xen was simply the wrong model for Linux. It works for VMWare because the are producing what is essentially their own closed source operating system and only using Linux as a boot loader. There is absolutely no reason to follow the same path with a FOSS operating system. Microsoft validated the KVM model by integrating virtualization into Windows Server showing that virtualization on top of a running as an integrated service is the next step in the evolution of the PC. I'm sure the VMWare and Xen way of doing things is the right way to do virtualization in certain high end configurations or environments, but the vast majority of virtualization is going to be as an addon to the OS to run rarely used services or legacy applications. In that area the KVM and Microsoft method are the right way to do it.

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 13, 2010 0:44 UTC (Sat) by gdt (subscriber, #6284) [Link]

Xen does use the operating system's drivers. But very little else.

It wasn't at all clear prior to KVM that using the operating system as the hosting platform was wise. The original VM -- on IBM mainframes -- couldn't do that as using MVS as the hosting platform would have been counter-productive. Sure, using an operating system had architectural appeal, but so do many notions that implement poorly. I'd say that there's a fair use of the retrospectoscope in this discussion.

As it turns out, using a good quality operating system as the hosting platform works well. Just as importantly, it is much simpler to administer. Administration costs are turning into a major lesson: virtualising a hundred machines is easy, managing those hundred instances is not.

One of the nice things about Linux is that it brings "enterprise" features into the realm of the desktop user. KVM makes is easy for a desktop user to use virtualisation for training and testing and has a fair user base for these purposes. This gives budding system administrators a depth of grounding in KVM that they do not get in Xen.

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 13, 2010 1:09 UTC (Sat) by nix (subscriber, #2304) [Link]

User-mode Linux was surely a proof of concept of sorts, as well.

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 14, 2010 2:14 UTC (Sun) by clump (subscriber, #27801) [Link]

Agreed, though UML is an interesting concept. It's a Linux kernel ported to userspace. It just so happens that you can start a userspace kernel to 'virtualize' in a sense. That said, it's not like the other virtualization technologies. It doesn't emulate, it's not paravirtualization, and it's not virtualization in the VMWare sense. Heck, it's not even a container or chroot.

As far as I know, nothing is like UML in terms of how it does what it does. Great little technology that I've used for years, but a major drawback is that you can't simply take a normal distro and turn it into a UML filesystem.

Many forget it was upstream in the Linux kernel years ago.

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 14, 2010 9:35 UTC (Sun) by nix (subscriber, #2304) [Link]

you can't simply take a normal distro and turn it into a UML filesystem.

Well, all you need to replace is the kernel. Not hard, given the existence of loopback mounts in the first place -- and given that UML stores its kernel outside the filesystem image in the first place, you don't need to touch the filesystem image as such at all.

UML and distribution tools

Posted Feb 15, 2010 12:40 UTC (Mon) by pboddie (subscriber, #50784) [Link]

Indeed. I have a set of scripts for doing this kind of thing with the Debian toolset:

http://www.boddie.org.uk/paul/UMLconfig.html

It's not very nicely done and could surely be better integrated with the toolset - like what pbuilder does in a chroot but using UML (and not being specifically about package building, obviously) - and I'm inclined to merge it with some other work that would avoid the need even to be root, although I think setting up the network bridges/tunnels requires root access, unfortunately.

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 19, 2010 17:00 UTC (Fri) by jengelh (subscriber, #33263) [Link]

>It doesn't emulate, it's not paravirtualization, and it's not virtualization in the VMWare sense. Heck, it's not even a container or chroot.

It is mostly a matter of how you see it. Container seems to be the most fitting. It shares resources with the host system which may be running other UML instances all parallel to each other.
But in order to distinguish it from things like OpenVZ containers, one would have to call OpenVZ a kernel-assisted container solution, while UML is an kernel-independent one (though it's probably not running on, say, Solaris, out of the box). Opinions?

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 19, 2010 19:19 UTC (Fri) by viro (subscriber, #7872) [Link]

Essentially it's a debugger with syscall emulation. And no, it won't run on Solaris as-is; at the very least you'll need to deal with signal handler stack frame layout differences and there's a bunch of assumptions along the lines of "layout expected by guest processes is identical to one we get from host when we deal with the signals". The same structure is used for both and I've no idea whether it would be easy to separate these two or not.

In principle, host OS dependent code is separated (arch/um/os-Linux), but separation is not particulary clean, to put it mildly.

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 13, 2010 1:43 UTC (Sat) by drag (subscriber, #31333) [Link]

Don't also forget that this virtualization stuff has been around for years.

IBM started doing it back in 1967 and it was a standard part of their enterprise hardware and software in the 1970's.

Vmware's stuff started out as academic research project, I believe.

Xen was just following the accepted route for hypervisors. It's a lovely idea. Similar to the Microkernel. You have this very simple software solution that sits between the hardware and your kernel and it allows you to run many multiple systems in parallel.

That was simply just how you were suppose to do it.

-----------------------

And Xen really did not copy Vmware. It follows the same 'Type 1 bare metal hypervisor' that is used for Vmware's ESX solution. But so does KVM. KVM turns the Linux kernel into a Type 1 hypervisor. Vmware certainly did not invent the concept and they were not the first ones to take advantage of it (IBM was). They just happen to be the first successful company to use it on x86.

The biggest difference is how they actually operate. X86 has a annoying feature were some small amount of instructions run differently if your running your code in Ring0 versus Ring3. These rings make up one of the basic security features of modern computers and all decent architecture uses them. But x86 has that different rings act slightly different problem. The traditional goal of virtualization on x86 with a "Type 1 Hypervisor" is to lift the kernel code running in Ring0 out and run it in higher rings and replace it with your hypervisor.

Vmware uses software dynamic recompiling solution to virtualizing x86. This approach means that they are using software to dynamically catch and rewrite the Ring0 instructions that your kernel uses and translates them to run in Ring3. It's fast because there really is not much difference between Ring0 and Ring3 in these regards, but it ends up being complicated and there are a lot of other issues involving memory management and I/O redirection and all that happy stuff.

Xen's approach is completely different. Instead of modifying the code dynamically they simply have you change how the kernel operates. It's a approach that works very well with Linux and would be impossible to do with Windows (due to Microsoft's disposition). It's very fast and seemingly elegant solution.

But they certainly did not copy Vmware to do that.

What does KVM do?
_nothing_. KVM depends on hardware virtualization to solve the Ring0 vs Ring3. This is why KVM cannot run on a lot of machines while Xen can... Of course this makes KVM simpler and using existing Linux facilities makes it even more simpler.

----------------------

Not a big deal, but I don't think at all that Xen is losing ground to KVM because Xen copied Vmware and did not innovate. That is the wrong lesson to learn from this. The modify-Linux-to-make- it-very-fast was a good approach and held a lot of promise. it fundamentally is a huge optimization that is not possible without open source operating systems. Without Intel and AMD's modification to the x86 architecture KVM would be worthless.

But now since we do have VT, VT-D, Pacifica, IOMMU, and other related items it turns out KVM is just a better design. Much simpler and uses already proven Linux infrastructure and a very mature Qemu code base to get what is needs done.

If you want to a lesson here the lesson is:
DO NOT REINVENT THE WHEEL. Reuse your code. Reuse everybody else's. Depend on proven code bases and successful projects. Do not write from scratch. Do not have NIH. Unabashedly copy and take advantage of everything available to you that was made by other people, is mature, and is proven in the real world. And, on top of that, work with those other people to make sure that your solution stays maintainable and they benefit from your copying also as much as possible. Without reusing Linux and Qemu AND working closely with Linux kernels and Qemu developers then KVM would not be were it is today.

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 13, 2010 2:00 UTC (Sat) by rahvin (subscriber, #16953) [Link]

My copying VMWare comment was meant in the respect of following the traditional microkernel methods that Vmware successfully implemented (I didn't say they originated them). IMO there were several big mistakes with Xen, the first and biggest was starting essentially from scratch. The Second was not taking that big step back and asking if their design was the right way to do it. And the third biggest mistake was not going back to mistake 2 when Linus told them they needed big changes if they ever wanted into the Kernel.

As I said, I'm sure Xen and the Vmware microkernel method make really good sense in part of the marketplace. Not being a VM expert I don't know what that market is but I would guess it's Data centers. The problem is that the vast majority of VM is going to be at the small level where VM's are legacy apps (such as XP mode on Windows7) or seldom used servers. The marketplace that's left isn't very big and I question whether Xen can survive against Vmware in that much smaller marketplace.

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 13, 2010 7:01 UTC (Sat) by drag (subscriber, #31333) [Link]

My copying VMWare comment was meant in the respect of following the traditional microkernel methods that Vmware successfully implemented (I didn't say they originated them).

If your simply talking about the archictectual differences, then I agree with you. But the goals basic concepts behind the designs for Xen/KVM/ESX are all pretty similar.

As I said, I'm sure Xen and the Vmware microkernel method make really good sense in part of the marketplace. Not being a VM expert I don't know what that market is but I would guess it's Data centers. The problem is that the vast majority of VM is going to be at the small level where VM's are legacy apps (such as XP mode on Windows7) or seldom used servers. The marketplace that's left isn't very big and I question whether Xen can survive against Vmware in that much smaller marketplace.

Oh the marketplace is tiny now, but it's going to be HUGE HUGE HUGE in the future. The near future. Virtualization is a killer feature. It's going to be so commonplace that people will think that it is just natural and couldn't know how to manage their systems in any other way.

The goal of Type 1 hypervisors is to totally separate your OS from your hardware. The OS becomes then a freeform thing.. nothing more then a container that houses your applications.

The capabilities that you gain when your OS is completely divorced from your hardware is _massive_ and will benefit pretty much every aspect of computing. Web servers, mobile devices, terminals, desktop, storage servers... You name it you'll see virtualization change how it's managed and developed.

One of the classic things you had to give up when switching from high cost big Unix systems to commodity hardware is management. Those big machines had all sorts of extra management and monitoring features built into it that increases the utility and reliability of them massively.

However a hypervisor can blow those things out of the water. Imagine installing SNort on your machine and actually have it useful. Being able to monitor, measure, and control all network I/O in and out of your machine. Also be able to the same thing for storage or any other I/O coming in and out of your machine.

Think about the usefulness of checking for malware on your Windows clients when you can intercept and control the I/O from their volumes. You gain the ability to pause and checkpoint your OS. To redirect output to a temporary file and take advantage of volume snapshots to create multiple live forks of your operating systems. Be able dump changes to the file system and go back to earlier changes. Be able to live migrate your OS from machine to machine without losing user sessions. Be able to migrate your system from data center to data center also.

Got troublesome developers that insist on root access or other privilaged access to the OS? Screw them. Give them root. Give them their own OS. Give them 40 operating systems to screw around with. So what? Let them all be root. They won't be able to do anything that would hurt you. They can access the machines from the network and you don't have to let the machines have access to anything outside their virtual network. Just sandbox them in were they can't do any damage. It's very simple. You don't need a physical network to have a network anymore. It can be all done virtually. Switches without switches, bridges without bridges, routers without routers. You can tunnel virtual ethernet networks over real networks or even non-ethernet networks.

Do hardware upgrades and repairs without rebooting your services. Add new machines and run operating systems on them without installing anything or even booting the OS up. Make your desktops and servers plug-n-play. Roaming profiles? What's what? USELESS. That is what that is. How about roaming operating systems? Have the OS be on whatever computer the user is using regardless of their physical location.

------------------

Ever heard of Coreboot? The project that replaces motherboard firmwares with Linux? Lets think about that for a whole too seconds... Lets thing.. Linux can boot the system quicker, with more reliability, and better configurations then a typical BIOS can do. ........ Well what else can Linux do? .....

Be a hypervisor, that's what.

You buy your machine. It's a PC or a Server. Whatever. You take the machine into were you work. You plug it into the network, plugin the power. Flip the on switch. You walk over to your workstation in the next room. Fire up the management console, update the firmware, add in the TLS packets or authenticate it against your kerberos or whatever you use for management.

Fire up your management console and fire down a Windows install or two. Throw a Linux server on it. Hell you don't even have to install anything. Just migrate a few systems over or let them 'load balance walk' their way over 'naturally'.

No sweat. No installation. Pretty close to being completely stateless. Your hardware becomes a abstract thing. A pool of hardware. A cluster of collections of CPUs and memory and storage media that your OS containers float around it. Hardware nodes can come and go... always flashing in and out of existence.. but your Debian System can live on forever!

:)

-------------------------

Xen I actually don't worry much about. People that are running OpenSolaris and FreeBSD need Xen as it is the only really effective full virtualizations available to them right now. And Xen does the paravirt thing, which is still special.

But Vmware is getting hit on both sides. Between Hyper-V and KVM they are going to have a very hard time.

The one thing I've learned about people that depend heavily on Microsoft (they use Office/exchange/aD/windows desktop/etc etc) in business is that Microsoft is godlike. Microsoft is not just Microsoft. It is THE MICROSOFT.

It does not matter that Hyper-V is slower, or that you can't do live migrations, or that the OS client support is not good because the virtualization is spotty. All that is immaterial. It is from Microsoft and thus is better. Vmware is now a plaything, a also ran; something only nerds spout on and on about. What do they know? Are they Microsoft, do they work for Microsoft, are they A+ certified? They know nothing. Microsoft is Microsoft and Microsoft is the real deal and makes the real tools. Microsoft knows what you want, Microsoft knows what you need. Just listen and obey and they _love_ big brother Microsoft for that.

In other words because Hyper-V is from Microsoft then people will switch over in mass. Hell, even with complete crap like Microsoft Virtual Server people still shoehorned it into their org and made it work for them despite all odds... and were happy about it. To hear some people talk about Virtual Server you'd swear that Microsoft invented the entire concept of virtualization.

So that is really going to hurt Vmware relatively quickly.

So that leaves Vmware with the more enterprisy group of folks coming from the Unix camp, right?

Well... not so much. Linux now can do 80-90% of everything Vmware can do out of the box. It can do it faster, be more flexible, and is massively cheaper. It's open source, friendly, and easy to use. It's also built into just about every single Linux system distributed in the last year or two.

Why buy Vmware ESX when Linux does it all by default?

Vmware is in a very very hard spot right now. It is not something that is really going to affect then immediately, but the pressure is just going to build and build and build. And I actually like Vmware. Vmware is now the #1 developer for Linux graphics drivers and infrastructure. They are now the biggest driving force behind Gallium and the modernization of the Linux graphics system. They have contributed to the Gnome project and improved all sorts of things. They are pretty decent folks given that they make a living selling proprietary software.

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 25, 2010 15:27 UTC (Thu) by appie (subscriber, #34002) [Link]

I think VMWare will do quite well what with the whole ecosystem of the supporting management software. Granted, the actual virtualization layer and hypervisors will become kind of commodity, but VMWare still has, and will probably continue to have for some time, the best tools to manage your virtualized environment.

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 14, 2010 22:18 UTC (Sun) by Cyberax (subscriber, #52523) [Link]

"Xen was just following the accepted route for hypervisors. It's a lovely idea. Similar to the Microkernel. You have this very simple software solution that sits between the hardware and your kernel and it allows you to run many multiple systems in parallel.
"

But now consider ACPI, on-the-fly system reconfiguration (like plugging in additional CPUs and RAM), guest system migration, monitoring, etc. And your hypervisor has to support all of it. So suddenly your bare-metal hypervisor becomes not such a beautiful idea at all.

Of course, your hypervisor might try to offload some tasks to a privileged partition (dom0) instead of trying to re-invent them, but then it raises a question - why would you need a hypervisor at all?

That's what happened in the XEN vs. KVM debate. KVM's design is just nicer because it doesn't need a specialized hypervisor.

Reliance of KVM on hardware virtualization is secondary. There are no fundamental reasons why KVM can't use dynamic recompilation.

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 14, 2010 22:55 UTC (Sun) by nix (subscriber, #2304) [Link]

Well, KVM per se is just a speedup hack for QEMU, which when KVM can't
kick in (no hardware support, source != target) already *does* do dynamic
recompilation. Obviously, even with that it's still a *lot* slower than
KVM (which most of the time doesn't need to do emulation at all).

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 15, 2010 23:57 UTC (Mon) by drag (subscriber, #31333) [Link]

KVM creates a new process model for Linux called 'guest'. It allows you to run Ring0 domain code
in Ring3 domain. That's what makes Linux now a bare-metal hypervisor.

It's not really anything much different then ESX or Xen does. It's just that Linux is much more
capable then those hypervisors. Xen still depends running a modified Qemu on Linux. ESX, also,
has it's own userspace portion that runs on Linux.

The major difference as far as Qemu and KVM is concerned is that the KVM folks have worked
closely with Qemu so they did not have to maintain their own fork for long.

Who Is Developing KVM Linux Virtualization? (ServerWatch)

Posted Feb 13, 2010 1:45 UTC (Sat) by rahvin (subscriber, #16953) [Link]

Hindsight is always 20/20 but the mistake I see with Xen was that everyone (RedHat, SUSE, IBM, etc) jumped in without anyone stepping back and asking if it was the right way to do it. It would be interesting if some of the original Qumanet employees came forward and let us know how they originated the idea of integrating the VM infrastructure into Linux. I'd be very curious to find out if it was their own brainstorming on a better solution or if the opposition Xen received from the Kernel developers was what triggered the KVM project.

IIRC the KVM patches were first proposed about a month after the Xen/Vmware kernel collaboration effort collapsed after a huge flame fest and Linus telling them they were both wrong and their patches required far to many changes. I've always wondered if that public flame out was what caused the "thinking outside the box" that Qumanet did to start coding KVM. Or if it was part of their development of SPICE and they already had all the KVM stuff worked out before they submitted the first patches essentially putting them months or even years ahead of Xen before their work went public.

Red Hat did delay

Posted Feb 15, 2010 20:50 UTC (Mon) by dmarti (subscriber, #11625) [Link]

Red Hat did delay offering Xen support: "Xen not ready for prime-time, says Red Hat" They came to Xen late, but they did make their move and buy Qumranet when they got the chance, picking up Andrea Arcangeli in the process AFAIK.