| Weekly edition | Kernel | Security | Distributions | Search |
| Archives | Calendar | Subscribe | Write for LWN | LWN.net FAQ |
The point of virtualization
The point of virtualization
Posted Oct 10, 2007 16:13 UTC (Wed) by giraffedata (subscriber, #1954)In reply to: The point of virtualization by roelofs
Parent article: Matt Domsch on Linux support at Dell
The virtualization layer doesn't slow the machine down, anyway. On a system built for virtualization, with only one virtual machine, you shouldn't notice a difference vs running directly on the real machine....until you get a TLB miss, anyway; then you pay double. See Ulrich's currently running series for details. ;-)
I believe on a system built for virtualization, with only one virtual machine, a TLB miss costs the same, and is equally likely, as on an equivalent system without a virtualization layer.
Of course, if you have a counterargument more specific than a 100 page tome, I might admit I'm wrong.
(Log in to post comments)
The point of virtualization
Posted Oct 11, 2007 5:09 UTC (Thu) by roelofs (subscriber, #2599) [Link]
Of course, if you have a counterargument more specific than a 100 page tome, I might admit I'm wrong.Let the healing begin! :-)
Memory part 3: Virtual Memory
Specifically, see the section discussing shadow page tables: without some of the upcoming hardware assists, you need to do two "physical to virtual" translations (within the context of the given kernel--i.e., a "physical" address for a guest kernel isn't truly physical), and that involves (on x86-64) four separate, sequential, non-hardware-prefetchable memory accesses each--for a total of eight--plus the memory access for the actual data. IOW, 5 accesses become 9, and each one is extremely costly in terms of clock cycles due to the no-prefetch issue.
ObCaveat: this is all from Ulrich Drepper's paper and talk (recently given at a local LUG). While I'm pretty sure I've summarized it accurately, he's the expert, not me; I might be misrepresenting some of it. Also keep in mind that the really bad TLB penalties kick in for certain (large) data-member and working-set sizes, and neither of us has made any statement (that I'm aware of, anyway) as to how typical such things are in real applications. (For the kind I mostly work on, I'm pretty sure they have the potential to be quite common. Actually quantifying it, however, is...not so easy.)
Greg
The point of virtualization
Posted Oct 12, 2007 17:38 UTC (Fri) by giraffedata (subscriber, #1954) [Link]
you need to do two "physical to virtual" translations (within the context of the given kernel--i.e., a "physical" address for a guest kernel isn't truly physical), and that involves (on x86-64) four separate, sequential, non-hardware-prefetchable memory accesses each--for a total of eight--plus the memory access for the actual data. IOW, 5 accesses become 9, and each one is extremely costly in terms of clock cycles due to the no-prefetch issue.
Ulrich doesn't mention physical to virtual translations. I presume you meant virtual to physical, since that's what would involve a TLB miss. Ulrich points out only one of those: the one through the shadow page table. The virtual page table is not involved in page translation. So it's the same speed with virtualization as without.
Leaving the topic of TLB misses, Ulrich does mention in this section that page table updates are more expensive, but I don't think that makes a noticeable dent in virtual machine speed. And he doesn't mention that some systems built for virtualization reduce that by doing away with the duplicate page tables.
without some of the upcoming hardware assists,
Those are probably included in my phrasing, "a system built for virtualization." And they're upcoming only for the Intel stuff. IBM's POWER (architecture of its Unix machines) has had it since about 2003; IBM's Z (360/370/390) probably has had some of these since the early 90s, when it started integrating virtualization with its processor products.
The point of virtualization
Posted Oct 12, 2007 22:49 UTC (Fri) by roelofs (subscriber, #2599) [Link]
I presume you meant virtual to physical...Yes, of course. Sorry for the typo!
The virtual page table is not involved in page translation. So it's the same speed with virtualization as without.
I don't understand why you think that. The guest app triggers a TLB (cache) miss, which means the guest kernel has to do a virtual-to-"physical" lookup in the page table, which involves one memory access for each level of the table. But its "physical" addresses are still virtual in the hypervisor kernel's space, which means you have to do a second virtual-to-physical lookup in the shadow page table, at a cost of three or four more memory accesses (for x86 and x86-64, respectively).
Those are probably included in my phrasing, "a system built for virtualization." And they're upcoming only for the Intel stuff.
Right--Ulrich explicitly stated that he was principally concerned with commodity x86-family hardware (and, to a lesser extent, Linux), and that certainly goes for me as well. I wouldn't be terribly surprised if IBM had some of this stuff back in the 1970s...
Greg
The point of virtualization
Posted Oct 12, 2007 23:02 UTC (Fri) by roelofs (subscriber, #2599) [Link]
Those are probably included in my phrasing, "a system built for virtualization." And they're upcoming only for the Intel stuff.OK, maybe I didn't go back far enough for context. I think I now see what you meant by "a system built for virtualization"--you were implicitly excluding the extra set of memory accesses due to shadow page tables in software-based virtualization. But I misread that since you were replying to the author's comment about existing Dell hardware--which explicitly does not include IBM mainframe technologies. And your original statement was a flat denial of any performance penalty due to virtualization ("The virtualization layer doesn't slow the machine down, anyway."). It was more that part to which I was replying.
Greg
The point of virtualization
Posted Oct 13, 2007 2:31 UTC (Sat) by giraffedata (subscriber, #1954) [Link]
I agree the answer to the question, "does virtualization slow systems down" is different if one means virtualization per se than if one means virtualization in practice.In the original comment, I thought I saw questioning of the very concept of virtualization -- i.e. why is everyone rushing toward this thing if it costs so much speed? So in addition to pointing out what I think the benefits are supposed to be, I also opined that it doesn't cost speed. I think the virtualization that people are presenting as the next big thing is virtualization done right, not necessarily what is practical today. E.g. companies are introducing new products specifically to enable virtualization.
The point of virtualization
Posted Oct 13, 2007 2:16 UTC (Sat) by giraffedata (subscriber, #1954) [Link]
The guest app triggers a TLB (cache) miss, which means the guest kernel has to do a virtual-to-"physical" lookup in the page table,
How would the guest kernel get involved? Page translation is done in the CPU. The guest app does a load instruction to an address he hasn't accessed in a while, so its translation is not in the TLB, so the CPU walks the shadow page tables to find to which physical page it refers, then loads the bytes into the register from there, then proceeds to the next instruction. The CPU doesn't even know the guest page table exists; the shadow page table maps the guest ap's virtual addresses to physical memory.
You seem to be saying TLB miss, but thinking page fault. When there's a page fault (the shadow page table does not contain a translation for the address in question), there is that multi-level lookup by the guest kernel and hypervisor, to ultimately add a translation to the shadow page tables. On systems that have guest page tables, that is.
The point of virtualization
Posted Oct 14, 2007 16:58 UTC (Sun) by roelofs (subscriber, #2599) [Link]
You seem to be saying TLB miss, but thinking page fault. When there's a page fault (the shadow page table does not contain a translation for the address in question), there is that multi-level lookup by the guest kernel and hypervisor, to ultimately add a translation to the shadow page tables.Yes, I think you've nailed it. I seem to have conflated the two issues.
Greg