LWN: Comments on "AlacrityVM"

Virtualization and InfiniBand

xoddam — 2009-09-02T08:02:56+00:00

Wish *I* got a System P processor standard with *my* virtual machine!

Virtualization and InfiniBand

giraffedata — 2009-08-29T17:13:25+00:00

You've given an example I actually know something about, so I can comment further. You're talking about the mechanism used on IBM's System P processors (which come standard with virtual machines) to allow a server in virtual machine S to present a SCSI disk device to client virtual machine C.

The server in S bases its disk devices on real SCSI disk devices (e.g. it splits a 10G real SCSI disk into 5 2G disks, one for each of 5 client virtual machines), and the actual data transfer is conventional DMA done by the real SCSI HBA to the memory of C, using hypervisor facilities specifically designed for this I/O server VM application.

AFAICS the only infiniband-related part of this is SRP (SCSI RDMA (Remote DMA) Protocol). SRP is how the program running in S initiates (by communicating with C) that DMA into memory C owns, much as a server at the end of an IB cable might set up to transmit data down the IB wire into the client's memory.

And as I recall, SRP is simple and not especially fast or low-latency -- just what anybody would design if he needed to communicate DMA parameters. A person could be forgiven for just reinventing SRP for a particular application instead of learning SRP and reusing SRP code.

Virtualization and InfiniBand

abacus — 2009-08-29T12:18:21+00:00

Note: there are already kernel drivers in the Linux kernel that use this concept for communication between a virtual machine and the hypervisor or another virtual machine. These drivers are ibmvscsi (initiator, runs in the virtual machine) and ibmvstgt (target, runs in the entity exporting the SCSI device). See also Virtual SCSI adapters for more information.

High-performance I/O in Virtual Machines

mcmanus — 2009-08-13T13:38:08+00:00

"the snazzy PCI express virtualization stuff let the guest safely talk directly to the hardware"

SR-IOV et al is really cool, but it still leaves open the problems of hairpin routing, firewall enforcement, etc where the alacrity approach really helps. There is talk of hardware answers to that too, but its further down the pipe. This kind of hardware is going to be substantially more expensive in the short term as well, so having more efficient answers in software is a plus for the ecosystem overall.

Also interesting, not long after this article was published there was a patch made available to virtio (for pure kvm) that reduces the trips through userspace for that code too: https://lists.linux-foundation.org/pipermail/virtualizati...

Virtualization and InfiniBand

jgg — 2009-08-13T04:54:14+00:00

10GBASE-T is not compatible with Cat5e, it needs Cat6 cabling. It is also still a pipe dream, who knows what process node will be necessary to get acceptable cost+power. All 10GIGE stuff currently is deployed with CX-4 (Identical to SDR IB) or XFP/SFP+ (still surprisingly expensive).

The big 10GIGE vendors are desperately pushing the insane FCoE stuff to try and get everyone to re-buy all their FC switches and HBAs since 10GIGe has otherwise been a flop. IB is 4x as fast, and 1/4th the cost of 10GIGE stuff from CISCO :|

High-performance I/O in Virtual Machines

jgg — 2009-08-13T04:43:42+00:00

The new IB HCAs when combined with the snazzy PCI express virtualization stuff let the guest safely talk directly to the hardware and this whole issue becomes fairly moot. I've heard some of the FCoE chips will be able to do the same thing too. Any serious deployment with shared storage will want to go that way.

High-performance I/O in Virtual Machines

abacus — 2009-08-08T11:31:14+00:00

In the past research has been carried out about high-performance I/O in virtual machines running on the Xen hypervisor and in a system equipped with an InfiniBand HCA. See also Jiuxing Liu e.a., High Performance VMM-Bypass I/O in Virtual Machines, USENIX 2006, Boston.

Virtualization and InfiniBand

abacus — 2009-08-08T11:00:11+00:00

As I wrote above, the physical limitations of IB are not relevant in the context of the AlacrityVM project. These limitations only apply to the physical layer of the IB protocol and not to the higher communication layers. By the way, the InfiniBand Architecture Specification is available online. And

Virtualization and InfiniBand

abacus — 2009-08-08T10:52:28+00:00

As I wrote above, implementing an IB driver would allow to reuse a whole software stack (called OFED) and the implementation of several communication protocols. Yes it is possible to develop all this from scratch, but that is more or less like reinventing the wheel.

Virtualization and InfiniBand

dlang — 2009-08-08T09:27:16+00:00

if you are talking a virtual interface, why would you use either?

define a driver that does page allocation tricks to move data between the client and the host for zero-copy communication. at that point you beat anything that's designed for a real network.

then you can pick what driver to run on top of this interface, SCSI, IP, custom depending on what you are trying to talk to on the other side.

Virtualization and InfiniBand

dlang — 2009-08-08T09:24:25+00:00

yes, IB has lower latencies than 10GB ethernet

10G ethernet isn't low latencies at all costs. it benifits/suffers from backwards compatibility issues.

it also allows for longer cable runs than IB

it's not that one is alwaysbetter than the other, it's that each has it's use

if you are wiring a cluster of computers and price is not an issue, then IB clearly wins

if you are wiring a building, than IB _can't_ do the job, but 10G Ethernet can, so it clearly wins

Virtualization and InfiniBand

abacus — 2009-08-08T07:42:02+00:00

Yes, 10 GbE has been designed for low latency. But look at the numbers: the best performing 10 GbE interface today (Chelsio) has a latency of 3.8 us [1] while recent IB interfaces have a latency of 1.8 us [2]. The difference is small but it matters when communicating messages that are less than 64 KB in size. And IB interfaces do not cost more than 10 GbE interfaces that support iWARP.
IB interfaces have a lower latency than 10 GbE interfaces because the whole IB stack has been designed for low latency while 10 GbE had to remain compatible with Ethernet.

References:

Virtualization and InfiniBand

abacus — 2009-08-08T07:24:59+00:00

What you wrote above about cabling is correct but completely irrelevant in this discussion. What I proposed is to use the IB API's (RDMA) and software stack (IPoIB, SDP, iSER, SRP, ...) for communication between a virtual machine and the host system. In such a setup no physical cables are necessary. An additional kernel driver will be necessary in the virtual machine however that implements the RDMA API and allows communication between guest and host.

Virtualization and InfiniBand

dlang — 2009-08-07T21:05:17+00:00

IB is faster and lower latency, but is significantly more expensive, has shorter cable length restrictions, (IIRC) many more wires in the cables (which make them more expensive and more fragile)

IB was designed for a system interconnect within a rack (or a couple of nearby racks)

ATA and SATA aren't general interconnects, they are drive interfaces

10 GbE is a fair comparison for IB, but it was designed to allow longer cable runs, with fewer wires in the cable (being fairly compatible with existing cat5 type cabling)

Virtualization and InfiniBand

giraffedata — 2009-08-07T20:31:46+00:00

has a higher throughput and a lower latency than any other popular storage or networking technology (IDE [ATA], SATA, 10 GbE, ...)

Nonetheless, ATA, SATA, and 10 GbE were all designed to have high throughput and low latency. So one can't say that being designed for high throughput and low latency sets IB apart from them.

So what does? Were the IB engineers just smarter? Did they design for higher cost of implementation? Did they design for implementation technology that wasn't available when the alternatives were designed?

Virtualization and InfiniBand

abacus — 2009-08-07T18:37:28+00:00

> > This is because the IB stack has been designed for high throughput and low latency.

> As opposed to what? Doesn't every protocol seek to be fast? Does IB make different tradeoffs than the alternatives?

IB is a technology that comes from the supercomputing world and that has a higher throughput and a lower latency than any other popular storage or networking technology (IDE, SATA, 10 GbE, ...). Key features of IB are support for zero-copy I/O (RDMA) and the possibility of performing I/O without having to invoke any system call, even for user-space processes.

Some impressive graphs can be found in this paper: Performance Analysis and Evaluation of Mellanox ConnectX InfiniBand Architecture with Multi-Core Platforms.

Note: I'm not affiliated with any vendor of IB equipment.

Virtualization and InfiniBand

giraffedata — 2009-08-07T18:04:15+00:00

This is because the IB stack has been designed for high throughput and low latency.

As opposed to what? Doesn't every protocol seek to be fast? Does IB make different tradeoffs than the alternatives?

Virtualization and InfiniBand

abacus — 2009-08-07T17:46:55+00:00

I'm surprised that although several virtualization implementations are looking at high-speed I/O that none of these is using the InfiniBand (IB) stack. With current IB hardware data rates of up to 2.4 GB/s are possible -- between different systems. This is because the IB stack has been designed for high throughput and low latency. Linux' IB stack already has implementations of networking and storage drivers. So implementing a single IB driver would allow virtualization software to reuse the drivers in the IB stack.

paravirtulised drivers?

ghaskins — 2009-08-07T13:09:24+00:00

> I seem to remember a virtulisation session at OLS 2008 where the discussion was
> both Xen and KVM moving to common paravirtualised drivers for guests to speed
> up I/O. Isn't this basically the same thing?

Its the same in terms of these are also PV drivers. Its different in that this is much faster PV
infrastructure than the currently deployed versions. Ideally we will be able to use the same
drivers and just swap out the inefficient part in the hypervisor side. As of right now, the drivers
are also different (venet vs virtio-net), but this may change in the future.

In addition, it is also infrastructure that allows us to do new kinds of PV operations, such as
supporting real-time guests.

(Note: the graphs posted are against the virtio-net based PV drivers that were probably the
result of that presentation you saw at OLS)

-Greg

AlacrityVM

ghaskins — 2009-08-06T13:18:32+00:00

> what kind of support does this require in the guest?

You load drivers in the guest for various IO subsystems (network, disk, etc)

> Is this another paravirtualized interface

Yes, though it is not entirely orthogonal. For instance, it is possible to tunnel existing PV protocols
over it (e.g. virtio-net). This means you swap out the low-level protocol (virtio-pci is exchanged
for virtio-vbus) but the higher layer PV drivers (virtio-net, virtio-blk) remain unchanged.

-Greg

AlacrityVM

ghaskins — 2009-08-06T13:12:37+00:00

The patches apply on top of KVM

paravirtulised drivers?

alex — 2009-08-06T10:21:58+00:00

I seem to remember a virtulisation session at OLS 2008 where the discussion was both Xen and KVM moving to common paravirtualised drivers for guests to speed up I/O. Isn't this basically the same thing?

AlacrityVM

dunlapg — 2009-08-06T10:04:31+00:00

And what kind of support does this require in the guest? Is this another paravirtualized interface (along the lines of Xen's frontend/backend interface)?

AlacrityVM

rahulsundaram — 2009-08-06T07:56:36+00:00

Is this a series of patches on top of KVM or a fork of some sort? Seems to missing some more history or context.