Virtualization and InfiniBand

Posted Aug 29, 2009 17:13 UTC (Sat) by giraffedata (subscriber, #1954) [Link]

You've given an example I actually know something about, so I can comment further. You're talking about the mechanism used on IBM's System P processors (which come standard with virtual machines) to allow a server in virtual machine S to present a SCSI disk device to client virtual machine C.

The server in S bases its disk devices on real SCSI disk devices (e.g. it splits a 10G real SCSI disk into 5 2G disks, one for each of 5 client virtual machines), and the actual data transfer is conventional DMA done by the real SCSI HBA to the memory of C, using hypervisor facilities specifically designed for this I/O server VM application.

AFAICS the only infiniband-related part of this is SRP (SCSI RDMA (Remote DMA) Protocol). SRP is how the program running in S initiates (by communicating with C) that DMA into memory C owns, much as a server at the end of an IB cable might set up to transmit data down the IB wire into the client's memory.

And as I recall, SRP is simple and not especially fast or low-latency -- just what anybody would design if he needed to communicate DMA parameters. A person could be forgiven for just reinventing SRP for a particular application instead of learning SRP and reusing SRP code.