User: Password:
Subscribe / Log in / New account

Virtualization and InfiniBand

Virtualization and InfiniBand

Posted Aug 8, 2009 7:42 UTC (Sat) by abacus (guest, #49001)
In reply to: Virtualization and InfiniBand by giraffedata
Parent article: AlacrityVM

Yes, 10 GbE has been designed for low latency. But look at the numbers: the best performing 10 GbE interface today (Chelsio) has a latency of 3.8 us [1] while recent IB interfaces have a latency of 1.8 us [2]. The difference is small but it matters when communicating messages that are less than 64 KB in size. And IB interfaces do not cost more than 10 GbE interfaces that support iWARP.
IB interfaces have a lower latency than 10 GbE interfaces because the whole IB stack has been designed for low latency while 10 GbE had to remain compatible with Ethernet.


  1. Chelsio about The Cybermedia Center at Osaka University.
  2. Performance Analysis and Evaluation of Mellanox ConnectX InfiniBand Architecture with Multi-Core Platforms.

(Log in to post comments)

Virtualization and InfiniBand

Posted Aug 8, 2009 9:24 UTC (Sat) by dlang (subscriber, #313) [Link]

yes, IB has lower latencies than 10GB ethernet

10G ethernet isn't low latencies at all costs. it benifits/suffers from backwards compatibility issues.

it also allows for longer cable runs than IB

it's not that one is alwaysbetter than the other, it's that each has it's use

if you are wiring a cluster of computers and price is not an issue, then IB clearly wins

if you are wiring a building, than IB _can't_ do the job, but 10G Ethernet can, so it clearly wins

Virtualization and InfiniBand

Posted Aug 8, 2009 11:00 UTC (Sat) by abacus (guest, #49001) [Link]

As I wrote above, the physical limitations of IB are not relevant in the context of the AlacrityVM project. These limitations only apply to the physical layer of the IB protocol and not to the higher communication layers. By the way, the InfiniBand Architecture Specification is available online. And

Virtualization and InfiniBand

Posted Aug 29, 2009 12:18 UTC (Sat) by abacus (guest, #49001) [Link]

Note: there are already kernel drivers in the Linux kernel that use this concept for communication between a virtual machine and the hypervisor or another virtual machine. These drivers are ibmvscsi (initiator, runs in the virtual machine) and ibmvstgt (target, runs in the entity exporting the SCSI device). See also Virtual SCSI adapters for more information.

Virtualization and InfiniBand

Posted Aug 29, 2009 17:13 UTC (Sat) by giraffedata (subscriber, #1954) [Link]

You've given an example I actually know something about, so I can comment further. You're talking about the mechanism used on IBM's System P processors (which come standard with virtual machines) to allow a server in virtual machine S to present a SCSI disk device to client virtual machine C.

The server in S bases its disk devices on real SCSI disk devices (e.g. it splits a 10G real SCSI disk into 5 2G disks, one for each of 5 client virtual machines), and the actual data transfer is conventional DMA done by the real SCSI HBA to the memory of C, using hypervisor facilities specifically designed for this I/O server VM application.

AFAICS the only infiniband-related part of this is SRP (SCSI RDMA (Remote DMA) Protocol). SRP is how the program running in S initiates (by communicating with C) that DMA into memory C owns, much as a server at the end of an IB cable might set up to transmit data down the IB wire into the client's memory.

And as I recall, SRP is simple and not especially fast or low-latency -- just what anybody would design if he needed to communicate DMA parameters. A person could be forgiven for just reinventing SRP for a particular application instead of learning SRP and reusing SRP code.

Virtualization and InfiniBand

Posted Sep 2, 2009 8:02 UTC (Wed) by xoddam (subscriber, #2322) [Link]

Wish *I* got a System P processor standard with *my* virtual machine!

Copyright © 2017, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds