Identifier locator addressing

Companies that run huge data centers have an obvious incentive to get the most performance possible out of each of their vast number of machines. Virtualization and live migration help by allowing tasks to be moved between machines so that each can be run at full utilization, but there is a problem: how do cooperating jobs find each other as they are moved across the data center? Numerous solutions to this problem exist; the 4.3 kernel will have another one, in the form of a technology called identifier locator addressing, or ILA.

ILA, which will work only with IPv6, is built on a simple idea: each task in the data center is assigned a unique identifier that is not tied to any specific location in the net. That identifier is built into that task's IPv6 network address; the networking subsystem then does the necessary magic to route packets properly between them, changing the routing as needed as the task moves between machines.

The details of how ILA works can be found in this draft RFC, written by Tom Herbert, who also happens to be the author of the ILA patches merged into the mainline for 4.3. In short, ILA splits the 128-bit IPv6 network address space into two 64-bit fields; one contains the identifier, the other the locator. The identifier is, as described above, a unique number identifying the task in the center. With 64 bits to play with, ILA can identify enough tasks to work in even the biggest data center — for the foreseeable future, at least. The identifier is not tied in any way to any specific physical machine in the data center. The locator, instead (stored in the upper 64 bits of the IPv6 address), uniquely identifies a physical interface on the network; a packet with an ILA address can be routed across the network using just the locator field.

A task wishing to communicate with another does not know that locator, though; all it knows is the identifier of the task it needs to talk to. This task will put a special "standard identifier representation" (SIR) prefix into the locator field, while the destination task's identifier goes into the lower 64 bits. The resulting "SIR address," which does not correspond to any actual system on the net, indicates to the networking subsystem that the address uses ILA and that the true locator must be filled in at transmission time. In practice, this SIR address will likely be obtained via a DNS lookup and need not be constructed by the task at all, of course.

The task will then open a network connection to the SIR address for the service it needs to contact. The networking stack cannot route the SIR address as-is, though, since that address doesn't correspond to any specific target on the net. Instead, it must find the real machine hosting the task with the given identifier and replace the SIR prefix with a proper locator corresponding to that system. It is thus almost like performing an ARP lookup on the identifier portion of the address. Once the real destination has been placed into the locator field, the packet can be sent on its way. The receiving system will, prior to handing the packet to the application, convert the ILA address back to a SIR address by putting the SIR prefix back into the locator field.

The SIR address will be used for the duration of the connection; it will continue to work even if the addressed task is migrated in the middle. That naturally means that the identifier lookup and SIR-prefix replacement must be done on each outgoing packet. It's worth noting that SIR addresses can be used for both endpoints of a connection, but it's not mandatory. The end result of all this should be a low-overhead mechanism for virtualization of network addresses within a data center. There is no encapsulation or other trickery required; it essentially comes down to a single address-translation step.

There is one little catch, of course: the kernel must somehow keep up with the proper locator value for each identifier of interest. As documented in the networking merge commit, the table of translations can be maintained by way of some extensions to the ip command. In practice, of course, nobody who needs a technology like ILA is going to mess around with ip commands; there will, instead, be some sort of central job-management system that maintains that mapping. How mappings (and changes) will be propagated through a data center is not addressed by the code in the kernel; that's a task for higher-level software. The good news is that mappings are not expected to change all that often (task migration is expensive, so it shouldn't be done more often than is strictly necessary), so the identifier-to-locator mapping can be effectively cached most of the time.

The ILA implementation in 4.3 appears to be a bit of a work in progress. It works, but it suffers a 10% performance penalty with respect to routing without ILA. The source of the slowdown seems to be known, and Tom has promised that it will be dealt with in a forthcoming patch set. There are also difficulties in the interaction with IPvlan [PDF] that should also be fixed in the future. Meanwhile, the core of the new feature is in the mainline and available for those who would like to play with it.

Among other things, ILA is a sign that IPv6 is finally coming into its own. It was not that long ago that IPv6 would not have been considered for performance-sensitive settings like data centers; it is easy enough to use an isolated IPv4 network and avoid the performance issues and application compatibility issues that came with IPv6. But most of those issues have been resolved, and the pressure to move toward IPv6 continues to increase. As technologies like ILA come along and make use of the greatly expanded IPv6 address space, IPv6 may increasingly come to look like the more fully featured alternative.

Index entries for this article
Kernel	Networking/IPv6

---

to post comments