The switchdev driver model

Over the years, Linux has grown to a position of dominance in many branches of computing, but there are still niche markets—some of them surprising—where it has yet to make a serious impact. One of those markets is in high-end networking hardware: the managed switches and high-density routers that connect many large-scale networks or data centers. Such hardware is often marketed as "Linux capable" or "built with Linux" but, in reality, has its functionality implemented only in a proprietary blob. In 2015, that situation led the kernel networking developers to adopt a new driver model called switchdev that is aimed at replacing those proprietary blocks with standard kernel interfaces. Early on, it may have seemed like a rather speculative move, but at the Netdev 1.1 conference in Seville, Spain, switchdev was front and center.

The market for the switches in question is dominated by a few companies: Cisco is the largest vendor by a comfortable margin, followed by Arista, Juniper, and several other smaller companies. These appliance vendors often advertise Linux support for their products, but the model they employ is, at best, a small Linux system connected to a proprietary application-specific integrated circuit (ASIC) that actually handles all packet processing in hardware. The Linux system boots to a shell prompt, where proprietary command-line utilities are used to manipulate the switch chip. Furthermore, those utilities are generally built around a binary-blob "software development kit" (SDK) provided by the chip vendor. Thus, for Linux to break through the appliance vendor's lock-in, the chip vendor's lock-in must be broken as well.

The design of switchdev originates in Open vSwitch project, and was first described by Jiří Pírko in a September 2014 patch. At the Netdev 0.1 conference in February 2015, the networking developers decided to expand and adopt switchdev as a general solution for hardware switch chips and to make a concerted effort to break the binary-blob "SDK" stranglehold.

An overview of switchdev can be found at Documentation/networking/switchdev.txt. Each physical port on a device is registered with the kernel as a net_device, as is done for existing network interface cards (NICs). Ports can be bonded or bridged, tunneled or divided into virtual LANs (VLANs) using the existing tools (such as bridge, ip, and iproute2). The advantage of a switchdev driver is that such switching constructs can be offloaded to the switch hardware. As such, the driver mirrors each entry in the forwarding database (FDB) down to the hardware, and monitors for changes.

The kernel's netlink facility issues a NETDEV_CHANGEUPPER notification whenever there is a modification to the "upper" (that is, software) side of the mapping, so that the driver can make the corresponding adjustment to the "lower" (that is, hardware) side. That allows users to manage the switch hardware with the standard complement of tools. On the flip side, switch hardware is also often capable of learning about the network, such as which MAC addresses are found on which VLAN/port combinations. The driver sends a notification (for example, SWITCHDEV_FDB_ADD when it learns a new {port, MAC, VLAN} tuple), which is then used to update the FDB.

Up until now, much of the emphasis has been on modeling layer-2 (Ethernet) switching in switchdev, but layer-3 (IP) offload is supported as well for devices that perform IP routing. Currently, that support is limited to IPv4, but IPv6 support is in the works.

Initially, the only device supported by switchdev was the "rocker" software switch for QEMU. But, in July 2015, support was introduced for the Mellanox SwitchX-2 ASIC. Mellanox has continued to fund in-kernel driver development since. At the low end of the spectrum, Broadcom BCM53xx switches (typically found in home router hardware) gained switchdev support, too, although the emphasis remained on the large switch devices and the vendors of the chips within.

As our coverage of the 2016 Netconf meetings (part 1, part 2) notes, switchdev is increasingly the layer at which new kernel networking features are being planned—such as single-root I/O virtualization (SR-IOV) or Virtual Routing and Forwarding (VRF). But in-kernel usage of the API and a single chip vendor hardly equates with breaking the existing switch-vendors' logjam.

What is significant, however, is a major telecom carrier throwing its weight behind switchdev. Damascene Joachimpillai from Verizon did just that in his keynote talk at this year's Netdev. Joachimpillai (or "DJ" as he is know around the community) runs the Network Architecture team for the network-applications division inside Verizon—the group that develops and deploys the carrier's own mobile apps, for everything from contacts synchronization to video streaming. The division runs several data centers in the United States, he said, but it is the switches and other "middle boxes" that impose the most significant requirements, rather than the servers.

To that end, he has grown dissatisfied with the user-space network-configuration tools provided by most switch vendors, and is backing switchdev development. Several dozen switches that each require a separate CLI application for management becomes a support nightmare, he said. Primarily, that is because the proprietary CLI tools do not make it easy to automate operations; his division needs to adjust network policies on the fly to adapt to changing traffic. The switch vendors are typically reluctant to make programmatic interfaces to their tools, and even those that do, he said, seem to change those interfaces with each product release.

Joachimpillai took numerous questions from the audience, most of which dealt with what a large carrier wants to see from switchdev and from the kernel in general. In reply, he cited uniformity as a critical feature, in contrast to the ever-changing interfaces of proprietary switch vendors' tools. He also noted that hardware offload was vital, and that Verizon tends to purchase hardware that can be upgraded later in the field. He hopes to choose hardware today, he said, that has good software support for features now that will be hardware-accelerated in future kernels.

An audience member asked whether Joachimpillai's division was doing any kernel development internally, or if using existing knobs was sufficient. He replied that, so far, the team has just been working with outside developers who were already working on projects that it was interested in. The kernel community has been quite responsive to change requests so far, he added, in particular with its recent work on improving IPsec support.

Throughout the rest of Netdev, it was clear that Verizon's endorsement of switchdev was seen as a big win for the project. Whether it prompts any proprietary switch vendors to open up their product lines certain remains to be seen—although, on the following day, Intel developers did announce that the company would be writing switchdev drivers for its own switch chips.

Together, Mellanox and Intel still only account for a fraction of the overall switch device market, but their migration to switchdev is likely to influence other small vendors in the short term. And, if big customers like Verizon start purchasing switch hardware based on switchdev support, the proprietary switch vendors will surely begin to feel the heat.

[The author would like to thank the Netconf and Netdev organizers for travel assistance to Seville.]