Varlink: a protocol for IPC

One of the motivations behind projects like kdbus and bus1, both of which have fallen short of mainline inclusion, is to have an interprocess communication (IPC) mechanism available early in the boot process. The D-Bus IPC mechanism has a daemon that cannot be started until filesystems are mounted and the like, but what if the early boot process wants to perform IPC? A new project, varlink, was recently announced; it aims to provide IPC from early boot onward, though it does not really address the longtime D-Bus performance complaints that also served as motivation for kdbus and bus1.

The announcement came from Harald Hoyer, but he credited Kay Sievers and Lars Karlitski with much of the work. At its core, varlink is simply a JSON-based protocol that can be used to exchange messages over any connection-oriented transport. No kernel "special sauce" (such as kdbus or bus1) is needed to support it as TCP or Unix-domain sockets will provide the necessary functionality. The messages can be used as a kind of remote procedure call (RPC) using an API defined in an interface file.

One of the foundations of varlink is simplicity. As outlined on the "ideals" page, the protocol is "not specifically optimized for anything else but ease-of-use and maintainability". To that end, interface definitions are text files, readable by both machines and humans, that describe the services a varlink endpoint will provide. The interface files are meant to be self-documenting and can be retrieved using the GetInterfaceDescription() method of the varlink service interface (org.varlink.service). As Hoyer describes, they are human-readable so that the interfaces can be discussed widely:

Hoyer shows a simple example that gets information from the /etc/passwd file:

    interface com.redhat.system.accounts

    type Account (
      name: string,
      uid: int,
      gid: int,
      full_name: string,
      home: string,
      shell: string
    )

    method GetAccounts() -> (accounts: Account[])

    method GetAccountByUid(uid: int) -> (account: Account)

    method GetAccountByName(name: string) -> (account: Account)

    method AddAccount(account: Account) -> (account: Account)

    error AccountNotFound ()

    error AccountCreationFailed (field: string)

All it takes is four lines of Python to retrieve and print the information for the "root" user (for example). There is also a varlink command-line tool (written in C) that can be used to make varlink calls. Bindings for other languages (C, JavaScript, Go, Java, and Rust) are also available, though some are just a proof of concept at this point.

As described so far, there is still a missing piece. Some service must provide a way to resolve names like "com.redhat.system.accounts" to a Uniform Resource Identifier (URI) corresponding to the running service. If the service is known, but is not running, something needs to start it. Both of those tasks can be handled by the varlink resolver.

Unlike other protocols, such as D-Bus, varlink makes no provision for sending things like file descriptors. It is simply for sending simple data types (numbers, strings, arrays, etc.) That means the messages can be transparently proxied or redirected elsewhere for servicing. As the ideals statement notes: "Varlink should be free of any side-effects of local APIs. All interactions need to be simple messages on a network, not carrying things like file descriptors or references to locally stored files."

Varlink is available in a GitHub repository. It is available under the Apache 2.0 license.

As part of the announcement, Hoyer makes a sweeping claim about the current API to a Linux system: it could all be replaced with varlink-based interfaces. In that statement, he includes kernel interfaces, such as ioctl() and other system calls, procfs, and sysfs; the Linux command-line interface; and various IPC mechanisms including D-Bus and Protobuf. There is a kernel module that allows varlink interfaces to be added to the kernel, but it is a little hard to see the kernel API being replaced, even if it was deemed desirable. It would be decades (if not longer) before the existing kernel interfaces could be removed, which would make for a maintenance headache at minimum.

Hoyer does wryly note the classic xkcd standards proliferation comic: "Of course varlink is the 15th xkcd standard here".

As nice as it might be to have a single, standard interface mechanism throughout the Linux system, that's not a likely outcome. However, varlink does seem like it may have its uses. One would guess that, rather than have each early boot daemon have "fallback IPC via unix domain sockets with its own homegrown protocol", it may make sense for (some) distributions to move to varlink. Given that the developers are from Red Hat, Fedora would seem like a plausible starting place.

Varlink is a fairly simple way to gather needed information or request that certain services be performed, though it doesn't provide the kinds of guarantees that D-Bus is supposed to require—or the increased performance that folks have been clamoring for. The amount of churn throughout the Linux ecosystem to support it "everywhere" would be enormous and the benefits to doing so are not obvious. As they say, however, the future is unwritten.