Offload-friendly network encryption in the kernel

The PSP security protocol (PSP) is a way to transparently encrypt packets by efficiently offloading encryption and decryption to the network interface cards (NICs) that Google uses for connections inside its data centers. The protocol is similar to IPsec, in that it allows for wrapping arbitrary traffic in a layer of encryption. The difference is that PSP is encapsulated in UDP, and designed from the beginning to reduce the amount of state that NICs have to track in order to send and receive encrypted traffic, allowing for more simultaneous connections. Jakub Kicinski wants to add support for the protocol to the Linux kernel.

The protocol

PSP is a fairly minimal protocol. It completely avoids the topic of how to do a secure key exchange, assuming that the applications on either end of a connection will be able to exchange symmetric-encryption keys somehow. This is not an unreasonable design decision, since IPsec does the same thing. There are several existing protocols for securely exchanging keys, such as Internet Key Exchange or Kerberized Internet Negotiation of Keys. Usually, those protocols are indifferent to the source of the symmetric keys. PSP is a bit different. To support hardware-offload, PSP requires that the NIC itself generate the keys for a PSP connection. The PSP architecture specification goes into detail about how NICs should do that.

The main requirement is that the NIC should be able to rederive the key for a session from a fairly limited amount of information — specifically, from a 32-bit "Security Parameter Index" (SPI) value and a secret key stored on the device. That key is generated by the NIC and used to derive the session-specific encryption key for each connection as needed, using a secure key-derivation function. Therefore, the SPI alone is not enough to decrypt a packet; this allows SPIs to be included in PSP packets, letting the NIC rederive the encryption key for a packet on the fly. In turn, this means that the NIC does not actually need to store an encryption key in order to receive and process a packet, greatly reducing the amount of memory necessary on the device.

Unfortunately, this requirement to rederive keys comes at a price — PSP connections are unidirectional. For real use cases where bidirectional communication is required, the application needs to set up a separate PSP connection for each direction. Since rederiving the key requires access to a device key stored on the NIC, only the receiving NIC can rederive it, so the transmitting device still needs to store the key somewhere. While that could be on the NIC, the PSP specification leaves open the possibility of a hardware implementation that requires the computer to send the encryption key to the NIC alongside any transmitted packets.

To encrypt packets, PSP uses AES-128-GCM or AES-256-GCM. These are both authenticated encryption with associated data (AEAD) schemes — they guarantee that the received data has not been tampered with (authentication) and bundle some encrypted data alongside some associated plain-text data. In PSP's case, this is used to implement an offset that allows the sender to leave the headers of a protocol encapsulated in PSP unencrypted, while still protecting the contents. Supporting only two modes of AES keeps the implementation complexity of PSP low.

PSP also has a packet layout designed to make parsing the packet in hardware more efficient, by providing explicit lengths in the header and using fewer optional headers than IPsec does. In combination with the unique key-derivation scheme, PSP ends up being more hardware-implementation-friendly than other encryption protocols like IPsec or TLS.

While Google is both the originator and largest current user of PSP, the protocol could potentially be useful to other users. Compared to other encrypted protocols, PSP requires a lot less on-device memory, letting it scale to larger numbers of connections. Because the protocol doesn't mandate a key-exchange standard, PSP is probably a good choice for an environment where the user controls both ends of the connection, but still wants to ensure that traffic can be encrypted.

The discussion

Despite how useful Google finds the protocol, kernel developers were dubious about adding yet another encryption protocol to the kernel, which already handles IPsec, WireGuard, TLS, and others. Paul Wouters expressed surprise that Kicinski wanted to add PSP to the kernel, when the IETF had declined to standardize the protocol on the basis that it is too similar to IPsec.

Steffen Klassert shared a draft that the IPsecME working group has been putting together that covers some of the same use cases as PSP. That may not be as helpful as it sounds, however, because there are already hardware devices implementing PSP, Willem de Bruijn pointed out. "It makes sense to work to get to an IETF standard protocol that captures the same benefits. But that is independent from enabling what is already implemented."

That answer didn't satisfy Wouters, who asked: "How many different packet encryption methods should the linux kernel have?" He said that waiting for protocols to be standardized provides interoperability, and chances to make sure a protocol is actually useful for more than one use case. PSP and IPsec can also use a lot of the same NIC hardware, he pointed out.

"I don't disagree on the merits of a standards process, of course. It's just moot at this point wrt PSP", de Bruijn replied. Klassert agreed that existing PSP users need to be supported, but thought that it was still important to work on standardizing a modern encryption protocol that meets everyone's needs. He invited Google to send representatives to the IETF IPsecME working group meeting to discuss the topic. De Bruijn said that the company would.

But some reviewers also had technical objections to Kicinski's patch set. The API he proposed allows user space to set a PSP encryption key on a socket, after which data transmitted through that socket will be encrypted. The problem is how exactly this interacts with retransmissions. PSP is encapsulated in UDP, and has no guarantee that data will arrive intact or in order, leaving that detail up to higher-level protocols. But Kicinski is mainly interested in PSP as a TLS replacement — which of course runs on top of TCP. PSP supports wrapping TCP, but just by encrypting individual packets, not by participating in TCP's retransmission logic.

Together, this creates some edge cases, de Bruijn pointed out. How is the kernel supposed to handle retransmissions of plain-text data after a PSP key has been associated with a socket? "Like TLS offload, the data is annotated 'for encryption' when queued. So data queued earlier or retransmits of such data will never be encrypted", Kicinski explained.

De Bruijn wasn't completely satisfied with that, pointing out that it still leaves edge cases. What happens if one peer upgrades the connection at the same time the other peer decides to drop it? "If (big if) that can happen, then the connection cannot be cleanly closed." In response, Kicinski suggested "only enforcing encryption of data-less segments once we've seen some encrypted data". De Bruijn agreed that might help, but was still worried that the kernel API could still be used in ways that break the correctness of the protocol, and suggested that some more thorough documentation might be appropriate.

In response, Kicinski put together some sequence diagrams showing how a PSP-secured socket is set up and torn down. De Bruijn thought that was a substantial improvement, but remained unconvinced that all of the edge cases had been handled.

Lance Richardson raised some questions about the kernel API as well, noting that there didn't seem to be any real support for rekeying a socket, something that PSP mandates every 24 hours. In a follow-up message, Richardson suggested that it might be as simple as keeping an old key around for a minute or two after rekeying. Kicinski agreed that made sense, and promised to add it to the next version of the patch set.

While there are lots of reasons to use PSP, and the presence of hardware that supports it is a good sign, the lack of a standard and questions around the implementation suggest that it may be some time before support is finalized. Still, in the future we could see yet another encryption protocol come to the kernel. The exact details of how that happens remain to be seen.