Moving past TCP in the data center, part 1

Even Homa is excessive for most cases. Way back when I worked in a large cloud company, we did some experiments with UDP-based transport for RPC.

The idea was dead simple: just throw away EVERYTHING.

The network layer simply used fixed-size jumbo Ethernet frames and a request could contain up to 4 of them. The sender simply sent them one by one. No retransmissions or buffering, so the code was dead simple.

The receiver simply reassembled the frames into a buffer. Since the frame sizes were fixed, only 1 data copy was necessary (or none, if a request fits into 1 packet). No NAKs, ACKs or anything was needed, just a simple timer to discard the data and fail the request in case of a timeout due to a lost packet.

Everything else was handled on the upper levels. E.g. a packet loss simply got translated into a timeout error from the RPC layer. Instead of a network-level retry, regular retry policy for the service calls was used. It worked surprisingly well in experiments, and actually had a very nice property of making sure that the network congestion pressure rapidly propagates upstream.

We had integrity control and request authentication as a part of the protocol (basically, hash the message along with the secret key that only you and the receiver know). Mostly to make sure the data is not corrupted in transit, which absolutely does happen at large enough scales.

The service for this protocol was used to move already encrypted data, so additional layers of encryption were unnecessary for us. But a more general protocol should definitely have support for it.

> Encryption at least prevents those with network super-user roles from snooping on server traffic, and those with super-user access on some server-roles from being able to snoop on traffic for other server-roles

Snooping at datacenter-scale is surprisingly not at all useful. You can observe only a small amount of traffic if you're trying to do it by physically splicing into cables, and if you can do it on the machines that run the network code, it's probably game over already.

You at least want encryption over any line you don't physically control (i.e. it is not inside a building you own), regardless of ownership/leasing rights, which in practice means you encrypt all inter-DC traffic. You also probably want integrity (signatures, certificates, etc.) for most if not all traffic, to make it more difficult for malware and other adversaries to move laterally within your systems and/or escalate their privileges. Encryption within the DC may not be strictly necessary if you have good physical security and you trust the networking equipment, but those are both big ifs.

NSA is not going to install spy boxes in your DC network switches. There's simply way too much traffic to analyze and the network complexity makes it infeasible to even do independently of AWS.

So NSA would simply officially ask Amazon to provide them with a covert way to access the data for a specific customer directly using the AWS services.

No it is not the consensus *at all*.

Google encrypt all its traffic at the Stubby layer.