Moving past TCP in the data center, part 1

The idea was dead simple: just throw away EVERYTHING.

The network layer simply used fixed-size jumbo Ethernet frames and a request could contain up to 4 of them. The sender simply sent them one by one. No retransmissions or buffering, so the code was dead simple.

The receiver simply reassembled the frames into a buffer. Since the frame sizes were fixed, only 1 data copy was necessary (or none, if a request fits into 1 packet). No NAKs, ACKs or anything was needed, just a simple timer to discard the data and fail the request in case of a timeout due to a lost packet.

Everything else was handled on the upper levels. E.g. a packet loss simply got translated into a timeout error from the RPC layer. Instead of a network-level retry, regular retry policy for the service calls was used. It worked surprisingly well in experiments, and actually had a very nice property of making sure that the network congestion pressure rapidly propagates upstream.