Moving past TCP in the data center, part 1 [LWN.net]

Moving past TCP in the data center, part 1

Posted Nov 3, 2022 17:42 UTC (Thu) by Cyberax (✭ supporter ✭, #52523)
In reply to: Moving past TCP in the data center, part 1 by paulj
Parent article: Moving past TCP in the data center, part 1

> And authentication without a MAC is effectively the same - and if you're going to MAC the traffic, you can as easily encrypt it

Not quite true for very high-traffic services.

> Bear in mind, in the large tech companies some of the server machines basically have a (host controlled) switch built-in to them, to allow multiple hosts in a brick/sled to share the same PHY to the network.

It's typically more complicated. A host can contain one or more PHYs and they typically go to the TOR (top-of-the-rack) device that does further routing/NAT/whatever.

to post comments

Moving past TCP in the data center, part 1

Posted Nov 4, 2022 10:28 UTC (Fri) by paulj (subscriber, #341) [Link] (2 responses)

You're thinking of the 'enterprise' type server world. I'm thinking of the much, much, much bigger FANG-type DC world. At at least 1 of the biggest DC operators in the world, the (newer) servers do not have an ethernet PHY. The host has a PCIe connection to a special shared-NIC (which is like a very cut-down, hacky L2+ switching ASIC). 3 or more hosts connect to the shared-NIC. The shared NIC then switches the host ethernet packets, which are sent over hardware specific messaging to the shared-NIC, onto the PHY.

There is one more PHY, to the BMC, which controls the chassis and can give serial access to the hosts (and some very slow, hacky IP/TCP to the host, IIRC).

There are some large enterprise "blade" type systems which also use onboard switching ASICS I think, but the ones I know of use more traditional switching ASICs, (e.g. HPE were using Intel RRC10k) with actual NIC blocks for each host included into the ASIC. So these actually look and work a lot more like a traditional network, with proper buffering and flow-control between the hosts and the switching logic and the upstream (the shared NICs above did not do this properly - least in earlier iterations - causing significant issues).

Moving past TCP in the data center, part 1

Posted Nov 4, 2022 10:32 UTC (Fri) by paulj (subscriber, #341) [Link]

Oh, and the reasoning for this is that PHYs are expensive. Specifically, the optics are and the power needed for higher-end optics along with the thermal budget/constraints (the DC cabling is also a headache, but more minor). Fewer PHYs, fewer high-end transceivers == cheaper + lower-power, more efficient DCs. At the scales the really really big DC operators work at anyway.

Moving past TCP in the data center, part 1

Posted Nov 23, 2022 22:25 UTC (Wed) by Rudd-O (guest, #61155) [Link]

Having worked in FANG I can attest to the correctness of what you're saying.

Cyberax's views would fit right into, let's say, a Google in the year 2011. In the year 2015 — after Snowden — it was already consensus that everything needed to be encrypted, no exceptions.

And so all traffic inside Google — anything at all using Stubby — is, in fact, encrypted. As you said, if you're gonna MAC the traffic, might as well crypt it too. All those primitives are accelerated in the CPU, and some in software now.