Moving past TCP in the data center, part 1 [LWN.net]

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 3:06 UTC (Wed) by Cyberax (✭ supporter ✭, #52523)
In reply to: Moving past TCP in the data center, part 1 by KaiRo
Parent article: Moving past TCP in the data center, part 1

If anything, QUIC is even worse because it mandates encryption which is not really needed in a DC.

The criticisms of TCP for DC applications are on point. Any sane RPC design already has to deal with retries, timeouts and request idempotency. So switching to a connectionless protocol is definitely an interesting way.

With a way to fall back onto reliable streaming protocol as needed (e.g. if an RPC call needs to return a large amount of data).

to post comments

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 4:13 UTC (Wed) by willy (subscriber, #9762) [Link] (7 responses)

A flow-control-free protocol is not exactly a new idea. Here's IL from plan9 (circa 1993; the "4th edition" in the URL is misleading): http://doc.cat-v.org/plan_9/4th_edition/papers/il/

They still saw an advantage to maintaining a connection in order to manage reliable service. I don't know that was the right choice, but I'm looking forward to reading about Homa's design decisions.

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 4:50 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

Even Homa is excessive for most cases. Way back when I worked in a large cloud company, we did some experiments with UDP-based transport for RPC.

The idea was dead simple: just throw away EVERYTHING.

The network layer simply used fixed-size jumbo Ethernet frames and a request could contain up to 4 of them. The sender simply sent them one by one. No retransmissions or buffering, so the code was dead simple.

The receiver simply reassembled the frames into a buffer. Since the frame sizes were fixed, only 1 data copy was necessary (or none, if a request fits into 1 packet). No NAKs, ACKs or anything was needed, just a simple timer to discard the data and fail the request in case of a timeout due to a lost packet.

Everything else was handled on the upper levels. E.g. a packet loss simply got translated into a timeout error from the RPC layer. Instead of a network-level retry, regular retry policy for the service calls was used. It worked surprisingly well in experiments, and actually had a very nice property of making sure that the network congestion pressure rapidly propagates upstream.

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 9:21 UTC (Wed) by amboar (subscriber, #55307) [Link]

This is roughly the strategy used by the DMTF's MCTP protocol for intra-platform communication between devices. It's kinda impressive in its simplicity and effectiveness

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 12:36 UTC (Wed) by paulj (subscriber, #341) [Link] (3 responses)

Sending back-to-back frames - at a low enough level to ensure no other station could try send in-between - was a trick SGI used in IRIX NFS to get performance - I think I remember reading this in comments on LWN before, perhaps in a reply to you mentioning the same thing before. :)

NFS was of course RPC over UDP based too.

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 21:12 UTC (Wed) by amarao (guest, #87073) [Link] (2 responses)

I remember time people said that udp for NFS is back and we should switch to tcp. Had something changed?

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 21:32 UTC (Wed) by joib (subscriber, #8541) [Link]

https://www.man7.org/linux/man-pages/man5/nfs.5.html#TRAN... has a discussion about the pitfalls of NFS over udp.

Should also be noted that current Linux kernels actually no longer support NFS over UDP.

Moving past TCP in the data center, part 1

Posted Nov 3, 2022 9:52 UTC (Thu) by paulj (subscriber, #341) [Link]

NFS has switched to TCP since /ages/ ago. However, in the beginning (and for quite a while) NFS ran its RPC protocol over UDP. Just, without any real flow-control, and poor loss handling. TCP provides flow-control and better loss-handling - though, still not great, (because of the impedance mismatch of TCP knowing only about the data as a byte-stream, and not being able to take the app. message boundaries into consideration), as per this talk.

Moving past TCP in the data center, part 1

Posted Nov 10, 2022 1:06 UTC (Thu) by TheJosh (guest, #162094) [Link]

Did this ever get used in production?

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 11:04 UTC (Wed) by paulj (subscriber, #341) [Link] (6 responses)

Large tech companies want encryption within their DC. They have worked hard at compartmentalising access levels. This still leaves some super-user roles able to snoop on network traffic. Encryption at least prevents those with network super-user roles from snooping on server traffic, and those with super-user access on some server-roles from being able to snoop on traffic for other server-roles (if they can get at the traffic, which they probably can due to internal service-IP advertising tools).

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 23:55 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

We had integrity control and request authentication as a part of the protocol (basically, hash the message along with the secret key that only you and the receiver know). Mostly to make sure the data is not corrupted in transit, which absolutely does happen at large enough scales.

The service for this protocol was used to move already encrypted data, so additional layers of encryption were unnecessary for us. But a more general protocol should definitely have support for it.

> Encryption at least prevents those with network super-user roles from snooping on server traffic, and those with super-user access on some server-roles from being able to snoop on traffic for other server-roles

Snooping at datacenter-scale is surprisingly not at all useful. You can observe only a small amount of traffic if you're trying to do it by physically splicing into cables, and if you can do it on the machines that run the network code, it's probably game over already.

Moving past TCP in the data center, part 1

Posted Nov 3, 2022 9:59 UTC (Thu) by paulj (subscriber, #341) [Link] (4 responses)

> Snooping at datacenter-scale is surprisingly not at all useful. You can observe only a small amount of traffic if you're trying to do it by physically splicing into cables, and if you can do it on the machines that run the network code, it's probably game over already.

If an attacker has access to machines, without role-authentication and encryption an attacker can potentially use that to widen the number of services and access-roles the attacker can manipulate. (And authentication without a MAC is effectively the same - and if you're going to MAC the traffic, you can as easily encrypt it).

Bear in mind, in the large tech companies some of the server machines basically have a (host controlled) switch built-in to them, to allow multiple hosts in a brick/sled to share the same PHY to the network. Also, the switches are basically (specialised) servers too. So an attacked could insert code in the switching agents to programme the switching hardware to selectively pick out certain flows, for later analysis to use for privilege widening/escalation - quite efficient.

Moving past TCP in the data center, part 1

Posted Nov 3, 2022 17:42 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

> And authentication without a MAC is effectively the same - and if you're going to MAC the traffic, you can as easily encrypt it

Not quite true for very high-traffic services.

> Bear in mind, in the large tech companies some of the server machines basically have a (host controlled) switch built-in to them, to allow multiple hosts in a brick/sled to share the same PHY to the network.

It's typically more complicated. A host can contain one or more PHYs and they typically go to the TOR (top-of-the-rack) device that does further routing/NAT/whatever.

Moving past TCP in the data center, part 1

Posted Nov 4, 2022 10:28 UTC (Fri) by paulj (subscriber, #341) [Link] (2 responses)

You're thinking of the 'enterprise' type server world. I'm thinking of the much, much, much bigger FANG-type DC world. At at least 1 of the biggest DC operators in the world, the (newer) servers do not have an ethernet PHY. The host has a PCIe connection to a special shared-NIC (which is like a very cut-down, hacky L2+ switching ASIC). 3 or more hosts connect to the shared-NIC. The shared NIC then switches the host ethernet packets, which are sent over hardware specific messaging to the shared-NIC, onto the PHY.

There is one more PHY, to the BMC, which controls the chassis and can give serial access to the hosts (and some very slow, hacky IP/TCP to the host, IIRC).

There are some large enterprise "blade" type systems which also use onboard switching ASICS I think, but the ones I know of use more traditional switching ASICs, (e.g. HPE were using Intel RRC10k) with actual NIC blocks for each host included into the ASIC. So these actually look and work a lot more like a traditional network, with proper buffering and flow-control between the hosts and the switching logic and the upstream (the shared NICs above did not do this properly - least in earlier iterations - causing significant issues).

Moving past TCP in the data center, part 1

Posted Nov 4, 2022 10:32 UTC (Fri) by paulj (subscriber, #341) [Link]

Oh, and the reasoning for this is that PHYs are expensive. Specifically, the optics are and the power needed for higher-end optics along with the thermal budget/constraints (the DC cabling is also a headache, but more minor). Fewer PHYs, fewer high-end transceivers == cheaper + lower-power, more efficient DCs. At the scales the really really big DC operators work at anyway.

Moving past TCP in the data center, part 1

Posted Nov 23, 2022 22:25 UTC (Wed) by Rudd-O (guest, #61155) [Link]

Having worked in FANG I can attest to the correctness of what you're saying.

Cyberax's views would fit right into, let's say, a Google in the year 2011. In the year 2015 — after Snowden — it was already consensus that everything needed to be encrypted, no exceptions.

And so all traffic inside Google — anything at all using Stubby — is, in fact, encrypted. As you said, if you're gonna MAC the traffic, might as well crypt it too. All those primitives are accelerated in the CPU, and some in software now.

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 20:36 UTC (Wed) by MatejLach (guest, #84942) [Link] (12 responses)

> it mandates encryption which is not really needed in a DC.

Is this still the general consensus even after Snowden? I am not saying it's a silver bullet against a dedicated attacker like the NSA, but I'd think that every additional obstacle in the way is going to raise the bar for dragnets just a tiny bit, no?

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 21:58 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (3 responses)

You at least want encryption over any line you don't physically control (i.e. it is not inside a building you own), regardless of ownership/leasing rights, which in practice means you encrypt all inter-DC traffic. You also probably want integrity (signatures, certificates, etc.) for most if not all traffic, to make it more difficult for malware and other adversaries to move laterally within your systems and/or escalate their privileges. Encryption within the DC may not be strictly necessary if you have good physical security and you trust the networking equipment, but those are both big ifs.

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 22:40 UTC (Wed) by Wol (subscriber, #4433) [Link] (2 responses)

My company - along with many others I suspect - is moving away from "trusted core". My works laptop now needs 2fa to authenticate to works servers - even when in the office and connecting over corporate infrastructure.

Given the breakins and stuff businesses have suffered, yes it's an absolute pain, but it means that if anybody does manage to break in to my laptop, it's now rather harder for them to jump to someone else and break into their laptop, moving up the chain ...

Cheers,
Wol

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 23:09 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (1 responses)

> My company - along with many others I suspect - is moving away from "trusted core". My works laptop now needs 2fa to authenticate to works servers - even when in the office and connecting over corporate infrastructure.

1. 2FA has nothing to do with encryption. 2FA is primarily about stopping phishing, and only used by humans (I was talking about machine-to-machine communication).
2. Unless your laptop is in the same physical building as all of the servers you will be interacting with, and your company has complete autonomy over that building (i.e. you're not leasing it out from someone who might have physical access), you're not using "trusted" lines in the sense I was referencing. I explicitly said this isn't about who owns or leases the lines. It's about who is able to physically touch and interact with the lines.

Moving past TCP in the data center, part 1

Posted Nov 3, 2022 7:56 UTC (Thu) by Wol (subscriber, #4433) [Link]

Yup. Might not be quite what you were talking about, but it's a general trend to restrict the trusted zone, even in a trusted network, such that there's minimal trust between any actors, even if you would assume that they are trusted actors.

Cheers,
Wol

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 23:58 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

NSA is not going to install spy boxes in your DC network switches. There's simply way too much traffic to analyze and the network complexity makes it infeasible to even do independently of AWS.

So NSA would simply officially ask Amazon to provide them with a covert way to access the data for a specific customer directly using the AWS services.

Moving past TCP in the data center, part 1

Posted Nov 3, 2022 9:48 UTC (Thu) by paulj (subscriber, #341) [Link] (1 responses)

The NSA may however have its employees go and work for your company under cover. It's pretty much a certainty that /multiple/ state intelligence agencies have agents working at the various large tech companies.

Minimising the trust and scope of access of what employees can access, to what they need to access for their immediate role, is a good thing, security wise.

Moving past TCP in the data center, part 1

Posted Nov 3, 2022 17:31 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

Sure. But the complexity of AWS is so overwhelming that you won't really be able to do much unless you want a very targeted attack and have access through many layers of security (physical and virtual).

Moving past TCP in the data center, part 1

Posted Nov 23, 2022 22:27 UTC (Wed) by Rudd-O (guest, #61155) [Link] (3 responses)

Counterpoint: NSA absolutely does install spyware into network switches.

Snowden already revealed this. There are extensive talks on the subject presented by Jacob Applebaum at CCC.

Moving past TCP in the data center, part 1

Posted Nov 23, 2022 23:43 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

Sure. So you installed spyware into a TOR switch. Then what? The data plane is too fast for any meaningful analysis with the puny switch CPU.

You maybe can mirror one port to another, but in an Amazon DC this means nothing. You'll just confuse some random EC2 hardware instance. You won't even be able to do much if you redirect the traffic to an instance that you control, because it's encrypted.

Moving past TCP in the data center, part 1

Posted Nov 24, 2022 9:54 UTC (Thu) by paulj (subscriber, #341) [Link]

"The data plane is too fast for any meaningful analysis with the puny switch CPU."

Except the L3 switch ASIC can be programmed to redirect only certain flows to said CPU. They can also be programmed to encap and redirect certain flows to other hosts. Indeed, they can be programmed to mirror packets (but I can't remember if the L3 ASICs commonly used at the super-large DCs can /both/ mirror and encap the same flow - if not, it's just a matter of time till they do).

So:

1. You don't need to analyse the entire data flow on the puny switch CPU, cause the /powerful/ switching ASIC can be programmed to do hardware tcpdumping (basically). Given the CPUs on these switches aren't /that/ puny (low to mid end Xeons), further analysis on host is quite feasible.
2. Even better, you can just redirect the flow you're interested in to a bigger server by encapping it (the server can resend the flow's packets out again so they're not missed).

Moving past TCP in the data center, part 1

Posted Nov 24, 2022 9:56 UTC (Thu) by paulj (subscriber, #341) [Link]

"You won't even be able to do much if you redirect the traffic to an instance that you control, because it's encrypted."

I note this is a good counter-argument to your own argument in another sub-thread that intra-DC traffic doesn't need to be encrypted. ;)

Moving past TCP in the data center, part 1

Posted Nov 23, 2022 22:26 UTC (Wed) by Rudd-O (guest, #61155) [Link]

No it is not the consensus *at all*.

Google encrypt all its traffic at the Stubby layer.

Moving past TCP in the data center, part 1

Posted Nov 3, 2022 15:32 UTC (Thu) by k3ninho (subscriber, #50375) [Link]

>mandates encryption which is not really needed in a DC.

nth-ing the pile-on about this. 'Zero Trust' is an important choice when you're sharing network links with unknown people or having malicious and incompetent staff mess things up in your datacentre. QUIC sucks because encrypted links are expensive to set up in the context of small message sizes in a datacentre.

K3n.

Moving past TCP in the data center, part 1

Posted Nov 10, 2022 18:32 UTC (Thu) by flussence (guest, #85566) [Link]

> it mandates encryption which is not really needed in a DC.

If your org's budget doesn't extend to CPUs with AES-NI instructions that operate at line rate and you're *sure* it's a closed system, there's nothing really stopping you from dropping in a highly optimised rot26 cipher, like SSH-HPN does.