Moving past TCP in the data center, part 1 [LWN.net]

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 20:13 UTC (Wed) by jenro (subscriber, #7024)
Parent article: Moving past TCP in the data center, part 1

Is it just me, who thinks some of these ideas sound rather strange?

If I summarize the article it seems to read like: TCP is not a good base for any kind of RPC-system, so ditch all of TCP and replace it with a shiny new thing.

Is it really true, that all or at least most traffic in a DC is some kind of RPC or request/reply system? Is it really true, that small packages are the ones that the whole DC needs to be optimized for? I highly doubt this.

I am an application developer. When I request data from any kind of information service - for example a database - I want the data to be complete and reliable. And to me that is far more important than to have some data faster. The nice thing about a TCP session is: as long as the session stays connected and error free it doesn't matter, if I retrieve 100 bytes or 100,000 bytes or 100,000,000 bytes, TCP guarantees, that I get all the data in the correct order.

On the other hand I have to work with market data feeds of different stock exchanges like NYSE or Xetra. They use UDP multicast to distribute the information. Dealing with this unreliable transmissions and recovering from lost packets is something you really don't want to do.

So let's wait for the second part of the article and than decide, if I want to give up my reliable TCP connections for Homa.

to post comments

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 21:13 UTC (Wed) by NYKevin (subscriber, #129325) [Link]

> Is it really true, that all or at least most traffic in a DC is some kind of RPC or request/reply system?

In my experience (as a Google SRE), yes, at least for the way that Google tends to architect its services, anyway.

> I am an application developer. When I request data from any kind of information service - for example a database - I want the data to be complete and reliable. And to me that is far more important than to have some data faster. The nice thing about a TCP session is: as long as the session stays connected and error free it doesn't matter, if I retrieve 100 bytes or 100,000 bytes or 100,000,000 bytes, TCP guarantees, that I get all the data in the correct order.

For a typical CRUD app, you're generally dealing with one tuple at a time, or at most a few dozen (e.g. for search results).

Yes, we do have batch jobs that look at many (or all) tuples and manipulate them in some way. No, those jobs are not top priority. Top priority is responding to incoming HTTP requests in a timely fashion. If that means throwing away hours of batch work because live serving needs the machine, so be it.

> On the other hand I have to work with market data feeds of different stock exchanges like NYSE or Xetra. They use UDP multicast to distribute the information. Dealing with this unreliable transmissions and recovering from lost packets is something you really don't want to do.

Sounds like you're in finance. This is a very different sort of industry to tech - your top priority is probably some variation of figuring out which way the market is moving, not responding to incoming HTTP. So things are going to look different on your side of the fence. That does not mean the article is wrong. It just doesn't apply to your particular situation.

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 21:19 UTC (Wed) by joib (subscriber, #8541) [Link] (2 responses)

> Is it just me, who thinks some of these ideas sound rather strange?

It's just you.

> Is it really true, that all or at least most traffic in a DC is some kind of RPC or request/reply system? Is it really true, that small packages are the ones that the whole DC needs to be optimized for? I highly doubt this.

No, the argument is that 'lots of small messages' is an important usecase that TCP struggles with, due to inherent features in the TCP design. Large transfers are certainly not unimportant, but OTOH, if you can fill your pipe with lots of small messages, then the special case of a few very large messages is kind of trivial.

> if I want to give up my reliable TCP connections for Homa.

So with TCP you get 'reliable stream', and with UDP you get 'unreliable datagram'. The argument here is that for a lot of applications, the correct primitive is neither of these, but rather a 'reliable datagram' kind of service. Which is what this Homa proposal is apparently providing. Incidentally, Amazon's SRD is another design that has converged to a roughly similar conclusion.

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 23:39 UTC (Wed) by willy (subscriber, #9762) [Link]

It sounds like it's reliable out-of-order datagrams which is different from SOCK_SEQPACKET (reliable in-order datagrams)

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 23:43 UTC (Wed) by rcampos (subscriber, #59737) [Link]

TIPC is another protocol that offers reliable datagram. It's merged in Linux since the 2.6 era at least.

It is a very interesting protocol for local networks, but don't know benchmarks about it.

Moving past TCP in the data center, part 1

Posted Nov 3, 2022 8:20 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> The nice thing about a TCP session is: as long as the session stays connected and error free it doesn't matter, if I retrieve 100 bytes or 100,000 bytes or 100,000,000 bytes, TCP guarantees, that I get all the data in the correct order.

This isn't such a great advantage for many datacenter applications. You have to design services very differently if they need to handle 1K or 100M of data.

In fact, ability to hide this leads to bad designs.

Moving past TCP in the data center, part 1

Posted Dec 2, 2022 20:57 UTC (Fri) by marcH (subscriber, #57642) [Link] (1 responses)

Do we need a new https://en.wikipedia.org/wiki/Fallacies_of_distributed_co... ?

Moving past TCP in the data center, part 1

Posted Dec 4, 2022 5:25 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

Absolutely. Including discussions of emergent behavior, such as retry storms.