Moving past TCP in the data center, part 1 [LWN.net]

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 11:15 UTC (Wed) by smoogen (subscriber, #97)
Parent article: Moving past TCP in the data center, part 1

How do you deal with inter DC traffic? It seems to be common these days to just move half your compute needs to a different DC for cost or similar reasons. These may even be different 'clouds', and you may end up moving stuff live depending on failure or rollout issues. Would this mean that applications would need to be dual stacked? As in 'use HOMA' for local and fail over to TCP for 'been moved to Canada'?

to post comments

Moving past TCP in the data center, part 1

Posted Nov 2, 2022 19:30 UTC (Wed) by NYKevin (subscriber, #129325) [Link]

It depends on how your traffic is routed:

1. If you're only going over fiber links that you own (or lease), then you can send whatever you want over those links, but if you start tampering with the IP layer or below, some routers and/or switches might be unhappy about that. Playing with the transport layer should be fine in most reasonable deployments.
2. If you're going over one fixed ISP's lines, then you can experiment and/or talk to them to figure out exactly how transparent they are to "weird" protocols. Most ISPs at least have the good sense to be transparent-ish to TCP, but some ISPs will be fully IP transparent and won't care what you do at the transport layer. This is probably easier if you are paying them a lot of money and/or have some sort of peering arrangement with them.
3. If you're going over the public internet, I imagine there's a lot of old gear that will do bizarre things if you try sending packets it doesn't recognize.
4. You can always encapsulate your non-TCP transport in UDP, although that adds enough overhead that it may not be worth it. Some ISPs may not treat UDP nicely, but IMHO you have good cause to complain if that happens.