Moving past TCP in the data center, part 1

Switches are the big pain here - they've created a world in which, because the switch can only forward or drop frames (where a router can also send errors back to the source, or possibly fragment frames), we can't increase packet size - a 100G switch that cannot interconnect to 10G networks is not useful, so the 100G switch uses 10G compatible frame size limits, and this repeats all the way down to a 100M switch wanting 10M compatible frame size limits.

One of the reasons IPv6 has the concept of link-local addressing is that design work on it started before switches were a thing in most computer networks - and the idea was that you'd have routers interconnecting network segments of different speeds (so your 100M FDDI network would go through a router to your 10M Ethernet segment, and thus devices on the FDDI network could use a higher MTU). If the industry as a whole had kept this idea alive, instead of having Ethernet switches interconnecting different speeds (10/100/1000/10G Ethernet all carrying identical frames), we'd have had routers, and we could have split the gains from faster wire speeds differently (because moving from one speed to another would go via a router that could send a packet too big message).

If we'd done that, we'd had have three choices:

1. Just improve serialization delay of the maximum packet size. This is what we actually did - we went from 1.2 ms for 10BASE-T to 0.12 ms for 100BASE-T to 0.012 ms for gigabit.
2. Just increase maximum MTU. We could have gone from 1,500 bytes to 15,000 bytes to 150,000 bytes for gigabit (with 100G allowing a maximum MTU of 15,000,000 bytes), maintaining serialization delay at 1.2 ms.
3. Split the difference. As an example tradeoff, we could have increased MTU 4x each generation, resulting in a 2.5x drop in serialization delay - 1500 bytes and 1.2 ms at 10M becomes 6000 bytes and 0.48 ms at 100M, which becomes 24,000 bytes and 0.2 ms at gigabit.