legacy MTU of 1500 bytes [LWN.net]

legacy MTU of 1500 bytes

Posted Nov 2, 2022 20:32 UTC (Wed) by stephen.pollei (subscriber, #125364)
In reply to: Moving past TCP in the data center, part 1 by shemminger
Parent article: Moving past TCP in the data center, part 1

I really wish that path MTU discovery worked better and default MTU would typically be much higher. We have networks that run a thousand times faster so using a factor of about ten to increase typical MTU seems reasonable enough. 16k at least, everywhere might be nice dream.

to post comments

legacy MTU of 1500 bytes

Posted Nov 3, 2022 10:50 UTC (Thu) by farnz (subscriber, #17727) [Link] (4 responses)

Switches are the big pain here - they've created a world in which, because the switch can only forward or drop frames (where a router can also send errors back to the source, or possibly fragment frames), we can't increase packet size - a 100G switch that cannot interconnect to 10G networks is not useful, so the 100G switch uses 10G compatible frame size limits, and this repeats all the way down to a 100M switch wanting 10M compatible frame size limits.

One of the reasons IPv6 has the concept of link-local addressing is that design work on it started before switches were a thing in most computer networks - and the idea was that you'd have routers interconnecting network segments of different speeds (so your 100M FDDI network would go through a router to your 10M Ethernet segment, and thus devices on the FDDI network could use a higher MTU). If the industry as a whole had kept this idea alive, instead of having Ethernet switches interconnecting different speeds (10/100/1000/10G Ethernet all carrying identical frames), we'd have had routers, and we could have split the gains from faster wire speeds differently (because moving from one speed to another would go via a router that could send a packet too big message).

If we'd done that, we'd had have three choices:

Just improve serialization delay of the maximum packet size. This is what we actually did - we went from 1.2 ms for 10BASE-T to 0.12 ms for 100BASE-T to 0.012 ms for gigabit.
Just increase maximum MTU. We could have gone from 1,500 bytes to 15,000 bytes to 150,000 bytes for gigabit (with 100G allowing a maximum MTU of 15,000,000 bytes), maintaining serialization delay at 1.2 ms.
Split the difference. As an example tradeoff, we could have increased MTU 4x each generation, resulting in a 2.5x drop in serialization delay - 1500 bytes and 1.2 ms at 10M becomes 6000 bytes and 0.48 ms at 100M, which becomes 24,000 bytes and 0.2 ms at gigabit.

legacy MTU of 1500 bytes

Posted Nov 3, 2022 17:39 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> Switches are the big pain here - they've created a world in which, because the switch can only forward or drop frames

The MTU should have been reified to the IP level. Not doing this was one of the most unforgivable errors in the IPv6 development.

In the ideal world all the IP packets would have two fields: forward minimum packet size and the reverse minimum packet size. A switch/router during packet forwarding sets the forward minimum size to their max size, if it's less than the current value. Reverse size is simply propagated back.

legacy MTU of 1500 bytes

Posted Nov 3, 2022 17:45 UTC (Thu) by guus (subscriber, #41608) [Link]

> The MTU should have been reified to the IP level. Not doing this was one of the most unforgivable errors in the IPv6 development.

Actually the same should be done for congestion control; it makes no sense that multiple TCP connections to the same host each implement their own congestion control loop when all packets go via exactly the same path. This is in effect what QUIC is doing: combine multiple reliable, in-order streams with a single congestion control loop. But of course that only is effective if you really use a single QUIC connection.

legacy MTU of 1500 bytes

Posted Nov 4, 2022 23:15 UTC (Fri) by stephen.pollei (subscriber, #125364) [Link] (1 responses)

I think that I'm in camp #3 "Split the difference". Some decrease in packet transmit latency seems good, but it isn't the only thing to optimize. I also remember 56k modems, where 1500 octets is 214 milliseconds worth of time.

In practice, Path MTU discovery can break in in interesting ways . People over block ICMP and perhaps a mechanism closer to what Cyberax suggested would be nice. With tunneling and MTU discovery having issues, CloudFlare at one point reduced their MTU to 1024 for ipv4 and 1280 for ipv6. It might have been nicer world in many ways if routers/switches supported 16k packets, but most people only used 12k normally. Extra 4k could be margin for tunneling and encapsulation. This is of course all IMHO.

legacy MTU of 1500 bytes

Posted Nov 5, 2022 9:47 UTC (Sat) by farnz (subscriber, #17727) [Link]

To an unfortunately large extent, broken Path MTU discovery is accepted on the Internet because you can pretty much assume that a 1280 byte MTU will be close enough to optimal that it makes no practical difference. If we had much larger MTUs - 6,000 bytes for 100M Ethernet, 24k for gig, 96k for 10G (yes, this needs jumbograms at the IP layer), WiFi probably at around 12k for 802.11n, and about 50k for 802.11ac, we'd have much more incentive to get PMTUD working well - the difference between 1280 bytes (minimum IPv6 frame size), and 12k for 802.11n is huge.