LWN: Comments on "So you think you understand IP fragmentation?"

So you think you understand IP fragmentation?

sammythesnake — Fri, 23 Feb 2024 15:48:32 +0000

Colour me sceptical on that idea. I imagine that in such a case there would be some vigorously partisan "discussions" of exactly where responsibility lies - is the router responsible for how the source/destination behaves when packets go missing?

I think there's a good argument that occasional missing packets is a normally expected behaviour of "the internet" - a whole lot of the specs for things like TCP/IP exist specifically because of that fact. When it happens unnecessarily, that's certainly a *performance* issue, but not a *security* issue in some random part of the internet, rather in any end-point that reacts by leaking information or whatever.

Any endpoint that can't stay as safe as a "connection failed" error really shouldn't be dealing with anything security related...

If an intermediary on the path *rewrites* stuff, that's a much harder thing to justify by this kind of argument, but even then I think the more reasonable next step is ensuring integrity/privacy via end-to-end encryption because the internet is a hostile environment full of baddies of all kinds, not just crappy middleboxen (e.g. a whole alphabet soup of state agencies who absolutely do not share my priorities with regard to my internet traffic(!))

So you think you understand IP fragmentation?

fest3er — Thu, 22 Feb 2024 06:50:26 +0000

Grok! I used to think that blocking all ICMP was A Good Thing™ until I read and understood that IPv4 cannot work without ICMP. Then I modified Smoothwall Express' iptables to accept ICMP messages while retaining the option to tell the kernel to block or allow PING from the internet.

So you think you understand IP fragmentation?

da4089 — Mon, 19 Feb 2024 10:32:20 +0000

It's great to see another article from Valerie in LWN!
I particularly remember the union mounts article/work, and it's hard to believe that it was ~14 (!) years ago.

So you think you understand IP fragmentation?

farnz — Fri, 09 Feb 2024 17:58:08 +0000

By 1990, it was already the case IME that communication was not possible if there were smaller MTUs in the path, unless you were lucky enough to have a path where everything was run by sensible netadmins (usually true of academia), or you were on dial-up (where you had the bottleneck MTU).

And one of the many issues back then was routers with multi-MTU paths that were configured explicitly to not fragment packets because it could overload the CPU; packets were either pre-fragmented, or were dropped. Add in people configuring routers to drop fragments "because security" (which got worse after the ping of death vulnerability was discovered, since that depended on buggy fragment handling), and fragmentation became useless.

The IETF, by limiting fragmentation to the endpoints, were reacting to the state of play in 1990, where many routers already didn't fragment, but dropped packets that were too big.

So you think you understand IP fragmentation?

farnz — Fri, 09 Feb 2024 17:40:23 +0000

That's the MAC MTU; the PHY MTU can be as large as 2,097,148 bytes in 802.11ac networks (noting that the PHY MTU depends both on static parameters like channel width, but also dynamic parameters like time it will take to transmit the frame). For 802.11ax, the PHY MTU is permitted to go as high as 6,500,631 bytes. Even as early as 802.11n (in 2009), the PHY MTU was allowed to go as large as 65,536 bytes under good conditions.

This is made useful with a much smaller MAC MTU by having aggregation options, so that a single PHY frame contains many MAC frames; the downside is that there is overhead for each and every MAC frame in the PHY frame, which would go down if the MAC frames were larger. There would still be overhead mapping MAC frames into PHY frames, so you wouldn't have as large a MAC MTU as the PHY MTUs, but there would be large MTUs involved.

So you think you understand IP fragmentation?

paulj — Fri, 09 Feb 2024 17:40:00 +0000

And that's not to say we should go back to data-plane fragmentation. Impossible, and it might not be technically the best solution anyway. But the current situation is a mess and unfortunate.

So you think you understand IP fragmentation?

paulj — Fri, 09 Feb 2024 17:39:01 +0000

Well, fragmentation being slow is not a problem. It's expected that frags will be slow-path - but at least communication is still possible. Piece-meal upgrades of the common MTU are at least /possible/ then.

Slow but working beats the mess we have today: We will never be able to default to >1500 MTUs, and even then we still don't have reliable networking (VPNs, etc.), and because of that the awesome networking tool of encapsulation is restricted in utility.

So you think you understand IP fragmentation?

farnz — Fri, 09 Feb 2024 17:34:01 +0000

My experience was that they were always present, and become more and more of an issue throughout the 90s, until they were basically making fragmentation unusable unless the path was either between two academic institutions or between my ISP at the time and an academic institution.

Additionally, long before middleboxes became widespread, the dataplane support already sucked; there were plenty of Cisco routers that could do forwarding in hardware, but did fragmentation in software on a slow path. Not a problem from home, where my modem was the bottleneck, but a very noticeable issue when at an academic institution where the "wrong" MTU could bring speeds down from megabits per second to tens of kilobits per second.

So you think you understand IP fragmentation?

Cyberax — Fri, 09 Feb 2024 17:22:51 +0000

Ha. My AP synthesizes ICMP Too Big packets. Layering violation, you say?

I have 9k MTU within my home network (and I get 10500 megabits over 10GB connections), and so far my WiFi has been behaving pretty well. I can get 1.5GBps download and 2.5GBps upload.

So you think you understand IP fragmentation?

paulj — Fri, 09 Feb 2024 17:21:00 +0000

Well, yeah, firewalls always suck. They were still uncommon even in the 90s though when I was first online IVR - i don't know about the 80s. The idiotic middle-boxes only seemed to really become common in the v late 90s and 00s, with the Internet going mainstream. Again, my vague, hand-wavy, recollection.

The data-plane level support was fine though, until IETF moved to deprecate, and then vendors of course did.

So you think you understand IP fragmentation?

farnz — Fri, 09 Feb 2024 17:12:58 +0000

I last saw fragmentation being reliable in the 1980s; my experience ever since then with IPv4 is that it's hugely unreliable, because all sorts of entities use middleboxes that drop all fragments (rather than forwarding them, or reassembling then forwarding), and thus it's basically useless.

So you think you understand IP fragmentation?

pizza — Fri, 09 Feb 2024 17:03:01 +0000

> Also, how do APs handle jumbo frames? I need to do some experiments...

Based on my (admittedly not recent), they don't handle it well. Indeed, despite wifi nominally supporting 2300ish byte MTUs, APs routinely fail with anything over 1500 bytes. Because reasons.

(That is is the main reason why I reverted back to 1500 byte MTUs on my home networks....)

So you think you understand IP fragmentation?

Cyberax — Fri, 09 Feb 2024 17:00:10 +0000

> Nearly all equipment has supported at least 9k MTU for a long while

Except for WiFi. Its PHY MTU is just 2300 bytes.

So you think you understand IP fragmentation?

Cyberax — Fri, 09 Feb 2024 16:11:58 +0000

> But this needs not just Cyberax's idea, but also a change from switching Ethernet frames to routing IP packets everywhere, so that the WiFi AP is expected to know about per-device MTUs.

Not necessarily. There are two ways this can be done without significant changes:

1. APs can just update the "forward MTU" field in IP packets, they don't need to be full routers for this. Yeah, it's a layering violation, but I doubt that people care too much about that sort of thing anymore.

2. MTU can be added to ARP/ND directly. So the sender will discover the L2 MTU of the destination when it does the initial L2 discovery. WiFi APs are responsible for ARP/ND already, so it even fits in well into the "proper" layered model.

Also, how do APs handle jumbo frames? I need to do some experiments...

So you think you understand IP fragmentation?

paulj — Fri, 09 Feb 2024 16:06:41 +0000

Fragmentation was reliable actually. Until we deprecated it and encouraged vendors to degrade/remove support. Way more reliable than what was meant to supercede it at least.

So you think you understand IP fragmentation?

paulj — Fri, 09 Feb 2024 16:05:04 +0000

Nearly all equipment has supported at least 9k MTU for a long while. The problem now is the network protocols we use have no reliable way to figure out how to use larger MTUs.

See my blog link in another comment in this article. It has quotes from an early paper on TCPIP from Kahn and Cerf explaining why it is important to have a reliable network mechanism to allow different MTU networks to inter-op. Unfortunately, we - collectively - failed to heed their wise words.

A reliable mechanism needs to be in-band. E.g., data-plane fragmentation. Side-band end-host solutions - i.e., relying on ICMP messages - have proven to be fragile. Pure end-host probing (i.e. Path-MTU Discovery, in protocol or out) is also inefficient, temporally unreliable, and fragile.

So you think you understand IP fragmentation?

Sesse — Fri, 09 Feb 2024 14:22:32 +0000

I guess a combination of hardware limitations and the specs not being clear about how much of the packet to return. In any case, it's just a fact of life now.

So you think you understand IP fragmentation?

farnz — Fri, 09 Feb 2024 14:20:55 +0000

Even IPv6 jumbograms have a length field, in the Jumbo Payload hop-by-hop header. This is a 32-bit number instead of a 16 bit number, but if you have the full IPv6 header, you'll get a length field to work with (either 16 bits, if not jumbo-sized, or 32 if jumbo-sized).

So you think you understand IP fragmentation?

DemiMarie — Fri, 09 Feb 2024 14:13:02 +0000

Is that an OS/router bug?

So you think you understand IP fragmentation?

smurf — Fri, 09 Feb 2024 13:52:02 +0000

No, the checksum will not always fail. The checksum protects against one-bit changes and similar shenanigans. It's not safe for truncation. One in 65536 truncated packets will pass the checksum test. That's way too much for a protocol that's supposed to be reliable.

However, the UDP header also contains … surprise … a length. As long as you don't send IPv6 jumbograms (length word: zero) you're thus still safe there.

So you think you understand IP fragmentation?

paulj — Fri, 09 Feb 2024 12:28:30 +0000

Yep. Mentioned that also in the blog post. This used to be a terrible problem some time ago (??? 15+ years ago?). We had a number of firewall vendors who had "block all ICMP" as their default policies - either cause the vendor was institutionally stupid, or because their customers were and demanded it (possibly facilitated by dumb tech industry reviewers?). It /seems/ to be a /little/ better now - i.e. firewalls have better defaults now.

However, you still have dumb IT people who buy firewalls and then click at stuff without really knowing what they're doing. "That option to block all ICMP, that must be more secure!", and some perverse security idiots too.

A lesson here for protocol designers however has to be that side-channel / out-of-band control/error channels are not a good idea.

So you think you understand IP fragmentation?

farnz — Fri, 09 Feb 2024 12:22:04 +0000

We have a chicken-and-egg situation; the equipment is not configured to support larger packets because path MTU detection is not reliable, fragmentation is not reliable, and the failure case if both of those don't work is random failures of higher level protocols like TCP. This means that there is no reason for anyone to support a larger MTU on Internet-facing equipment, since you're likely to have issues where a required path between two points drops large MTU packets.

In addition, if your MTU is too large, you will frequently experience an RTT delay where something on the path sends an ICMP Too Big your way, and you have to reduce the detected path MTU; in Cyberax's proposal, you can determine the current path MTU with small packets (like those used to establish a TCP connection), and thus not pay that penalty unless the path is changing from larger MTU to smaller MTU during the lifetime of a single connection.

And it's worth noting that we already have examples of devices where there's a large MTU at PHY level, and we aggregate MAC level packets to fill a single PHY packet; it's called WiFi. Having a good way to handle variable MTU (which would include WiFi APs being able to change the path MTU on you, because the client has moved) would reduce overheads. But this needs not just Cyberax's idea, but also a change from switching Ethernet frames to routing IP packets everywhere, so that the WiFi AP is expected to know about per-device MTUs.

So you think you understand IP fragmentation?

intelfx — Fri, 09 Feb 2024 11:50:08 +0000

> We no longer would be limited by just 1500 bytes

I thought that we are limited by 1500 bytes because the Internet equipment does not support / is not configured to support larger packets. Even if you implement perfect discovery, that won't magically fix the equipment, no?

So you think you understand IP fragmentation?

LtWorf — Fri, 09 Feb 2024 11:31:38 +0000

Python is generally easier I guess for this sort of things. It is installed on all linux systems I presume.

It should be possible to do using just the standard library too.

And it has setsockopt and all of that, so it should work. If it isn't enough you can just do libc calls easily.

But personally I would not rewrite it.

So you think you understand IP fragmentation?

paulj — Fri, 09 Feb 2024 11:09:30 +0000

Some of the most popular L3 switching/routing ASICs by default now use the flow-label for IPv6 ECMP flow grouping, *not* the (address,port) 4-tuple (which is the flow-ID for v4). E.g. more modern Broadcom Tridents. Anything that cares about ECMP is going to have to set the flow-label. I'm sure popular TCP stacks must do already - given ASICs using it by default. I can't double-check or find an authoritative reference right now, but I'm pretty sure Linux assigns a random flow-label to a TCP connection if the app hasn't set one. So apps generally don't need to care - the kernel has it covered.

So you think you understand IP fragmentation?

paulj — Fri, 09 Feb 2024 11:01:44 +0000

Ah, ok. That isn't quite clear from the article. :) E.g., the bit that says:

> "They were confident that the software already only sent packets that couldn't be fragmented, so all we had to do is send the right size of probe packets, using a built-in ping feature, and record the response. Imagine our surprise when the packet captures turned out to be full of fragmented packets."

The reader here can reasonably conclude from "all we had to is send the right size of probe" that we are talking about software under your control to do probing. And "surprise when the packet captures turned out to be full of [frags]" suggests surprise at seeing frags, which would imply the software /had/ set DF - for otherwise frags on large packets are not that surprising.

Perhaps a bit more word-smithing there? Something like "Surprise at seeing [frags], even where [condition where an expert wouldn't always predict frags]"? Or some other change there?

Great seeing detailed networking articles on LWN, and thanks for the tool! More please. :)

So you think you understand IP fragmentation?

vaurora — Fri, 09 Feb 2024 10:34:43 +0000

I appreciate you spelling out your reasoning and experience with ICMP ping-based PMTUD. We came to similar conclusions. Doing DPLPMTUD inside an encrypted VPN protocol is far easier than using anything ICMP-based: intermediate routers can’t inspect and filter, we can send pad data, and we can’t get spoofed. But I am absolutely going to be using your script to double check my future work. :)

So you think you understand IP fragmentation?

Sesse — Fri, 09 Feb 2024 08:02:40 +0000

Most OSes and/or routers don't include enough of the original packet in the ICMP response to do that, unfortunately. So you don't know which backend to send the ICMP packet to, and you don't really want to flood it to all of them.

So you think you understand IP fragmentation?

DemiMarie — Fri, 09 Feb 2024 05:58:38 +0000

Is this because the load balancer doesn’t look at the part of the packet that was included in the ICMP response?

So you think you understand IP fragmentation?

shemminger — Fri, 09 Feb 2024 03:51:44 +0000

The big problem I have seen is that many environments set routers to block/drop all ICMP packets.
That includes Echo (ping), Time Exceeded, and fragmentation.
It makes dealing with thise kind of black holes hard.

So you think you understand IP fragmentation?

auerswal — Thu, 08 Feb 2024 22:07:49 +0000

Regarding not sending all probes in parallel in my script:

On the one hand, ICMP Echo Responses are often rate limited. Sending all probes in a burst thus likely results in some missing responses because of the rate limit, not because of requests dropped due to too small PMTU. At least that was my experience back when I started writing the script. ;-)

If the largest probes in a "plateau search" are sent first, the first arriving probe (so not yet rate limited) has a good probability of being the largest probe fitting inside the PMTU. But even that may not be true if probes take different paths.

On the other hand, I do remember times when there was little bandwidth available and do not want to send one big burst as fast as possible. Some form of packet pacing is just more considerate with respect to other users of the network (perhaps I have too often encountered situations where adding packet pacing via some network device configuration shenanigans "solved" "network" problems of applications…). I am also comfortable waiting a bit for my manually triggered PMTUD to finish, which may not be the case for some VPN product user who might complain over long startup times.

Sending (1280+1400+1500=4180) bytes in one burst is still less than the TCP IW10 burst without pacing. (1280+1400+1500+8000+9000=21180) is even over the TCP IW10 maximum initial burst, but hopefully the larger packets would not get far, e.g., stay inside a data center. Together with a cooperating target (the other side of a VPN tunnel) this seems OK. (Using a 1280 bytes floor is a sign of the times, with more and more IPv6 all around. :-) )

I wrote my PMTUD script to work around a VPN missing a reliable built-in PMTUD mechanism. Therefore I use ICMP Echo although both rate and packet size limits are commonly encountered over the Internet. I do not need a special responder, just a server I want to reach via VPN that answers pings without size limit. As such I really like that you added PMTUD to the VPN product, and also put some thought into it. :-)

So you think you understand IP fragmentation?

Cyberax — Thu, 08 Feb 2024 21:33:49 +0000

I'm honestly not familiar with that. I know that a lot of use-cases were proposed throughout the years (QoS, routing hints, etc.) but I'm not aware of any real uptake. Load balancers can't depend on it, because it's controlled by clients.

So you think you understand IP fragmentation?

vaurora — Thu, 08 Feb 2024 18:48:29 +0000

Great shell script implementation of PMTUD! (That is not a sentence I was expecting to write.)

I do agree that the concept of "probe this pre-defined table of MTU sizes" is obvious, which is why it is in the RFC. I just haven't yet seen a PMTUD search algorithm that sends all the probes simultaneously, which teeeeeeechnically your script doesn't seem to do either. ;) I'm still excited to see the implementation.

I'm not sure why sending all the probes simultaneously is not commonly done. My theory is that people get stuck in a stream-oriented TCP-style mental model when they design PMTUD stuff. One of my colleagues thinks it's a holdover from when long-distance network links were much lower bandwidth.

So you think you understand IP fragmentation?

vaurora — Thu, 08 Feb 2024 18:29:20 +0000

I see! No, everyone I talked to *believed* that the application was already setting the DF bit on all the packets, and I believed them. It wasn't until the PMTU discovery code failed that I did the packet dump and realized the application was not setting DF.

My takeaway from that was that multiple network experts were writing code assuming that the packets weren't fragmentable when they actually were, and no one noticed for years because things just worked anyway.

The solution on slide 9 is for fragquiz, which is a packet generator, not an implementation of DPLPMTUD (which I pronounce "dpblublubblubbbbd").

So you think you understand IP fragmentation?

vaurora — Thu, 08 Feb 2024 18:15:30 +0000

The "hard way" refers to how I found out that (a) I had left my VPN on, (b) my VPN did not send Time Exceeded. I'm not fond of the "I didn't change anything, so how did I break it???" feeling.

However, given that the purpose of this article is to demonstrate that people often don't understand networking as well as they think, including me, it's a reasonable conclusion! My main takeaway is that all kinds of networking oddities are happening in the background and I'm blissfully unaware of them unless I write code to check it.

So you think you understand IP fragmentation?

paulj — Thu, 08 Feb 2024 18:06:16 +0000

Well, you sent >path-MTU packets and observed fragments, and this was surprising, right? Which means you must have had DF set on the packets - as is common, particularly if probing for pMTU? And your slide deck on the fragquiz, on slide 9 describing the solution says "Set the don't fragment socket option" - which makes sense, given what you're trying to measure (?).

And you observed fragments, despite having set DF, right? Which would be surprising, naturally.

Otherwise, you did /not/ set DF, sent various size packets and observed fragments - which is... not surprising... ?

I'm confused now. ;)

So you think you understand IP fragmentation?

vaurora — Thu, 08 Feb 2024 17:52:53 +0000

> If routers *are* fragmenting packets that have the DF bit set that is... wow. WTF? That's a pretty serious bug. Has the author been able to figure out which vendors' products are doing this?

Sounds like I've written something unclear since I haven't observed that! If you let me know which part sounds like that is happening, I'll try to reword.

So you think you understand IP fragmentation?

auerswal — Thu, 08 Feb 2024 17:51:47 +0000

Thanks for an interesting article!

I plan to look at the "fragquiz" program and might even try it, when it is released under some free software license and no longer requires root (on GNU/Linux).

> in the real world there are a small number of likely packet sizes

This was also mentioned in RFC 1191[1] from 1990. The actual values obviously change through time. The 1492 bytes entry from the RFC 1191 plateau table is still relevant today for some consumer-grade Internet connections.

> What we did differently is this: we sent ALL of the possible packet sizes at the same time.

I would have called this "obvious" when testing a small number of packet sizes (thus I used this idea in 2018 in a little shell script[2] for personal use, modulo a rate limit to increase the chances of obtaining useful results). ;-)

Br,
Erik

[1] https://www.rfc-editor.org/rfc/rfc1191.html
[2] https://github.com/auerswal/sft/blob/master/pmtud

So you think you understand IP fragmentation?

vaurora — Thu, 08 Feb 2024 17:42:32 +0000

Just a pedantic note for other readers that IPv6 does support fragmentation - at the source only. This is to avoid breaking applications that, for example, send UDP packets that are larger than even the local MTU and expect them to be fragmented by the OS.

Preventing fragmentation in IPv6 is harder than you think if you have any form of encapsulation going on. Sure, you can rely on a 1280 byte minimum MTU, but if you're tunneling and adding another few bytes to every packet, your applications still expect to be able to send a 1280 byte packet without fragmentation. But your encapsulated packet is now 1288 bytes (or 1320 or whatever).

So you think you understand IP fragmentation?

vaurora — Thu, 08 Feb 2024 17:36:12 +0000

Thanks for the bug reports! Fixed.

Do you have suggestions for what language you would like to see fragquiz written in? Annoyingly I need to set some pretty low-level OS-specific socket options so that needs to be possible.