Bottomley: Creating a home IPv6 network

Posted Sep 20, 2020 19:36 UTC (Sun) by jem (subscriber, #24231)
In reply to: Bottomley: Creating a home IPv6 network by Cyberax
Parent article: Bottomley: Creating a home IPv6 network

>It's the same dumb design, but made much worse.

You keep saying this, but provide no explanation to why the IPv6 design is much worse. Or why it is a dumb design in the first place, and what would be a better design.

>IPv6 allows mixed-sized networks on the same link. With various interesting interactions. Nobody uses that, but it's supported.

Everybody uses that. Link-local is fe80::/10. What are the "interesting interactions"?

>And this makes simple discovery protocols to become much more complicated and involved, even though they use the same broadcast in the end (Ethernet has no native multicasting!).

Again, you do not substantiate how the protocols become much more complicated and involved. This smells like FUD. And Ethernet does have native multicast. Unless you have a shitty network card, the card does its best to filter out multicast packets the host has not subscribed to, at the card level.

>You have to do all the same subneting stuff for IPv6, even on the beginner level (to explain how prefixes work).

All a beginner needs to know is the 64+64 split. My point was that learning to administer IPv4 networking is much more difficult.

>It's really not. IPv6 turned out to be inferior to IPv4 in operational issues,

Here we go again...

> and it doesn't provide anything terribly important for users.

How about a working Internet for all current and future users? The net keeps growing and is quickly outgrowing itself. There is a limit on how much IPv4 NAT can help alleviate the problems, and looking at how much IPv6 has been deployed worldwide already, one can ask whether the Internet at the current scale would even be possible without it.

Bottomley: Creating a home IPv6 network

Posted Sep 20, 2020 20:03 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link] (10 responses)

> You keep saying this, but provide no explanation to why the IPv6 design is much worse. Or why it is a dumb design in the first place, and what would be a better design.
OK. In IPv4 to get to the TCP ports info you just need to read the IPv4 header length, walk that many words and that's it. There's no flexibility.

In IPv6 you have to read the next header, jump to that header, parse it, get the next next header. Possibly many times, as there are no recursion limits.

> Everybody uses that. Link-local is fe80::/10. What are the "interesting interactions"?
This is an implementation detail (btw, link-local addresses are yet another complication that is absent in IPv4). Consider this situation: you have /96 and /120 subnets on one link. How would hosts talk to each other?

Can you honestly tell without looking into Wiki how on-link cache works? Can you honestly tell that it's easier or better than a fixed subnet in IPv4? Do you know what tools can be used to examine that cache? What about negative cache entries? What happens if they overflow?

If you still think you know how it works, here's another question: why does a host _have_ to listen to RA messages if it's using DHCPv6? And what would it do with to its battery life?

I personally implemented an IPv6 stack for a low-RAM battery-powered device so I know for sure that it's WAY more complicated than IPv4. Doubly so if you want to make it robust and efficient.

> All a beginner needs to know is the 64+64 split. My point was that learning to administer IPv4 networking is much more difficult.
Incorrect. A beginner will need to know about PD at the very least, including subnetting within the network /64 part. It's all the same knowledge as required for IPv4.

> How about a working Internet for all current and future users? The net keeps growing and is quickly outgrowing itself. There is a limit on how much IPv4 NAT can help alleviate the problems
NAT effectively multiplies the IPv4 address space by about 100x. So there is no practical limit on the number of IPv4 users. All it does is breaking end-to-end connectivity, but this is not a big problem anymore.

Don't get me wrong, I think that IPv6 is needed to avoid at least some of the bogosity of CGNAT. But it doesn't change my opinion that IPv6 is a pig that needs a lot of thrust to fly.

Bottomley: Creating a home IPv6 network

Posted Sep 21, 2020 5:54 UTC (Mon) by jem (subscriber, #24231) [Link] (9 responses)

>OK. In IPv4 to get to the TCP ports info you just need to read the IPv4 header length, walk that many words and that's it. There's no flexibility.

Ok, I get your point. But then the TCP headers are not at a fixed address after all, like you said earlier. Your ASIC will have to read the header length and add it to the start of the packet. Then you will also have to read the Type header field to check that the payload is TCP. Or at least the hardware has to check that both fields contain the common values (no options, type TCP).

>In IPv6 you have to read the next header, jump to that header, parse it, get the next next header. Possibly many times, as there are no recursion limits.

The fast path is: check the Next Header to see if it contains the value 6 (TCP). If it does, you'll find the TCP header at a true constant offset. If the packet contains extension headers i guess you will probably have to do some extra processing anyway.

>NAT effectively multiplies the IPv4 address space by about 100x. So there is no practical limit on the number of IPv4 users. All it does is breaking end-to-end connectivity, but this is not a big problem anymore.

Even if you can multiply the address space by 100, are you sure NAT overall can scale that far? T-Mobile opted for IPv6-only in the handset, combined with 464XLAT. They are now on a path to a future that is only getting better as more services are available directly over IPv6, like Google (including YouTube), Facebook, Netflix, etc.

And why is breaking end-to-end connectivity not a problem anymore? The demand for public addresses has gone up because everybody wants to move to the cloud. RFC 1918 addresses can only go that far, which made Facebook switch to IPv6 for their internal network.

Bottomley: Creating a home IPv6 network

Posted Sep 21, 2020 6:16 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (8 responses)

> Your ASIC will have to read the header length and add it to the start of the packet. Then you will also have to read the Type header field to check that the payload is TCP. Or at least the hardware has to check that both fields contain the common values (no options, type TCP).
Yup. Modern high-throughput load ballancers actually do that while the packet is being received.

> The fast path is: check the Next Header to see if it contains the value 6 (TCP).
There are no fast paths in core routers. That's a very hard-learned lesson - anything on the "slow path" just becomes a DDoS attack vector or is useless.

> Even if you can multiply the address space by 100, are you sure NAT overall can scale that far?
Yep. So far it's scaling just fine. Now that all major ISPs have CGNAT hardware, the push to move to IPv6 is basically gone.

> T-Mobile opted for IPv6-only in the handset, combined with 464XLAT. They are now on a path to a future that is only getting better as more services are available directly over IPv6, like Google (including YouTube), Facebook, Netflix, etc.
You might note that T-Mobile hasn't bothered pushing IPv6 to their hotspots. In the end, 464 XLAT doesn't save much in hardware cost, it just allows to move IPv4 to the edges of the network and move the core onto IPv6. It pays off in case of humongous networks like T-Mobile that simply don't have enough private IPv4 space for their _internal_ addressing.

> And why is breaking end-to-end connectivity not a problem anymore? The demand for public addresses has gone up because everybody wants to move to the cloud. RFC 1918 addresses can only go that far, which made Facebook switch to IPv6 for their internal network.
Cloud is indeed a major consumer of IPv4. It's more than 10% of the total IPv4 address space by some measures. However it can be compressed quite a lot, a typical cloud application doesn't actually _need_ IPv4 if it uses cloud-provided load ballancers (that can use SNI). So I personally don't foresee huge amount of pressure from that side.

Mind you, I would love to switch everything to IPv6. I make sure that all my applications support it. I just don't see it happening any time soon (i.e. the next 10 years).

While a lot of this is due to inertia and probably would have happened with any successor of IPv4, the general awfulness of IPv6 certainly doesn't help a bit.

Bottomley: Creating a home IPv6 network

Posted Sep 21, 2020 9:44 UTC (Mon) by farnz (subscriber, #17727) [Link] (7 responses)

A core router should not be looking at the packet contents anyway, so would not be looking at the Next Header/Type field to begin with. In a load balancer setup, you'd do what you do in IPv4 and drop all packets with extension headers/options, because they're not relevant - IPv6 did at least make a lot of effort to ensure that "drop all packets with extension headers" is a reasonable default behaviour for all devices.

And 464XLAT does save quite a bit in hardware cost on modern fast networks; it moves some heavy data consumers (Netflix, YouTube) into stateless routing instead of NAT. Hotspot from my phone on EE uses IPv6 as a result to save hardware costs at the network end; they've not bothered doing separate hotspots because the total usage of hotspot devices (as opposed to phones and hotspots from phones) is so low that the hardware savings are outweighed by support costs.

I do agree that IPv6 suffers from second-system syndrome; however, it also suffers from being designed before the dot-com boom, and failed to take three important changes (that happened after it was designed!) into account (ordered below from least important to most important):

The rise of switched Ethernet as the one true link layer for almost everything; with the exception of LTE and 5G phones, everything a normal person uses that has an IP stack is running that stack over an Ethernet link layer, rather than the variety of link layers in use in 1995 (bear in mind that IPv6 was designed concurrently with 100M Ethernet at a point when competing link layers like FDDI were already doing 100M links, and SDH was providing 150M payload links with scalability planned to 10G and beyond). Heck, in the 1990s, when IPv6 was designed, people were happily doing things like routing from an FDDI server network to Ethernet (hubs or 10BASE2/10BASE5) for the campus network; Ethernet switches were still newfangled tech, and if you wanted multiple broadcast domains, you used Ethernet bridges to interconnect hubs. It would not have seemed unreasonable to believe that once IPv6 was established, we'd switch to a link layer designed around efficient IPv6 carriage.
The end of NSFNET (which happened in parallel to IPv6 design) would mean that switching IP protocol version was no longer a carefully orchestrated fast action. We did the transition to IPv4 in a year by setting deadlines and having a flag day after which pre-IPv4 packets would not be routed. IPv6 has not had this advantage, as commercial providers have never had a reason to switch off IPv4.
Hardware routing that neither parses the entire header nor looks at just the addresses; the IPv4 Router Alert option didn't exist when IPv6 was being designed, since routing didn't have a hardware fast path that just looked at addresses. This one is less forgivable; it's clear that hardware routing was in development at the time, and IPv6 has several features (no fragmentation at the router, flow labels, hop-by-hop options must be the first options if present) which show that the IPv6 designers were aware of the needs of hardware routing, and they could easily have done more to ensure that IPv6 packet parsing in hardware was fast even if it looked at more than just addresses.

Bottomley: Creating a home IPv6 network

Posted Sep 21, 2020 16:35 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

> In a load balancer setup, you'd do what you do in IPv4 and drop all packets with extension headers/options, because they're not relevant - IPv6 did at least make a lot of effort to ensure that "drop all packets with extension headers" is a reasonable default behaviour for all devices.
Why is it reasonable?

> And 464XLAT does save quite a bit in hardware cost on modern fast networks; it moves some heavy data consumers (Netflix, YouTube) into stateless routing instead of NAT.
Why? 464XLAT requires just as much NAT state on the outer edge of the network as anything else, you need it to map 4-tuples onto IPv6 addresses.

> Hotspot from my phone on EE uses IPv6 as a result to save hardware costs at the network end; they've not bothered doing separate hotspots because the total usage of hotspot devices (as opposed to phones and hotspots from phones) is so low that the hardware savings are outweighed by support costs.
There are no large mobile operators in the US that do IPv6 for phone-based hotspots, I should have explained more carefully.

Bottomley: Creating a home IPv6 network

Posted Sep 21, 2020 17:14 UTC (Mon) by farnz (subscriber, #17727) [Link] (5 responses)

> > In a load balancer setup, you'd do what you do in IPv4 and drop all packets with extension headers/options, because they're not relevant - IPv6 did at least make a lot of effort to ensure that "drop all packets with extension headers" is a reasonable default behaviour for all devices.
> Why is it reasonable?
It's reasonable because extension headers are not normally used - their presence implies something unusual going on, just as IP options do in IPv4.

> > And 464XLAT does save quite a bit in hardware cost on modern fast networks; it moves some heavy data consumers (Netflix, YouTube) into stateless routing instead of NAT.
> Why? 464XLAT requires just as much NAT state on the outer edge of the network as anything else, you need it to map 4-tuples onto IPv6 addresses.
Because those heavy data consumers are IPv6-enabled already, so they require zero NAT state, just stateless IPv6 routing. You only need NAT state for things that go to IPv4, not for IPv6 enabled services.

> > Hotspot from my phone on EE uses IPv6 as a result to save hardware costs at the network end; they've not bothered doing separate hotspots because the total usage of hotspot devices (as opposed to phones and hotspots from phones) is so low that the hardware savings are outweighed by support costs.
> There are no large mobile operators in the US that do IPv6 for phone-based hotspots, I should have explained more carefully.
I'm surprised - my contacts at EE tell me that they've cloned their setup from their colleagues at T-Mobile USA; they haven't yet been able to IPv6 enable iPhone hotspots because of issues relating to routing from WiFi/USB to cell, but it's in progress with Apple helping them work out what's going wrong and where, so that it can be fixed.

Bottomley: Creating a home IPv6 network

Posted Sep 21, 2020 17:40 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

> It's reasonable because extension headers are not normally used - their presence implies something unusual going on, just as IP options do in IPv4.
I don't think IPv6 designers thought that they are optional, the fragmentation header definitely was designed to be used. It's basically an emergent behavior because people have been ignoring extension headers.

> Because those heavy data consumers are IPv6-enabled already, so they require zero NAT state, just stateless IPv6 routing. You only need NAT state for things that go to IPv4, not for IPv6 enabled services.
That helps, but not as much as you'd think. A single CGNAT box can sustain around 500 million connections at around 300GBps ( https://www.a10networks.com/wp-content/uploads/A10-DS-Thu... ). This is enough for about 10000 customers at around $10 per customer one-time investment.

Once it's installed, the financial pressure to move to IPv6 is basically gone.

The main advantage for T-Mobile was improving the internal network management, because they were running out of private (and squatted) IPv4 space. But this is not an issue for smaller ISPs.

> I'm surprised - my contacts at EE tell me that they've cloned their setup from their colleagues at T-Mobile USA; they haven't yet been able to IPv6 enable iPhone hotspots because of issues relating to routing from WiFi/USB to cell, but it's in progress with Apple helping them work out what's going wrong and where, so that it can be fixed.
You CAN get IPv6 for phones, but hotspot traffic goes through NAT on the phone device (and then through 464 XLAT). Apparently, adding another IPv6 circuit to each phone will double the price for ISPs.

Bottomley: Creating a home IPv6 network

Posted Sep 21, 2020 18:14 UTC (Mon) by farnz (subscriber, #17727) [Link] (3 responses)

Sure, but emergent behaviour is what we see in IPv4 as well - e.g. the lack of use of IPv4 options is for the same reasons, as they were definitely meant to be used, too. And the IPv6 designers did do a reasonable amount of work to ensure that routers etc didn't need to worry about extension headers - e.g. fragmentation is an endpoint thing only, not a router thing. While I don't think they did enough, it's clear that they were thinking in terms of avoiding the need for routers to comprehend more than the fixed header.

Also, while that's $20 (because you need a HA pair for mobile network use) per customer, the box is still 10x more expensive than a plain IPv6 router that has no connection count limit, and a much higher throughput; when you can reduce the number of CGNAT boxes by 50% because the high volume traffic is gone, you save a lot of money. Maybe not on a small network where 10,000 is a lot of customers, but it's a big saving on larger networks.

And on my Google Pixel 2, hotspot traffic doesn't go through NAT on the phone device for IPv6 - there's a /64 assigned to my handset, it takes one address (which it uses on the cell interface), and the rest of the /64 is used for other devices on the hotspot. This is still a single IPv6 circuit so doesn't require its own EPS bearer (which is what doubles the costs), but it means I have real IPv6 while tethered, and 464XLAT on the phone (using NAT) for IPv4.

Bottomley: Creating a home IPv6 network

Posted Sep 21, 2020 19:32 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> Also, while that's $20 (because you need a HA pair for mobile network use) per customer
$10 includes HA. And it actually gets cheaper as you scale up, because you only need to keep a couple of spare units to pick up the load if one unit fails. Kinda like RAID-5 vs RAID-1.

> the box is still 10x more expensive than a plain IPv6 router that has no connection count limit, and a much higher throughput; when you can reduce the number of CGNAT boxes by 50% because the high volume traffic is gone, you save a lot of money. Maybe not on a small network where 10,000 is a lot of customers, but it's a big saving on larger networks.
The thing is, you still need CGNAT anyway because a large part of the Internet is not accessible by IPv6. And it has to be sized to handle the worst case - lots of clients going to an IPv4-only site and watching 4K videos. There are still plenty of them: Amazon Prime Video, Twitch, TikTok, Disney+, etc.

This really is a painful problem and right now IPv6 is a net financial loss, not gain. This is reflected in world-wide IPv6 deployment patterns, with poor countries not getting it (they can't afford additional operational expense).

> And on my Google Pixel 2, hotspot traffic doesn't go through NAT on the phone device for IPv6 - there's a /64 assigned to my handset, it takes one address (which it uses on the cell interface), and the rest of the /64 is used for other devices on the hotspot.
That's nice! I've tried that with my Pixel 4 and I'm not getting IPv6 from either T-Mobile or AT&T.

Bottomley: Creating a home IPv6 network

Posted Sep 22, 2020 10:32 UTC (Tue) by farnz (subscriber, #17727) [Link] (1 responses)

So, having had a chance to chat to my network contact, he broadly agrees with you on IPv6 on the Internet side, but says that the real driver of IPv6 and 464XLAT is not Internet data - it's VoLTE. For poorer countries, he suggests that the newer network kit that's just entering the second-hand market is going to drive IPv6 on mobile there.

Firstly, VoLTE is not reliable if the VoLTE network is IPv4 only - it has to be an IPv6 network using globally unique addresses (i.e. from your RIR assignment) to work reliably. Reliable VoLTE roaming has driven up call durations on charged roaming destinations (thus more revenue for those operators). However, this implies that you're running IPv6 network-wide for VoLTE; as the marketing blurb for your CGNAT box points out, modern CGNAT boxes are as good at NAT64 as they are at NAT64, so you can reduce your operational costs by running an IPv6-only network, rather than having an IPv4 network for data traffic and an IPv6 network for voice.

Secondly, they're now seeing cheap kit on the market that, instead of having an ATM interface (whether real, or emulated over Ethernet) for 2G/3G voice, handles 2G/3G voice by translating it to IMS voice (using IPv6) over the Ethernet it already has for 4G. So, if you set up an IPv6 network for VoLTE, you get 2G/3G voice moved off legacy ATM onto your VoLTE network.

Put those two together, and you've got a powerful opex reduction case for a poor country - when you add 4G and VoLTE for whatever reason (whether to take advantage of the cheap handsets built for Reliance Jio, or because it gives you a marketing point, or just to pick up some sweet roaming revenue), you can switch to an all-IPv6 Ethernet backbone, remove a lot of power-hungry legacy kit relating to old 2G and ATM networks, and then do IPv6 and 464XLAT because it's cheaper than maintaining an IPv6 network *and* an IPv4 network.

From EE's point of view (rich country), the reason to do IPv6 on the cellular network was two-fold:

You have to run a reliable IPv6 network between your IMS servers and the handsets anyway for VoLTE. You might as well run user data traffic on that network, too, using QoS to get it out the way of voice, and giving you extra load to make your IPv6 network metrics clearer (since there's traffic on the IPv6 routing when there's no voice in use on a cell, which means that a problem path is visible now, not when a voice call happens).
A significant fraction of user traffic (25% today, apparently) now bypasses the CGNAT (which is in a central location) and goes on local break-out; this means both stateless routing for EE, and a way for companies who want to control the user experience to bypass large swathes of EE's backhaul network by taking traffic over in more locations.

Bottomley: Creating a home IPv6 network

Posted Sep 22, 2020 19:35 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

This is really good to hear! I worked in telecom back when it was a morass of ATM and RSVP, and it's great that IPv6 is gaining traction.

And it also appears that large networks have a motivation to move to IPv6 simply because there's not enough space in IPv4 for them. I wonder if "market consolidation" in the ISP area is what will finally push IPv6 over the hump.