LWN: Comments on "TCP small queues and WiFi aggregation — a war story"

MTU

farnz — Mon, 25 Jun 2018 17:40:52 +0000

You do, but they're not using IEEE standard Ethernet (jumbo frames implies not IEEE standard) - and WiFi standards (including frame aggregation) are written to get high performance when using IEEE standard Ethernet.

Hence frame aggregation rather than high MTUs - a high MTU for performance means being outside the IEEE standard, while a 1500 MTU allows you to be inside the standard.

MTU

raven667 — Mon, 25 Jun 2018 17:28:18 +0000

That seems bogus, I'm pretty sure I have many pieces of equipment using jumbo frames, whatever IEEE has in their written specs or whatever they "like".

MTU

farnz — Mon, 25 Jun 2018 16:20:40 +0000

Nope. The issue is that you cannot have an MTU above 1500 on Ethernet without breaking the IEEE specs for Ethernet and for WiFi. You are simply not allowed a jumbo MTU on the Layer 2 link, and the IEEE won't accept changes to 802 series standards that increase the user MTU beyond 1500.

IPv6 is not relevant here - it's an IEEE decision because even in IPv4, with router fragmentation allowed, the IEEE doesn't like it.

MTU

raven667 — Mon, 25 Jun 2018 16:01:06 +0000

That's what I first though about this but the issue is not that you can't have jumbo 9000 byte MTU on the local layer2 link, either Ethernet or WiFi (although I think WiFi standard only supports 1500), which would be negotiated between endpoints as part of the TCP MSS, it's that intervening links over the Internet at some point are likely to only permit 1500 byte frames, so somewhere along the line you would need to fragment the layer3 packets, which is not allowed for IPv6 which relies on PMTU discovery to find the correct MSS/MTU that can cleanly make it through the whole path, leading to needing aggregation on the first WiFi hop for efficiency, to make up for not requiring upstream routers to fragment/reassemble. You could probably sidestep this if you had a jumbo-clean path, and you weren't concerned about stations monopolizing airtime with large frames, but that's unlikely unless you control the whole infrastructure between endpoints.

MTU

farnz — Mon, 25 Jun 2018 09:16:15 +0000

Nope; the answer is IEE 802.3 Ethernet. WiFi (IEEE 802.11) is designed to transparently interoperate with 802.3 Ethernets. The IEEE has declared that the Ethernet MTU is fixed at 1500 bytes[1]; this implies that WiFi per-frame MTUs are also fixed at 1500 bytes. Given that it is a hard requirement for WiFi that the frame MTU is no more than 1500 bytes, you need things like aggregation to get a decent speed.

If larger frames were permitted on 802.11, then you would not be able to bridge 802.11 with IEEE standard 802.3; while it's common to support jumbo frames on Ethernet, this is technically a non-standard extension, and IEEE standard 802.11 can't assume that any Ethernet it is connected to will permit jumbo frames.

[1] While the IEEE 802.3 MTU is 1500 bytes, they also now require all equipment to handle frames of up to 2000 bytes in total size, to allow for headers, checksums, VLAN tags etc. WiFi is similar - 2304 byte maximum MSDU frame, of which 1500 bytes maximum is user MTU, and the other 804 bytes are reserved for VLAN tags etc.

MTU

meuh — Sat, 23 Jun 2018 18:16:23 +0000

I asked myself why not increasing the IP MTU (Maximum Transmission Unit) instead of relying on WiFi Frame Aggregation behavior ? I guess the answer is IPv6: as IPv6 doesn't expect routers to fragment packets, a typical TCP connection over the Internet involve packets no larger than 1280 bytes. So the WiFi hardware has to handle such small packets, hence the Frame Aggregation feature.

TCP small queues and WiFi aggregation — a war story

mtaht — Fri, 22 Jun 2018 23:40:58 +0000

the qualcomm work was first documented by a good ccc talk, the video for which I cannot find right now.

But: https://osmocom.org/projects/quectel-modems/wiki

and the slides from that talk: https://fahrplan.events.ccc.de/congress/2016/Fahrplan/sys...

TCP small queues and WiFi aggregation — a war story

kronat — Fri, 22 Jun 2018 09:52:49 +0000

> Some chipsets (like quantenna's) actually wedge an entire linux stack into their chip.
> qualcomms's LTE modems do also.

Do you have a reference for these statements? I am investigating a similar problem in 3GPP networks. Thanks!

TCP small queues and WiFi aggregation — a war story

cagrazia — Fri, 22 Jun 2018 08:27:03 +0000

> I'd be curious how fast UDP on the other operating system went, to know if it topped out at the same 100Mb/s.

We did a very quick TCP and UDP test on a Windows 8 machine, with a kind of iperf tool (honestly, it was tricky and messy to configure): throughput results were oscillating between 90 and 95 Mbps, while the TCP RTT was very unstable as well as the UDP jitter.

TCP small queues and WiFi aggregation — a war story

cagrazia — Fri, 22 Jun 2018 08:12:01 +0000

> Which USB/WiFi 802.11ab/g/n dongles were used?

The exact chipsets we used in our tests are the Atheros AR9271 (ath9k_htc driver), the Atheros AR9580 (ath9k), and Atheros QCA9880v2 (ath10k). For space constraints, we presented here only the results coming from the ath9k_htc device, but we had a similar positive outcome with all the three chipsets.

TCP small queues and WiFi aggregation — a war story

mtaht — Thu, 21 Jun 2018 19:05:48 +0000

I am pleased to say that hooks into qualcomm's 802.11ac firmware have appeared sufficient to mostly mitigate the bufferbloat problem they had there with the code at the mac80211 layer. Certainly I have hope their internal firmware will improve.

If I had any one wish for "smart firmware", it would be only that there was enough smarts in the wifi and lte hardware/firmware to handle no more than 4ms worth of queuing and real time processing, and let the kernel handle the rest within its constraints for interrupt latency.

BQL accomplished this for ethernet (sub-1ms there, actually)

fq_codel for wifi ( https://www.usenix.org/system/files/conference/atc17/atc1... ) gets it down to two aggregates, which can take up to ~5ms each, at any achieved "line" rate.

We can do better than this with better control of txops on the AP, and certainly the algorithms above can exist, offloaded, in smarter hardware, which is happening on several chipsets I'm aware of.

PS The new sch_cake actually can run ethernet, shaped to 1gbit, at lower latencies than anything that uses BQL - at a large cost in cpu overhead, but not as much as you might think - it works at that rate on a quad core atom, for example.

http://www.taht.net/~d/cake/rrul_be_-_cake-shaped-gbit-qu...

vs sch_fq:

http://www.taht.net/~d/cake/rrul_be_-_fq-quad-long-smooth...

The principal advantage of cake, here, even though now capable running at speeds and latencies like this, is to defeat other black box token bucket shapers on a link, however, at much lower rates, with a corresponding reduction in cpu cost to (at sub-100mbit) the level of "noise".

TCP small queues and WiFi aggregation — a war story

excors — Thu, 21 Jun 2018 16:43:16 +0000

Power efficiency seems important too - you don't want to wake up Linux on the big high-performance main CPU just so it can choose to ignore a packet, when you could run that logic in the firmware on the already-awake and just-fast-enough CPU in the networking chip instead.

And portability - you don't want to maintain multiple separate copies of your million lines of driver code (plus regression tests etc) for Linux, Windows, Apple, Fuchsia, the several obsolete Linux versions your customers still use, etc, when it's much easier to put almost all the code in firmware so the OS-specific driver is just a thin wrapper. Plus the Linux community will likely be much happier if you upstream that wrapper and leave the firmware opaque, than if you attempt to upstream your million lines of cross-platform-ish code that doesn't follow the Linux coding style and has an ugly abstraction layer for OS-specific bits.

None of that stops you making the thick firmware open source, though.

TCP small queues and WiFi aggregation — a war story

mtaht — Thu, 21 Jun 2018 15:59:25 +0000

Thinner firmware for 802.11ac would be nice, particularly in an age when the main
cpu chip is multicore and much faster than anything built into the wifi chip.

but wifi has some really hard real-time constraints that require dedicated cpus onboard, and once you start doing that the temptation to wedge all your functionality there has thus far been overwhelming for vendors.

So far as I know qualcom's 802.11ac devices use a proprietary R/T OS inside.

Some chipsets (like quantenna's) actually wedge an entire linux stack into their chip.

qualcomms's LTE modems do also.

TCP small queues and WiFi aggregation — a war story

mtaht — Thu, 21 Jun 2018 14:47:46 +0000

From eric dumazet and toke on the bloat mailing list:

From eric dumazet
On 06/21/2018 02:22 AM, Toke Høiland-Jørgensen wrote:
> Dave Taht <dave.taht@gmail.com> writes:
>
>> Nice war story. I'm glad this last problem with the fq_codel wifi code
>> is solved
>
> This wasn't specific to the fq_codel wifi code, but hit all WiFi devices
> that were running TCP on the local stack. Which would be mostly laptops,
> I guess...

Yes.

Also switching TCP stack to always GSO has been a major gain for wifi in my tests.

(TSQ budget is based on sk_wmem_alloc, tracking truesize of skbs, and not having
GSO is considerably inflating the truesize/payload ratio)

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/...
tcp: switch to GSO being always on

I expect SACK compression to also give a nice boost to wifi.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/...
tcp: add SACK compression

Lastly I am working on adding ACK compression in TCP stack itself.

TCP small queues and WiFi aggregation — a war story

josh — Tue, 19 Jun 2018 17:50:30 +0000

> The nominal transfer rate of the dongles is 150Mb/s, but what we saw on the screen was disappointing: an upload iperf connection, no matter which options were used, was able to reach only 40Mb/s. Using another operating system as a client, we were able to achieve 90Mb/s, leaving out a problem with the server. [...] To stress-test the equipment, we started a UDP transmission at a ludicrous speed. Not so surprisingly, we arrived almost at 100Mb/s.

I'd be curious how fast UDP on the other operating system went, to know if it topped out at the same 100Mb/s.

TCP small queues and WiFi aggregation — a war story

johan — Tue, 19 Jun 2018 13:29:52 +0000

I wish the kernel devs had better testing (in this case regression testing).
If they had they would likely have found issues like these a bit earlier.
Obviously it's a very hard task though considering how many devices there are to test.

TCP small queues and WiFi aggregation — a war story

shiftee — Tue, 19 Jun 2018 08:04:01 +0000

Excellent article with a nice positive outcome.

Dongles based on this chipset are available
https://www.thinkpenguin.com/gnu-linux/penguin-wireless-n...
and
https://www.olimex.com/Products/USB-Modules/MOD-WIFI-AR92...

If I remember correctly there was a FOSS enthusiast working for Atheros who convinced them to release the firmware code but he has since left the company

TCP small queues and WiFi aggregation — a war story

Beolach — Tue, 19 Jun 2018 06:56:19 +0000

I don't know what specific dongles they used, but the main thing to look for is the chipset used, which as mentioned in the article were Atheros chipsets supported by the ath9k / ath9k_htc drivers. Even if you get a different brand of dongle, if it has an Atheros chipset supported by these drivers you should be able to get similar results. As the article mentioned, these ath9k chipsets are fully open-source, both w/ the driver software & the firmware, which allows much easier & faster improvements like this. Dave Taht also used ath9k chipsets for his CeroWRT Bufferbloat / Make-WiFi-Fast projects, for this same reason.

Sadly, to the best of my knowledge there are no 802.11ac chipsets w/ both open-source drivers & firmware - I believe even the ath10k 802.11ac chipsets have closed firmware blobs. :-( I would love to hear about any I've missed.

TCP small queues and WiFi aggregation — a war story

pabs — Tue, 19 Jun 2018 03:35:20 +0000

Which USB/WiFi 802.11ab/g/n dongles were used?