LWN: Comments on "BBR congestion control"

BBR congestion control

zlynx — Mon, 07 Jan 2019 16:54:41 +0000

Sorry for the confusion there, but TCP does not work when tunneled inside TCP. It may appear to work but its feedback mechanisms are fundamentally broken.

Use TCP sessions inside of a UDP OpenVPN tunnel.

Proxy tunnels like SSH and SOCKS are different because they do not tunnel the actual TCP packets. Proxies unwrap the TCP sessions and build new ones.

BBR congestion control

bircoph — Mon, 07 Jan 2019 16:36:58 +0000

> You may be having a OpenVPN problem.

I tested OpenVPN in exactly the same environment with 4 different TCP congestion control algorithms on sender's side: RENO, BIC, CUBIC and BBR. (I also tested changing algo on receiver's side, but this changes almost nothing.) First three of them behave rather alike: RENO, BIC and CUBIC achieve 4.6-5.0 MB/s (with CUBIC being the best of three), but BBR provides a steady drop to ~400 KB/s. This is absolutely unacceptable. And this is unlikely to be an application problem, since it works well with other congestion control algorithms. So something is very wrong with BBR or its implementation.

> One thing I just thought of though. You are definitely not using OpenVPN in TCP mode, right?

Of course I'm using TCP. It is pointless to test TCP congestion control with an UDP application. Why I'm using OpenVPN over TCP? Because that's how the server is configured and I can't change that: both endpoints are under my control, but the server is not. That's the reality I have to face and work with.

BBR congestion control

zlynx — Sun, 06 Jan 2019 22:43:48 +0000

You may be having a OpenVPN problem. I honestly haven't tried it that way.

But today I needed to copy a disk image around anyway so I tried it from my laptop over WiFi to my NAS. Both are running Fedora 29, both with BBR enabled.

Something you should know about FQ though. The WiFi system has dropped that. It no longer uses qdiscs at all. It's internal to the WiFi system and is a fq_codel variant designed for WiFi. It is supposed to work the same as FQ for BBR, however.

Doing my transfer using rsync with --progress it averaged right around 20 MB/s aka just over 200 Mbps. The speed went up and down but never dropped to the floor as in your example.

One thing I just thought of though. You are definitely not using OpenVPN in TCP mode, right? Using any VPN through a TCP tunnel is a horrible idea, and especially bad on WiFi. You get all of TCP's problems compounded as packet losses and delay cause both TCP sessions to react, overreact, and miserably fail.

BBR congestion control

bircoph — Sun, 06 Jan 2019 21:26:42 +0000

I gave BBR a try on 4.19.13 kernel, FQ traffic scheduler is enabled as well.
Tests were made on two endpoints (desktop and laptop) connected via openvpn.

With Cubic on sending side I have ~3 MB/s initiall which steadily rises to ~5 MB/s and holds there. 5 MB/s hits a CPU limit on laptop due to sophisticated encryption used.

With BBR I have initial speed ~5MB/s which drops steadily to ~400 КB/s and holds there.

So maybe BBR is good for specific datacenter setups or lab environment, but it is a failure for real life end-user hardware on common wired or wireless networks. At least for now. Maybe some bug in the kernel?

BBR congestion control

flussence — Sat, 19 Aug 2017 16:30:59 +0000

BBR is a sender-side change, so all end users get the benefit if a server upgrades. If it required client changes it'd be nearly worthless; Android vendors never keep their kernel up to date.

BBR congestion control

plasma-tiger — Thu, 17 Aug 2017 13:18:44 +0000

With this algorithm in place, will a handheld mobile user see any difference in web working faster?

BBR congestion control

dps — Sat, 17 Dec 2016 00:04:22 +0000

A while ago I looked at *lot* of research about bandwidth, delay, etc measurement. Some of the results where impressive but none of the tools developed and much else is no longer accessible. The only major exception is nttcp, which debian has but is hard to find elsewhere.

BBR seems to be using some similar techniques to tools whih measured rsponse tines for two back to back ICMP echo request packets, some which claimed good results. Very asymmetric links and 100:1 contention has made the problem harder.

data rate, not bandwidth

giraffedata — Wed, 05 Oct 2016 18:03:39 +0000

"Throughput" is the shorter word you're looking for.

In this case, I think "throughput" would have been almost as confusing as bandwidth, because we're dealing with the special case that the data rate into the pipe is greater than the data rate out of it (because of that meddlesome buffering by routers). I think of throughput as a steady state thing covering the whole pipe.

data rate, not bandwidth

marcH — Sun, 02 Oct 2016 04:26:41 +0000

> It's supposed to say, "the actual rate of data...

"Throughput" is the shorter word you're looking for.

Where is one supposed to run this?

marcH — Sun, 02 Oct 2016 04:23:54 +0000

Video isn't just YouTube - it's also Skype, Hangouts etc. The latter are much more vulnerable than the former to network issues, espcially latency issues like bufferbloat. When reading the description of BBR above I felt like someone in the back kept screaming "adaptive video codecs".

BBR congestion control

simoncion — Sat, 01 Oct 2016 00:27:42 +0000

By relying on the fact that IPv6 requires admins to permit at least _some_ ICMP packets through their firewalls? (And maybe the -somewhat morbid- hope that sysadmins of the "Hackers can't be allowed to _ever_ see any of my ports, ever!!1" persuasion are leaving the field to enjoy their retirement?)

BBR congestion control

pjardetzky — Fri, 30 Sep 2016 05:51:12 +0000

Reminds me of this rate-based work from the early 90s. http://ccr.sigcomm.org/archive/1995/jan95/ccr-9501-keshav...

BBR congestion control

mtaht — Thu, 29 Sep 2016 19:34:39 +0000

I don't see how to get past this for a newer icmp message.

"Widespread deployment of ICMP filtering makes it impossible to
rely on ICMP Source Quench messages for congestion control."

BBR congestion control

forthy — Thu, 29 Sep 2016 18:45:12 +0000

Net2o doesn't use round trip delay for rate (it keeps measuring this, though), the sender is sending out bursts, and the receiver measures how fast these bursts arrive. The way back is not included in the measurement, so the measured time is from first to last packet of a burst (and also the timestamp of the first packet in a burst, to measure delay).

Note that net2o can do a lot of things TCP can't, because it is redesigned from scratch, and therefore the ack packets can contain more information than a TCP ack.

BBR congestion control

Cyberax — Thu, 29 Sep 2016 09:13:32 +0000

Source quench was meant to be a request, not a hint. And it was deployed way back when the TCP stacks were exceedingly primitive.

A semi-authenticated ICMP message that is treated as a hint might be a much better idea now.

BBR congestion control

mtaht — Thu, 29 Sep 2016 05:10:40 +0000

see the history of icmp source quench

https://tools.ietf.org/html/rfc6633

Where is one supposed to run this?

gdt — Wed, 28 Sep 2016 00:22:58 +0000

Recall that it is the transmitter that makes the decisions in TCP congestion control, the receiver simply Acks. Where the handset or Chromebook is predominately a consumer of data then its choice of TCP congestion control algorithm isn't going to make a difference.

BBR congestion control

nnovoice — Tue, 27 Sep 2016 09:45:53 +0000

Does BBR not care for tcp rmem and wmem values? I have been trying to set small values to the tcp rmem and wmem buffers so as to counter the bufferbloat issues on 3G and 4G links. Also trying to use fq_codel.
Also, http://blog.cerowrt.org/post/bbrs_basic_beauty/ link says better bandwidth and sawtooth is also dead, which is really good if it happens. Itching to try BBR, but have to go from 4.1 to 4.9. Long way to go!

BBR congestion control

anton — Mon, 26 Sep 2016 12:20:57 +0000

Net2o also estimates the available bandwidth through the round-trip time, as described in these Slides from 2012. It would be interesting to see a more detailed comparison between BBR and net2o flow control.

Out-competed by more aggressive algorithms?

mtaht — Sun, 25 Sep 2016 18:55:40 +0000

That is a very good explanation. I will be writing more on my blog, after I run a few thousand more tests, and think about things more. I've already mildly mis-interpreted one dataset, publicly, and I'd like to "do it right" - and for that matter, wait for the paper.

That said, BBR and cubic CAN co-habitate in a conventional drop tail system - this plot - showing a BBR, cubic, BBR, and cubic flow, starting 3 seconds apart, in sequence, shows that.

http://blog.cerowrt.org/flent/bbr-comprehensive/latecomer...

(interpretation, I welcome corrections!)

The first BBR flow gets a reasonable RTT estimate, and when cubic hits the link, BBR falls off to what it thinks is a reasonably "fair share" of the link, while cubic grabs what it can, and then gets crushed by the second BBR flow entering the link (which does not get an accurate early RTT estimate) - BUT - eventually, it does, and all flows of both CCs eventually achieve something close to parity.

There's a lot of carnage going on in the background to get to this point - and this is a nasty test, intentionally so! And I did most of the data collection in the wee hours of the morning (and in the interest of science made all my data public)

I keep the blog and a goodly portion of the flent data (including, rarely, packet captures) on github. I track queue depth, emulate several RTTs, multiple kinds of devices (notably cable modems), and also put multiple AQM and fq technologies in the path as well.

https://github.com/dtaht/blog-cerowrt

I think - based on what I just seen - I need to go add conventional policers to the test matrix. Sigh. And then theres, you know, making wifi fast, which is what I should be finishing up instead of going gaga over BBR.

Out-competed by more aggressive algorithms?

Cyberax — Sun, 25 Sep 2016 04:29:56 +0000

There's another effect here. Lots of routers try to share bandwidth fairly between various flows, so BBR and Cubic flows will get the same bandwidth if they both are 100% loaded. But in this case Cubic will oscillate back and forth while BBR will remain mostly stable.

Out-competed by more aggressive algorithms?

flussence — Sun, 25 Sep 2016 02:20:21 +0000

Let's see if I understand all this right... (mtaht will probably correct me if not :)

Cubic is aggressive in the sense that it has permanent acceleration, and the only thing keeping its velocity in check is a simple negative feedback loop: always trying to stuff more data into the network, until *after* the network itself becomes congested and starts rejecting (or dropping) packets.

Due to that design it spends most of its time significantly under maximum capacity, ramping up until a congestion spike hits. And due to *that* effect being in the wild for so long, network middleboxes have evolved to buffer tons of data even when they can't forward it right away, because a higher initial rate spike for a short-lived connection (e.g. HTTP) would be perceived as "faster".

BBR uses a lot of cleverness to guess the network path's effective bandwidth based on ACK delays and then self-regulates slightly below that, updating its estimate over time. If a Cubic stream is sharing the same pipe, it'll still be oscillating between over- and under-loaded, but because it's mostly under, the BBR stream will see "spare" capacity and gradually eat the difference until it has significantly more than 50% of the total throughput.

BBR congestion control

Cyberax — Sun, 25 Sep 2016 01:59:22 +0000

Realistically though, the global network topology doesn't usually change drastically throughout the life of a typical TCP connection. It's not possible to obtain accurate guaranteed throughput number, but simple estimates might help.

Something like ICMP "Packet Too Big" message, but for bandwidth. So if a router sees an "Advised Throughput Capacity" message with a value greater than it can sustain, it re-sends the message with a lower value. Kinda like extended ECN.

Of course, endpoints will be free to ignore this hint.

data rate, not bandwidth

giraffedata — Sun, 25 Sep 2016 00:57:12 +0000

I had a hard time understanding this at first because of the incorrect use of the word "bandwidth." Bandwidth is capacity, so phrases such as, "the actual bandwidth of data delivered to the far end" don't make sense.

It's supposed to say, "the actual rate of data delivered to the far end," and BBR is simply about comparing the rate of data delivered to the far end to the rate sent from the near end, and when you've raised the latter to the point that it's greater than the former, you've discovered the bandwidth of the connection.

To be even more clear, I would say "the actual rate of data arriving at the far end."

Out-competed by more aggressive algorithms?

giraffedata — Sun, 25 Sep 2016 00:44:42 +0000

No, BBR crushes cubic in its earlier phases, in my preliminary tests.

I'm not sure what you mean by this. Do you mean a node gets more bandwidth if it uses BBR than if it uses CUBIC, in a typical network? Or that in a network in which one node uses BBR and another user CUBIC, the BBR gets more bandwidth? Or something else?

The actual question is about a node using BBR in a network in which other nodes use an older, more aggressive policy. That may be a moot question. The older policies one finds in a network aren't more aggressive. CUBIC in particular is not terribly aggressive. Even older policies do things like cut their sending rate by half when they see the first dropped packet, which is not aggressive at all.

BBR congestion control

hechacker1 — Sat, 24 Sep 2016 21:23:53 +0000

Thanks,

It was a good post. I hoping to see it in ubuntu server soon since I use a squid proxy for my own network.

BBR congestion control

shemminger — Sat, 24 Sep 2016 17:13:52 +0000

End-to-End bandwidth information does not exist in the Internet. Circuit switch networks died years ago.
Only in restricted research networks and closed networks has it ever been possible. Even then the dynamic nature of available bandwidth means that any data given to end points would be out of date.

Out-competed by more aggressive algorithms?

mtaht — Sat, 24 Sep 2016 12:51:52 +0000

No, BBR crushes cubic in its earlier phases, in my preliminary tests.

Out-competed by more aggressive algorithms?

runekock — Fri, 23 Sep 2016 10:43:01 +0000

From reading this, I would fear that sharing bandwidth with an older, more aggressive algorithm will result in the BBR flow getting squeezed out. Sort of a networking parallel to Gresham's law.

BBR congestion control

Cyberax — Fri, 23 Sep 2016 09:37:46 +0000

ATM (and really, other synchronous networks) were the first step. The next one was RSVP ( https://en.wikipedia.org/wiki/Resource_Reservation_Protocol ).

I worked for a telecom that tried to deploy it. It actually looked awesome when it worked - you could get a channel across a network with guaranteed throughput and no jitter. And because telecoms usually build networks in rings, it could even survive individual fiber cuts without losing a single packet.

The catch phrase, of course, "when it worked". RSVP-capable hardware cost a LOT more than regular one and when it breaks finding the cause often involved restarting large parts of the network.

Eventually it was abandoned and they simply overprovisioned networks, segregating purely data flows into low-priority buckets.

BBR congestion control

diegor — Fri, 23 Sep 2016 09:22:24 +0000

This is what they tried to do in the past, and they came with ATM. ATM before a connection is opened, verified that every router have enough bandwith for you connection, and that bandwidth is allocated to your connection.

ATM was pushed by many big telecomunication company, but at the end failed. Without router pre allocating bandwith, there is no real improvements in estimate "bandwith information".

Note that in a resiliant network, where every packet of a connection can be routed in a different way, it's just delusional to think that you can better track bandwith allocation for every connection on router, respect to what you can do on the endpoint.

Every complex problem, have a nice solution, easy to understand, that doesn't work.

BBR congestion control

gdt — Fri, 23 Sep 2016 03:32:00 +0000

at least ten such algorithms in the Linux Kernel

Having all the congestion control algorithms available on a platform allows simple reasoning about the real-world performance of the algorithms: the only difference is the algorithm; the operating system, hardware and network are constant.[1] Stephen and others' implementation of plugable congestion control modules was by itself responsible for an appreciable advance in the state of the art, as it made in-the-field comparisons of algorithm performance easy to do: you didn't have to additionally have to reason about the comparative networking attributes of FreeBSD and Windows. This doesn't mean that all the algorithms provided by Linux are a good choice, far from it. However it is still useful to ship these algorithms as points of reference to identify regressions in newer algorithms.

As for user complexity, the default algorithm is a reasonable choice. There's no reason for a user to know other choices exist. If you wish to make a different choice then having all the alternatives available is a good thing, as often whichever alternative is "best" depends upon your circumstances. The default algorithm has been altered in the past, and I expect will be again. However given the possibility of a fault causing congestion collapse, all maintainers of widely-used operating systems take pains to ensure that algorithms and their implementations have proven themselves before they are used as a new default.

--------

[1] "all", as far as patent law allows.

BBR congestion control

jcm — Fri, 23 Sep 2016 02:03:45 +0000

What is being done to solve the real problem (of providing estimated bandwidth information to end points on a network)? Sure, there's all kinds of things that could go wrong and all kinds of lies (equipment needs to propagate this, and provide updates in a timely fashion, etc.) but what is actually being done to try?

Too often, I see problems in this world being solved by cleaver technical solutions because "utopia will never happen, so we'll just do this...". It might take a decade to provide a nicer solution that works at internet scale, which is all the more reason to get that moving now. And it's amazing how you can force people to use standards if you get big enough players pushing it through to help.

BBR congestion control

Sesse — Thu, 22 Sep 2016 23:46:24 +0000

Sure, but the question was about codel, so I used that as an example.

In a sense, it's a bit icky that sch_fq uses the qdisc hooks to do TCP pacing, but given how many failed attempts there were at implementing paced TCP in Linux before Eric Dumazet came along and just did it, and how amazingly well it works (even without BBR, it's a great win), I'll survive that it takes up the “slot” from fq_codel. (Also, it's a bit confusing that it's called sch_fq; it should really be called sch_packet_pacing, because I can't imagine another reason to run it.)

BBR congestion control

shemminger — Thu, 22 Sep 2016 23:18:56 +0000

From Eric Dumazet:

fq_codel is stochastic, so it wont work very well on hosts with 1,000,000 flows or more...

fq_codel is aimed for routers, while sch_fq targets hosts, implementing pacing at a minimal cost (one high resolution timer per qdisc)

BBR congestion control

mtaht — Thu, 22 Sep 2016 20:43:22 +0000

sesse - *fq*-codel is for the routers along the way, IMHO. :)

Given BBR's characteristics, the "FQ" portion helps most with legacy CC mechanisms, but I did just run a long string of tests showing the BBR + fq_codel is very good, much better than either alone.

I'll try to write up those results in the next week or so, but the
flent dataset is here: http://blog.cerowrt.org/flent/bbr-comprehensive.tgz if you want to browse and compare.

Where is one supposed to run this?

farnz — Thu, 22 Sep 2016 15:28:41 +0000

Any TCP endpoints - desktops, servers etc. Routers don't need to if they're just routing, but they'll benefit for things like SSH.

Where is one supposed to run this?

blitzkrieg3 — Thu, 22 Sep 2016 15:17:19 +0000

Based on developed by Google, and the fact that it's supposed to work better with wifi connected devices, I'm thinking Android handsets and Chromebooks.

Where is one supposed to run this?

giggls — Thu, 22 Sep 2016 12:39:31 +0000

Which machine should run this rather than Linux based Servers?

My (Linux based) Router or Desktop?

Sven

BBR congestion control

Sesse — Thu, 22 Sep 2016 10:38:47 +0000

BBR isn't a traffic control algorithm (like CoDel is), it's a TCP congestion control algorithm (like CUBIC is).

Think of it this way: BBR is for your endpoints. CoDel is for every router along the way. (The waters get slightly muddied if your server doubles as a router, though.)