Bufferbloat: Dark Buffers in the Internet (ACM Queue) [LWN.net]

Bufferbloat: Dark Buffers in the Internet (ACM Queue)

Posted Dec 7, 2011 11:40 UTC (Wed) by mtaht (✭ supporter ✭, #11087)
In reply to: Bufferbloat: Dark Buffers in the Internet (ACM Queue) by dlang
Parent article: Bufferbloat: Dark Buffers in the Internet (ACM Queue)

Dlang:

You are doing a great job here of explaining things, but I have to correct you on one MAJOR point.

"however, if your 1G network is connected to a 1M network, then your server buffer size should be 1/1000 the size to maintain the correct latency, but your desktop has no way of knowing that there is a 1m link somewhere in the middle."

Um, no. The closest we know to a correct figure for buffering is the square root(flows) * bandwidth* delay product of the next hop.

Bandwidth as humans measure it is X Mbit/sec, and as computers do, it
bits/nanosec and this distinction trips us up. Also the 1G network generally has very low delay, and the 1Mbit network very high.

I recently shot myself in the foot here myself, I was doing some shell scripting that assumed a linear relationship of buffers to speed for tc, and those estimates got very wrong, quickly. It rather bugs me that there is no sqrt() call in the shell, you have to simulate one using echo "sqrt(the bdp)" | bc -i or something like that.

Assuming delay is a constant, (and delay is not!), doing some tons of square roots
for practice of common figures, tossing in nearly random numbers for the above values, straightened out my assumptions and thinking and code considerably.

http://www.bufferbloat.net/projects/bloat/wiki/Equations

http://en.wikipedia.org/wiki/Bandwidth-delay_product

http://www.cc.gatech.edu/~dovrolis/Papers/buffers-ton.pd

(Log in to post comments)

Bufferbloat: Dark Buffers in the Internet (ACM Queue)

Posted Dec 7, 2011 11:52 UTC (Wed) by mtaht (✭ supporter ✭, #11087) [Link]

And I should probably mention that whilst I'm fiddling with sqrts at this point, incorporating the next hop delay term into your thinking about this stuff is far more important mathematically.

Bufferbloat: Dark Buffers in the Internet (ACM Queue)

Posted Dec 7, 2011 12:17 UTC (Wed) by mtaht (✭ supporter ✭, #11087) [Link]

Meh. I can't believe how much I trip myself up on this. Who knows, maybe I've been getting it wrong all this time, too....

Let me try again.

the total amount of buffering in the txqueue + tx ring buffer portion of the stack needs to be not much longer than the the BDP to the *next hop*.

BQL appears to solve the tx ring portion of the problem thoroughly, at least on ethernet.

Figuring out how many streams can co-exist in the txqueuelen set of buffers above the tx ring, and when to start dropping packets there, is an AQM problem, about which much debate exists. The next-hop BDP*sqrt(flows) thing is, well, debatable, but getting the effective txqueue's length down to where that portion of the AQM debate can take place again, seems doable with the time in queue idea floating about.

The total amount of buffering in tcp's algorithms, which do their own buffering internally, that is required for the end-to-end queue to be handled, is dependent on the BDP, and I'm going to flat out wave hands and say that AQM can help there a lot, and typically has very 'interesting' problems with streams of different RTTs.

Bufferbloat: Dark Buffers in the Internet (ACM Queue)

Posted Dec 7, 2011 12:36 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]

If the buffer size is root(flows) * bandwidth * delay and the bandwidth is 1/1000 of what it was before (with all else being equal) doesn't that make the required buffer size 1/1000 as well?

I'm missing something here, but I don't see how the sqrt piece matters when we are talking about the bandwidth changing

Bufferbloat: Dark Buffers in the Internet (ACM Queue)

Posted Dec 7, 2011 13:04 UTC (Wed) by mtaht (✭ supporter ✭, #11087) [Link]

In your example, you missed the delay component.

'All else' is not equal in your example.

And I was mentally going from the 1Mbit link UP to the 1000Mbit link, where
delay factors in a lot. Your typical 1Mbit internet link can have an inherent next-hop delay of 1-60ms on wired technologies which is a significant component of that portion of the BDP. Wireless is far worse,
of course.

And I was still kicking myself about the sqrt part from my mis-spent weekend. And I conflated the three together in trying to explain myself.

I really shouldn't post stuff before my third cup of coffee. I may just delete what I tried to post and start over. If there is a way to explain it better (if you can explain it back to me!) I'm either going to make another pot of coffee or go to bed and pull a pillow over my head. Or both.