| |||||||||||||
Bufferbloat: Dark Buffers in the Internet (ACM Queue)Bufferbloat: Dark Buffers in the Internet (ACM Queue)Posted Dec 6, 2011 16:42 UTC (Tue) by dlang (✭ supporter ✭, #313)In reply to: Bufferbloat: Dark Buffers in the Internet (ACM Queue) by marcH Parent article: Bufferbloat: Dark Buffers in the Internet (ACM Queue)
the problem is that figuring out the right size queue to use is really hard to do
If you have a 1Gb/sec connection across the country, you need to have a large buffer on your machine.
the default buffer sized in the kernel are sized for this sort of thing.
however, if your 1G network is connected to a 1M network, then your server buffer size should be 1/1000 the size to maintain the correct latency, but your desktop has no way of knowing that there is a 1m link somewhere in the middle.
in this situation (1G - 1M - 1G links) you need the routers connecting to the 1M link to have small buffers and drop packets
in practice, things are actually worse than this
you have a laptop (1g) connecting to to your firewall/access point (1G) connecting to your DSL modem (10M or 100M) connecting to the ISP (1M) and similar setup on the other end.
you don't have any control over the buffers on the DSL modem, and the buffers there are much larger than they should be, and genrally not configurable, so you can easily fill them up and generate the high latency.
to work around this, you need to shrink the buffers on the firewall to have it drop packets sooner, or on your laptop to have it not generate the packets.
AQM is needed to have these devices detect that there is a problem and shrink the buffers in response.
everybody knows that the current buffer sizes are far too large, but there isn't a clear answer to the question of what the buffer size should be.
variable link speeds (wireless or cable 'turbo mode') greatly complicate this issue.
(Log in to post comments)
Bufferbloat: Dark Buffers in the Internet (ACM Queue) Posted Dec 7, 2011 8:55 UTC (Wed) by marcH (subscriber, #57642) [Link]
> however, if your 1G network is connected to a 1M network, then your server buffer size should be 1/1000 the size to maintain the correct latency, but your desktop has no way of knowing that there is a 1m link somewhere in the middle.
No: in this case the queue(s) in your server do not matter because they will be empty most of the time. Packets will only stack up at the bottleneck (as the name implies).
> in this situation (1G - 1M - 1G links) you need the routers connecting to the 1M link to have small buffers and drop packets
Yes.
If every link makes sure not to buffer more than 100ms or so, then bufferbloat goes away in 90% of the traffic cases (10% are left for the researchers to have fun with). Let's fix the most obvious problems first.
Bufferbloat: Dark Buffers in the Internet (ACM Queue) Posted Dec 7, 2011 10:45 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
you are right that the buffers on the 1G machine will always be empty, you need to (eventually) detect packet loss and then throttle the sending speed
the reality is that there is a lot of equipment out there that you are not going to be able to get replaced for several years, and part of that reason is that the vendors are still building equipment with buffers that are way too large because they still aren't doing testing that shows them the problem. this sort of publication and formal paper is what's required to get them to notice the problem and start the process of fixing it.
Bufferbloat: Dark Buffers in the Internet (ACM Queue) Posted Dec 7, 2011 12:05 UTC (Wed) by marcH (subscriber, #57642) [Link]
> you are right that the buffers on the 1G machine will always be empty, you need to (eventually) detect packet loss and then throttle the sending speed
TCP does that at the source. TCP is ACK-clocked on whatever is the current bottleneck thanks to the congestion or receiver window (whichever is smaller). No need to throttle TCP anywhere else.
Bufferbloat: Dark Buffers in the Internet (ACM Queue) Posted Dec 7, 2011 12:17 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
as I understand the effects of bufferbloat, the fact that these over-large buffers are queuing the packets instead of dropping them is breaking the TCP ack clocking mechansim
Bufferbloat: Dark Buffers in the Internet (ACM Queue) Posted Dec 7, 2011 13:26 UTC (Wed) by marcH (subscriber, #57642) [Link]
> ... is breaking the TCP ack clocking mechansim
... which does not mean TCP will go mad and send even more packets and clog whatever queues even more. I would bet it is actually the opposite.
Anyway, I only had the "reasonable queue size @ bottleneck" case in mind in my previous post. In this "normal", non-bufferbloated case ACK-clocking works fine and there is no need for externally throttling TCP anywhere else than at the bottleneck's queue when it fills up.
*In theory* not just TCP but every other protocol should be a good citizen and follow TCP's lead in respect to congestion/throttling: http://en.wikipedia.org/wiki/Datagram_Congestion_Control_...
I would not be surprised if TCP is actually the worst guy for falling in and filling bufferbloat traps. There is some irony in that considering it is usually considered the best congestion citizen *in the lack of bufferbloat*.
Bufferbloat: Dark Buffers in the Internet (ACM Queue) Posted Dec 7, 2011 16:39 UTC (Wed) by martinfick (subscriber, #4455) [Link]
> I would not be surprised if TCP is actually the worst guy for falling in and filling bufferbloat traps. There is some irony in that considering it is usually considered the best congestion citizen *in the lack of bufferbloat*
I had the same thought myself and was wondering if TCP needs to be fixed to take latency into account? From my lame understanding it seems to only care about throughput, isn't that why the bittorrent folks came up with their own UDP based solution? So, while I am all for fixing buffers wherever possible, shouldn't there be more discussion about fixing TCP?
Bufferbloat: Dark Buffers in the Internet (ACM Queue) Posted Dec 7, 2011 22:58 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
actually, what is happening is that TCP (eventually) gets the acks for the packets that it sends, so it speeds up to try and send more traffic.
eventually the delay gets so large that it times out before the acks arrive and the speed collapses.
If you look at the graphs in Getty's paper, you will see exactly this sort of picket-fence for what he is calling 'goodput', which is the amount of traffic that is actually getting to the destination (with the rest of the available bandwidth being taken up by 'badput', which is packets that are going to be dropped by the time they get to the destination because they are either too old, or they are retransmissions of packets that are still in flight, and so will be duplicates by the time they get there)
Bufferbloat: Dark Buffers in the Internet (ACM Queue) Posted Dec 7, 2011 11:40 UTC (Wed) by mtaht (✭ supporter ✭, #11087) [Link]
Dlang:
You are doing a great job here of explaining things, but I have to correct you on one MAJOR point.
"however, if your 1G network is connected to a 1M network, then your server buffer size should be 1/1000 the size to maintain the correct latency, but your desktop has no way of knowing that there is a 1m link somewhere in the middle."
Um, no. The closest we know to a correct figure for buffering is the square root(flows) * bandwidth* delay product of the next hop.
Bandwidth as humans measure it is X Mbit/sec, and as computers do, it
I recently shot myself in the foot here myself, I was doing some shell scripting that assumed a linear relationship of buffers to speed for tc, and those estimates got very wrong, quickly. It rather bugs me that there is no sqrt() call in the shell, you have to simulate one using echo "sqrt(the bdp)" | bc -i or something like that.
Assuming delay is a constant, (and delay is not!), doing some tons of square roots
http://www.bufferbloat.net/projects/bloat/wiki/Equations
Bufferbloat: Dark Buffers in the Internet (ACM Queue) Posted Dec 7, 2011 11:52 UTC (Wed) by mtaht (✭ supporter ✭, #11087) [Link]
And I should probably mention that whilst I'm fiddling with sqrts at this point, incorporating the next hop delay term into your thinking about this stuff is far more important mathematically.
Bufferbloat: Dark Buffers in the Internet (ACM Queue) Posted Dec 7, 2011 12:17 UTC (Wed) by mtaht (✭ supporter ✭, #11087) [Link]
Meh. I can't believe how much I trip myself up on this. Who knows, maybe I've been getting it wrong all this time, too....
Let me try again.
the total amount of buffering in the txqueue + tx ring buffer portion of the stack needs to be not much longer than the the BDP to the *next hop*.
BQL appears to solve the tx ring portion of the problem thoroughly, at least on ethernet.
Figuring out how many streams can co-exist in the txqueuelen set of buffers above the tx ring, and when to start dropping packets there, is an AQM problem, about which much debate exists. The next-hop BDP*sqrt(flows) thing is, well, debatable, but getting the effective txqueue's length down to where that portion of the AQM debate can take place again, seems doable with the time in queue idea floating about.
The total amount of buffering in tcp's algorithms, which do their own buffering internally, that is required for the end-to-end queue to be handled, is dependent on the BDP, and I'm going to flat out wave hands and say that AQM can help there a lot, and typically has very 'interesting' problems with streams of different RTTs.
Bufferbloat: Dark Buffers in the Internet (ACM Queue) Posted Dec 7, 2011 12:36 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
If the buffer size is root(flows) * bandwidth * delay and the bandwidth is 1/1000 of what it was before (with all else being equal) doesn't that make the required buffer size 1/1000 as well?
I'm missing something here, but I don't see how the sqrt piece matters when we are talking about the bandwidth changing
Bufferbloat: Dark Buffers in the Internet (ACM Queue) Posted Dec 7, 2011 13:04 UTC (Wed) by mtaht (✭ supporter ✭, #11087) [Link]
In your example, you missed the delay component.
'All else' is not equal in your example.
And I was mentally going from the 1Mbit link UP to the 1000Mbit link, where
And I was still kicking myself about the sqrt part from my mis-spent weekend. And I conflated the three together in trying to explain myself.
I really shouldn't post stuff before my third cup of coffee. I may just delete what I tried to post and start over. If there is a way to explain it better (if you can explain it back to me!) I'm either going to make another pot of coffee or go to bed and pull a pillow over my head. Or both.
|
|||||||||||||