Not logged in
Log in now
Create an account
Subscribe to LWN
LWN.net Weekly Edition for May 23, 2013
An "enum" for Python 3
An unexpected perf feature
LWN.net Weekly Edition for May 16, 2013
A look at the PyPy 2.0 release
LCA: Vint Cerf on re-engineering the Internet
Posted Jan 25, 2011 10:40 UTC (Tue) by gdt (subscriber, #6284)
Yep, exactly. Re-reading my posting I wish I'd spent more time making clearer the distinction between (1) TCP buffers in end systems and (2) IP buffers on the egress interfaces of routers. The leaflet was about (1), Vint obviously thought from the context of the question that the leaflet was about (2).
Posted Jan 26, 2011 1:25 UTC (Wed) by mtaht (✭ supporter ✭, #11087)
One core problem of bufferbloat is that devices and device drivers are currently doing too much *uncontrolled* buffering. The TCP/IP stack itself is fine, however, 1) once a packet lands on the txqueue, it can be shaped, but rarely is.
2) Far worse, in many cases, especially in wireless routers, once a packet gets off the txqueue it lands in the driver's queue, which can be quite large, and can incur huge delays in further IP traffic.
Once the device driver's buffers are filled, no amount of traffic shaping at a higher level will help. Death will not release you.
Here's a specific example:
Linux's default txqueuelen is 1000. Even after you cut that to something reasonable, it then hits the device driver's buffers. The driver I'm specifically hacking on (the ath9k, patches available here: https://github.com/dtaht/Cruft/raw/master/bloat/558-ath9k...
) defaults to 507 buffers, (with some limiting factors as to the size of the 10 queues applied) for which it will retry to send up to 13 times.
Assume your uplink is 3Mbit/sec, what's your maximum latency?
Assume your uplink is 128Kbit/sec, what's your maximum latency?
I'm not going to show the math here, it's too depressing. If you have an ath9k, try the above patch. there's one for the IWL going around. Many ethernet devices support ethtool...
The difference in overall network responsiveness with the crude ath9k patch above is amazing, and I've still got a long way to go with it.
This paper: http://www.cs.clemson.edu/~jmarty/papers/PID1154937.pdf
strongly suggests that a maximum of 32 uncontrolled IP buffers be used in the general case (with 1 being an ideal). It also raises some other strong points.
There's plenty of experimental data out there now too, and experiments you can easily perform on your own devices.
Every device driver I have looked at defaults to uncontrolled buffers far in excess of that figure, usually at around 256 buffers, even before you count txqueuelen.
The key adjective here for coping with bufferbloat is to reduce uncontrolled" buffering, starting with the device driver(s) and working up to various means of traffic shaping and providing adequate feedback to TCP/ip streams (packet drop/ECN etc) to keep the servo mechanism(s) working.
Posted Jan 26, 2011 3:02 UTC (Wed) by jthill (guest, #56558)
Posted Jan 26, 2011 4:11 UTC (Wed) by mtaht (✭ supporter ✭, #11087)
In the case of a home gateway, my comcast's business class service is running at about 3Mbit/sec on the uplink.
Huge dark (unmanaged) buffers in the device affect latency really badly - not just for TCP/ip, but for stuff that would ordinary jump to the head of the queue - udp, dns, voip, gaming, NTP... ... and in some cases are so big as to break TCP/ip almost entirely.
We've been sizing device buffers as if it was all on gigE backbone networks. Nor have we been using reasonable AQM. I urge you to try the experiments mentioned earlier.
Posted Jan 28, 2011 6:45 UTC (Fri) by The_Barbarian (subscriber, #48152)
Posted Jan 29, 2011 5:12 UTC (Sat) by mtaht (✭ supporter ✭, #11087)
Posted Feb 11, 2011 16:43 UTC (Fri) by mcgrof (guest, #25917)
Posted Feb 11, 2011 17:08 UTC (Fri) by mcgrof (guest, #25917)
Posted Jan 26, 2011 21:15 UTC (Wed) by mcoleman (guest, #70990)
Posted Jan 25, 2011 11:25 UTC (Tue) by marcH (subscriber, #57642)
Confusing these two very different buffering roles (TCP versus below TCP) is a huge mistake.
On the other hand, I wonder why a leaflet was needed at all. TCP auto-tuning is supposed to have fixed this problem already?
Posted Jan 25, 2011 11:32 UTC (Tue) by marcH (subscriber, #57642)
I take some of that back. In theory, they are not related. In practice, you can have nasty interactions between the two. In any case, they are totally different beasts.
Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds