LCA: Vint Cerf on re-engineering the Internet [LWN.net]

LCA: Vint Cerf on re-engineering the Internet

Posted Jan 26, 2011 1:25 UTC (Wed) by mtaht (✭ supporter ✭, #11087)
In reply to: LCA: Vint Cerf on re-engineering the Internet by gdt
Parent article: LCA: Vint Cerf on re-engineering the Internet

Not having seen the leaflet I don't know what, specifically, to address.

One core problem of bufferbloat is that devices and device drivers are currently doing too much *uncontrolled* buffering. The TCP/IP stack itself is fine, however, 1) once a packet lands on the txqueue, it can be shaped, but rarely is.

2) Far worse, in many cases, especially in wireless routers, once a packet gets off the txqueue it lands in the driver's queue, which can be quite large, and can incur huge delays in further IP traffic.

Once the device driver's buffers are filled, no amount of traffic shaping at a higher level will help. Death will not release you.

Here's a specific example:

Linux's default txqueuelen is 1000. Even after you cut that to something reasonable, it then hits the device driver's buffers. The driver I'm specifically hacking on (the ath9k, patches available here: https://github.com/dtaht/Cruft/raw/master/bloat/558-ath9k...
) defaults to 507 buffers, (with some limiting factors as to the size of the 10 queues applied) for which it will retry to send up to 13 times.

Assume your uplink is 3Mbit/sec, what's your maximum latency?
Assume your uplink is 128Kbit/sec, what's your maximum latency?

I'm not going to show the math here, it's too depressing. If you have an ath9k, try the above patch. there's one for the IWL going around. Many ethernet devices support ethtool...

The difference in overall network responsiveness with the crude ath9k patch above is amazing, and I've still got a long way to go with it.

This paper: http://www.cs.clemson.edu/~jmarty/papers/PID1154937.pdf
strongly suggests that a maximum of 32 uncontrolled IP buffers be used in the general case (with 1 being an ideal). It also raises some other strong points.

There's plenty of experimental data out there now too, and experiments you can easily perform on your own devices.

http://gettys.wordpress.com/2010/12/02/home-router-puzzle...

Every device driver I have looked at defaults to uncontrolled buffers far in excess of that figure, usually at around 256 buffers, even before you count txqueuelen.

The key adjective here for coping with bufferbloat is to reduce uncontrolled" buffering, starting with the device driver(s) and working up to various means of traffic shaping and providing adequate feedback to TCP/ip streams (packet drop/ECN etc) to keep the servo mechanism(s) working.

(Log in to post comments)

LCA: Vint Cerf on re-engineering the Internet

Posted Jan 26, 2011 4:11 UTC (Wed) by mtaht (✭ supporter ✭, #11087) [Link]

In the case of a wireless card in your laptop or a wireless link you can be running at speeds ranging from 300Mbit down to 1Mbit/sec, or less.

In the case of a home gateway, my comcast's business class service is running at about 3Mbit/sec on the uplink.

Huge dark (unmanaged) buffers in the device affect latency really badly - not just for TCP/ip, but for stuff that would ordinary jump to the head of the queue - udp, dns, voip, gaming, NTP... ... and in some cases are so big as to break TCP/ip almost entirely.

We've been sizing device buffers as if it was all on gigE backbone networks. Nor have we been using reasonable AQM. I urge you to try the experiments mentioned earlier.