The debloat-testing kernel tree

Memory is cheap and adding more packet buffers to a networked system helps improve throughput a very small bit. Since ISPs and hardware/software vendors focus on total throughput, they've got an incentive to add lots of packet buffering.

The problem though is that huge amounts of packet buffering prevent TCP from dealing with congestion appropriately by throttling connections: the larger buffers means that TCP has to wait much longer than it should to detect congestion. As with all feedback control loops, increasing the delay seriously harms the system. As a result, we end up with systems where total throughput might be high but latency under load becomes absurd. So you can't transfer a large file while running a VOIP call: the file transfer will get good throughput, but all the packets will see absurd latencies so that VOIP is no longer feasible. That's the real problem: we all need to be using latency-under-load in addition to total-throughput as our metrics for network performance.

The problem is compounded by having excess buffers damn near everywhere. They're in your router, in your ISP's terminal equipment, in your NIC hardware and probably in a few places in the kernel.

Probably something like TCP/IP does not assume that latency is a good measurement of available bandwidth. Over a high latency link it's not going to be a reliable indicator of anything. Packets can take different paths, arrive out of order, and all sorts of fun stuff like that.

The fact that latency may change randomly may mean that your running on a different path over the internet or a dozen other things. If your not losing packets it should be a safe indication that you can continue to send at the current rate*.

* That is unless your router has a gigantic bit bucket and can cache everything, of course. Maybe Linux should have a dial on it that says 'Ok you have hit 100% network bandwidth, now dial it back 20% to avoid potential issues if you see latency changes' or something like that.

So you end up using 1 more buffer per RTT. The RTT does continue to increase, and increase, until you've filled the buffer at the bottleneck (in each direction). You can see this in the TCP traces I took.

But having added soooo much latency (many times the natural RTT), the servo system that is TCP can no longer be stable. Bad Things Happen (TM). No servo system can be stable with those delays; things oscillate.

Remember, you should have seen a packet drop much sooner; you don't because of these bloated buffers. At some point, TCP thinks the path may have changed, and goes searching for a potential higher operating point, and goes back to multiplicative increase.

So the buffers then rapidly fill and drop lots of packets, and you get these horrible bursts (that you can see in my traces), with periods of order 10 seconds.

I still don't understand all of what I see in those traces; I need to sit down with a real TCP expert. Why the pattern is a double peak, I don't understand.

In a sense you are right. The TCP/IP congestion control algorithms used on the internet today ("reno", "newreno", "bic", "cubic") are all based on detecting dropped packets. Alternatives have been proposed, such as "vegas", that are based on detecting increasing latency when buffers fill up.

The problem is that nobody has figured out yet how to mix drop based algorithms with delay based ones: if vegas and cubic streams share the same connection, vegas backs off as soon as the buffers fill up but cubic keeps going until packets start dropping. This leaves the vegas streams starved - no good. So delay based algorithms have never been able to take off, given the lack of a clean transition mechanism.

Technically it seems easier to fix the drop based algorithms by making sure you don't let buffer grow too much before dropping the first packets; but this is a fix that needs to be deployed in all routers rather than just in the endpoints.

So ECN is not a panacea. I suspect I am bufferbloated by the buffers in my (ISP-supplied) zyxel ADSL router, but it appears impossible to inspect, let alone change, the size of those buffers. (There is no useful manual for this router's command line interface, so it is very hard to tell. Typical, really: hundreds of pages documenting the hopelessly uninformative web interface, five pages describing almost nothing about the command line. I could experiment, only if I get it wrong enough I could lock myself out of the router and lose my network connection forever. No way.)

(btw, when people figure out the fix for this, if it involves changes to traffic shaping, can someone package the whole thing up in something like the long-maintenance-dead wshaper? The whole traffic shaping stuff in Linux is woefully underdocumented: I tried reading lartc.org but it just left me more confused than when I started, and there appear to be no useful manpages yet, thanks Alexey. So I just leave wshaper to do its job and hope, even though bits of it appear to throw errors on recent kernels. Consider me a normal geek who isn't a TCP/IP specialist. If I can't do it, no non-geek is going to have a hope.)