Gettys: Whose house is of glasse, must not throw stones at another

Reading the fine article is probably helpful here. I'm guessing that even if he thought he was discovering something new here, one of these people probably was able to fill him in. I guess.

Many of the puzzle pieces were handed to me (unassembled) by Comcast.

As always, we are on the shoulders of other giants: the area of congestion avoidance was explored with a depth of understanding I admire deeply by the likes of Van Jacobson, Sally Floyd, and many, many others. If I’ve done anything important here, it has been recognizing that the problem is occurring in other parts of the end-to-end system than “conventional” internet core routers, where it was pretty fully explored in the 1980′s and 1990′s.

And chance is very important: aiding me was knowing some of the players here, so that when I smelled smoke, they could diagnose the fire, giving me the confidence to dig deeper and look more widely. So in part, it’s being in a particular place at a particular time.
- Jim

Just to go through some numbers: you have a local network, with an RTT of < 1ms. Your remote app generally responds in < 10ms.

If you drop a SYN packet, the retransmit delay if you don't get a SYNACK is 3 *seconds*. You've now blown your expected response time by 300x.

If you drop the last packet in a request or response, you have to wait for the other side not to send an ack. The expected time to wait for an ack is supposed to be informed by the RTT estimator, except that the *minimum* (RTO_MIN) is pegged to 200ms in linux, so you have a 200ms delay in the request/response, even on a 1ms RTT network! You've now blown your expected response time by 20x.

Note that both those delays are impossibly large compared to the expected RTT. With those kinds of numbers, it's a Really Really Bad Thing to drop packets...it's no wonder people decide to buffer them for a few ms instead!

The only time there's no issue with dropping packets is when it's in the middle of a large data transfer stream: SACK will take care of that, and the packet will be retransmitted with no delay to the overall transfer.

There is some research about reducing (or eliminating) RTO_MIN -- let the retransmit time be informed purely by the RTT estimator. But one issue is that delayed-ack is a 40ms timeout (itself huge!), which means that reducing the RTO_MIN below that value will cause many packets to be sent twice (only twice, because an immediate ack is sent after the second transmission of the same packet is received). And, RTO_MIN is controlled by the sender, while delayed-ack is controlled by the receiver.

Sigh.

Dropping packets can be good - hence schemes developed to specifically drop packets. eg: RED, BLUE, BLACK, GREEN, PURPLE, WHITE. (Does anyone notice a pattern here?) If two packets would otherwise collide, dropping one rather than losing both will improve overall throughput.

However, the problem is to drop the right packets. Not all packets are equal. As far as I can tell, packet dropping schemes don't attempt to introduce anything but the simplest of bias. Which may indeed be all you can do. If analysis takes longer than the resend, you don't gain with the cleverer bias.

There are, of course, other mechanisms. ECN handles throttling, for example.

As far as over-large buffers are concerned, it is important to distinguish between two different methodologies - those buffers that try and buffer everything and those which are sub-divided somehow (one per conversation, one per classification, etc).

It's also important to distinguish between various buffering schemes (flush on fill, continuous drain, real-time, etc). If a buffer is polled regularly (so it acts as more of a queue than a classical buffer), or where there are certain guarantees as to the minimum and maximum time a packet can be held without either being sent or being dropped, then you've a very different behaviour than when the buffer has to be entirely filled and then dumped in one go.

As far as Linux is concerned, it has ample code for QoS and windowing controls. There's also the Web100 code for adding various tweaks. There is absolutely ZERO excuse for any distribution not providing a kernel that is sub-standard. If there is to be any laws passed regarding Internet traffic, I want it illegal to ship incompetently-configured networking software, with a mandatory minimum of 100 hours meta-moderating Slashdot.

Conclusion: "We believe that the buffers in backbone routers are much
larger than they need to be — possibly by two orders of magnitude." (2004!)

Another good starting point into this old and really vast literature:
http://www.cisco.com/web/about/ac123/ac147/archived_issue... (2006)

This is often hide on many computers because the Ethernet on their machine is not the bottleneck and therefore the traffic is queued elsewhere.

You don't see bufferbloat on Windows at 100Mbps, as they have deliberately tuned their TCP to never exceed about 85Mbps (XP, Vista, and Win 7), so in usual use on the most common switch still in use, Windows won't exhibit buffer bloat (since the ethernet bandwidth is higher than the NIC, the bottleneck moves somewhere else).

I suspect that Microsoft ran into bufferbloat, but didn't fully understand it. I'd love to talk to the right engineers and find out. Bufferbloat (from comments in one of their tech notes) appears to have played havoc with the responsiveness of media playing), and so they rate limited their network stack (by default). If you put much latency into the loop, the media player/media server control loop won't function well. Since usually in other situations the bottleneck is in some other device (home router or broadband device), they exonerated Windows and moved on.

Or maybe they were brighter than we are; but then I think they would have gone for a more general fix. The engineers/scientists there are really good, and they now have a first rate TCP stack (not true in XP days). So I doubt they would have stopped there if they had analyzed the situation correctly.

But, now that I actually understand what's going on, it turns out that for my situation a one *character* kernel change suffices! -- http://thread.gmane.org/gmane.linux.kernel.wireless.gener...

Man, this has been frustrating me for *years*. I hope it gets fixed for real soon!

Deep buffers means that the medium is always utilized as much as possible, which gives better throughput numbers at the expense of latency. But that latency doesn't really crop up until you try to use multiple, differing data streams.

My company's clients didn't like it when I pointed this out to them, but to quote a certain engineer, "I kinna change the laws of physics!"

Don't get me wrong, I'm glad to see this bit of sleuthing presented with such visibility, but... it seems to me like one of those dirty open secrets that everyone knew about but nobody completely thought through.

I'm not sure what we can really do about this though.. ("Oh no, this new kernel is slower than the old one, it sucks") ...at least until the PowersThatBe are made to care about multi-stream performance.

The only way to have good latency and throughput at the same time is QoS. Considering this is not yet a solved problem on your local workstation <http://lwn.net/Articles/404993/>, good luck with that on the Internet.

This being said, I tend to agree that the pendulum has been swung way too far in many devices/drivers. One simple way to put this excess into crude light and explain it in simple terms, even to non experts, is to stop describing queue sizes in terms of packets or bytes but in terms of milliseconds instead.

But my tests are not extensive, and I've heard of sporadic reports of much higher loss rates, so far non-reproducible.

Over a year ago, when I was last chasing home network disease in my house (documented in an older blog entry), my tests were showing much higher loss rates than the 1-3% I got out of the traces I've taken. Now, that may have been lightning damage in either my router's NIC or the cable modem; but then again, it might have been bufferbloat provoking TCP, if the timing was pessimal. I can't easily reproduce that test configuration, as all the hardware involved is now dead due to lightning.

Nick Weaver also reported to me seeing higher loss rates last summer when he reproduced my tests on his home service, but since then has not seen the tens of percent losses again (neither has he been testing).

So it is entirely possible that there is some state where TCP is yet less sane the buffers induce sometimes.

But to prove it and understand the cause, we need TCP traces. Without them, it is anecdotal evidence at best, and I carefully claim the documented 1-3% I've observed in my traces.

Please do investigate: if you can catch higher loss rates, *please* do a packet capture and let me know!!!!
- Jim

P.S. it has often been very difficult to monitor your overall home network, since you haven't been able to easily get a good tap between a home router and the broadband device. I stumbled across the following device which is a small switch which provides port mirroring and runs at a gigabit, and bought one (they are $150 or so). So I can finally watch all my traffic! Sometime soon, I'll try to see what all the spyware in web browsers are trying to do....

See: http://www.dual-comm.com/

Unfortunately, many devices have no provision for QOS, including much/all of broadband (this is certainly true for the cable modem/CMTS link).

So QOS is *not* a panacea; it may be a useful tool, but it is *not* sufficient by itself. Some sort of queue management is fundamentally necessary.

It is hard but not totally impossible, you can cheat TCP into some form of "inbound shaping". The trick is to artificially create an inbound bottleneck on your premises. Even when this bottleneck is downtream, as long as it is somewhat "narrower" than any other bottleneck on the path it will be the first to drop packets which will half the TCP congestion window of senders. By keeping the queue size of the artificial bottleneck small you can preserve latency.

Of course this does not work at all if you are hammered down by oblivious inbound UDP traffic.

> I see three real solutions:
> (a) uncontended bandwidth, so you can shape appropriately. anyone thinking *that's* going to happen for consumer internet is clearly deluded..

I am not sure what you mean by "uncontended". No matter where the bottleneck between A and B is, TCP is designed to reach it. See the Cisco reference I posted above.

> (b) some way of having the modem tell the router about backpressure (and yes, that's in the ethernet standards, but they never *do* it) and disable or reduce its buffering.

Backpressure is just pushing the problem one step further. And it typically adds a Head Of Line blocking problem when multiplexing.

> (c) have the modem itself support QOS levels and fair queueing.. then its buffers can be put to good use without screwing up latency. seems like an easy one until you realize that it's not that easy - you have to configure the right ports, protocols, etc,

I think the configuration complexity of this is what killed alternatives to IP. In other words, IP won because it did not care about QoS, in accordance with the End to End principle.

The story I've heard is that a particular Taiwainese ODM shipped a pile of code used in home routers a while ago that keels over and dies if it sees a ECN bit set. Various ISP's and big institutional networks and the like have been avoiding service calls by clearing the ECN bit.

Research is underway to see if we can go ahead and deploy it.

If we can, it's almost certainly the case that ECN should get marked anytime the buffers begin to grow (before packet loss occurs).

I've seen bufferbloat in many applications (including the X server).

Apps are just as prone to bufferbloat as the network.

Think end to end: you have to manage all buffers to control latency. The trick is figuring out how to manage them efficiently and correctly. This turns out to be subtle and sometimes hard, and there is some real research to do. It is particularly difficult when you are in situations where predicting the available bandwidth is difficult (e.g. wireless, graphics). One size never fits all.

- Jim

You can see in my traces periodic behavior; if there are tendencies (as there often are in large systems) for such behavior to synchronize, you can get really bad aggregate time based congestion behavior.

Best, of course that the underlying flows don't start exhibiting periodic behavior so they won't participate in such synchronization.

A future posting will be my fears for the future.

We've effectively destroyed congestion avoidance by this buffering, for TCP and other protocols.

So I have to fear possible congestion collapse. The more subtle form of this is this self synchronizing aggregate behavior.

And the traffic pattern of the Internet is now finally shifting as Windows XP finally gets retired. Just because we don't see (with the exception of 3g networks) bad congestion collapse today, doesn't mean we may not see it in the future, as the overall traffic shifts toward TCP's that run with window scaling more able to congest all links. And time based behavior is more subtle than the simple congestion collapse case.

Maybe I'm paranoid (having experience the NSFnet collapse), but I do lose sleep about it.
- Jim