Gettys: Bufferbloat demonstration videos

That sounds better than the current situation where we have retransmissions and we're *not* dropping the packets. That's waste.

I'm not understanding your problem though. If packets are being dropped fairly, then everything is fine. Suppose your bulk transfer transfers 100 times as much data as your voip connection then there would be 100 times as many packets dropped in your bulk transfer than your voip connection, which seem like it shouldn't be a problem, because the transfer will slow down till there are no drops, at which point your voip connection has no drops either.

But you're right, if you want to guarantee 0.00000% loss, then the only solution is to not saturate the pipes.

Remember, the more streams going through a pipe, the smaller the buffer can be, because any loss will only affect each stream slightly.

Well, that's why we need proper AQM at the home router, and at ISP's routers.

It's fairly trivial to prioritize VOIP, DNS, and gaming traffic so they get low latency queues and rarely dropped packets. After all, these low latency sources of traffic actually consume very little in bandwidth (though they do consume packet space!).

I do this already with my home router running a custom version of openwrt. I can prioritize certain classes of traffic to effectively never drop, while punishing long lived, bursty traffic by causing it to drop, or causing it to get held in the queue.

At my home network, bufferbloat isn't really an issue with small enough queueing as to never occur a large latency.

But, my ISP's provided modem has its own buffering that I can't do anything about, except to reduce my own upload and download below that of my provisioned speed.

After all my tweaking, I managed to get my buffering down to about 150ms at the worse loaded case, as before it could get into the seconds.

I'm excited to see development of home router firmware that should "just fix" most of the offending problems at home. Also those changes are going into the Linux kernel, so perhaps servers in time will adopt those techniques.

actually, VoIP and many other multimedia protocols tend to handle gaps in the data better than they handle significant delays.

This is why the ATM protocol was designed with no retransmission capability and buffering was strongly discouraged for example.

I think the idea (please someone correct me if I'm wrong) is that reasonable packet loss/retransmission actually functions as a very crude form of queue management. Even time sensitive applications like VOIP can tolerate some packet loss, but they cannot tolerate huge packet delay being induced by buffer bloat.

Obviously, what is "reasonable" packet loss depends on a lot of things, including the application, it's required maximum latency, and how much delay the buffer bloating actually introduces.

> I fear this is a no win situation, as reducing buffers may well cause the routers and such to drop old packets and cause retransmissions. Not a major issue for file transfers, it will just slow down. But voip or similar will start to stutter once the packet drops becomes high enough.

As I understand it, this is exactly this idea that packet drops are Evil (capital E Evil) that led to bufferbloat. Packet drops are like pain, a warning signal that you may be doing something that may harm you. In case of TCP it means you're going too fast. So it may sound undesirable, and apparently many people thought so, but you don't want to stop feeling pain altogether.

While it might have been nice, 25 years ago, to foresee this and make the congestion avoidance algorithm delay sensitive, I fear we can't get there from here (for game theory reasons). I'd love to be wrong, but I suspect any delay sensitive transport is going to lose relative to a non-delay sensitive transport. So I don't see how to get there from here.

The fundamental issue here is allowing the buffers (of whatever size) to get filled in the first place. Then it doesn't matter what size the buffers are. AQM can prevent that; but RED won't work in many places we have to deal with today, and we don't have a replacement for it yet. There is hope here, but not quite ready to try to do code...

In the meanwhile, understand that the default buffer sizes are often hugely too big (say, by an order of magnitude), and tuned for the fastest the hardware can ever go (when you are actually using it at a fraction of its possible bandwidth). Even simple mitigation can help alot.

> it's to replace TCP with one of the (phenomenally successful) alternatives that doesn't depend on packet loss, or proxies for it, as an indicator of congestion.

Is there such an protocol, ie one that gives us something beyond what ECN + TCP already provides, and preserves the basic IP tenet of putting the intelligence at the edges? What is it?

It is true the ECN hasn't been deployed widely enough to be reliable. But as jg says here, the reason isn't technological, it's to do with incentives of deploying it versus not deploying it - which can be analysed using game theory. But we haven't lost all hope of implementing it. We have an inflection point coming up, IPv6. Hopefully all IPv6 capable routers will implement ECN reliably.

Using packet loss to tell a sender to slow down is the only viable option, because packet loss is the only thing that imposes pain on the sender. Anything that does not impose pain on the sender can and will be ignored by a sender that wants to consume an unfair proportion of the link.

The alternative is delay based TCP algorithms (Vegas and its descendants). These have never been widely deployed for two reasons: first, any delay based algorithm loses in a fight for bandwidth against a loss based algorthm. The good guy (Vegas) ends up losing to the bad guy (Reno) in any contended queue in the core routers. Secondly, delay based algorithms are too sensitive to other traffic on the reverse path or normal delays (like IRQ coalescing). The Round Trip Time measurement has a lot of variation and attempts to filter than variation out is hard except in the case of long lived flows (ie benchmarks).