Weekly Edition
Return to the Kernel page
| |||||||||||||
Increasing the TCP initial congestion window
The TCP slow start algorithm, initially developed by Van Jacobson, was one
of the crucial protocol tweaks which made TCP/IP actually work on the
Internet. Slow start works by limiting the amount of data which can be in
flight over a new connection and ramping the transmission speed up slowly
until the carrying capacity of the connection is found. In this way, TCP
is able to adapt to the actual conditions on the net and avoid overloading
routers with more data than can be accommodated. A key part of slow start
is the initial congestion window, which puts an upper bound on how much
data can be in flight at the very beginning of a conversation.
That window has been capped by RFC 3390 at four segments (just over 4KB) for the better part of a decade. In the meantime, connection speeds have increased and the amount of data sent over a given connection has grown despite the fact that connections live for shorter periods of time. As a result, many connections never ramp up to their full speed before they are closed, so the four-segment limit is now seen as a bottleneck which increases the latency of a typical connection considerably. That is one reason why contemporary browsers use many connections in parallel, despite the fact that the HTTP specification says that a maximum of two connections should be used. Some developers at Google have been agitating for an increase in the initial congestion window for a while; in July 2010 they posted an IETF draft pushing for this change and describing the motivation behind it. Evidently Google has run some large-scale tests and found that, by increasing the initial congestion window, user-visible latencies can be reduced by 10% without creating congestion problems on the net. They thus recommend that the window be increased to 10 segments; the draft suggests that 16 might actually be a better value, but more testing is required. David Miller has posted a patch increasing the window to 10; that patch has not been merged into the mainline, so one assumes it's meant for 2.6.39. Interestingly, Google's tests worked with a number of operating systems, but not with Linux, which uses a relatively small initial receive window of 6KB. Most other systems, it seems, use 64KB instead. Without a receive window at least as large as the congestion window, a larger initial congestion window will have little effect. That problem will be fixed in 2.6.38, thanks to a patch from Nandita Dukkipati raising the initial receive window to 10 segments. (Log in to post comments)
Increasing the TCP initial congestion window Posted Feb 10, 2011 3:59 UTC (Thu) by butlerm (subscriber, #13312) [Link] There are a couple of other things that should be done:First, the minimum retry time out should be reduced to approximately 1 ms, or whatever the timer accuracy allows. On local area networks a MIN_RTO of 1 second is ridiculous. Second, the syn retry timeout should be reduced from 1 second to 200 ms or so. The only reason to have it so high is if the bottleneck link is a 56 kbps modem or something. It would be great if there were route attributes that allow these numbers (or something that allows them to be derived) to be further customized on a route specific basis. In particular to increase them on the default route in the case of a unusually slow link, and to decrease them for the route to the local LAN.
Increasing the TCP initial congestion window Posted Feb 14, 2011 3:23 UTC (Mon) by ivannovick (guest, #72897) [Link]
Are you talking about TCP_RTO_MIN? I seem to believe this is hard coded in the kernel to 200 Milliseconds. Under the workload I have tested, huge gains were had by just changing this value to 10 miliseconds.
It would be great if this was configurable or lowered ... as the 200 miliseconds seems to cause a real bottleneck in some cases.
Does anyone know if this has been proposed or is being made configurable?
Cheers,
Increasing the TCP initial congestion window Posted Feb 10, 2011 6:49 UTC (Thu) by mtaht (✭ supporter ✭, #11087) [Link]
If google sent less data, it would also solve their problem nicely, without endangering TCP/ip.
The useful components in a google search result would fit into a single packet.
I would like it if google's search was more like DNS in structure and brevity.
Increasing the TCP initial congestion window Posted Feb 10, 2011 11:22 UTC (Thu) by cortana (subscriber, #24596) [Link]
I would like to see the opposite: a new DNS RR to allow DNS responses to carry advertising information!
Increasing the TCP initial congestion window Posted Feb 10, 2011 15:13 UTC (Thu) by jengelh (subscriber, #33263) [Link]
Google once was fast, now the major part of /index.html is just javascript fat.
Increasing the TCP initial congestion window Posted Feb 10, 2011 18:29 UTC (Thu) by mtaht (✭ supporter ✭, #11087) [Link]
As if breaking NXDOMAIN everywhere wasn't enough?
One DNS provider (name.com), in addition to returning ptrs to ad sites on missing domain names, is ALSO returning completely invalid pointers to IPv6 addresses for the same:
d@cruithne:~$ host no.such.subdomain.bufferbloat.net
I've been meaning to discuss this with them once I can summon the needed diplomacy and tact. So far, no luck.
Be careful what you wish for.
Increasing the TCP initial congestion window Posted Feb 10, 2011 22:14 UTC (Thu) by cortana (subscriber, #24596) [Link]
Of course, my suggestion was made in jest. However... given that ISPs are already breaking DNS by replacing NXDOMAIN responses with forged records... would we not all be better off if they were allowed to do so by the protocol? As long as there was a flag in each RR indicating that it is an advertising result, users of web browsers could opt in/out of receiving them, and other programs could ignore them altogether. At least we'd be in a better situation to that which we are in now. Of course, DNSSEC presumably fixes all of this, but not in favour of those who are incentivized to present the forged advertising responses.
Increasing the TCP initial congestion window Posted Feb 10, 2011 16:19 UTC (Thu) by tstover (subscriber, #56283) [Link]
spot on.
I'm still waiting for gopher 2.0 with, say, links in utf-8 text.
Increasing the TCP initial congestion window Posted Feb 10, 2011 17:15 UTC (Thu) by mtaht (✭ supporter ✭, #11087) [Link]
At the risk of over-promoting myself, I got fed up with google's default search a few months back and (re)wrote a command line client, called gnugol, to make my life simpler and faster - and far less annoying again.
It works the command line and in emacs.
The latency improvement is remarkable. The S/N ratio is nice too.
Originally, it was a client/server protocol that ran over udp, I've been thinking of resurrecting the server piece, possibly using sctp, as I think latency can be further improved with a persistent connection for search.
Google's search IS so fast that in many cases the RTT for initial TCP setup dominates the entire session - I can get a 50ms response for a query located 2ms from google, and 200ms (at best) for one from Colorado.
I would certainly like more people to give the idea and tool a shot.
Gnugol also supports bing and a few other search engines - I thought what I wrote here about stackoverflow's interface, vs gnugol's to be pretty damning.
http://nex-6.taht.net/posts/Screen_Space/
bufferbloat Posted Feb 10, 2011 22:28 UTC (Thu) by Lennie (subscriber, #49641) [Link]
Some time ago I would think this was a good idea. A small increase aint all that bad. But now that I know about bufferbloat, I think we have bigger problems to fix.
bufferbloat Posted Feb 10, 2011 23:34 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
actually, they are both problems, usually (but not always) impacting different situations.
If you are on high-speed connections, then the the congestion window issue is most significant for you.
if you have an overloaded link at some point in your path, then bufferbloat is your biggest problem.
bufferbloat Posted Feb 11, 2011 0:28 UTC (Fri) by Lennie (subscriber, #49641) [Link]
What I was thinking is, browsers open many connections to webservers on many different domains and when we increase this window we will be sending a lot of data at a time.
This could induce or increase the effect of bufferbloat
bufferbloat Posted Feb 11, 2011 6:31 UTC (Fri) by butlerm (subscriber, #13312) [Link] The reason why multiple connections are useful is not so much to give you an unfair advantage (over yourself?) in the congestion control wars - it is rather because HTTP over TCP is a single stream protocol, which causes the serialization of all requests and responses to/from the same server. That is not remotely ideal.Google has a proposal to fix this with a mid layer multi-stream protocol between HTTP and TCP. Even better would be a proper multi-streaming transport protocol like SCTP, but not so intelligent NAT devices appear to have killed that option for the time being. Meanwhile, Google Chrome appears to restrict the number of connections it uses pretty severely, which makes it noticeably slower than Internet Explorer on some applications which are not at all bandwidth constrained.
bufferbloat Posted Feb 11, 2011 10:08 UTC (Fri) by Lennie (subscriber, #49641) [Link]
Yes I know about SPDY, I don't see it happening any time soon though. Because the specification process seems to be going really slow. Atleast from what has been published.
Also it is a model which might not yet fit wel with some current webservers. It won't be an easy module to add to Apache for example. A proxy could work though.
Ironicly of all the currently deployed browsers, IE6 and IE7 from Microsoft are actually open the least amount of connections and the one with the most connections: ie8 (and I presume ie9)
Proxies, multi-streaming, wireless, bufferbloat Posted Feb 11, 2011 13:41 UTC (Fri) by mtaht (✭ supporter ✭, #11087) [Link]
This is where I think a proxy running over sctp to the gateway has some potential - it could run http over a multi-streaming protocol like sctp.
https://lists.bufferbloat.net/pipermail/bloat/2011-Februa...
bufferbloatnpn Posted Feb 12, 2011 6:53 UTC (Sat) by dtlin (✭ supporter ✭, #36537) [Link] Look forEnableSpdy in chromium/src/chrome/browser/browser_main.cc for 90% of Google Chrome users, the browser will switch protocols to SPDY whenever the server advertises the capability.All secure Google services advertise SPDY as an alternate protocol (using the next_protocol_negotiation extension of TLS). That's a lot of real-world users.
bufferbloatnpn Posted Feb 12, 2011 12:04 UTC (Sat) by Lennie (subscriber, #49641) [Link]
That is all very nice, but it is useless if it is not properly described and put forward as a standard. Atleast that one part is on the IETF site:
http://tools.ietf.org/html/draft-agl-tls-nextprotoneg-01
I'm talking about this one, it's just sitting there (atleast that is what it looks like to me):
http://www.chromium.org/spdy/spdy-protocol
Without something at IETF or something which looks mostly final why would other browser vendors and webserver people implement it ? Without atleast that, it is useless.
This is the last status:
http://groups.google.com/group/spdy-dev/browse_thread/thr...
Who knows, maybe this is the reason why:
http://groups.google.com/group/spdy-dev/browse_thread/thr...
He is just to busy and there isn't anyone else to help him ?
;-)
bufferbloatnpn Posted Feb 16, 2011 17:22 UTC (Wed) by mbelshe (guest, #72948) [Link]
Ha!
Well, I'm glad you guys are interested. I assure you that we're still hard at work on it. It just takes time.
This year, we'll be much more active and look to get IETF support. Unfortunately, some things just can't be rushed.
Mike
bufferbloatnpn Posted Feb 17, 2011 9:28 UTC (Thu) by Lennie (subscriber, #49641) [Link]
For something as basic as the use of TCP for HTTP, I prefer to do it right instead of working around the problems. HTTP has always had this 'bug' that it only allowed one request at a time over HTTP (pipelining exists but did not find widespread use because backwardscompatibility).
SPDY sounds to me like the right way, instead of adding more TCP-connections all with a larger initial windows size.
Bufferbloat is a different problem, but adding more TCP-connections with larger initial window sizes wouldn't help either.
SPDY seems to also be about delivering fast TLS. That would help a lot, we've seen to many problems with cookies getting picked up by others.
Maybe we don't want to use it for certain static files, atleast that is what people think when they see what is happening with Google SSL/Google Images. Maybe it is just a loadbalancer/TLS-sessions problem, it breaks TLS Session Resumption ?
If browsers that use SPDY are shown to be faster and people flock to it, it will also help spread the support for SNI. Although I think people who use IE on XP are probably not speedfreaks anyway.
But the more reasons people have not use IE on XP the better. :-)
bufferbloat Posted Feb 16, 2011 17:18 UTC (Wed) by mbelshe (guest, #72948) [Link]
The TCP congestion control heuristics are likely dated. Using an initial cwnd of 4KB is just too low for most users, and the data to increase this limit is compelling (see Chu/Dukkipati data). HTTP clients/servers are using multiple connections to achieve higher initial cwnds on the initial burst.
SCTP does centralize cwnd across all open streams, but it uses the same core algorithms as TCP and fails to address the initial burst. Unless the initial burst is increased, SCTP is slower than HTTP over lots of TCP connections out of the gate. This is measurable and demonstrable.
Oh, and SCTP doesn't work over NAT. So still waiting for IPv6.
Increasing the TCP initial congestion window Posted Feb 14, 2011 1:32 UTC (Mon) by showell (subscriber, #2929) [Link]
I agree TCP slow start is crucial part of the control mechanisms that exist in networking. Unfortunately these protocols have not kept up with the wireless revolution. This feature leads to bandwidth oscillation and buffer starvation in mobile data networks like WCDMA/HSPA and LTE.
Some optimisation of this protocol was started a while ago by Nortel prior to its breakup but I'm unaware of any further work.
It would benefit a large number of people if some smart TCP hacker could find a way to meet the congestion control required to keep our internetwork links running and at the same time optimise the wireless experience.
Increasing the TCP initial congestion window Posted Feb 19, 2011 21:56 UTC (Sat) by mtaht (✭ supporter ✭, #11087) [Link]
"It would benefit a large number of people if some smart TCP hacker could find a way to meet the congestion control required to keep our internetwork links running and at the same time optimise the wireless experience"
See thread at: https://lists.bufferbloat.net/pipermail/bloat/2011-Februa...
old fashioned web proxies and split tcp.
Increasing the TCP initial congestion window Posted Feb 24, 2011 14:15 UTC (Thu) by amh (guest, #1902) [Link]
Thanks for the reference! Several comments in this discussion were interesting, but I don't have the facilities here to do a google search on the various obscure names/etc (see?) which came up.
|
|||||||||||||