LWN: Comments on "TCP window scaling and broken routers"

TCP window scaling and broken routers

PolyPeter — Thu, 11 Feb 2010 10:23:00 +0000

After an automatic Windows Vista update at the local pc the uploads works! And can just conclude that Windows Vista is really crappy!

TCP window scaling and broken routers

PolyPeter — Mon, 18 Jan 2010 11:01:35 +0000

What I did was of course:
echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
(turning OFF tcp window scaling)

TCP window scaling and broken routers

PolyPeter — Mon, 18 Jan 2010 10:49:13 +0000

My setup

I have developed web application, now with more than 1000 members.
The web application runs on my own server with Linux Redhat.
The members can upload images (jpeg) to a gallery.

My problem

3 members complain that they cannot upload images. On other web pages they can upload images just fine. They all have a Windows Vista machine.
The upload works just fine from the members network if they use Windows XP.
When uploading we have tested with af flash component where its possible to follow the uploaded bytes. The upload stops running at app. 64KB every time (64KB is described in the article!).
Other members with a Windows Vista machine does not have a similar problem

My conclusion

The upload problem only exist if the member have a broken router and at the same time have OS where TCP Window Scaling is enabled by default (for example Vista)

A partial solution

After I found this article i tryed did this:
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling

After I did that, it solved the problem for one of the members! But only one. The two other members still have the problem...
Now I wonder what else I should do?

Hotel access systems

zlynx — Wed, 09 Apr 2008 17:02:07 +0000

Microsoft went to quite a lot of trouble to auto-detect network routes which fail to work with
window scaling, IPv6 and whatever else, and work around the problems.

Linux relies on the user to set things.  I believe there are some iptables rules you can apply
to selectively disable scaling and ECN.  However, I don't know of any distros with scripts
that detect problems and apply those rules automatically.

Hotel access systems

shemminger — Wed, 09 Apr 2008 04:27:12 +0000

A common vendor of hotel access control systems (ibahn) seems to be particularly problematic.
I have already experienced that in 3 separate cases the wireless or wired access does not work
with window scaling.

Probably doesn't work with Vista either, unless Microsoft has turned off window scaling in
Vista SP1.

TCP window scaling and broken routers

shemminger — Thu, 07 Feb 2008 21:12:51 +0000

The problem is firewall's that want to enforce window sizes but are too stupid and try to do
this without tracking the state of window scaling of the connection.

I will pick out OpenBSD as particularly broken in that regard, and they haven't fixed it.

TCP window scaling and broken routers

jsl123 — Thu, 18 Oct 2007 15:35:55 +0000

Just for the records...

Vista has the same problem too! (Found this with broken ssh connections)
A possible solution is described here:
http://www.tech-recipes.com/rx/1744/vista_tcp_cannot_comm...
(beware of line breaks in the above URL)

HTH. Salut, Joerg

TCP window scaling and broken routers

dcam — Tue, 21 Aug 2007 07:17:02 +0000

Putting net.ipv4.tcp_window_scaling=0 into /etc/sysctl.conf and rebooting worked for me

TCP window scaling and broken routers

ssabchew — Mon, 16 Jul 2007 08:01:42 +0000

I made an work around as I decrease the TCP outgoing buffer:
from
net.ipv4.tcp_rmem = 4096 87380 4194304

to this
net.ipv4.tcp_rmem = 4096 87380 207520

TCP window scaling and broken routers

pcharlan — Thu, 14 Dec 2006 00:15:48 +0000

With kernel 2.6.17.13 or higher, you can also do:

THEIR_IP=1.2.3.4
MY_GATEWAY=5.6.7.8

ip route add $THEIR_IP/32 via $MY_GATEWAY window 65535

which only limits window scaling for that destination without interfering with your other connections.

[It has been a while since the original article, but this still shows up first in Google when searching for "linux tcp window scaling broken router", so perhaps this will help someone.]

TCP window scaling and broken routers

walken — Thu, 01 Dec 2005 23:06:23 +0000

That sounds like a good idea, but - is there any way to get iptables to do what you describe ? From my own little netfilter experience, I know how to pass, drop or reject packets, but not how to filter bits (well, I think there is an option to do that with ECN, but what about OTHER must-be-zero bits) or how to drop arbitrary unknown tcp options.

Sounds a bit hypocritical for linux developers to complain about firewalls in the field if their own firewalling functionality does not allow this either.

But then again I'm not a netfilter expert so I could be mistaken.

TCP window scaling and broken routers

gene_wood — Fri, 10 Dec 2004 00:02:38 +0000

I just wanted to thank the author for detailing this phenomenon. I've been banging my head against this for about 2 months. I implemented the workaround you described and everything works perfectly.

The symptoms that I was experiencing are detailed here.

TCP window scaling and broken routers

schabi — Wed, 21 Jul 2004 18:31:05 +0000

"this case the nasty firewalls zero out the bits in the unknown option and people are complaining"

It's different. With ECN, the router had two different, valid options: Leave the bits in the flag word as they are, or clear them and thus deleting the option. ECN was designed carefully enough that both ways worked. Blocking or dropping the packed is no option.

The Window scaling is not bits in the flag word, but an separately added option field. There, the firewall has two valid options: let the packet pass as it is, or remove the window scaling option field entirely. Communication continues to work with both options. Fiddling around inside the header field and wildly mangling the values is no option.

TCP window scaling and broken routers

ron.flory@adtran.com — Mon, 12 Jul 2004 20:42:22 +0000

careful: hacker != cracker

hacker (hobbyist) is not necessarily bad.
cracker is always bad.

Please see: http://info.astrian.net/jargon/terms/h/hacker.html

No

Ross — Sat, 10 Jul 2004 02:53:10 +0000

You misunderstand the standard: "must be zero" means that they must be
set to zero for software adhering to this version of the standard. They
absolutely must be ignored by software adhering to that version of the
standard. If they are dropped when not zero then the implementation is
broken because it is not upwards-compatible with other implementations.
TCP/IP is specifically designed to be upwards compatible. If you don't
understand options you are supposed to ignore them, otherwise there is
no point in having an extendable protocol.

But in any case, even for those who can't read RFCs properly, there was a
draft RFC at the time and it's now official. So there are absolutely,
positively no excuses anymore but there are still lots of broken vendors
and even more unpatched routers and firewalls (one only wonders what
security problems these systems must have).

Does Linux have that much clout?

giraffedata — Sat, 10 Jul 2004 00:45:34 +0000

It has been a few years, now, that Linux has a strong enough presence in the networking world that it can get away with taking this sort of position

Even if Linux in general has a strong enough presence, does kernel.org Linux have it? The rejection in question is only for kernel.org Linux, which almost nobody runs. The Linuxes that matter -- Red Hat, Suse, etc. probably will not follow suit, since their customers want their computers to talk the the existing Internet more than they want to take a stand against bad router owners.

TCP window scaling and broken routers

dlang — Sat, 10 Jul 2004 00:19:20 +0000

let's see with ECN people were up in arms about those nasty firewalls that blocked backets that did unknown things with the undefined bits in the header (even before ECN was an approved spec) and said that they should have zerod out the bits instead of blocking the packet.

in this case the nasty firewalls zero out the bits in the unknown option and people are complaining

putting these two togeather it sounds like what people really want is for the firewalls/routers to just let everything through and not try to enforce anything.

why am I not surprised that this doesn't "just happen"

This sounds a lot like the IE / Mozilla / Opera / etc dilema to me

HeathPetersen — Fri, 09 Jul 2004 19:37:11 +0000

For a few years now, I've expected web sites to support non-IE web browsers. If that hasn't happened, why should this?

Don't get me wrong, I am outgraged at the commercial entities forcing these things down out throats. I just don't expect quick resolutions to their errors.

Broken routers and firewalls

Ross — Thu, 08 Jul 2004 20:15:57 +0000

"If ... the situation is left as it is, pressure on the router
manufacturers should get the problem fixed relatively quickly.
... Linux has a strong enough presence in the networking world
that it can get away with taking this sort of position."

Really? Is that why all of the broken firewalls stopped blocking packets
with ECN bits? Well, all of them except for a few tiny obscure places like
Sun, Sprint, CitiBank, Cornell, SAE, ISOC, Iomega, US DoJ, Wells Fargo, and
Checker Auto Parts :)

But seriously, while I hope this does force vendors to fix their broken
code I just don't have a lot of faith that it will work.

I _still_ find websites behind broken firewalls which stop all ICMP
packets, including "must fragment" errors. This doesn't just affect Linux
users. Well I can't reliably visit some of those sites (iptables PMTU
clamping helps considerably). The same thing with ECN. I once went to the
trouble of actually calling a network admin at Southwest Airlines to help
them fix the problem. It worked, in less than one week they had patched
their router, but now it is broken again.

The basic problem is that it doesn't affect them, and they have little
incentive to fix it. There's no clear communications channel to get the
information to the people who need it.

If you want to report ECN problems here's a good resource:
http://urchin.earth.li/ecn/
(follow the link to the "ECN Hall of Shame")

TCP window scaling and broken routers

ekj — Thu, 08 Jul 2004 07:49:57 +0000

64KB, the max with zero scaling ain't going to be enough for good performance in many situations.

For example, if your ping-time to the destination is 250ms, which is fairly typical for me for accessing US sites from Germany over DSL, then a 64KB window is going to be full if my available bandwith is 2Mbit/s or more. That's not a particularily rare condition.

When downloading from my Uni (which has triple redundant 622MB/s atm-links) the limiting factor is typically the 100Mbit ethernet, at that speed a 64KB window would be full for any transfer where the ping-time is more than 5ms, which is essentially everywhere.

I'm with the kernel-hackers on this one. A router that doesn't handle larger window-sizes properly is broken and needs to be fixed or replaced as soon as possible, anything else will just be band-aids.

TCP window scaling and broken routers

Baylink — Thu, 08 Jul 2004 04:24:17 +0000

> It has been a few years, now, that Linux has a strong enough presence in the networking
world that it can get away with taking this sort of position.

I wonder if the same thing is true of the hackers who maintain the large clusters of said
machines: can we 'down tools and tell em go to Hell' if they persist in misnaming criminals as
hackers? :-)