TCP cookie transactions

In the currently ongoing Linux kernel merge window, for the kernel which will become 2.6.33, a new TCP feature has been added. TCP cookie transactions [PDF] are meant to eliminate various kinds of attacks, such as denial of service, while making the TCP connection handshake use fewer resources. One of the main motivations for cookie transactions is to avoid some problems that have cropped up in rolling out DNSSEC (Domain Name System Security).

DNSSEC responses are substantially larger than those of DNS, large enough that they have outgrown the default UDP datagram size of 512 bytes. UDP is generally used for DNS today, but large responses from DNSSEC over UDP result in multiple IP fragments. While it is perfectly reasonable to break up UDP packets that way, there are a large number of Network Address Translation (NAT) routers and firewalls that do not properly handle multiple UDP fragments.

When a DNS response is not received—or not received properly—a DNS resolver will typically retry the request over TCP. Because TCP is connection-oriented, there is a handshake that goes on to establish that connection before any data gets transferred. Normally, servers need to save some state between the two client packets that constitute the handshake. When handling an enormous number of requests, as the DNS root servers will for example, the storage of the state information adds up quickly. In addition, the well-known SYN-flood attack sends just the first packet of the handshake, often from a spoofed IP address, and never replies to the server to complete the connection. Enough "half open" connections can exhaust the server's resources, leading to a denial of service.

SYN cookies were created to defend against SYN flood attacks, and have been in the Linux kernel since 1997 when those attacks were raging. But, as Perry Metzger, William Allen Simpson, and Paul Vixie describe in their TCP cookie transactions (TCPCT) paper linked above, SYN cookies are only used when a system is under attack. They are a clever hack that uses the TCP sequence number to allow servers to defer using resources until they receive the second handshake packet from the client. Crucially, SYN cookies did not require client support, so they could be deployed unilaterally on the server side.

Various other mechanisms have been proposed to handle these problems over the years but, as outlined in the paper, failed to completely solve the problem. TCPCT sets out to do just that. It adds a new TCP option that contains a much larger, cryptographically secure cookie that is sent by the client in the initial handshake (SYN) packet. The server can then create a cookie for the reply that only it can decode. When the client uses that cookie in its second handshake (the third overall of the three-way handshake), the server can recover all of the information it needs to establish the connection from the cookie.

In addition, TCPCT allows for a limited amount of data to be sent in the request from the client and reply from the server, which allows for a query/response like DNS to be handled as part of the connection establishment. In those cases, the connection is torn down as soon as it is established.

TCPCT also addresses another problem that heavily used servers often have: port exhaustion. The TCP protocol requires that there be a timeout before port numbers are reused so that old messages that get delivered do not get confused with those of a newly-established connection. This is the TIME_WAIT timeout (usually four minutes) that is often annoying to those who restart server programs frequently (at least those without the SO_REUSEADDR socket flag). There are a limited number of ports available (nominally 64K, but at least 1K are reserved), an active server may have all of its free ports in the TIME_WAIT state. Because TCPCT can distinguish new and old connections based on the cookie data, it no longer has to wait on the server side. Only clients need wait out the TIME_WAIT period.

Obviously, TCPCT requires client support, and it will be some time before most operating systems have that support. As is often the case, Linux is out ahead of the pack by supporting TCPCT in the mainline. But even for Linux, it will be quite some time before 2.6.33 kernels make their way out to users via their distributions. Given that, widespread DNSSEC deployment seems quite a few years off, something that is a bit disheartening given all of the recent DNS server issues.