Inferring TCP sequence numbers [LWN.net]

By Jake Edge
January 3, 2013

Over the past few years, seemingly harmless internal kernel state that is available to user space has allowed attackers to gain useful information to facilitate their attacks. The TCP DelayedACKLost counter (exported to user space via the /proc/net/netstat virtual file) is yet another. It can be used to determine the sequence number of a TCP connection, which can lead to packet injection or connection hijacking.

TCP sequence numbers have proven to be problematic over the years. Sequence numbers essentially count the number of octets transferred in each direction in a TCP connection. Knowing the sequence number, along with IP address and port number, allows an attacker to interfere with a connection by spoofing (or faking) packets that appear to have come from the other endpoint. Endpoints will reject packets that contain invalid sequence numbers, so those numbers must be known to an attacker. Originally, sequence numbers were easily predicted, so, in the mid-1990s, RFC 1948 specified that the initial sequence number (ISN) for each side of a TCP connection should be randomized.

Randomization should make sequence numbers difficult for attackers to guess, but various ways to infer the sequence numbers for a connection have come about over the years. In this case, researchers Zhiyun Qian, Z. Morley Mao, and Yinglian Xie discovered a way to quickly determine the sequence number for an open connection using the DelayedACKLost counter as a side channel. The researchers presented a paper [PDF] with their findings at the ACM Conference on Computer and Communications Security back in October. In addition, Qian posted a summary of the problem to the kernel netdev mailing list in December:

The vulnerability would allow [a] local malicious program to gain write access to TCP connections of other applications. An example attack scenario (on android) would be "an attacker uploads a seemingly benign app to the google play, when run at the background, it can inject malicious HTML payload into a webpage open by the browser".

The problem is caused by the common TCP stats counters (the specific counter I found is DelayedACKLost) maintained by the kernel (but exposed to user space). By reading and reporting such counters to an external attacker (colluded), the aforementioned attack can be accomplished.

As described in their paper, the researchers found a way to use the DelayedACKLost counter as a reliable side channel to quickly narrow in on the sequence number. By having a client-side application relay the counter value to a collaborating host, which can then send probe packets to the client, a binary or N-way search can be done to determine the sequence number. In their test cases, the researchers were able to infer the sequence number using four to five round trips between the client application and its collaborator, which could complete in as little as 50ms.

The key to this search is a bug in the way the Linux kernel handles packets with incorrect sequence numbers. If a packet is received that has a sequence number "less than" that which is expected, the DelayedACKLost counter is incremented—regardless of whether the packet is an acknowledgment (ACK) or not. The calculation that is done to determine whether the number is less than what is expected essentially partitions the 32-bit sequence number space into two halves. Because DelayedACKLost does not get incremented if the sequence number is larger than the expected number, it can be used in a search to narrow in on the value of interest.

DelayedACKLost was once used to decide if the network stack should use delayed ACKs or not. It is meant to count missing incoming delayed ACKs. In normal usage, it is rarely incremented, so an attacker can get a "clean" signal by simply sending packets with various sequence numbers and observing their effects on the counter. The paper describes using an N-way search technique that splits the sequence number space into a small number of equal-sized bins, sends a different number of packets with sequence numbers from each bin, and uses the value of the counter to find the sequence number with fewer round trips between the client and the probing host.

Beyond just inferring sequence numbers, though, the paper lays out a number of ways to use that information to interfere with real connections. The first of those is to inject data into an existing connection. By using the inferred sequence number, the "off-path" (i.e. not a man in the middle) attacker can spoof a packet to the client. The example used is an HTTP response from some internet server. If that server takes "long enough" to respond, the sequence number can be determined and a fake response can be sent before the real server responds. That turns out to be the pattern for Facebook, for example, and the researchers were able to inject JavaScript into a client session to update the user's status. In addition, if an HTTP connection is used for multiple requests, later responses (after the sequence number has been determined) can be spoofed.

A similar technique, using the same counter, can determine the client-side sequence number. That number can be used by an attacker to send spoofed packets in the other direction, so client requests to the server can be faked.

Beyond that, two types of connection hijacking are possible. A passive hijacking can be done on kernels older than 3.0.2 because there were only 24 bits of randomness in the ISN. When the connection is established to the remote server, the off-path attacker immediately sends spoofed "reset" (RST) packets to the server with sequence numbers covering the entire 24-bit client ISN space. It then commences to determine the server's ISN using the DelayedACKLost counter and sends a spoofed response once it has done so.

For more recent kernels, there is still an active hijacking that can be performed by pre-calculating the client ISNs for a range of port numbers. By "port jamming" the other ports (i.e. using those port numbers so they aren't available), the malware can ensure that the outgoing connection originates from a known, pre-calculated port. ISNs change at a known rate, so the attacker can calculate the ISN based on what it originally determined and how much time has passed since the determination was made. The trick is to know that a connection will be made to the server "soon". Ensuring that is the "active" piece of the puzzle.

Both hijacking techniques require that the server have some kind of stateful firewall that discards "out of state" packets. Otherwise, RST responses from the server on a connection it thinks is closed will confuse the real client (which thinks the connection is still open). It turns out that many internet services (e.g. Facebook, Twitter) do have such firewalls.

All of the techniques described in the paper were put to use in fairly "real world" scenarios. An Android application was used on the client side to monitor the counter, while other servers acted as the off-path collaborator. Many Android devices are based on 2.6.x, so they are vulnerable to either of the two hijacking techniques. Beyond the Facebook-status-updating malware mentioned above, they were able to create ways to "phish" for Facebook credentials using connection hijacking.

The fix for the problem on Linux is to check that the ACK bit is set on the packet before deciding that it is a lost delayed ACK. Attackers cannot just switch to turning on the ACK bit on the probes, because they would need the sequence number in the other direction to use as the "ACK number" in the packet. Eric Dumazet has posted a patch to discard the packets that would trigger this bug, but there is more needed. The longer-term fix will require moving most of the packet safety checks to an earlier point in packet handling. That way, fewer kinds of bogus packets will get far enough into the networking stack to cause this kind of observable state change.

One other possible solution would be to remove access to the DelayedACKLost counter for non-privileged programs. That would likely be difficult to do in Linux, as it has become part of the kernel's ABI, but it is something to consider for the future. In their paper, the researchers pointed that out:

(1) Even though OS statistics are aggregated and seemingly harmless, they can leak critical internal network/system state through unexpected interactions from the on-device malware and the off-path attacker. Similar observations have been made recently in a few other studies as well, e.g., using procfs as side channels. Our study reveals specifically that the packet counters can leak TCP sequence numbers. (2). Our systems today still have too much shared state: the active TCP connection list shared among all apps (through netstat or procfs); the IP address of the malware's connection and other apps'; the global packet counters. Future system and network design should carefully evaluate what information an adversary can obtain through these shared state.

There certainly are dangers from exposing internal kernel state to user space—sometimes those dangers don't manifest themselves for quite some time. Doing so has its benefits, though, for users and developers. It is a bit of a tricky balancing act—one that is made more difficult by the "no ABI changes" policy in the kernel. In this case, though, it seems that a solution has been found without changing the ABI. While the examples given in the paper may seem somewhat trivial, the techniques could certainly be used in ways that are far more damaging than an embarrassing Facebook status update.

(Log in to post comments)