Weaponizing middleboxes

Middleboxes are, unfortunately in many ways, a big part of today's internet. While middleboxes inhabit the same physical niche as routers, they are not aimed at packet forwarding; instead they are meant to monitor and manipulate the packets that they see. The effects of those devices on users of the networks they reign over may be unfortunate as well, but the rest of the internet is only affected when trying to communicate with those users—or so it was thought. Based on some recently reported research, it turns out that middleboxes can be abused to inflict denial-of-service (DoS) attacks elsewhere on the net.

Though it lacks the catchy nickname and logo that have come to dominate security research reporting over the last few years, the "Weaponizing Middleboxes for TCP Reflected Amplification" web site describes a potent threat. The researchers, Kevin Bock, Abdulrahman Alaraj, Yair Fax, Kyle Hurley, Eric Wustrow, and Dave Levin, found flaws in the TCP/IP implementation of various middleboxes that allow DoS attacks via amplification. The middleboxes they studied are those deployed by government organizations for the purposes of censorship, but "even benign deployments of firewalls and intrusion prevention systems in non-censoring nation-states can be weaponized using the techniques we discovered".

It is important to note that the problems found are in the TCP/IP implementations of these middleboxes and not in the protocol itself. Those problems could be fixed, perhaps, but the researchers are skeptical that will happen:

Completely fixing this problem will require countries to invest money in changes that could weaken their censorship infrastructure, something we believe is unlikely to occur.

Background

Amplification is a technique used in denial-of-service attacks that allows an attacker to send a relatively small amount of data which causes a larger amount of data to reach the victim. Typically, this is done by spoofing the IP address of the victim while making a request to a service that will respond with a larger reply. The ratio between the sizes of the response and request is known as the "amplification factor"; historically, those have ranged from low single digits to as much as 50,000x for the memcached amplification flaw that was fixed in 2018.

In the past, DoS amplifications have used UDP because TCP requires a three-way handshake to establish a connection. Spoofing a victim's IP address and trying to establish a TCP connection does not work because the information in the SYN-ACK response from the destination (i.e. step 2 of the handshake) gets sent to the victim. That means that the attacker does not receive what is needed to complete the handshake, thus the connection never gets established. But the researchers found that the censorship middleboxes ignore the fact that there is no TCP connection established and respond (to the victim IP) with an HTML page indicating that the content is censored (i.e. a "block page").

Middleboxes are often not TCP-compliant by design: many middleboxes attempt [to] handle asymmetric routing, where the middlebox can only see one direction of packets in a connection (e.g., client to server). But this feature opens them to attack: if middleboxes inject content based only on one side of the connection, an attacker can spoof one side of a TCP 3-way handshake, and convince the middlebox there is a valid connection.

Methodology

In order to find packet sequences that would elicit the misbehavior, the researchers developed Geneva, which is a program that employs a genetic algorithm to "automatically discover new ways to evade censorship". The genetic algorithm is used to refine the results, "but at its core, Geneva is a packet-level network fuzzer". They gathered a list of 184 censorship middleboxes from around the world using data gathered by the Quack tool from Censored Planet. Then they turned Geneva loose with a goal of finding amplifications: "We modified Geneva’s fitness function to reward it for making the elicited response as large as possible, and then trained Geneva against all 184 middleboxes."

That fuzzing resulted in finding five packet sequences that would elicit amplified responses, as well as five ways to increase the amplification for a small subset of the middleboxes, which could be used in targeted attacks. The packet sequences abuse the TCP protocol in various ways, which should result in the packets being ignored, but the middleboxes tested would instead respond with a block page.

As might be guessed, these censoring middleboxes need to find something "objectionable" in the packet before they will respond with a block page. Using the data gathered by Quack, the researchers put together a list of the five most-blocked domains. As it turned out, those domains covered five different subject areas that governments like to censor (e.g. pornography, file sharing, social media). Those domains were used in well-formed HTTP GET requests in a packet (or packets) that caused the censor boxes to send the block page.

In a fairly eye-opening experiment—one that would, of course, be impossible for IPv6—the researchers scanned the entire IPv4 address space 35 separate times. They used the five domains of "interest", plus no domain and example.com as controls, with each of the five packet sequences and recorded the amplification found for each. Below is a graph from the web site that shows the amplification factors found:

Amplification

Generally, the amount of amplification from a country's censorship middlebox is fairly low, though there are exceptions. "Most of these nation-states are weak amplifiers (the Great Firewall of China only offers about 1.5x amplification, for example), but some of them offer more damaging amplifications, such as Saudi Arabia (~20x amplification)." Certain types of attacks can actually lead to effectively infinite amplification, however, either because of a routing loop where the time-to-live (TTL) value is not decremented correctly or one where the victim system's default response sustains the attack:

We found amplifiers that, once triggered by a single packet sequence from the attacker, will send an endless stream of packets to the victim. In our testing, some of these packet streams lasted for days, often at the full bandwidth the amplifier’s link could supply.

A factor that makes these attacks even more pernicious is that when the censorship middleboxes block a connection to a host, they spoof the IP address of that host as the source address in their response. That way, it appears that the response is coming from the host and not from the middlebox. Normally, an amplified response has the source IP address of the amplifier, which might make it easier to filter when a DoS attack is underway. That is not the case here:

The real challenge of nation-states is that their censorship infrastructure usually processes all traffic entering or exiting the country. This means that unlike other amplification attacks, where the source IP address of the traffic received by the victim is the amplifier itself, every IP address behind a middlebox can appear as the source of traffic. Said another way: every IP address within an amplifying nation-state can be an amplifier.

The full damage that attacker could do using this technique is not known. The amplification factor is only one part of the equation, as it also depends on the bandwidth available to the amplifier, which the researchers were unwilling to try to measure:

Unfortunately, this is the hardest to study ethically. To measure the maximum capacity of a given amplifier, we would have to completely saturate the link for each network, which could have real negative consequences on the users of that network. For now, the true capacity available to attackers from this attack is unknown.

This mechanism has not been seen in the wild, at least yet, but the researchers are working with companies that monitor and mitigate distributed DoS (DDoS) attacks in order to keep an eye out for it—and assist in mitigation. In September 2020, the flaws were reported to "several country-level CERTs [computer emergency response teams], DDoS mitigation services, and firewall manufacturers" in an effort at responsible disclosure. But, as noted, the changes needed to actually fix the problems would potentially threaten the reasons for deploying the middleboxes (i.e. censorship), so are rather unlikely to ever occur.

There is a lot more information available, including the full paper, which was presented at the 30th USENIX Security Symposium in August; it received a Distinguished Paper award from the conference. A YouTube video of the presentation is also available. Beyond that, there are GitHub repositories for the code used in the research and for Geneva.

[Thanks to Paul Wise for giving us a heads-up on this topic.]