Continued attacks on HTTP/2

On April 3 security researcher Bartek Nowotarski published the details of a new denial-of-service (DoS) attack, called a "continuation flood", against many HTTP/2-capable web servers. While the attack is not terribly complex, it affects many independent implementations of the HTTP/2 protocol, even though multiple similar vulnerabilities over the years have given implementers plenty of warning.

The attack itself involves sending an unending stream of HTTP headers to the target server. This is nothing new — the Slowloris attack against web servers using HTTP/1.1 from 2009 worked in the same way. In Slowloris, the attacker makes many simultaneous requests to a web server. Each request has an unending stream of headers, so that the request never completes and continues tying up the server's resources. The trick is to make these requests extremely slowly, so that the attacker has to send relatively little traffic to keep all the requests alive.

In the wake of the Slowloris attack, most web servers were updated to place limits on the number of simultaneous connections from a single IP address, the overall size of headers, and on how long the software would wait for request headers to complete before dropping the connection. In some web servers, however, these limits were not carried forward to HTTP/2.

In 2019, there were eight CVEs reported for vulnerabilities exploitable in similar ways. These vulnerabilities share two characteristics — they involve the attacker doing unusual things that are not explicitly forbidden by the HTTP/2 specification, and they affect a wide variety of different servers. Web servers frequently have to tolerate clients that misbehave in a variety of ways, but the fact that these vulnerabilities went so long before being reported is perhaps an indication that there are few truly broken clients in use.

Two of these vulnerabilities in particular, CVE-2019-9516 and CVE-2019-9518, can involve sending streams of empty headers, which take a disproportional amount of CPU and memory for the receiving server to process compared to the effort required to generate them. Nowotarski's attack seems like an obvious variation — sending a stream of headers with actual content in them. The attack is perhaps less obvious than it seems, given that it took five years for anyone to notice the possibility.

Continuation flooding

HTTP/2 is a binary protocol that divides communications between the client and the server into frames. Headers are carried in two kinds of frame: an initial HEADERS frame, followed by some number of CONTINUATION frames. The continuation flood attack involves sending a never-ending string of continuation frames, with random header values packed inside them. Because they are random, these headers are certainly not meaningful to the receiving server. Despite this, some servers still allocate space for them, slowly filling the server's available memory. Even servers which do place a limit on the size of headers they will accept usually choose a large limit, making it relatively straightforward to consume their memory using multiple connections.

Another wrinkle is that HTTP/2 requests are not considered complete until the last continuation frame is received. Several servers that are vulnerable to this attack don't log requests — failed or otherwise — until they are complete, meaning that the server can die before any indication of what happened makes it into the logs. The fact that continuation flooding requires only one request also means that traditional abuse-prevention tools, which rely on noticing a large number of connections or traffic from one source, are unlikely to automatically detect the attack.

Nowotarski listed eleven servers that are confirmed to be vulnerable to the attack, including the Apache HTTP Server, Node.js, and the server from the Go standard library. The same announcement stated that other popular servers, including NGINX and HAProxy, were not affected.

It is tempting to say that all these attacks — the eight from 2019, and now continuation flooding — are possible because HTTP/2 is a complex, binary protocol. It is true that HTTP/2 is substantially more complicated than HTTP/1.1, but every version of HTTP had had its share of vulnerabilities. The unfortunate truth is that implementing any protocol with as many divergent use cases as HTTP is difficult — especially when context is lost between designers and implementers.

The designers of HTTP/2 were well aware of the potential danger of DoS attacks. In July 2014, Roberto Peon sent a message to the ietf-http-wg mailing list talking about the potential for headers to be used in an attack:

There are three modes of DoS attack using headers:
1) Stalling a connection by never finishing the sending of a full set of headers.
2) Resource exhaustion of CPU.
3) Resource exhaustion of memory.

[...]

I think #3 is the interesting attack vector.

The HTTP/2 standard does not set a limit on the size of headers, but it does permit servers to set their own limits: "A server that receives a larger header block than it is willing to handle can send an HTTP 431 (Request Header Fields Too Large) status code." Yet despite this awareness on the part of the protocol designers, many implementers had not chosen to include such a limit.

In this case, fixing the vulnerability is relatively straightforward. For example nghttp2, the HTTP/2 library used by the Apache HTTP Server and Node.js, imposed a maximum of eight continuation frames on any one request. However, this vulnerability still raises questions about the security and robustness of the web-server software we rely on.

HTTP/2 is a critical piece of the internet. It accounts for somewhere between 35.5% and 64% of web sites, depending on how the measurement is conducted. There are several tools to help implementers produce correct clients and servers. There is a publicly available conformance testing tool — h2spec — to supplement each individual project's unit and integration tests. nghttp2 ships its own load-testing tool, and Google's OSS-fuzz provides fuzz testing for several servers. These tools hardly seem sufficient, however, in light of the ongoing discovery of vulnerabilities based on slight deviations from the protocol.

The continuation flood attack is not particularly dangerous or difficult to fix, but the fact that it affects so many independent implementations nearly nine years after the introduction of HTTP/2 is a stark wakeup call. Hopefully we will see not only fixes for continuation flooding, but also increased attention on web server reliability, and the tests to ensure the next issue of this kind does not catch us by surprise.