Apache attacked by a "slow loris"
Posted Jun 24, 2009 16:17 UTC (Wed) by NAR (subscriber, #1313)
Posted Jun 25, 2009 9:52 UTC (Thu) by Darkmere (subscriber, #53695)
Posted Jun 24, 2009 16:20 UTC (Wed) by mcmanus (subscriber, #4569)
Obviously you can't use all of the vmm map for thread stacks - but you can use a lot of it. And if 32KB is not enough, then its a state problem not a threading problem (i.e. an async design scenario would need to stash the state somewhere too)). Plus apache has that hybrid process/thread model, so each process has its own vm map.
Maybe 100K threads is hyperbole if you're all in one process, but 50+K seems quite reasonable from a memory management standpoint and makes a thread much less of a scarce resource than it is in the default apache config (2 or 3 hundred iirc).
I'm more curious if Linux kernel/libc could handle the creation and scheduling of so many threads. If not then that's kind of a setback from the direction of things when 2.6 was being released (see the link I provided last post), and if it can it seems a much easier approach than rearchitecting the application.
Fundamentally, isn't this multiplexing the kind of thing the OS should be doing for you efficiently?
Posted Jun 24, 2009 16:31 UTC (Wed) by smurf (subscriber, #17840)
Apache is generally configured so that the maximum number of _real_ work threads (i.e. including all that state) doesn't cause the system to swap excessively. A slowloris connection eats much fewer resources than that, but Apache doesn't know that and thus reaches the configuration's limit far too quickly.
Posted Jun 24, 2009 17:03 UTC (Wed) by mcmanus (subscriber, #4569)
But it does seem to me that if what Apache is trying to prevent is memory exhaustion, then it should be doing admission control based on memory allocated instead of using max clients as a poor stand-in. Especially as the two don't correlate well at all even in normal (i.e. non DoS) situations.
Posted Jun 25, 2009 5:29 UTC (Thu) by quotemstr (subscriber, #45331)
Now, you can increase these limits in order to defeat the attack. There's the problem: Apache's resource consumption can be too high to service the number of simultaneous connections required. For prefork servers, the problem is especially severe: each simultaneous request requires its own process. Now, any serious operating system can handle thousands of processes; that's not the problem. You can even have thousands of Apache processes, since Apache workers are actually pretty light, memory-wise, and all the code and some of the data structures are shared among all workers.
More an issue is the mod_* idiocy that embeds an interpreter in each worker process*. Then, when you have thousands of processes, you run into a problem far more severe than virtual address space exhaustion: swap thrashing, poor performance, machine lockups, and OOM killing sprees.
Using threads, in principle, helps the problem. Each thread shares the same process, meaning the additional memory occupied by each new simultaneous connection is just the memory needed to keep track of that connection; if all the threads share a single copy of the interpreter and other data structures, I bet you can avoid the slowloris problem entirely by simply setting connection limits high enough.
A small variant of this approach is to use multiplexed I/O, as in lighttpd; that's like using the threaded approach above, but you don't need a thread stack for each simultaneous connection. In practice, if you make the thread stack small enough, you can still fit thousands in a 32-bit virtual address space.
The problem with threads is that you need threads, though! The free world's most inexplicably-popular web scripting language, PHP, doesn't have a thread-safe interpreter. That limitation us to another solution: divorce the heavyweight state from the HTTP handler, and use something like FastCGI to communicate to the interpreter. That way, you can have as many (cheap) HTTP connection handlers (threads, processes, or state machines) as you'd like, and still limit the number of heavyweight interpreters you need. (Using a separate processes for the heavy lifting is better than mixing all the threads together anyway; see below.)
To deal with slowloris attacks, just batch all the headers up in a request before sending the whole request on to the heavyweight process that actually does something meaningful with it. (That's what mod_fastcgi does.) That way, a slowloris-using attacker can pound away at the lightweight worker thread (or process), and only when a complete request is read will the system actually commit to using a significant resource, one of the heavyweight FastCGI servers.
lighttpd can't use embedded interpreters; it's state machine precludes that. Instead, people generally set it up to use FastCGI: guess why lighttpd is not vulnerable to slowloris attacks. If you configure Apache appropriately, you can make it work similarly.
(Before you say, "but wait! FastCGI is pretty slow!" -- that's simply not true. The HTTP user-agent's communication to the server is orders of magnitude slower than the local communication between Apache and a FastCGI server. The FastCGI communication doesn't add any meaningful time to each request, so it doesn't increase latency. And since each application is sequestered into its own process instead of running in each worker process in Apache, total memory use can actually be lower than the mod_* model.
As if that weren't a good enough reason to use FastCGI (or something similar, like scgi), it's also a lot easier to keep track of your web application's system resource usage as distinct from the web server's. You can actually measure the CPU consumption of, say, squirrelmail without conflating it with Apache's.)
Oh, and there's an even better reason: nothing forces a FastCGI server to run as the same user as the web server. Finally, you can put an end to anything remotely related to the web having to be owned by apache. Each application can have its own user, and be limited to its own little corner of the machine just like any other kind of network-facing daemon. Really, mod_* is just an ugly hack that's completely unnecessary today.)
Copyright © 2017, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds