Moving to Python 3 [LWN.net]

Moving to Python 3

Posted Feb 17, 2011 8:26 UTC (Thu) by rqosa (subscriber, #24136)
In reply to: Moving to Python 3 by kleptog
Parent article: Moving to Python 3

That way can scale poorly, because there must be at least one thread per FD.

Using an epoll-driven main loop and a pool of worker threads (with one work queue per worker thread) makes the amount of threads become independent from the amount of FDs, so you can adjust the amount of threads to whatever gives the best performance. It also has the benefit of avoiding the overhead of thread-start-on-FD-open and thread-quit-on-FD-close, since you can reuse the existing threads. (Make it so that any idle thread will wait on a semaphore until its work queue becomes non-empty. Also, rather than using epoll directly, use libevent, so that it's portable to non-Linux systems.)

(Log in to post comments)

Moving to Python 3

Posted Feb 17, 2011 8:41 UTC (Thu) by rqosa (subscriber, #24136) [Link]

> at least one thread per FD

Forgot to mention this in my previous post: the "one thread/process per FD" pattern is the main design issue that made possible the Slowloris DoS attack, which LWN covered 2 years ago.

Moving to Python 3

Posted Feb 17, 2011 20:50 UTC (Thu) by kleptog (subscriber, #1183) [Link]

I think you misread my post: I have a fixed number of FDs and a fixed number of threads. Each FD has a completely different purpose and protocol and so an event loop is not really practical. You have to get several different components of the system (which don't know of each other's existence) to work through a single event loop. Sure it's possible, but threads are a nice way to isolate them.

Of course in the general case you are right, a service like a webserver should try to reduce the number of threads. But also in the special case of CPython it's pointless to use more threads, since the GIL prevents more than one thread running at a time anyway.