Unladen Swallow 2009Q2 released [LWN.net]

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 23:33 UTC (Wed) by rriggs (guest, #11598)
In reply to: Unladen Swallow 2009Q2 released by drag
Parent article: Unladen Swallow 2009Q2 released

When you have 16 worker threads all consuming work from a producer thread, the amount of coding you have to do for sockets (which I use as well) is a bit cumbersome (serializing the data, select() or poll() loops, waitpid(), etc.) given the clean interface one has with Queue and Thread.

Sometimes serializing the data (pickle/unpickle or what ever makes sense) takes up too much overhead.

Memory usage is rarely an issue. I've got enough RAM, enough CPU; I just need my favorite language to make the use of the resources I have a bit simpler.

to post comments

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 11:28 UTC (Thu) by pboddie (guest, #50784) [Link]

When you have 16 worker threads all consuming work from a producer thread, the amount of coding you have to do for sockets (which I use as well) is a bit cumbersome (serializing the data, select() or poll() loops, waitpid(), etc.) given the clean interface one has with Queue and Thread.

There's a list of probably appropriate solutions for you on the Python Wiki; my humble offering, pprocess, tries to hide the awkward socket management behind fairly straightforward abstractions, as (I believe) do other solutions including the 2.6/3.x standard library offering, multiprocessing.

There are also libraries like Kamaelia which take a more radical approach to doing concurrent programming, and these certainly shouldn't require you to manage the tedious details.