Unladen Swallow 2009Q2 released [LWN.net]

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 20:54 UTC (Wed) by drag (guest, #31333)
In reply to: Unladen Swallow 2009Q2 released by rriggs
Parent article: Unladen Swallow 2009Q2 released

I like using forks and sockets for IPC.

Linux does 'COW' with memory for forks so that as the number of processing 'threads' go up the memory overhead does not match. The only unique code would be stuff that gets created after the fork, which is probably going to be unique to each process/thread anyways if you do your job right and load up the modules you need prior to the fork().

I don't know how helpful that approach would be. Probably not much, but it's what I like doing.

to post comments

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 23:33 UTC (Wed) by rriggs (guest, #11598) [Link] (1 responses)

When you have 16 worker threads all consuming work from a producer thread, the amount of coding you have to do for sockets (which I use as well) is a bit cumbersome (serializing the data, select() or poll() loops, waitpid(), etc.) given the clean interface one has with Queue and Thread.

Sometimes serializing the data (pickle/unpickle or what ever makes sense) takes up too much overhead.

Memory usage is rarely an issue. I've got enough RAM, enough CPU; I just need my favorite language to make the use of the resources I have a bit simpler.

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 11:28 UTC (Thu) by pboddie (guest, #50784) [Link]

When you have 16 worker threads all consuming work from a producer thread, the amount of coding you have to do for sockets (which I use as well) is a bit cumbersome (serializing the data, select() or poll() loops, waitpid(), etc.) given the clean interface one has with Queue and Thread.

There's a list of probably appropriate solutions for you on the Python Wiki; my humble offering, pprocess, tries to hide the awkward socket management behind fairly straightforward abstractions, as (I believe) do other solutions including the 2.6/3.x standard library offering, multiprocessing.

There are also libraries like Kamaelia which take a more radical approach to doing concurrent programming, and these certainly shouldn't require you to manage the tedious details.