Weekly Edition
Return to the Development page
| |||||||||||||
Rigo: Multicore Programming in PyPy and CPython
PyPy developer Armin Rigo describes
his vision for parallel programming in higher-level languages.
"We often hear about people wanting a version of Python running
without the Global Interpreter Lock (GIL): a 'GIL-less Python'. But what we
programmers really need is not just a GIL-less Python --- we need a
higher-level way to write multithreaded programs than using directly
threads and locks. One way is Automatic Mutual Exclusion (AME), which would
give us an 'AME Python'."
(Log in to post comments)
Rigo: Multicore Programming in PyPy and CPython Posted Aug 10, 2012 10:42 UTC (Fri) by dps (subscriber, #5725) [Link]
AME sounds like BSP (Bulk Synchronous Parallel), which divides programs into independent supersteps separated by global barrier and no communication except at the end of a superstep. There is an efficient free C library that implements BSP and does not require any compiler support whatsoever. It supports linux, windows, etc and TCP, MPI, shared memory, etc.
The determininistic nature of BSP avoids turning 5 line programs into 10 pages of monograph with the simplifying assumption that x=x+1 is atomic.
An implementation of BSP can actually do the communication earlier than the end of a superstep provided this does not affect anything until the next superstep. Many problems require a very small number of supersteps, so the impact of the global barriers is not severe.
I find it hard to buy the idea that AME can't be implemented in CPython.
Rigo: Multicore Programming in PyPy and CPython Posted Aug 10, 2012 12:41 UTC (Fri) by njs (guest, #40338) [Link]
BSP requires programmers to be explicit about communication. The goal of the STM stuff he's talking about is to preserve the shared-everything threaded memory model for inter-thread communication, and automatically detect and serialize these "communication events" (i.e., arbitrary memory accesses).
I have no idea why this is the goal, because shared-everything is horrible for programmers and implementors alike. I guess that makes it a fun engineering problem. It's like if Oulipo designed runtime environments.
Rigo: Multicore Programming in PyPy and CPython Posted Aug 10, 2012 16:13 UTC (Fri) by drag (subscriber, #31333) [Link]
> I have no idea why this is the goal, because shared-everything is horrible for programmers and implementors alike. I guess that makes it a fun engineering problem. It's like if Oulipo designed runtime environments.
Isn't the major advantage of threading versus forking in Linux the ability to have low-overhead IPC over things like shared memory and the such?
Otherwise what is the advantage over forking?
Just curious.
Rigo: Multicore Programming in PyPy and CPython Posted Aug 10, 2012 18:08 UTC (Fri) by njs (guest, #40338) [Link]
If we're using terms like BSP, STM, shared-everything, etc., then we're talking about what semantics your runtime provides, not how it's implemented in terms of OS primitives.
Right now Python on Linux has two easy options for parallelism. You can fork(), which gives you shared-nothing parallelism (which is great, very easy to reason about) with somewhat cumbersome and expensive IPC (serialized objects copied explicitly over sockets or shared memory, explicit chunk-of-shared-bytes via mmap, that sort of thing). Or you can use "threads", and get shared-everything parallelism (which is horrible) with fast IPC, except the GIL kind of kills that.
But you can imagine PyPy implementing fork()-like shared-nothing semantics *with* fast IPC. It'd use pthreads underneath, but with each thread having a totally different Python namespace, and when you passed an object to another thread it would just mark it copy-on-write instead of actually making a copy. This is how languages designed for parallelism, like Erlang or Rust, are intended to work.
Implementing this on top of CPython would be harder, and you'd have to have a whole transition period while people audited their existing C modules to adapt them to handle GIL-less operation and the new copy-on-write stuff, but it'd be possible. And seems more likely to me than designing a new C-compatible language like Armin is suggesting. I doubt it's worth it for a relatively minor IPC speedup, though...
Rigo: Multicore Programming in PyPy and CPython Posted Aug 16, 2012 14:55 UTC (Thu) by martin.langhoff (subscriber, #61417) [Link]
Yes! Having an Erlang-ish map() in Python, that transparently dispatches the work to a number of processes/threads would be fantastic.
Rigo: Multicore Programming in PyPy and CPython Posted Aug 10, 2012 12:31 UTC (Fri) by geertj (subscriber, #4116) [Link]
I've been passively following the PyPy project for many years now. Armin Rigo, the founder of the PyPy project, is one of the smartest software engineers that i've ever encountered in any open source project. He's a true visionary and if he is bullish on STM that means to me it has great potential.
Rigo: Multicore Programming in PyPy and CPython Posted Aug 17, 2012 10:32 UTC (Fri) by fdr (subscriber, #57064) [Link]
I will second this. There was a long period where PyPy was very slow, and he was quite assured in his vision that it would one day be fast, just give him some time. Years later -- I had lost some hope, I'll admit -- promise delivered.
Whether it is psyco (the specializing JIT for Python) or greenlet (the relatively crazy stack-copying coroutine library that is one of the underlying assets of gevent) or PyPy, every project he has undertaken has been drawn out *and* successful.
Maybe this one won't be. But if anyone has the smarts and the grit to make STM practical, it is Armin.
Rigo: Multicore Programming in PyPy and CPython Posted Aug 14, 2012 18:05 UTC (Tue) by mwsealey (guest, #71282) [Link]
Doesn't Stackless pretty much get the same thing done though?
|
|||||||||||||