A viable solution for Python concurrency

Posted Oct 15, 2021 1:18 UTC (Fri) by njs (subscriber, #40338)
In reply to: A viable solution for Python concurrency by fw
Parent article: A viable solution for Python concurrency

This has been a major goal of a lot of cpython contributors for maybe 5? years now. Lots of internal refactoring, etc. See PEP 554.

I don't really see the point – subinterpreters aren't meaningfully more efficient than subprocesses, and they're way more complicated and fragile. But some people are more hopeful.

A viable solution for Python concurrency

Posted Oct 15, 2021 1:35 UTC (Fri) by njs (subscriber, #40338) [Link] (1 responses)

Also, the biggest challenge for GIL removal is legacy C extensions. And C extensions are inherently shared across the whole process.

For python code, the VM itself can take care of isolating subinterpreters from each other – for python code this is "free".

But for C code, each C extension has to manually implement subinterpreter isolation for its own internal state. So here, the subinterpreter model doesn't really simplify anything – in fact it makes it more complicated.

A viable solution for Python concurrency

Posted Oct 15, 2021 3:19 UTC (Fri) by NYKevin (subscriber, #129325) [Link]

Well, it depends on how those C extensions have been implemented. If they have large amounts of global, mutable state, then this is going to be a nightmare because you will need to make separate copies of that state for each subinterpreter, and then you have to plumb context pointers into everything, which is probably not easy.

As far as I can tell, the most likely problem for "well designed" C extensions (those which do not have substantial global mutable state) is the PyGILState_* API, which doesn't (currently) support sub-interpreters (and by my read, it is unlikely to gain such support without an API break, because there's no plausible way for it to infer which PyInterpreterState* you want it to use). You mostly only need to use those functions if you're calling into the Python interpreter from foreign threads (i.e. threads which were not created by Python). The simplest solution is probably for the Python people to provide a version of those functions which accepts an explicit PyInterpreterState* argument. Currently, for example, PyGILState_Ensure() just grabs "the" interpreter state out of a global variable.[1] This still would require some plumbing on the C extension side, of course, because extensions would still need to actually pass those arguments. In the meantime, you can use PyThreadState_New() and friends to do the same job by hand, but it's a lower-level API and more cumbersome to work with (especially if you want to tackle the reentrancy issues which PyGILState_* is intended to solve for you).

[1]: https://github.com/python/cpython/blob/main/Python/pystat...