LWN: Comments on "Subinterpreters for Python"

Subinterpreters for Python

NYKevin — Sun, 17 May 2020 06:20:30 +0000

My apologies, it was not my intent to blame either the extensions or the core. My point was that, even though what you describe would be a superior design, that's not the API we have to live with.

Subinterpreters for Python

mathstuf — Fri, 15 May 2020 20:37:54 +0000

> I find it unfortunate that so many C extensions have tacitly assumed there can be only one Python interpreter in the whole process, but it appears that this assumption is endemic to large swaths of the community. It seems imprudent to provide a stdlib facility which will break popular extensions such as NumPy, and I suspect that this will block the PEP's adoption in the short term.

If the CPython API weren't designed around there being only one interpreter, maybe this could all have been avoided. I still don't get why an extra pyctx parameter wasn't added to every C API at some point (Python3 seems like it would have been a perfect time). The existing ones can become macros (with trampolines for ABI compat if that mattered) that pass the existing global context around. Wean yourself off that and now CPython isn't encouraging static globals anymore.

Granted, I'm sure it's a lot of work and Python already burnt too many bridges for such an API break to happen again and be accepted.

But I'd not be so quick to blame the extension developers when the core doesn't have it's act together either.

Pygolang

kirr — Fri, 15 May 2020 08:35:33 +0000

> Uhh... What? Go's channels can be trivially expressed ...

For the reference:

https://pypi.org/project/pygolang/

and in particular GIL-avoidance mode:

https://pypi.org/project/pygolang/#cython-nogil-api

Subinterpreters for Python

njs — Thu, 14 May 2020 21:06:48 +0000

If you want Go-style *semantics* (shared-everything + channels to pass objects around), then Python already has that, though threads or async libraries.

If you want Go-style multi-core efficiency, though, without the GIL messing things up... then that's harder. CPython is stuck with using the GIL to protect access to Python objects. Therefore, if you want each subinterpreter to have its own GIL, then that means you can never pass Python objects between subinterpreters.

So the reality is that subinterpreters are like subprocesses: each new subinterpreter has to re-load all modules from scratch, passing objects between subinterpreters requires pickling/sending bytes/unpickling, etc. PEP 543 uses the words "CSP" and "channel" a lot, but this was never going to look like Go.

Subinterpreters for Python

enchantner — Thu, 14 May 2020 19:37:19 +0000

I wish all that had performance which is somewhat comparable with Go or at least GNU Parallel. Basically, forking and dealing with pipes is just a bunch of syscalls. I get why these guys did what they did, but I think it could be better.

Subinterpreters for Python

NYKevin — Thu, 14 May 2020 16:44:48 +0000

Yes, and they already did that hard work for you in the stdlib: https://docs.python.org/3/library/multiprocessing.html#ex...

Subinterpreters for Python

enchantner — Thu, 14 May 2020 13:01:53 +0000

Well, it depends on how you implement them. True, the simplest implementation is just a mutex, but in a dynamic Python world with GC and refcounts, especially with multiprocessing, it also would require some "pass-the-ownership" semantics to work properly, like std::move in C++.

Subinterpreters for Python

jezuch — Thu, 14 May 2020 12:39:44 +0000

> prevents prevents multiple

This typo is even self-explanatory! :)

(Sorry, couldn't resist)

Subinterpreters for Python

Conan_Kudo — Thu, 14 May 2020 12:38:55 +0000

There is also an experimental GraalVM implementation of Python 3.7, but yeah, the landscape of Python implementations is kind of barren with Python 3 at the moment.

Subinterpreters for Python

anselm — Thu, 14 May 2020 10:46:17 +0000

Tcl offered a language-level subinterpreter feature quite similar to this (including a facility to use subinterpreters that had all their dangerous commands removed, to execute untrusted Tcl code) in the late 1990s/very early 2000s. At the time nobody else had anything of the sort and it was pretty nifty.

Subinterpreters for Python

k3ninho — Thu, 14 May 2020 08:58:50 +0000

> based on string eval
Not even pickle'd for efficient serialisation between subinterps.

I've not read the proposal, but it being absent from this article suggests there's no aysnc-yield pairing either for saying 'evaluate until this other component is ready to progress global tasking state'. That's something that V8 and ES-16 made easy to manage with (resolve, reject) promises.

K3n.

Subinterpreters for Python

flussence — Thu, 14 May 2020 08:42:50 +0000

Smells like Perl 5.8.0's interpreter threads. Or Java 1.1's green threads? “Concurrent” Javascript via iframes in Internet Explorer 6? Perl 6's threading model fifteen years ago?

I'm flabbergasted that this is all Python can muster in 2020. Cooperative multitasking based on string eval.

Subinterpreters for Python

Cyberax — Thu, 14 May 2020 05:50:36 +0000

Realistically, IronPython or Jython are not even footnotes right now. Heck, Jython doesn't even support Py3.

There are exactly two real-world implementations of Python: CPython and PyPy. Both will have to be modified heavily to support GIL-less world, with subinterpreters being a somewhat lesser evil.

Subinterpreters for Python

Cyberax — Thu, 14 May 2020 05:46:38 +0000

Uhh... What? Go's channels can be trivially expressed in languages with threading support, like C++ or Java. There's nothing at all special about Go channels, except that they are built-in into the language.

Subinterpreters for Python

eric.saint.etienne — Thu, 14 May 2020 05:09:15 +0000

Changing the standard library to circumvent the limitation of one interpreter, CPython, even if it's the most used as of today, seems wrong.
Subinterpreters wouldn't help with performances on interpreters with no GIL like Jython or IronPython and possibly Python for GraalVM.

Subinterpreters for Python

Rudd-O — Thu, 14 May 2020 05:04:53 +0000

Makes me sad. This was the best opportunity to steal most of Go's best feature — in-process CSP/channels — with a very simple API.

Can the same thing be done with multiprocessing or threading? No, not really, as the semantics of those mechanisms aren't quite right.

Tragic torpedoing of a good idea.

Subinterpreters for Python

NYKevin — Thu, 14 May 2020 00:40:18 +0000

I am rather skeptical of this idea. Quoting Nick Coghlan from the "Rationale" section of PEP 554:

> [I] expect that communicating between subinterpreters is going
> to end up looking an awful lot like communicating between
> subprocesses via shared memory.
>
> The trade-off between the two models will then be that one still
> just looks like a single process from the point of view of the
> outside world, and hence doesn't place any extra demands on the
> underlying OS beyond those required to run CPython with a single
> interpreter, while the other gives much stricter isolation
> (including isolating C globals in extension modules), but also
> demands much more from the OS when it comes to its IPC
> capabilities.

I must admit that this comes across as a bit abstruse to me. All modern operating systems have something which resembles a pipe or socket, and for the most part, they also have a reasonable way of sharing memory between processes (admittedly, the Windows way is a bit odd, but it certainly exists). Perhaps Nick was thinking of some other form of IPC, but if so, I cannot divine it from what he has written. A great deal of IPC-related complexity has already been implemented in the multiprocessing module (it has a number of primitives for copying objects across the process boundary).

I find it unfortunate that so many C extensions have tacitly assumed there can be only one Python interpreter in the whole process, but it appears that this assumption is endemic to large swaths of the community. It seems imprudent to provide a stdlib facility which will break popular extensions such as NumPy, and I suspect that this will block the PEP's adoption in the short term.

Subinterpreters for Python

geofft — Wed, 13 May 2020 23:26:38 +0000

From the very bottom of the PEP, it sounds like all the necessary changes to the CPython API/implementation have already been merged? (Or is it just that the work has been done in a fork but not yet merged?) If the CPython changes can be merged by themselves, does the "interpreters" module need to be part of the standard library, or can it be a CPython extension module with the same functionality?

That seems like it would help with the concern of breaking other extension modules. "NumPy doesn't work with this other random extension module I found on PyPI" is much easier for the developers to dismiss (and much less likely to be filed, at all) than "NumPy doesn't work with this thing in Python 3.9 core."

(Also given the "removing dead batteries" PEP and the points it makes about maintenance burden, implicit endorsement of things in the standard library, etc., it seems worth keeping new batteries out until they're fully charged.)