Speeding up CPython

Posted Dec 17, 2020 0:43 UTC (Thu) by mathstuf (subscriber, #69389)
In reply to: Speeding up CPython by NYKevin
Parent article: Speeding up CPython

That's all well and fine, but the problem I see is the fact that it is *global*. I can't even spin up two in-process interpreters to try and do per-thread interpreters to do some local Python code processing. If Python3 had gone and added a `pyctx` call to every API call so that the global state could at least be /scoped/ in some way, that'd have at least allowed someone to go and try a GIL-less context a lot easier. I also think PyPy would also have had a much easier time mocking up a context for easier C API use rather than a bunch of un-namespaced top-level names and direct struct member access being the ABI. Additionally, it makes any kind of hopes for a Python sandbox completely hopeless since I can't hide sensitive modules from just one part of the code in a meaningful way.

Speeding up CPython

Posted Dec 17, 2020 1:02 UTC (Thu) by atnot (guest, #124910) [Link] (2 responses)

Exactly this is being done right now under the "subinterpreters" umbrella. Global state has already been removed from most of the interpreter and you can run multiple of them in one thread. However, last I checked, this ability has not been usefully exposed to programs so far.

Speeding up CPython

Posted Dec 17, 2020 2:30 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (1 responses)

That is good news to hear. I think I read about it here on LWN (either an article or from comments), but I think the lack of release notes about how we could use it has degraded the memory. Thanks for the reminder :) .

Speeding up CPython

Posted Dec 17, 2020 16:28 UTC (Thu) by ms-tg (subscriber, #89231) [Link]

Fascinating that the CPython “subinterpreters” proposal sounds similar to the Ruby 3.0 “ractors” implementation.

In the Ruby concept as I understand it today, only “frozen” (fully immutable) objects will be possible to send across ractors (which are like subinterpreters) in place - any mutable data must be essentially cloned to send to another ractor.

I wonder if Python and Ruby parallel evolution here will lead to a similar new focus on immutable structures on both languages?

Speeding up CPython

Posted Dec 17, 2020 11:57 UTC (Thu) by renejsum (guest, #124634) [Link] (3 responses)

I don't understand why PyPy is not pushed as the default VM for Python. It's written in RPython, it's fast, it's compatible....

Speeding up CPython

Posted Dec 17, 2020 12:52 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (2 responses)

How do I embed a PyPy interpreter into an executable? Without an API to do so, CPython is the only answer there (or IronPython/Jython if your app is in those language families).

I do see https://doc.pypy.org/en/latest/embedding.html but it being marked as deprecated without a link to a replacement isn't promising…

Even then, distribution of an embedded Python environment is a nightmare with CPython, but it's a (mostly) known one at this point. Unless PyPy makes that vastly easier, I don't want to have to slay another dragon :/ .

Speeding up CPython

Posted Dec 24, 2020 10:13 UTC (Thu) by njs (subscriber, #40338) [Link] (1 responses)

The link could be more prominent, but that page does link to the new and supported pypy embedding interface:

https://cffi.readthedocs.io/en/latest/embedding.html

Speeding up CPython

Posted Dec 26, 2020 0:14 UTC (Sat) by mathstuf (subscriber, #69389) [Link]

This is fine, but not suitable (at all) for what I'm talking about. It looks like for any Python library we'd want to provide, we'd have to write some cffi code. No one is going to do that for matplotlib, numpy, scipy, or the umpteen other libraries we get asked to provide in our package.

Speeding up CPython

Posted Dec 17, 2020 13:40 UTC (Thu) by anselm (subscriber, #2796) [Link] (1 responses)

I can't even spin up two in-process interpreters to try and do per-thread interpreters to do some local Python code processing. If Python3 had gone and added a `pyctx` call to every API call so that the global state could at least be /scoped/ in some way, that'd have at least allowed someone to go and try a GIL-less context a lot easier.

John Ousterhout's Tcl had that figured out in the early 1990s. Those were the days.

Speeding up CPython

Posted Dec 22, 2020 23:12 UTC (Tue) by nkuitse (guest, #62915) [Link]

Amen to that!