Progress on no-GIL CPython

Posted Oct 11, 2023 14:15 UTC (Wed) by ericproberts (guest, #139553)
In reply to: Progress on no-GIL CPython by ringerc
Parent article: Progress on no-GIL CPython

It's a major backwards-incompatible change given the number of python libraries written in C. A major version increase would indicate python users might need to do some extra work to upgrade, even if just bumping up dependency versions -- and, let's be real, some libraries will just fall out of use instead of become nogil-compatible, so upgrading won't *necessarily* be straightforward. No need to pretend that isn't what this is.

And there's nothing wrong with major version upgrades that require most users to do some work to upgrade. It's just the previous upgrade made python scared of its own shadow.

Progress on no-GIL CPython

Posted Oct 11, 2023 16:28 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (2 responses)

IMHO it's not time for that *yet.* A major version bump will, rightly or wrongly, be interpreted as a promise that nogil will definitely land in mainline at some point in the near future, and that doing so will necessarily break backcompat. Based on how it is described in this article, I don't think the work is far enough along yet to make either of those promises:

* It may turn out that nogil only offers a performance win in a narrow set of circumstances that are atypical of most Python code, and so nogil Python becomes a separate, non-default mode of the interpreter, and relatively few programs actually use it in practice.
* It may turn out that a future version of the interpreter is able to create some sort of coarse-grained locking scheme that is "good enough" to be backwards compatible with "most" C extensions, perhaps requiring at most a simple recompile.
* It may turn out that nogil is so costly and infeasible that the work is dropped entirely. It doesn't currently look like it's trending in that direction, but development timelines are notoriously hard to predict.
* It may turn out that a major version bump is the best way to express nogil's effect on the Python ecosystem, but they can cross that bridge when they come to it.

Progress on no-GIL CPython

Posted Oct 11, 2023 18:34 UTC (Wed) by iabervon (subscriber, #722) [Link] (1 responses)

It seems entirely plausible that every C module written for 3.5 and recompiled for each later version will end up working fine with a Python that makes the no-gil implementation the normal one, but that loading such a module makes multithreaded performance even worse than 3.5 with respect to contention, due to causing everything to require a global and less subtle lock. On the other hand, programs using libraries not updated to deal well with the new interpreter probably don't use threads and don't care.

Progress on no-GIL CPython

Posted Oct 11, 2023 19:20 UTC (Wed) by NYKevin (subscriber, #129325) [Link]

> [...] but that loading such a module makes multithreaded performance even worse than 3.5 with respect to contention, due to causing everything to require a global and less subtle lock.

That is not a compatibility break. That is a quality of implementation issue. If the code still works and gives the right answer in a somewhat-believable* amount of time, then it is compatible. The question is whether this is a good or a bad tradeoff to make, not whether it will break anything. Offering two versions of the interpreter (or better yet, a flag/environment variable) would be one way to mitigate that issue, but they might instead reasonably decide that maintaining both versions is not worth the effort.

* I'm not going to precisely define "somewhat believable amount of time," but if performance on a modern system is comparable to the previous version on hardware released within the 21st century, then that's probably good enough. If it runs like a PDP-11, then that's probably too slow.