Speeding up CPython

That's all well and fine, but the problem I see is the fact that it is *global*. I can't even spin up two in-process interpreters to try and do per-thread interpreters to do some local Python code processing. If Python3 had gone and added a `pyctx` call to every API call so that the global state could at least be /scoped/ in some way, that'd have at least allowed someone to go and try a GIL-less context a lot easier. I also think PyPy would also have had a much easier time mocking up a context for easier C API use rather than a bunch of un-namespaced top-level names and direct struct member access being the ABI. Additionally, it makes any kind of hopes for a Python sandbox completely hopeless since I can't hide sensitive modules from just one part of the code in a meaningful way.

I use python a lot and I have been using Go a bit and I never have a Go program that doesn’t take up all the cores, pretty equally. I toss work to be done into a channel and spin up a configurable number of go routines, most often more than CPUs, and it all uses my 8 or or 16 or more core machine quite nicely. So yeah using all the cores takes parallelism in the algorithms but it’s not that hard and makes things so so fast. And go isn’t far behind in dev productivity, but is behind.

> Amdahl's law places a sharp upper bound on possible benefits from multithreading. You cannot simply divide the runtime of a program by the number of cores; the specific algorithm used must be amenable to concurrency.

I think that V8 js engine challenges Amdahl's Law. V8 is in Chromium (and derived web browsers), the Node.js server-side execution environment and more. The use-case is focused on high network latency relative to local data access rates and local processing time, so V8 races to idle with as many parts executed in parallel.

This is all in aid of getting a web page in front of a user with as little delay as possible between having data to work on and that work being complete. The up-front costs of parcelling up the work are paid as 'cost of doing business' for V8 and so the wide core counts, large memory spaces and billions of available instructions per second get used to arrive at an end result as fast as possible.

The optimisation problem that is constrained by Amdahl's Law contains the assumption that the algorithm could not be stated more efficiently before you spread out to parallel workers. This doesn't apply for the web ecosystem -- the web page and its bundled libraries are inefficient and the network latency is the worst aspect of the delay to the user experience.

This teaches us that we might harvest faster computation with enhanced parallelism.

K3n.

Testcase: Running 100 threads each sending messages to the next using queue.Queue. Thread 0 sends 1000 messages to Thread 1, which send them to thread 2 etc. until they are back at Thread 0
Machine: MacBook Pro i9 8 cores + 8 hyper threads

Cpython: Uses one core 50-60%
Jython: Uses all 8+8 cores at 100% (and is much faster)

Jython uses fine grained locking, so it seams possible to do. I understand that WAY more money has been thrown at the development of the JavaVM than CPython, and that is probably the biggest difference.

The main point of removing the GIL today is that, since the interpreter takes the GIL for each instruction and then re-takes it for the next, it introduces an atomic operation each time in the single thread case, an additional off-core cache ping (or pong) when there are multiple threads, and further two kernel round trips (to the scheduler, no less) when the lock is contested. So its price in programs that multithread for I/O multiplexing is much, much larger than in single-threaded programs; but even in the optimal case it's outright pointless to bang a global lock since clearly multiple threads aren't used.

Python's GIL is what happened when people who didn't properly know how went ahead and multithreaded something anyway. (BKL, anyone? It's good for Linus, it's good for Guido?) The counterargument involving fine-grained locking is similarly silly: consumers of a given data item can easily be changed over to a different static lock of lesser scope that, when it's necessary to take both, is dominated by the GIL. If it were only about the GIL, it'd only take a properly-designed locking hierarchy to enable concurrency between threads accessing unrelated data.

So it's not hard to do. You just need to know how, and the "what" is nearly anything that doesn't do pingpong for every instruction.

What is hard to do is to convert existing multithreaded Python code to a brave new world where individual instructions aren't atomic under the GIL. One might even suggest that a reasonable substitute for the GIL would be an userspace threading library since those can be hugely efficient (even more so on recent cores), preempt predictably[0], require next to no extra kernel interaction, and achieve dramatically higher throughputs when less processing is done to data arriving through a multiplexed set of channels than kernel-side thread switching (and worse, inter-core ping-pong!) costs; and these gains keep increasing as cores multiply.

[0] this is particularly appropriate for an interpreter, which can just call "pth_yield()" or something every 5000 instructions or so. This produces significant cache and scheduling advantages.

The problem with Python is that you quite often have something like a web-server. It can naturally be ran as multiple processes, so that each interpreter gets its own CPU.

This works fine. But then you want to add an in-memory cache and that's where the problems start. If each process gets its own cache, then you have a massive RAM waste. And that's the only easy option (before you go off into using Redis or memcached with all the associated issues).

It's not really any better for a lot of computational code as well. In my company we have Python code that runs computations for each hour of the year, almost completely independently of each other. It was quite liberating to switch from Python to C++ and get true multi-threading, and a huge performance boost.

> The GIL affects fewer Python programs than you might think.

I have fallen for this argument. Unfortunately.

This has been the argument for the GIL for as long as I can remember: Just optimize the inner loop in something like C++. This works if you have a tool that does some costly computation (solving of linear equations, FFT, scene rendering, ray tracing) at the core with some Python code doing the stuff around that is not time critical.

That is definitely a big domain and includes a lot of scientific computing (where Python seems to shine).

Unfortunately I maintain an application written in Python that does not have such a core but deals with lots and lots of small objects representing parts of a FE-model. And it is slow. When we started, we had small datasets so we did not worry too much - we'll just rewrite the relevant implementation details in C.

Only after I hit the performance wall I noticed that I basically had to rewrite most of the classes in C to replace the slow Python code. But since everything is held together by Python collections, manipulating those objects would either require holding the GIL or doing our own locking. Also, we would lose our database integration (which is using SQLAlchemy, no idea how to adapt this to C++ objects).

So we tried to use caching to avoid repeated computations. Which means we spent a lot of development time in fixing cache invalidation problems. Still the system was slow and customers (with 24 core systems) told us that they can't understand how the software performs that bad while not even loading the system. Just parallelize...

As rewriting in another language was out of the question, we implemented multi process concurrency. At last this improved things a bit (especially the user interface stays responsive even if the system is doing something), but at a hefty price. multiprocessing was unusable because this is a GUI application. Using fork wrecks the X11 connection, and the forkserver approach was not available in Python 2 when we started. So we wrote our own multi process IPC library.

I still love Python for writing scripts, the language is extremely expressive. But I am now wary about using it for applications that have to do simple actions on many diverse and small objects.