Making CPython faster

Posted Jun 3, 2021 23:01 UTC (Thu) by Paf (subscriber, #91811)
In reply to: Making CPython faster by anton
Parent article: Making CPython faster

Perhaps another way to think about this, rather than forcing towards a binary choice between single threaded optimization or multi-threaded code.

Multi-threading is a really powerful option to get greater performance, especially (and increasingly) on modern hardware.

It is not necessarily opposed to single threaded optimizations. (Eg, OpenBLAS probably sees a large percentage of its usage, perhaps a majority, in *multithreaded* programs!)

And, basically, it sucks that Python has this option mostly closed to it. It doesn’t make Python a bad language, it’s not fatal, etc. But it is certainly not a positive aspect of the language (even if the GIL enables many of the good things).

Making CPython faster

Posted Jun 3, 2021 23:08 UTC (Thu) by Paf (subscriber, #91811) [Link] (2 responses)

Also, to take a slightly less even handed view:
There are *tons* of cases that multithread pretty well, where optimized single threaded code can be repeated across multiple cores. (Again, single threaded performance optimization and multi threading are not necessarily opposed, though they do conflict in many cases.)

But they’re talking about hopefully eventually achieving a 5x speedup. This is great, and I want them to do it. But as someone with a partial scientific computing background... I sure wish I could use the other 15 cores on my desktop in CPython too! To say nothing of the 64 cores on individual nodes of the compute cluster.

There are a ton of computational folks who’d love to do more of their work in Python, but do not/cannot effectively because of the lack of parallelism in CPython.

I’m not saying it should be priority A1. But let’s not pretend it isn’t a hindrance!

Making CPython faster

Posted Jun 4, 2021 7:15 UTC (Fri) by LtWorf (subscriber, #124958) [Link] (1 responses)

use the multiprocessing module?

Making CPython faster

Posted Jun 4, 2021 17:24 UTC (Fri) by NYKevin (subscriber, #129325) [Link]

You don't even need to go that far. Most of the serious scientific computing community is already using NumPy or SciPy, and those are not subject to the GIL because they run the big expensive parts in native code.

Even if Python dropped the GIL tomorrow, those libraries would continue to be used. Native code is so much faster than interpreted code that it's not worth even trying to make Python fast enough to compete with them.