GIL removal and the Faster CPython project

Posted Aug 3, 2023 15:02 UTC (Thu) by zorro (subscriber, #45643)
In reply to: GIL removal and the Faster CPython project by maniax
Parent article: GIL removal and the Faster CPython project

I don't really understand the controversy. If instead of writing

    for i in range(len(list)):
        foo(list[i])

I can write

    parallel for i in range(len(list)):
        foo(list[i])

and get an N x speedup, then that is huge. No sub-interpreters, locking or message passing in sight.

If no-GIL allows me to parallelize my loops in a simple way so that my CPU cores don't go to waste then I'm all for it.

GIL removal and the Faster CPython project

Posted Aug 3, 2023 15:16 UTC (Thu) by mb (subscriber, #50428) [Link] (2 responses)

>No [...] locking or message passing in sight.

That depends on whether your list contains items that point to global mutable state or if your function accesses global mutable state directly.

In Python there is no way to statically check this.
Therefore, it's not as simple as marking a loop for parallel execution.
And that is exactly where subtle breakages can and will occur.

In Rust it is statically checked to the extent that the compiler forces you to add dynamic checks or locks if it can't be checked statically.
That's why such easy to use parallelism is easy and safe to implement in Rust.
Most subtle breakages are extremely unlikely. Some types of bugs are outright impossible.

GIL removal and the Faster CPython project

Posted Aug 3, 2023 15:28 UTC (Thu) by zorro (subscriber, #45643) [Link] (1 responses)

Sure, you need to know what your are doing. But many workloads involve stateless item-by-item data processing loops, and speeding them up by a factor N would be as simple as this in a no-GIL world. I don't see why this technology should be denied to Python developers.

GIL removal and the Faster CPython project

Posted Aug 3, 2023 17:15 UTC (Thu) by bluss (guest, #47454) [Link]

Work is ongoing on per-GIL subinterpreters and it will soon allow using independent interpreter threads, which should make it relatively easy to achieve this. For example using concurrent.futures.