| Weekly edition | Kernel | Security | Distributions | Search |
| Archives | Calendar | Subscribe | Write for LWN | LWN.net FAQ |
Changes ahead for Python
Changes ahead for Python
Posted Sep 13, 2007 9:30 UTC (Thu) by gypsumfantastic (subscriber, #31134)Parent article: Changes ahead for Python
One area that is not being changed, but is a source of frustration for some, is the "global interpreter lock" (GIL), which only allows one thread at a time to operate on any Python objects or call out to C language extensions.
Wait... It's not possible to write multi-threaded code in Py3k?
(Log in to post comments)
Changes ahead for Python
Posted Sep 13, 2007 10:06 UTC (Thu) by gouyou (subscriber, #30290) [Link]
You can write multi-threaded code, it's just get serialized during execution, and run on a single processor, pretty much like Java's green threads. And, as stated in the article, it's only in CPython, Jython and IronPython do not have this limitation.
It's not that bad.
Posted Sep 13, 2007 19:14 UTC (Thu) by dw (subscriber, #12017) [Link]
Only executing bytecode is "serialized": the GIL is generally released when a thread is invoking a system call, or by C extension modules when they know they don't need any interpreter state. Basically, you can have 100,000 threads concurrently blocking on read(); it's only when one of those calls returns that the lock needs taken again.
As repeated many zillions of times throughout the Internet, this is pretty acceptable for almost any application except heavily compute-bound programs, in which case the author would probably be wanting to use a language more suitable for high performance computational work to begin with.
It's not that bad.
Posted Sep 13, 2007 20:23 UTC (Thu) by oak (subscriber, #2786) [Link]
> this is pretty acceptable for almost any application except heavilycompute-bound programs, in which case the author would probably be wanting
to use a language more suitable for high performance computational work
Or separate processes (and shared memory e.g. via mmap if more data needs
to be communicated)?
It's not that bad.
Posted Sep 15, 2007 20:19 UTC (Sat) by moxfyre (subscriber, #13847) [Link]
As repeated many zillions of times throughout the Internet, this is pretty acceptable for almost any application except heavily compute-bound programs, in which case the author would probably be wanting to use a language more suitable for high performance computational work to begin with.Actually, if you're doing numerical work, like decomposing big matrices and vectors and such... Python is just great!!
That's because of the NumPy extension. It is a C extension, and when doing heavy computations it will happily use multiple threads. For example, I can make a huge random matrix and compute its eigenvalues and eigenvectors:
from numpy import * big = random.random([1000,1000]) # matrix of random values in the [0,1] interval eigval, eigvec = linalg.eig(big)So you can write your algorithm in Python, but all the heavy-duty number crunching will happen in multi-threaded code in C.
If your Python program is CPU-bound but not because of number crunching, then I can see the GIL being more vexing. The case of CPU-bound web applications is probably very frustrating, since there's no easy way to factor out the CPU-bound stuff into an easily parallelizable C extension :-( Anybody have any thoughts on that?
Changes ahead for Python
Posted Sep 15, 2007 17:51 UTC (Sat) by hazmat (subscriber, #668) [Link]
cpython threads map to os level threads. however due to the gil, only one thread per process may be executing python code. python c extensions may release the gil before invoking apis, a database adapter will for example release the lock before executing a query, to allow other threads a chance to execute. the internal management of the gil allows fine tuning of how often the threads are switched via sys.checkinterval api.