User: Password:
|
|
Subscribe / Log in / New account

Moving to Python 3

Moving to Python 3

Posted Feb 14, 2011 4:33 UTC (Mon) by ssmith32 (subscriber, #72404)
In reply to: Moving to Python 3 by cmccabe
Parent article: Moving to Python 3

As the other comment mentioned - I/O stuff is common.

Where I first started running up against problems was in a GUI that had a central append-only (which eased the locking constraints) data structure, and a corresponding tree of threads. Each thread could generate a new node(s) on the data structure, which might need more analysis by more threads. The operations on the data structure that needed to be thread safe had the thread safety encapsulated in the data structure. It just fell naturally into a threaded design.

As far as being mutex-oriented programming - with an appropriately designed shared data structure, I've rarely ran into that problem. There's usually a couple operations that you think hard about, get right, encapsulate, and move on. OOP really complements this well.

As far as the GIL-specific issue I ran into - with the GIL, your python interpreter runs on a single core, and all threads in that instance of the interpreter run on that core. So it makes multi-core kind of useless for heavily threaded programs. If you go the multi-process route, you get multiple interpreters. But like I said, sometimes threads can be nice :|


(Log in to post comments)

Moving to Python 3

Posted Feb 14, 2011 19:39 UTC (Mon) by cmccabe (guest, #60281) [Link]

> Where I first started running up against problems was in a GUI that had a
> central append-only (which eased the locking constraints) data structure,
> and a corresponding tree of threads. Each thread could generate a new
> node(s) on the data structure, which might need more analysis by more
> threads. The operations on the data structure that needed to be thread
> safe had the thread safety encapsulated in the data structure. It just
> fell naturally into a threaded design.

In my experience, when you start sharing a lot of data between threads, you can take one of two approaches. You can have a giant lock that covers all threads. This is sort of like the old BKL or the GIL itself. One giant lock is simple, but it limits scalability a lot. Alternately, you can have many different little locks. This is great for performance and scalability, but hard on the poor programmers. Debugging becomes very difficult because runs are not repeatable. Code reuse is impaired because everything is tangled in this web of locks and you can't easily move code around.

When you have many little locks, the usual approach is to impose absolute ordering, so that if you take lock A before B at any point, you must always take A before B. That's what the kernel does. It seems to be the best strategy, but again, why impose this on yourself when you don't have to? Just don't share state unless you have to.

It's sad that programmers have been trained to think that threads are "simple" and "natural" and processes are "hard." That's the exact reverse of reality. I blame Windows and its high per-process overhead.

Moving to Python 3

Posted Feb 15, 2011 11:15 UTC (Tue) by mpr22 (subscriber, #60784) [Link]

Data sharing looks easy in threadland, as long as you don't look too closely. Processes make you worry about files and sockets and pipes and god-knows-what, and J. Newbie Programmer thinks that all looks like a lot more work than just having threads that can all see all the data.

(See also: "Why do I have to go through this big scary graphics library with lots of things to set up? Why can't I just stuff pixels into the graphics card?")


Copyright © 2017, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds