User: Password:
|
|
Subscribe / Log in / New account

Moving to Python 3

Moving to Python 3

Posted Feb 10, 2011 15:48 UTC (Thu) by fandingo (guest, #67019)
In reply to: Moving to Python 3 by ssmith32
Parent article: Moving to Python 3

What's wrong with the Multiprocess library. There are all kinds of things like mutex locks and process safe data structures (like Queue) that will work across processes.
Using multiprocess, it's possible for a Python programmer to use "multi-threaded" techniques, with the underlying implementation being processes instead of threads.

I've been usign multiprocess with Python3 for a while, and I don't find that there are any limitations. What specific problems keep you from using it?

The GIL needs to go, but there's a lot of ways to work around it.


(Log in to post comments)

Moving to Python 3

Posted Feb 10, 2011 23:58 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

>What's wrong with the Multiprocess library.

It's not threading. My friend write a SIP switch in Python and the major bottleneck is SIP message parsing. Right now they use a thread pool and C library to parse them in several threads.

Pickling/unpickling results to pass them between processes would defeat the whole idea.

Moving to Python 3

Posted Feb 11, 2011 1:21 UTC (Fri) by cmccabe (guest, #60281) [Link]

Python seems like a weird choice for writing a SIP switch. It seems like you would want to write that sort of thing in C, for much the same reason apache is written in C.

Then again, I've never worked with SIP, so maybe I'm off base here.

I do think there's a desire for a language that's lower level than Python/Ruby/Perl, but higher level than C. Java and C++ sort of fill that role, but poorly. I'm hoping that I'll get a chance to test out Google Go to see if it works for that.

Moving to Python 3

Posted Feb 11, 2011 1:46 UTC (Fri) by nybble41 (subscriber, #55106) [Link]

You may want to look into D: http://www.digitalmars.com/d/2.0/index.html

Like C++, but better. Native performance, safe dynamic arrays/strings, Unicode support, garbage collection (if you wish), namespaces, classes, exception handling, interfaces, value types, metaprogramming, compile-time evaluation, and more. The main limitation at the moment is that it remains a new language under heavy development, but it's far enough along for real programs.

Moving to Python 3

Posted Feb 11, 2011 19:47 UTC (Fri) by cmccabe (guest, #60281) [Link]

D is kind of interesting. Wikipedia says it's existed since 1999, but apparently the 1.0 release was only in 2007.

To me the big question is why I should use D over, say, Java. D has "Java-style single inheritance with interfaces and mixins" according to Wikipedia. If you're going to be Java-style, why not just use Java? I'm sure that D's syntax is probably slightly better than Java's. (That's not a difficult achievement.) But does that slight improvement justify throwing away all the existing libraries and code?

On the other hand, Google Go doesn't have inheritance at all. It has duck typing, enforced at compile-time. Its approach to concurrency is not mutexes and condition variables, but channels. I think these two improvements alone justify the switch to another language.

I am curious how the D and Google Go runtimes compare to the JVM. My experience has been that the JVM runs code relatively fast, but starting the JVM itself takes up a huge amount of resources. In comparison, equivalent C++ or C programs have a lower up-front cost. This is one of the reasons why Java web hosting is still more expensive than PHP web hosting, even today. Java has a high up-front performance cost but then does better than PHP over time.

Moving to Python 3

Posted Feb 11, 2011 21:02 UTC (Fri) by nybble41 (subscriber, #55106) [Link]

D is far from "Java with slightly better syntax". It compiles to native binary code, like C or C++, and can link directly against C and C++ library functions. Its metaprogramming features give you all the flexibility of C++-style class and function templates, including compile-time "duck typing". It has a multithreading model based on message-passing, not mutexes or conditions, with explicit sharing and process-wide immutability built into the type system. Object invariants and pre-/post-condition annotation are available to support design-by-contract.

Moving to Python 3

Posted Feb 12, 2011 1:33 UTC (Sat) by cmccabe (guest, #60281) [Link]

> D compiles to native binary code, like C or C++

Well, technically gcj can compile Java to native binary code. However, Java in general has poor integration with non-Java code-- I can't deny that.

> Its metaprogramming features give you all the flexibility of C++-style
> class and function templates, including compile-time "duck typing"

Saying D has duck typing because it has templates is a little misleading. Nobody would seriously say that C++ has duck typing, and it also has templates. For most of your classes in either C++ or D, you're still going to be worrying about inheritance hierarchies and doing "big design up front" which seems more like the Java way of doing things, not the Ruby way.

Speaking of metaprogramming... one nice thing about Google Go is that because there's no inheritance hierarchies, there's no dynamic_cast. One less ugly piece of clutter.

The message passing stuff in Phobos is interesting. It seems that the integration into the language is just at the library level, though, rather than being an integral part of the syntax as in Go.

Overall, the more I learn about D, the more I see it as a "better C++". Like C++, it tries to include everything *and* the kitchen sink. Thankfully multiple inheritance didn't make the cut this time, but most of the other clutter did. (And just like C++, there are some weird omissions-- like reflection.) Google Go, on the other hand, seems to be a more elegant and minimalist language, kind of like C. I like that. But, these are just first impressions, and I guess I might change my mind later.

Moving to Python 3

Posted Feb 12, 2011 3:44 UTC (Sat) by nybble41 (subscriber, #55106) [Link]

> Saying D has duck typing because it has templates is a little misleading. Nobody would seriously say that C++ has duck typing, and it also has templates.

Why not? The following is valid D code:

void callMethod(T)(T object)
{
  if (object.property)
    object.method();
}
...
callMethod(new ClassA);
callMethod(new ClassB); // unrelated to ClassA

Any type which has the required property and method can be passed to the template function. I would say that looks exactly like duck typing. You can do the same thing in C++ with template functions, and in fact the STL makes extensive use of such functions.

If you end up using interfaces and/or inheritance instead that is only because the strongly-typed approach (which Go does not support) has advantages over duck typing, including better compile-time error checking and runtime performance.

> Speaking of metaprogramming... one nice thing about Google Go is that because there's no inheritance hierarchies, there's no dynamic_cast. One less ugly piece of clutter.

One could say that this is only because there is an implied dynamic_cast (but with worse performance, since it requires a runtime lookup) at every location where an object is used.

> The message passing stuff in Phobos is interesting. It seems that the integration into the language is just at the library level, though, rather than being an integral part of the syntax as in Go.

While I fail to see any reason to prefer extraneous syntax over a library, there is in fact some syntax in D intended to support multithreaded programming: the 'immutable' and 'sharable' type keywords, for example, plus the fact that global variables occupy thread-local storage by default. Assurance that shared state cannot be mutated behind the scenes (not just through a given reference, like 'const', but *anywhere* in the process) allows large messages to be passed efficiently and safely.

Moving to Python 3

Posted Feb 13, 2011 1:55 UTC (Sun) by cmccabe (guest, #60281) [Link]

> [For golang] One could say that this is only because there is an
> implied dynamic_cast (but with worse performance, since it requires a
> runtime lookup) at every location where an object is used.

You are confused. Go has static typing. As in, checked at compile-time. Code that misuses types will not compile.

Think of it this way: if you refer to a type T in a C++ template, and then use T::foo, you're not doing a dynamic_cast. You're just using the normal type system. If T does not have a foo method, the code will not compile. Similarly, in Go, you will get a compile-time, not runtime, error, if you try to use methods on an object that don't exist.

Also, you seem to be confused about C++ as well. dynamic_cast *always* "requires a runtime lookup" on the RTTI (runtime type information) in C++. If you do not compile with support for RTTI, you cannot use dynamic_cast. C++ templates, on the other hand, are a completely compile-time mechanism. I don't know why you would say that templated code is not as "strongly-typed" as other C++ code.

> While I fail to see any reason to prefer extraneous syntax over a library,
> there is in fact some syntax in D intended to support multithreaded
> programming: the 'immutable' and 'sharable' type keywords, for example,
> plus the fact that global variables occupy thread-local storage by
> default. Assurance that shared state cannot be mutated behind the scenes
> (not just through a given reference, like 'const', but *anywhere* in the
> process) allows large messages to be passed efficiently and safely.

There's a lot of syntax in D, but is it the right syntax? I think we are going to have to agree to disagree.

I also would like to note in passing that C/C++ have the const, restrict and __thread keywords, which could be used to provide exactly the same message-passing interface you describe in D. In fact, there have been a lot of them created over the years.

Moving to Python 3

Posted Feb 13, 2011 9:33 UTC (Sun) by nybble41 (subscriber, #55106) [Link]

> You are confused. Go has static typing. As in, checked at compile-time. Code that misuses types will not compile.... I don't know why you would say that templated code is not as "strongly-typed" as other C++ code.

Perhaps I am confused after all. I fail to see how Go can be statically typed, with, in particular, specific types for each function parameter, as well as "duck typed", where any type which provides certain methods will be accepted. If types must be known at compile-time then what would be the point of "duck typing"?

Note, too, that it is entirely possible to misuse types in a "duck typed" language, even with compile-time checking. Since there is no information regarding the connections between types (namely inheritance), any type which provides the right method signatures will be accepted, whether or not they were intended to be used in that fashion. This is where explicit interfaces and inheritance can prove very useful as a way of preventing semantic errors.

Go code and C++/D templates are not as strongly-typed for the reasons I stated above: namely, any type with a matching signature will be accepted, even if that use of the type's members is incorrect or even undefined. Classes with inheritance are more strongly typed because they explicitly state that they implement the expected interface--the semantics, not just the type signatures.

> Also, you seem to be confused about C++ as well. dynamic_cast *always* "requires a runtime lookup" on the RTTI (runtime type information) in C++.

RTTI is only required to test that the object you are casting is derived from the class you are casting to. As such, it is a very simple test. In Go the compiler must either generate separate code for each type (like C++/D template functions) or else look up the member offset or function pointer by name for each object at runtime, which seems to me quite likely to be far more expensive.

> I also would like to note in passing that C/C++ have the const, restrict and __thread keywords, which could be used to provide exactly the same message-passing interface you describe in D.

The 'const' and 'restrict' keywords in C++ are not equivalent to 'immutable' in D. Immutable data is guaranteed to never change, and this is enforced by the compiler. You can't pass mutable data to a function expecting immutable, for example. (You *can* pass mutable data to a function which accepts 'const' data, in addition to immutable data.) The 'restrict' keyword is just a declaration of intent. If you declare a pointer 'const' and 'restrict' then the compiler will optimize the code on the assumption that the data cannot be altered via another pointer, but it is up to the programmer to ensure that this is actually the case. I do not think it insignificant, either, that in D you have to specifically declare data as shareable between threads (again enforced by the compiler, unlike in C/C++), whereas in C/C++ all global data is shared by default and you must use __thread to make it private. In D thread-safety is the default; in C/C++ it is a rarely-used compiler-specific extension. (C++0x will supposedly add TLS as a standard storage class.)

Moving to Python 3

Posted Feb 13, 2011 21:12 UTC (Sun) by cmccabe (guest, #60281) [Link]

> Perhaps I am confused after all. I fail to see how Go can be statically
> typed, with, in particular, specific types for each function parameter, as
> well as "duck typed", where any type which provides certain methods will
> be accepted. If types must be known at compile-time then what would be the
> point of "duck typing"?

Golang's philosophy is that inheritance is evil. Not "multiple inheritance is evil" (that is Java's philosophy), or "inheritance is often less useful than composition" (that's Scott Meyers' philosophy in Effective C++). Just "inheritance is evil."

Why is inheritance evil? Well, it forces you to do a lot of work up front before you start writing code. A lot of that work is just writing boilerplate code like Singletons, abstract base classes, Factories, Adaptors, etc. This leads to longer and less readable code. Changing the inheritance hierarchy is difficult after you've written the code. Moreover, unless the code is totally trivial, you will *have* to change the hierarchy in response to changing requirements and new insights into the design that you'll have over time.

The dirty little secret of C++ is that code written in the high-level, object-oriented style often tends to be longer than code written in the old-fashioned C style. It starts to smell like Java.

For a good criticism of Java, and deep inheritance hierarchies in general, see:
http://steve-yegge.blogspot.com/2006/03/execution-in-king...

[snip discussion of TLS, const, and restrict]

You seem to have a good understanding of const and restrict. Your analysis is correct. I'm glad to hear that __thread will be standardized soon. pthread_getspecific is slow on Linux.

Moving to Python 3

Posted Feb 15, 2011 13:05 UTC (Tue) by marcH (subscriber, #57642) [Link]

> Perhaps I am confused after all. I fail to see how Go can be statically typed, with, in particular, specific types for each function parameter, as well as "duck typed", where any type which provides certain methods will be accepted. If types must be known at compile-time then what would be the point of "duck typing"?

The point is: no need to design and maintain a type *hierarchy*. This point looks orthogonal to the static versus dynamic debate.

See also "Structural Typing".

Moving to Python 3

Posted Feb 11, 2011 10:48 UTC (Fri) by mgedmin (subscriber, #34497) [Link]

Anthony Baxter had an entertaining presentation about writing VoIP code in Python back in 2004, titled "Scripting Language", My Shiny Metal Arse. Apparently, it is fast enough.

Moving to Python 3

Posted Feb 11, 2011 6:36 UTC (Fri) by njs (guest, #40338) [Link]

The multiprocessing module can use shared memory to pass state:
http://docs.python.org/library/multiprocessing.html#shari...

I've never used it, but in principle it should be pretty much equivalent to threads.

Moving to Python 3

Posted Feb 13, 2011 20:42 UTC (Sun) by spaetz (subscriber, #32870) [Link]

What's wrong with multiprocess. According to the docs, some parts don't work on *bsd. Which breaks intrroperability.


Copyright © 2018, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds