Weekly Edition
Return to the Development pageRecent Features
| |||||||||||||
Van Rossum: Python is not too slow (InfoWorld)
InfoWorld has a short interview with Guido van Rossum, the creator of Python. In it, he talks about Python 3, Unicode, the Global Interpreter Lock (GIL), and more. "At some point, you end up with one little piece of your system, as a whole, where you end up spending all your time. If you write that just as a sort of simple-minded Python loop, at some point you will see that that is the bottleneck in your system. It is usually much more effective to take that one piece and replace that one function or module with a little bit of code you wrote in C or C++ rather than rewriting your entire system in a faster language, because for most of what you're doing, the speed of the language is irrelevant."
(Log in to post comments)
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 21:42 UTC (Fri) by hpa (subscriber, #48575) [Link]
So apparently the solution to Python being slow is not to improve Python but to not use it.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 21:48 UTC (Fri) by kragil (subscriber, #34373) [Link]
Yeah, that argument is really lame. If you compare CPython with V8 you can see that so much more could be done. Guido is just lazy ;-)
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 23:34 UTC (Fri) by Sho (subscriber, #8956) [Link]
It's not like nothing is going on in the Python community with regard to performance work. If you're interested in a JIT'ed Python give PyPy a look, which had a _huge_ presence at this year's PyCon.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 7:52 UTC (Sat) by drag (subscriber, #31333) [Link]
It will be great when pypy gets better compatibility with python modules.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 13:06 UTC (Sat) by rbrito (subscriber, #66188) [Link]
Indeed.
One program is which is very straining to my notebook right now is offlineimap (it makes the notebook's fan go to the maximum when it is sync'ing my folders, even when I am using the `-q` switch---which stands for quick mode).
I tried using pypy with it (it is packaged in Debian) and, for some small programs of mine, the improvements were noticeable indeed.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 15:46 UTC (Sat) by spaetz (subscriber, #32870) [Link]
hi, if offlinemap strains your cpu, you might want to use the sqlite backend rather than the plain text backend. it helps save yourf cpu...
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 21:59 UTC (Fri) by JohnMorris (subscriber, #73531) [Link]
It's just mixed language programming, which people have been doing for decades. It's even in the Linux kernel, which is mostly in C but occasionally dips into assembler. Or look at the gcc / intel cc extensions for accessing the SSE instructions. Other than the fact that the compiler is taking care of the register allocation you might as well be programming in assembler, right in the middle of your C/C++ code.
Python for the high level stuff with descent into C for speed critical parts (after profiling) has worked out very well for me in a few projects.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 23:34 UTC (Fri) by Ed_L. (guest, #24287) [Link] Other than the fact that the compiler is taking care of the register allocation you might as well be programming in assembler...Compilers will do that
It does make sense, yes Posted Mar 17, 2012 20:45 UTC (Sat) by oldtomas (guest, #72579) [Link]
Yeah, it's an old hat: Emacs AutoCad...
It even got a shiny name "multi-language paradigm" (Ousterhout, ca. 1998).
To be real fun, the interface to the "low-level language" should be usable. Tcl is a good example. So is Lua. Don't know about Python.
Few languages qualify in my view to cover the whole spectrum. No, Python aint one. Nor is Java (gasp!). May be Common Lisp.
It does make sense, yes Posted Mar 19, 2012 17:34 UTC (Mon) by k8to (subscriber, #15413) [Link]
Well, the old hookups from python to c was pretty tedious (at the time it was considered OK, but the world moved).
But then we got swig, which still sucked but was automated.
But nowadays we've pyrex and ctypes, and at least on most platforms like windows and linux you can have the linkage to system libraries created for you dynamically at runtime all python-style from pythonic syntax that's quite easy to write. If your stuff has to run on like AIX and HPUX then there may be some limitations with this brand of magic.
It does make sense, yes Posted Apr 3, 2012 20:45 UTC (Tue) by JanC_ (guest, #34940) [Link]
> But nowadays we've pyrex and ctypes
And Cython[1] (a more-advanced Pyrex fork) and PyPy[2].
[1] http://cython.org/
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 6:00 UTC (Sat) by cmccabe (guest, #60281) [Link]
I use Python as a shell script replacement. It's good for things like automating deployment of applications, doing monitoring, automating common tasks, and so forth. In that role, speed really doesn't matter that much.
Google uses Python for their cluster management stuff, if I remember correctly. So apparently they have much the same attitude.
My biggest complaint about both Python and Ruby is that they keep changing things, in non-backwards-compatible ways. The languages lack any sane way to do library versioning-- it's inherent in the dynamically typed approach. Rather than realizing this and trying to minimize change, they just charge ahead with all kinds of program-breaking incompatible API changes, and just let you pick up the pieces. Bash may suck, but at least it always sucks the same.
So people have decided that it's a great idea to use things like virtualenv to peg every library to a highly specific version-- because obviously that's the ideal solution.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 13:28 UTC (Sat) by hanwen (subscriber, #4329) [Link]
There are many people (including some very senior ones) at google that think Python is a liability rather than an asset.
The biggest concern is maintainability: once a python program gets big, the dynamic typing makes refactoring very hard since you can't change function signatures and be sure that you updated all call sites. Testing helps remediate that, but for large systems, tests use lots of mocking and do not represent the final system. Also running tests to find trivial typos is wastes valuable time.
A second concern is performance: while python may work for proof-of-concept systems,
Even tools that "automate common tasks" accrue lots of dependencies over their lifetime, making them large and slower than they need be. As the author of a large python program (and, sadly, as a former Python fan) that falls under "automating common tasks"section, I wouldn't even recommend it as a shell-script replacement nowadays.
Fortunately, Go is shaping up to be an excellent alternative to Python: it's concise and quick to develop in, but it compiles to native code, resulting in performant and more maintainable systems.
Han-Wen
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 18, 2012 1:16 UTC (Sun) by jwb (guest, #15467) [Link]
Just the startup time gains of Go over python are worth it. I have some automation tools that need 10-20 seconds of CPU just to parse in all the python they need, and that's after I karate-chopped that code to bring it down from 90 seconds. So far I've never written a Go program that needed more than the blink of an eye to get started.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 18, 2012 8:07 UTC (Sun) by mb (subscriber, #50428) [Link]
> I have some automation tools that need 10-20 seconds of CPU just to parse in all the python they need
Seriously. What the heck are you doing wrong?
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 17:36 UTC (Mon) by k8to (subscriber, #15413) [Link]
Maybe the main.py is enormous? Python doesn't create a .pyc (compiled bytecode) for the entry script for whatever reason.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 20:49 UTC (Mon) by pboddie (subscriber, #50784) [Link]
True. It sounds like the system employs large quantities of generated source code. Lovely!
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 18, 2012 21:00 UTC (Sun) by NAR (subscriber, #1313) [Link] The biggest concern is maintainability: once a python program gets big, the dynamic typing makes refactoring very hard since you can't change function signatures and be sure that you updated all call sites. [...] Also running tests to find trivial typos is wastes valuable time.I had exactly the same concerns when I first started to work with Erlang. However, if my Erlang compile&test cycle is faster then the C++ compilation, finding typos via testing does not wastes time. And in many cases the Erlang way was faster - the test finishes by the time the C++ linker finishes its job. About the refactoring: I don't know about Python, but for Erlang there are tools that help in refactoring all of the code, but in practice even without tool help and with 100+k lines of code we can quite safely refactor code, the automated tests will find any errors that we miss. And we have to test anyway.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 5:18 UTC (Mon) by cmccabe (guest, #60281) [Link]
The problems with C++ compilation are well-known and don't represent the problems of statically typed languages in general. Java, for example, compiles very quickly, as does Go.
Unit tests aren't really a replacement for static typing. The systems that I prefer to work on use both type checking and testing.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 18:50 UTC (Mon) by foom (subscriber, #14868) [Link]
Well, Java doesn't *actually* finish compiling your program until at least a few minutes into the execution of your program!
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 19:17 UTC (Mon) by elanthis (guest, #6227) [Link]
There are compilers that can do full ahead-of-time compilation of Java. And C#. And various other static languages.
C++'s compilation speeds have almost nothing to do with being statically typed. It comes down to a combination of the pre-processor include system used for hooking files together and the prevalence of templates. The combination of the two means that every .cpp file can have megabytes and megabytes of identical code that has to be recompiled for each source file, only to have most of it thrown away by the linker.
The ahead-of-time optimization passes of a C++ compiler are rarely the biggest bottleneck in compilation. There's talk of getting a module system in the next C++, which will help tremendously with compilation speeds. In the meantime, learning how to use precompiled headers effectively will drastically improve compile times.
I have a couple large C++ game engines that can compile with full optimizations in less time than many small UNIX tools, largely just because I put in the little bit of time it takes to get a sane and optimized build environment. (hint: autotools, scons, cmake, and so on were not invited to the party)
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 20:45 UTC (Mon) by HelloWorld (guest, #56129) [Link]
> (hint: autotools, scons, cmake, and so on were not invited to the party)
Well, who was?
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 21:59 UTC (Mon) by cmccabe (guest, #60281) [Link]
> I have a couple large C++ game engines that can compile with
> full optimizations in less time than many small UNIX tools, > largely just because I put in the little bit of time it takes > to get a sane and optimized build environment. (hint: autotools, > scons, cmake, and so on were not invited to the party)
It is true that careful programmers can often achieve reasonable build times with C++ by minimizing unecessary dependencies. Sometimes you have to sacrifice some perfomance too, by using things like the pImpl idiom.
I think it's unfair to stigmatize cmake as slow. CMake actually compares very favorably with hand-written makefiles. autotools and scons are indeed slow, and should basically not be used by anyone.
There are ways to speed up your autotools build. One of them is to combine all of your Makefile.am files into one giant file (modularity? What's that?). If you write your makefile.am in the natural way, though, it will be very slow. And even with optimizations it takes 15 seconds just to get rolling after you type "make."
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 20, 2012 11:31 UTC (Tue) by RCL (guest, #63264) [Link]
Joining several .cpp files into single one (e.g. by creating files that consist purely of
#include <blah.cpp>
also known as "Unity builds") helps C++ compilation times as well (and also allows for better optimization for non-globally optimizing compilers).
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 21, 2012 12:55 UTC (Wed) by jwakely (subscriber, #60262) [Link]
But they break anon namespaces and entities with static linkage and cause the entire world to be rebuilt if you touch one file. They're a hack, an abhorrent one, to work around the lack of a decent build environment and sensible control of dependencies.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 14:24 UTC (Sat) by gmaxwell (subscriber, #30048) [Link]
It's the only form of python-slowness prevention shown to be 100% effective!
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 21:49 UTC (Fri) by slashdot (guest, #22014) [Link]
Not too slow?!
Python's implementation is absolutely horrible performance-wise, and Mr. van Rossum ought to be ashamed of it, if anything.
Look at LuaJIT for a properly implemented scripting environment, and at the JavaScript implementations in modern web browsers for passable ones.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 22:01 UTC (Fri) by zooko (subscriber, #2589) [Link]
I wonder if the first three commenters read the article before posting. (I did.)
What Guido is saying is exactly correct, in my experience. The Python culture is not like the Java culture of "100% Pure" in which re-using extant libraries written in native (C/C++) code is frowned on. Instead, the Python culture has always been pragmatic and polyglot, and it is common to re-use extant C/C++ libraries when appropriate, or to write a new bit of C/C++ code to compute the inner loop of your Python program. That latter technique is what Guido is talking about.
In practice, "Python" always means "Python plus as much C/C++ as you need".
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 23:36 UTC (Fri) by pboddie (subscriber, #50784) [Link]
The trouble is that it's difficult to persuade people to pick up Python if the persuasion goes something like, "Well, if you need it to go faster you can always write it in C." For people who already know C or are thinking of learning it, the response is then, "Why don't I just write it in C to start with?" Yes, I've had this conversation. The response is not quite soundbite-sized and is thus unsatisfactory for some audiences.
The argument that there are other factors in program performance is a valid one. However, this shouldn't be used to justify closing the door on the fundamental runtime or generated code performance. Fortunately, systems like PyPy and a variety of other tools are keeping that particular door open.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 6:03 UTC (Sat) by cmccabe (guest, #60281) [Link]
Maybe you should try Golang. It's also garbage collected and uses duck typing, but has a static type system. It also was explicitly designed with performance and scalability to multiple cores in mind. Use the right tool for the job.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 18, 2012 15:03 UTC (Sun) by pboddie (subscriber, #50784) [Link]
Well, my remark wasn't really describing any personal dilemma of my own, but I have looked briefly at Go and also at Dart. The type system in Go appears to be interface-oriented as opposed to supporting the completely informal duck typing that Python permits, but that isn't necessarily a bad thing: although you can argue that it requires stating the obvious - merely observing which attributes or methods an object uses in a method or function should provide the same information - having a declaration of conformance to an interface is good for documentation and validation purposes.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 1:56 UTC (Mon) by agrover (subscriber, #55381) [Link]
take a look at Mozilla's Rust: http://www.rust-lang.org/ . It looks better than Go, to me anyways, although not as far along.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 7:27 UTC (Mon) by ssmith32 (subscriber, #72404) [Link]
Thanks for the reminder!
Rust does look nice.. my second look at it after a while.. Just looked at Go again recently as well. Rust does seem to fit a little better with my mental models and processes. Hopefully that's a good thing :)
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 22:11 UTC (Mon) by cmccabe (guest, #60281) [Link]
The two languages are similar in a lot of ways. However, Go is actually being used in production (by Google)-- as far as I know, nobody actually uses Rust. Also, the Go standard library is a lot bigger than Rust's.
Rust's idea of making everything immutable sounds like a great idea and it appeals to a lot of folks (especially people with a functional programming background) but how practical is it really? The jury is still out, I think. Maybe at some point I'll give it another look, however.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 20, 2012 2:19 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]
Rust supports optional mutability, it's immutable only by default. I actually wish more languages would adopt this mentality.
Mutability is evil (EVIL, I say!!!), but sometimes it's required. So making it optional is a good design choice.
And there are successful languages without mutable variables (Erlang comes to mind).
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 20, 2012 9:06 UTC (Tue) by cmccabe (guest, #60281) [Link]
Ok, people have been bugging me about Rust for long enough. So I'm actually going to compare Rust and Go somewhat. Looking at the FAQ (from http://www.slideshare.net/pcwalton/rust-all-hands-winter-...), it seems like Rust made a few interesting decisions that differ from Go. Keep in mind that I am writing this in early 2012, and things may change. Everything here is current to the best of my knowlege.
Allocation: Rust has three different types of allocation -- "interior types" which are stored on the stack, "unique types" which are scoped, and "boxed types" which are GC'ed. Go only has one type of allocation and the compiler figures out what to do for you. Go does have the "defer" statement, which you can use to get something very like C++ scoped destructors or Rust "unique types." This somewhat reflects the C++ background of the Rust developers. C++ has always given you lots of different flavors of allocation. Is this a good thing? Um... for a high-level language, probably not.
Rust goes through great lengths to avoid global garbage collection. Go currently has a stop-the-world garbage collector. I think this reflects the fact that Mozilla, with their interest in big, user-interface intensive programs, developed Rust, whereas Google, with their interest in server stuff, is backing Go. I don't think this is a big long-term problem for Go. Google has shown us with Dalvik and V8 that incremental garbage collectors are a practical way to solve this problem.
Both Rust and Go have structural subtyping. Go has a very intuitive system for creating modules and encapsulation (in my opinion.) I'm not as familiar with Rust's systems in these areas, so I can't comment on them. Both Rust and Go provide mechanisms to structure programs in terms of many small tasklets that share state by communicating rather than with mutexes, locks, and shared state.
As mentioned above, Rust has this concept of immutable data, which Go lacks. In Rust, mutable data needs to be allocated in a special way, on a special heap (all other data is immutable.) Currently there is a flaw in the Rust type system which allows supposedly immutable data to be modified, but hopefully this will soon be fixed. [ See https://github.com/mozilla/rust/wiki/Proposal-to-address-... ] You can get most of the benefits of immutable data in Go by passing around values rather than pointers. (In Go, channels can take either values or pointers.) It is good to avoid gratuitous mutation, but I find myself wondering whether the Rust approach will be clunky in practice.
I've suffered through a lot of pain being "helped" by the C++ type system, which forbids, for example, passing a vector<char*>* to a function which expects vector<const char*>*. So I am a little bit gun-shy of trying to enforce these types of things through a complex type system. For now, I'll just say this: time will tell if Rust's approach is viable.
Rust also disallows null pointers, which I personally do not agree with. Maybe it's my background as a C programmer, but I like having a special value to represent "none." I always find myself using clunky workarounds to emulate NULL when it is not present in the language-- especially in languages without exceptions (but more about that later.)
Rust has generics, and Go does not. The creators of Go have publicly stated that generics might be added in the future, but nobody has a timeframe. Generics are probably the thing that I most miss in Go.
Both Rust and Go try to encourage you NOT to use exceptions as a control flow mechanism. Rust has "fail," and Go has "panic," but neither of them is a traditional exception mechanism like you might find in Java. They don't allow you to associate much with the exception except a string. This seems like a good choice to me-- flow control via exceptions is icky. It will surprise some people, though.
Rust has this concept of "typestate," which is kind of like the Linux kernel's BUILD_BUG_ON or Boost's static_assert, but built-in. This is something that I really like. Things like this can be retrofitted on to the language after the fact-- like sparse does for C, or annotations do for Java-- but it is better to put it to the compiler itself.
Language change: Rust has changed a lot from 2010 to 2011, and seemingly in backwards-compatibility breaking ways. Even the syntax changed a lot. This reflects a lack of actual users and a more academic attitude. Will Rust finally stabilize in the future, or keep ever-changing like D has? Hopefully the former.
Code maturity: The Rust slide deck from late 2011 says "we're missing garbage collection... and the compiler isn't yet production-quality." (This is a direct quote from the slide deck-- please don't flame me about this.) So it seems irresponsible to point people looking for something to use in production towards Rust, until those points are fixed. On the other hand, Google is using Go in production right now.
I hope I've been fair here!
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 20, 2012 10:10 UTC (Tue) by khim (subscriber, #9252) [Link] Google has shown us with Dalvik and V8 that incremental garbage collectors are a practical way to solve this problem. What Google actually showed is that RFC 1925 is still in effect: while GC is totally unsuitable for UI (that's why company which knows how to create attractive UI deprecates it), but yes, if you'll spend enough time with profiler and if your system is powerful enough (say, 10x as powerful as the task on hand really needs) then GC may work for UI. It remains to be seen if Go will actually be ever usable for UI. Rust has changed a lot from 2010 to 2011, and seemingly in backwards-compatibility breaking ways. Go did that too. This is not a crime: C++, for example, introduced a lot of backward-incompatible changes in it's early days. Only years after introduction (when there was already huge amount of code written) it stopped doing this. And C++11 introduces breaking changes again.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 20, 2012 13:02 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]
Rust has been produced specifically to minimize the need for garbage collection and the impact of the collector. Once one understands this, the design decisions for Rust become completely clear and intuitive.
Unique types allow deterministic destruction of resources - you are guaranteed not to have memory/resource leaks with them. Unfortunately, the next step (refcounted handles) can not make such guarantees. So they've stopped at unique types.
Dividing the RAM into independent thread-local heaps helps immensely - Erlang does it for exactly the same purpose, for example. Experience has taught us that truly pauseless 'global' garbage collectors are possible, but they either have a big overhead or require hardware assists (there's a company that even creates special hardware for high-speed GCs!).
The split between immutable and mutable types in Rust is interesting. But I don't think it can be simplified much. And no, Go doesn't have anything like it - you can pass mutable data using channels (which is actually an advantage in Go's programming style).
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 20, 2012 18:17 UTC (Tue) by njs (guest, #40338) [Link]
IIUC, the fundamental difference between Go and Rust is that both make multi-threading a core feature of the language, but Go threads are shared-everything and Rust threads are shared-nothing. (Or very roughly in C terms, Go threads act like pthreads, and Rust threads act like fork.) Go encourages you not to program in a style where your threads stomp on each other's data-structures willy-nilly, but there's no protection built into the language itself. OTOH, AFAICT Rust doesn't even *have* mutexes, because it has safer ways to share state.
Personally this difference is enough to pretty much rule out Go as an option so long as Rust ends up being viable at all, no matter what other clever syntax and type systems and stuff they have... but of course YMMV.
> Rust also disallows null pointers, which I personally do not agree with. Maybe it's my background as a C programmer, but I like having a special value to represent "none." I always find myself using clunky workarounds to emulate NULL
I don't remember the details, but I'm pretty sure it's easy and idiomatic in Rust to define a "maybe<X>" type: a value which is either "none" or else has a real pointer-to-X in it. The trick is that unlike C, now you *can't* dereference such a pointer without checking for NULL-ness when "unpacking" it. This might seem like a burden, but of course it only applies to those pointers which *might* be NULL, which are exactly the ones that you have to check. So they're not trying to make this clunky; basically it's just like C, except now the compiler will keep track of when you need to check for NULL and when you don't. Boom, no more segfaults.
> Rust has this concept of "typestate," which is kind of like the Linux kernel's BUILD_BUG_ON or Boost's static_assert
Typestates are *substantially* more powerful than BUILD_BUG_ON, because typestates make compile-time assertions about program flow. A simple example of a typestate assertion would be "strings which come from the network and then later end up being passed to the filesystem *must* go through the utf8 sanitizer at some point in between". If you accidentally add a code path that violates this invariant, the compiler will tell you.
Generally a lot of things in Rust are designed around the idea a good language should help you write correct code.
(Disclaimer: Graydon's a friend, and I reviewed an early version of the Rust spec, but I haven't been involved or followed the project much since.)
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 20, 2012 19:24 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]
To be fair, shared-everything concurrency can be much more efficient than shared-nothing message passing in certain tasks (just look at those RCU articles!).
So I'd certainly like at least some controllable amount of sharing with explicit locking. But message-passing seems to be much easier to write for.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 20, 2012 20:10 UTC (Tue) by njs (guest, #40338) [Link]
RCU doesn't imply shared-everything, just shared-a-few-things-under-very-carefully-controlled-conditions :-). There's nothing stopping them from adding an experts-only, carefully-bounded escape hatch in the standard library. Or from using RCU under the covers to implement something with those semantics -- Rust doesn't literally use 'fork', language runtimes can be very tricky about how they implement things. OTOH you can't add memory isolation in a library :-).
But yeah, it's possible that in some situations, code in Rust will be slower than the best possible concurrent implementation that exploits details of the CPU's cache coherency model etc. For me this is a totally acceptable trade-off, but again YMMV.
(Anyway, it doesn't look like Go has any primitive memory barrier operations, so your only safe concurrency options there are mutexes and channel sends. No RCU.)
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 21, 2012 16:36 UTC (Wed) by cmccabe (guest, #60281) [Link]
Golang has atomic operations, which imply memory barriers, and also pointers. See http://golang.org/pkg/sync/atomic/ This is something that actually exists now, not just in theory.
As far as I can see, you should be able to use the CompareAndSwapUintptr operation to get the update-side memory barrier you need for RCU. Then you should be able to have reader threads read the pointer value normally, without a memory barrier, and get whatever they get.
In Go, you have garbage collection, so you can forget all about fooling with grace periods and so forth. Just update the pointer.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 21, 2012 16:53 UTC (Wed) by khim (subscriber, #9252) [Link] In Go, you have garbage collection, so you can forget all about fooling with grace periods and so forth. Just update the pointer. Not exactly. Quite often things like RCU are used for performance-critical tasks. The fact that language is built around GC means that now you not only need to worry about grace periods but you must convince GC to [roughly] obey them. YMMV: in some cases it may all “just work”, in some other cases you'll spend huge amount of time taming GC (and then everything will break once GC will be changed in your implementation). In fact non-optional GC is my biggest gripe with go.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 21, 2012 17:05 UTC (Wed) by njs (guest, #40338) [Link]
Ah, thanks! I missed those because they aren't mentioned on the Go memory model page: http://golang.org/doc/go_mem.html
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 11:04 UTC (Sat) by anselm (subscriber, #2796) [Link] For people who already know C or are thinking of learning it, the response is then, "Why don't I just write it in C to start with?" Yes, I've had this conversation. The response is not quite soundbite-sized and is thus unsatisfactory for some audiences. What's wrong with »Because those 90% of your program which are not in fact time-critical will be that much easier to write and maintain«? Also, you never know, the Python code may actually be fast enough in practice. In any case, it is a good idea to prototype the whole thing in Python to begin with to (a) see whether it will work at all, and (b) provide a performance baseline for later optimisation work. As Donald E. Knuth famously said, »Premature optimisation is the root of all evil.«
Python is too slow, but that's not even the problem Posted Mar 17, 2012 12:15 UTC (Sat) by tialaramex (subscriber, #21167) [Link]
Sure, maybe 90% of your program isn't time critical and can thus be maintained by Python programmers.
But there is a substantial opportunity for cost inefficiency from requiring extra skillsets in your programming team. The very expensive C++ programmers you hired, who can write templated code that's simultaneously fast, correct and re-usable, are wasted writing noddy beginner-level Python for a week when there is no C++ work on the Gantt chart. A project that takes 2 Python programmers two weeks not only doesn't take one week if you throw 4 Python programmers at it, it can take six months if all you have is a C++ programmer.
In a hobby Free Software project this sort of thing can just strangle you. You do not get to hire the "right" people, you have to take what you get, and if what you've got is six C++ programmers then writing the non-critical paths in Python makes no sense. If the project's progenitor happens to love Python but none of its subsequent maintainers do, either the Python code is slowly removed or the project bitrots.
Now it so happens that the best programmer in my company happen to be skilled in both C++ and Python. But that wasn't a hiring decision, it was a happy accident. He's a perfectly good Java and C programmer too, I hope we're paying him very well indeed. So for us we have this type of flexibility. But despite that we rarely use it within a program. Programs that need raw bit-banging performance are in C and C++, programs that don't are in Java, scripts that bolt stuff together are in Perl or Python. And we still run into manpower problems where we have a developer ready to work on a high priority task, but their skillset doesn't match the task needed.
Python is too slow, but that's not even the problem Posted Mar 17, 2012 18:33 UTC (Sat) by robert_s (subscriber, #42402) [Link]
*ahem* You're both ignoring Cython, which allows you to compile a slightly-restricted subset of python to C. It's quite easy to write your inner loop functions in Cython.
Python is too slow, but that's not even the problem Posted Mar 19, 2012 0:26 UTC (Mon) by jmalcolm (guest, #8876) [Link]
> cost inefficiency from requiring extra skillsets in your programming team. The very expensive C++ programmers you hired
Ok. So your argument is that you should not use Python is that you may need SOME expensive C++ programmers to optimize critical paths? Is using ONLY C++ programmers a better solution?
If Python coders are cheaper, this sounds like an argument FOR using Python.
Python is too slow, but that's not even the problem Posted Mar 19, 2012 10:04 UTC (Mon) by tialaramex (subscriber, #21167) [Link]
No. I'm arguing exactly what I wrote. The more skillsets you need for your project the more likely you are to hit bottlenecks where you can't use the available people very effectively, and that means a higher overall cost. Whether the results are worth it will vary from project to project.
It's not just Python vs C++, the opportunity to screw this up will arise in other areas. Maybe your program could be faster if it used a custom network protocol instead of the stuff provided. You could hire a specialist to do that work. Maybe your program would benefit from a novel indexing strategy in the database. You could hire a specialist for that too. Does that make good sense? It depends.
One-language programmer == bad idea Posted Mar 19, 2012 1:46 UTC (Mon) by david.a.wheeler (subscriber, #72896) [Link]
If your software developers can only program in one language, they are already a problem, and it'd be best to deal with that problem first. That's like saying, "My carpenters only know how to use a saw".
No one language can be all things to all people. Most large programs include multiple languages, even if they appear otherwise at first (e.g., they start growing specialized interpreters).
Calling out to other programming languages for specific tasks is often a very *efficient* way to use human time. Use a language that's easy and fast to program in for most stuff, and then dive down when you need to. In many problems there are a few hotspots; spend most of the expensive (human) time on those, and not on stuff that doesn't matter. This is old (and still valid) advice).
Clearly, Python's slow speed and its poor support for multithreading makes it inappropriate for some tasks. It'd be sensible to improve that in Python implementations. But that doesn't mean it's inappropriate for all tasks. The biggest problem with Python right now is the really awful way they've handled the 2-to-3 transition. The CPython developers have abandoned python 2, but almost no one is moving to python 3, and they're incompatible. Note that PyPy does not even *try* to implement python3, and most of the available python libraries are for python2, not python3.
The problem here is that the CPython developers have confused implementations of the language with the language itself. They should have had a single implementation of both Python variants, so that people could gradually move up as the language changes (as they'd always done previously). Historically, it was easy to write code that ran on multiple versions of Python; once that failed to be true, python got a lot less useful. I like python, but I think the 2-vs-3 gap is a way more important problem than its (lack of) speed.
One-language programmer == bad idea Posted Mar 19, 2012 2:23 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
> The problem here is that the CPython developers have confused implementations of the language with the language itself.
I don't think this is the problem. Python has multiple implementations, so they are pretty careful to keep the language implementation from being the language definition.
The problem here is the Python mantra "there must only be one way to do something"
if you find that the way you implemented to do something is wrong, you can't add a better way and keep the old way, because there would now be two ways of doing something.
In the interview, he says that a large part of the reason for doing Python3 was that Python2 had grown more than one way to do some things.
you can't both keep backwards compatibility and eliminate the 'extra' ways to do something.
One-language programmer == bad idea Posted Mar 19, 2012 6:05 UTC (Mon) by ssmith32 (subscriber, #72404) [Link]
Hmm.. except CPython is defined as the reference implementation, which does make it the standard:
https://en.wikipedia.org/wiki/Reference_implementation_%2...
The main problem I run into with the "use a different interpreter" argument is that CPython is installed everywhere, and other implementations nowhere. Which means you constantly have to install other implementations.. and install them in a side-by-side setup that doesn't affect other programs that need CPython for whatever reason (like needing particular versions, etc). Basically, nice in theory, but, assuming your program isn't just something for one special machine, using other interpreters ends up be a huge PITA.
And, yes, threads are nice to have sometimes (they are not universally "evil" - and I seen some terribly evil, twisted programs from people who just discovered python & twisted, and got religion, but no implementation skills). And quite nice threaded programs from people who know what they're doing. So the GIL does suck, in the end. And the whole "need to support the uniprocessor" case is long gone..
One-language programmer == bad idea Posted Mar 22, 2012 23:52 UTC (Thu) by samroberts (subscriber, #46749) [Link]
Two different and mildly incompatible object systems.
xrange and range
ElementTree and cElementTree (one is fast, the other is the default)
string module, string methods
python has lots of different ways to do things.
One-language programmer == bad idea Posted Mar 19, 2012 11:08 UTC (Mon) by tialaramex (subscriber, #21167) [Link]
But I didn't say our software developers can only program in one language. Any more than they can only speak one natural language. Nevertheless, having acquired some skill in more than one language does not somehow magically furnish you with ability in _every_ language. That's how we end up with nonsense like "C/C++" pretending two very different languages are really just the same. Moreover, it doesn't substitute for expertise in a single language. Knowing how to write an idiomatic Hello World in six languages doesn't make you able to implement a fast multi-threaded bloom filter in any of them.
One-language programmer == bad idea Posted Mar 22, 2012 22:24 UTC (Thu) by zuki (subscriber, #41808) [Link]
> The CPython developers have abandoned python 2, but almost no one is
> moving to python 3, and they're incompatible. Not really. It is true that python2 is still much more popular, but almost all big and important python projects have python3 versions. Numpy is finally available for python3 in debian as of a few day ago, which makes the deployment of python3 packages for many people much easier.
> Note that PyPy does not even *try* to implement python3, and most of the
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 15:20 UTC (Sat) by iabervon (subscriber, #722) [Link]
I think the right argument is in the other direction: even if you write the part of the program that does the real work in C, you should write the high level flow control in Python, because it'll make it much easier to use your C code for other purposes.
On the other hand, I think you should keep the real work out of your Python code not just for performance reasons but also because Python doesn't have good encapsulation, which means that the size of the total Python code is a factor in how hard it is to maintain the program as a whole.
But I think that Python is an ideal bridge between UNIX-style C code that does one thing well and monolithic interactive applications that do all of the steps of a complex task.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 22:07 UTC (Fri) by ovitters (subscriber, #27950) [Link]
Suggest not to be so critical. I have no problems with the speed of Python. Mr. van Rossum provided a nice language and that's a great contribution.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 22:35 UTC (Fri) by slashdot (guest, #22014) [Link]
I do.
In the shootout.alioth.debian.org tests, Javascript programs using V8 take 5 seconds, while equivalent Python programs take 50 seconds.
This is not just slow, it's utterly embarassing performance, and simply unusable for any non-trivial computation.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 23:31 UTC (Fri) by Sho (subscriber, #8956) [Link]
That's an apples-vs.-oranges comparison. V8 is a JIT, CPython is a pure interpreter. Do you benchmark PyPy?
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 7:29 UTC (Sat) by Cato (subscriber, #7643) [Link]
I think the point is that V8 and similar advanced JIT implementations have changed the game for this type of language. If a language doesn't have performance similar to V8 (say within a factor of two), and therefore requires a mixed-language approach with C/C++, it will be less productive on many projects than a language with an extremely fast JIT.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 10:58 UTC (Sat) by anselm (subscriber, #2796) [Link] My applications don't require extreme speed. On my projects, productivity generally hinges more on being able to develop and deploy new code and fix bugs than on using the fastest language environment around. This is why I use Python, which as far as I am concerned is the better language by a wide margin, instead of Javascript wherever I can. If execution speed should ever become a serious issue I'll look at PyPy.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 18:38 UTC (Sat) by robert_s (subscriber, #42402) [Link] This is not just slow, it's utterly embarassing performance, and simply unusable for any non-trivial computation. Yeah that's why nobody uses it for serious computation, I - oh wait.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 6:12 UTC (Mon) by ssmith32 (subscriber, #72404) [Link]
And who uses NumPy for "serious" numerical computation?
Like on a HPC cluster? Doing physics or weather simulations or something? Not a rhetorical question, I actually don't know.
I thought must serious numerical computation was in C and MPI on high-end clusters.. but I really have no idea. Heard of NumPy several times. Never heard of it being used for serious numerical computation, though.
Then again, someone who consulted for JPL once told me they do insane amounts of stuff in Excel spreadsheets.. so... not sure if science geeks using it for serious computation would mean anything anyways :D
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 12:29 UTC (Mon) by tnoo (subscriber, #20427) [Link]
Numpy is not usually used for "serious" computation, but it is so insanely convenient to use for everything below "serious", that many scientists use it for the "quick and dirty" data analysis and simple models.
Python/Numpy/Scipy really has an advantage over the competition (Matlab) that Python is a universal, fully object-oriented programming language, is extremely flexible, and is very easy to use for evolutionary programming.
Once the code is fully functional, maybe performance bottlenecks play a role, but profiling and refactoring some code to run a compiled module is usually sufficient.
The last step, which is probably rarely done, is to implement everything from scratch with performance as the main objective, and thus choosing the fastest language/library available.
But doing this last step before the evolutionary exploration of ideas is not possible, and for that purpose Python & Co is ideal.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 21, 2012 13:47 UTC (Wed) by jbh (subscriber, #494) [Link]
Another example is fenics (fenicsproject.org) which generates compiled extensions on the fly, specialised for the given differential equations. It is potentially faster than any general c library -- c++ templates may achieve the same, but the complexity would be much higher.
It's very nice to be able to prototype with help from numpy/matplotlib/etc, and use the same code (without matplotlib :-) on a cluster. And yes, this is real physics, on up to thousands of processors.
(Disclaimer: I am occasionally involved in dolfin/fenics development)
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 13:07 UTC (Mon) by magi (subscriber, #4051) [Link]
I write serious HPC code in Fortran 2003 and MPI/OpenMP ;-) For the rest I use python + numpy + matplotlib. Python might not be the fastest but it is very convinient. matplotlib is the nicest (2d) plotting library I know of. Out of interest I have tried out solving the Laplace equation using Fortran, pure python and python+numpy - python is slow but python+numpy is very close to the fortran implementation.
At the moment we are trying to convince people to move from matlab to python.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 21, 2012 23:50 UTC (Wed) by dashesy (subscriber, #74652) [Link]
Recently I convinced my wife to use SciPy and Numpy instead of MATLAB. At first she was reluctant but setting up Spyder (with ipython 0.10), Pylab and Matplotlib was convincing enough. As an added bonus I set up the MKL compiled Numpy which is comparable in speed with MATLAB for most of the useful calculations.
IMO, MATLAB is specially bad when it comes to writing anything more than a simple script. I have seen people passing a global cell array to every single function! Python on the other hand is a real generic language witch happens to be easy for scientists too. In science when using simple scripts performance is guaranteed with NumPy (MKL); most of the calculations can be vectorized and for-loops can be avoided if carefully designed.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 20:55 UTC (Sat) by Tobu (subscriber, #24111) [Link] The shootout maintainer decided he was busy enough benchmarking one implementation per language, and didn't pick the fastest Python implementation (CPython instead of PyPy). PyPy is currently about 5 times as fast as CPython.
More Python fragmentation Posted Mar 18, 2012 13:00 UTC (Sun) by man_ls (subscriber, #15091) [Link] My understanding is that PyPy causes further fragmentation in an already fragmented language: Python 2.x is not compatible with Python 3.y, and only a subset of Python runs in PyPy. Also, extensions in C might not work in PyPy. So you may end up with a few different versions of popular programs depending on where they run: (Python 2.x, Python 2.y) x (fast in CPython, fast in PyPy), which means you may have to provide four different versions of a program.What van Rossum doesn't seem to understand is that while this fragmentation is not good for Python programs, it is very damaging to the library ecosystem. The "little piece of your system" may not be in your program, but in one of the libraries you are using. At which point you are out of luck, or you have to enter a rabbit hole not apparent from the surface.
More Python fragmentation Posted Mar 18, 2012 14:54 UTC (Sun) by Tobu (subscriber, #24111) [Link] PyPy isn't a Python subset, it is a compatible alternative implementation of the same language. One PyPy developer did work on the faster but restricted approach (Armin Rigo on Psyco) before getting involved in PyPy. Of the incompatibilities between CPython and PyPy, the one that matters in practice is the absence of refcounting. Python 3 compatibility is being worked on. So is compatibility with the C api, which has been reworked to be less abstraction-leaky in Python 3. When C is used to accelerate bottlenecks, a pure-python fallback for PyPy can perform faster than the C version. People are also working on implementing enough C-api compatibility that Cython-generated bindings work out of the box; this is competing with another approach that makes Cython generate pure python + ctypes bindings. Some parts of the ecosystem are harder to crack, especially scientific Python, but that is being worked on by rewriting the numpy core. As far as compatibility being a maintenance burden: I tend to write Python 2 code, which is compatible with CPython2 and PyPy, and let distribute invoke 2to3 to convert my packages when installed in a CPython3 environment. Running a testsuite across multiple implementations and versions of Python is very simple using a tool called tox.
More Python fragmentation Posted Mar 18, 2012 15:23 UTC (Sun) by pboddie (subscriber, #50784) [Link]
No, PyPy is meant to be a fully-compatible implementation of Python, and with support for CPython extensions, a drop-in replacement for CPython. Moreover, there are concrete plans for PyPy to support Python 3 as well as Python 2.
Most compatibility issues with alternative Python implementations are purely due to library availability, and if there hadn't been such a focus on CPython and migrating code to C for improved performance, this would be less of an issue. In fact, some code is being migrated back to Python to take advantage of PyPy's just-in-time compiler. The remaining issues are arguably a result of people taking advantage of CPython implementation details, often in an ill-advised manner or "because they're there".
Make no mistake: the principal cause of fragmentation in the Python realm is that of the introduction of Python 3. Fortunately for the language designers, PyPy will probably come to the rescue of Python 3 because the improved performance will no longer be confined to Python 2 programs. And I can easily see PyPy, not CPython 3, being the preferred runtime for Python in the not too distant future because of this.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 0:41 UTC (Mon) by jmalcolm (guest, #8876) [Link]
Your statement lacks considerable nuance and context.
Does the Alioth test bench resemble your program? Most programs?
Let's say you are writing an app that is memory bound or I/O bound and nowhere near CPU bound. Most web apps fit this description and a big factor in their performance is network latency. For such an app, the choice of language may make only a small difference to performance but a big difference to programmer productivity.
There are many, many examples of systems that win in the market against systems that "perform" better.
Now what if the increased programmer productivity translates into more time for profiling and optimizing the design and infrastructure? These could easily lead to a faster system overall despite working with a base system that benchmarks more slowly.
Saying that something is "unusable" because of performance is silly at the best of times. Saying it about a system that is extensively used is even sillier.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 23:13 UTC (Fri) by hazmat (subscriber, #668) [Link]
Even ignoring the use of extensions for hot code, i'd suggest trying the same benchmark with pypy, which adds modern js browser techniques, optimistic jit based on speculative tracing to the python language. There's also some interesting work being done on getting jython to use pinvokedynamic on the jvm, which can yield considerable speedups, and has no GIL.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 22:01 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]
Every useably fast interpreted language has some functions that are implemented in compiled code.
For example, both Perl and Python have modules available that are compiled code, and when you use the functions in these modules you are calling out to the C (or whatever) code.
From what I understand (as I have not needed to implement a module with these speed requirements), Python makes it especially easy to do this.
In many ways, this gets back to the age old issue "Premature optimisation is the root of all evil", and that you need to profile your resulting program to find the bottlenecks and then improve those.
For most programs, there is a huge amount of logic and code that is not performance critical, and a small amount that is. Being able to use a flexible interpreted language like Python for the non-critical code and only needing to write the performance critical section in a faster (usually lower level) language can be a big win.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 8:50 UTC (Sat) by khim (subscriber, #9252) [Link] In many ways, this gets back to the age old issue "Premature optimisation is the root of all evil", and that you need to profile your resulting program to find the bottlenecks and then improve those. Funny how everyone likes to recall this small piece of the whole discussion when confronted with unbearably slow something. But the full quote is “We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil” and the same guy opined “In established engineering disciplines a 12 % improvement, easily obtained, is never considered marginal and I believe the same viewpoint should prevail in software engineering”. 10x-100x slowdown imposed by python usage is by no means “small inefficiency”, it's enough to put in in “portable bash replacement” bucket.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 11:55 UTC (Sat) by anselm (subscriber, #2796) [Link] and the same guy opined “In established engineering disciplines a 12 % improvement, easily obtained, is never considered marginal and I believe the same viewpoint should prevail in software engineering”. The interesting question in this context is »12% improvement compared to WHAT«. »Premature optimisation« is what happens when you intentionally make your code more convoluted because you have a gut feeling that »this will be faster« without actually having bothered to find out how fast the simpler version would have been. There is nothing wrong with optimising your code to reap the »12% improvement, easily obtained«, but this presupposes that (a) you have benchmarked your code first to establish a baseline to compare against, and (b) the optimisations you do make for that 12% gain actually fall within the realm of »easily obtained«. Many people jump through burning hoops to try to make their code »fast« from the get-go when a much simpler implementation would have been ready a lot sooner, would be a lot easier to maintain, and would have been adequate, speed-wise, in the first place. These are the people who come up with buggy implementations of quicksort in order to sort an array that will never contain more that 10 elements, because »everybody knows quicksort is faster than insertion sort«.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 28, 2012 0:31 UTC (Wed) by vonbrand (subscriber, #4458) [Link] The big problem is what 12% we are talking about. Most of the time a 12% improvement in programmer productivity would happily be paid for with 10 times slowdown of the code. Not all people using e.g. Java are terminally incompetent...
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 16, 2012 23:01 UTC (Fri) by arjan (subscriber, #36785) [Link]
now I sure would be interested to learn if Python can drop + retake the GIL for small-ish pieces of code, with the aim of getting benefit of HTM on machines that support that...
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 8:33 UTC (Sat) by ferringb (subscriber, #20752) [Link]
The question there is more if you're dealing in python objects; if so, then you need the GIL. If not, dropping the gil is easy/peasy and most cpy extensions do that (particularly around syscall blocks).
If you're doing strictly native python, then chunking it into processes is the usual route- can be annoying at times, but the forced separation also does wonders for crappy/entangled code bases....
That said, I don't quite see your notion of hardware transactional memory here beyond possibly liking buzzwords ;)
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 14:48 UTC (Sat) by arjan (subscriber, #36785) [Link]
HTM allows you to run "locked" sections in parallel as long as you don't have actual data conflicts
it's basically a way to make a lock scale in that sense.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 2:58 UTC (Sat) by davide.del.vento (guest, #59196) [Link]
> because for most of what you're doing,
> the speed of the language is irrelevant
Well, sometimes. Claiming that in general is disingenuous.
I daily deal with software whose profile is utterly flat. Some of them would greatly benefit from being ported (or partially ported) to Python. And yes, I'm looking at PyPy, but unfortunately there are some issues there, more often than not caused by CPython implementation details (e.g. numpy and GIL), not Python language itself.
So I agree that for some things Python is pretty good and for those things it doesn't need to be faster. But for other things, it does and "not using it" (as the first commenter said) it's not a good solution to the problem (and yes, I read the full article before posting)
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 5:07 UTC (Sat) by russell (subscriber, #10458) [Link]
I've just completed a multi year replacement of a C++ based air traffic simulator with one written in python. The python simulator can simulate 50x more radars and 166x more aircraft on the same hardware. It also offers more functionality and contains a lot fewer lines of code. Using python allows me to get my head into the problem space more than the language space. That resulted in a much more efficient and functional program design that more than made up for the speed drop in the language.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 7:46 UTC (Sat) by drag (subscriber, #31333) [Link]
Sssh. Try not to interrupt the ranting with facts.
Nevermind the fact that programs are only as fast as the programmers choice in algorithms allow them to be. Also lets completely ignore that one of of the major strengths of python is the vast library of modules and functionality that you can get along with it. And, of course, it's irrelevant that people have been making it scale for years and writing fantastic software using python completely despite the presence of the global lock in the interpreter.
If we just ignore everything then poor old Guido just tuck his head into the sand for the shame of creating such a worthless programming language and piss-poor implementation.
:P
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 8:10 UTC (Sat) by epa (subscriber, #39769) [Link]
No doubt Python helped you concentrate on the problem, but it's not quite fair to compare an older program in C++ with a rewrite in language X. If you had rewritten it in C++ starting from scratch you would most likely have got a good speedup too - perhaps even more than 100x.
I do agree that the right metric is not 'language X is five times faster than Y' but rather 'for a given number of programmer hours spent, the program you end up with in X is faster than that you'd get using Y'.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 18, 2012 22:16 UTC (Sun) by russell (subscriber, #10458) [Link]
Yes, a well written C++ program would be faster. However, the other ugly side of programming, budget, would have probably prevented me investing the effort required.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 12:58 UTC (Sat) by slashdot (guest, #22014) [Link]
That's probably because the C++ program was utter crap.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 16:55 UTC (Sat) by drag (subscriber, #31333) [Link]
duh. Why would you ever try to rewrite a program that wasn't?
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 23:08 UTC (Sat) by rgmoore (✭ supporter ✭, #75) [Link] Sure, but there are reasons programs turn out to be crap. One of them is that people spend so much time on the trees that they ignore the forest, and that's a much bigger risk if your basic programming elements are branches and leaves than if they're oak and spruce. This is the biggest, most common argument in favor of high level languages. By programming at the highest level, you can focus on the problem rather than getting bogged down in implementation details. That lets you make sure you have the program architecture right before spending a lot of time worrying about how fast it's going to run.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 10:45 UTC (Sat) by BackSeat (subscriber, #1886) [Link]
Is speed the only thing that matters?
For me, the power and clarity of Python are the most important factors. Speed issues, in the main, are trivial to fix; maintaining code at a later date is not always so easy, and the choice of language can be a significant factor in how easy it is.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 16:59 UTC (Sat) by drago01 (subscriber, #50715) [Link]
> Is speed the only thing that matters?
No but it does matter.
> maintaining code at a later date is not always so easy, and the choice of language can be a significant factor in how easy it is.
That mostly comes down to the developer(s) and how well structured the code is the language is a secondary factor here.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 17:53 UTC (Mon) by k8to (subscriber, #15413) [Link]
The language is a huge factor, much larger (comparing things like python and C++) than local code quality except for pathological cases.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 12:40 UTC (Sat) by edlenz (guest, #12021) [Link]
Python is a very powerfull language, with a bunch of libraries. Mpi4py, numpy and scipy for example, can be used to develop very fast code. Some tools like psyco can speed up Python code A LOT. Also, pypy proves that one thimg is the language, other is the implementtion.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 17:10 UTC (Sat) by jengelh (subscriber, #33263) [Link] Try implementing $data =~ s/([^\x07\x08\x0a\x0d\x20-\x7e])/sprintf "\\%03o", ord $1/egs; in Python. python2 just is behind it terms of runtime. (But perhaps the Python folks would never even do that...): def to_octal(s):
return "\\%03o" % int(ord(s.groups()[0]))
data = re.sub(r"([^\x07-\x0d\x20-\x7e])", to_octal, data)
Mixed-language programming is a pain Posted Mar 17, 2012 20:30 UTC (Sat) by endecotp (guest, #36428) [Link]
I went through a phase of mixed-language programming. It included mixing tcl and C (or [incr tcl] and C++) using SWIG, and mixing SQL with C++. Neither experience was good:
- Marshalling non-trivial data between the different languages can be painful when they don't match perfectly.
- It's rarely just "one little piece" of the system that needs to be fast; it will be a selection of inner loops in different places, and as the application evolves, they will change.
- You end up structuring your system along the language-divide, when that is not the natural grain of the design.
- It requires more "book learning" before you can start. (And since the features needed for mixed-language programming are inevitably "advanced" features of each language, you need to be an "advanced" user of all of them.)
My conclusion now is that it is better to choose one language that is going to be sufficiently performant, and use it for everything. I've chosen C++, but there are other possibilities.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 17, 2012 20:31 UTC (Sat) by Tobu (subscriber, #24111) [Link] There are many options to speed up Python for computation. PyPy is good at JITting while staying 100% Python-compatible, Numpy is good at data parallelism, numexp at optimising and vectorising array expressions, Pandas at very large, cache-oblivious arrays, Cython can speed up a bottleneck with compilation of a slightly-restricted dialect as well as wrap C libraries, and scipy.weave does compilation of inline blocks.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 18, 2012 7:11 UTC (Sun) by gmaxwell (subscriber, #30048) [Link]
A lot of tools to learn— and none provide 100% protection from python runtime slowness like python abstinence does. :)
Personally, I complain about python mostly because I tried the "my time is more valuable than the computer so I'll do all my one-time use code in python instead of C" route — only to discovery that my admittedly simplistic python code was often unusably slow compared to my C— often hundreds to as much as 2000x times slower. Now, sure— my one time use code (often optimization problems, and OKAY at writing fast C) is probably not the most representative case... but it's enough to disprove, in my eyes, the blanket claims that python is fast enough. PyPy helps, but it's still often miserable.
It makes me sad though— writing a bunch of boilerplate code in C gets boring.
But having to learn a bunch of tricks and tools to make python acceptably fast would substantially diminish the the "my time" vs "machine time" advantage. ... and python has other problems— the lack of static typing means you get burned by idiotic runtime errors that you'd never hit in C, high memory utilization, etc. Oh well. Hopefully pypy will eventually get the performance close enough that the other issues will be outweighed by the advantages.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 18, 2012 10:04 UTC (Sun) by HelloWorld (guest, #56129) [Link]
> the lack of static typing means you get burned by idiotic runtime errors that you'd never hit in C
Now just imagine what life would be like with a real, sensible type system instead of the pile of crap that the C type system is. *scnr*
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 19, 2012 18:10 UTC (Mon) by ballombe (subscriber, #9523) [Link]
You mean Haskell ?
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 18, 2012 10:32 UTC (Sun) by drago01 (subscriber, #50715) [Link]
> It makes me sad though— writing a bunch of boilerplate code in C gets boring.
Well C any python aren't the only languages in existence ;)
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 18, 2012 19:28 UTC (Sun) by deepfire (subscriber, #26138) [Link]
Well, very-high-level, very dynamic languages with high-quality, portable, JIT-ting, honestly threaded, natively-compiling, partially-typechecking implementations _do_ exist.
And yes, some of those are even perfectly integrated with Emacs.
I guess that the collective subconscious values the light-weight syntax of Python higher than the (to me, anyway, obviously) more important things.
As an aside, the Python libraries, while numerous, do not seem to be of such high quality to me.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 18, 2012 20:12 UTC (Sun) by Zizzle (guest, #67739) [Link]
I think the attitude of "write the critical bits in C/C++" may actually hold back the performance of Python in the long term.
They are stuck with their binary ABI.
If everyone was writing pure Python and all the libs were in python, it would be possible to come out with a new JIT and speed everything up.
As it stands, with every significant project using lumps of binary, programmed in C or whatever, there is no hope of a new Python engine that will just work.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 18, 2012 22:56 UTC (Sun) by faassen (guest, #1676) [Link]
Good point about C modules holding back alternative Python implementations. I do think it can; you see people using Python on the app engine complain they can't use certain C modules for instance. But I don't think a binary ABI is the problem; people *can* recompile.
Actually PyPy does have hacks to allow it to use some CPython C modules, though you'll have to recompile them. It's far from perfect right now but it does work.
There's also Cython, which is an extended Python which can compile down to fast C in many cases and is also used to interface with C libraries. This avoids the Python API for the module developer, and has already been shown to do this by supporting both Python 2 and Python 3 with the same Cython codebase.
Van Rossum: Python is not too slow (InfoWorld) Posted Mar 21, 2012 18:05 UTC (Wed) by zooko (subscriber, #2589) [Link]
By the way I just finished publishing benchmarks of the "pycryptopp" library that I maintain. It provides a simple and Pythonic interface for secure hashing, stream ciphers, and digital signatures. Its implementation re-uses C and C++ code written by better programmers than me, because that's a very secure and efficient way to provide this sort of functionality.
https://tahoe-lafs.org/pipermail/tahoe-dev/2012-March/007...
|
|||||||||||||