Unladen Swallow 2009Q2 released [LWN.net]

The Unladen Swallow 2009Q2 release is out. Unladen Swallow is an accelerated Python implementation; LWN covered it in May. It now uses LLVM to compile Python functions to machine code, and appears to be coming along nicely, even if the developers do not yet recommend its use in production situations. "Unladen Swallow 2009Q2 passes the tests for all the third-party tools and libraries listed on the Testing page. Significantly for many projects, this includes compatibility with Twisted, Django, NumPy and Swig."

to post comments

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 13:20 UTC (Wed) by juanjux (guest, #11652) [Link] (28 responses)

I didn't knew about this project. A JIT compiler for Python using LLVM, with Google engineers and a goal to get 5x the current performance and be integrated into mainline Python (which, with Guido working for Google, shouldn't be too dificult.) All this for the end of 2009.

Just WOW!

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 13:44 UTC (Wed) by kragil (guest, #34373) [Link] (23 responses)

Yeah, this could make Python the an even more desirable dev platform on Linux (not sure Unladen Swallow will even work on windows though).

Core parts of Gnome and KDE use Python already and that will only increase with Zeitgeit etc.

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 14:16 UTC (Wed) by drag (guest, #31333) [Link] (22 responses)

Yeh... if Unladen Swallow can be made production, get the improved performance, get the memory usage under control, and effectively replace the CPython implementation then that should end up being quite a boon for the entire language.

And as a side effect it should reduce quite a bit of the need/desire for things like Mono or Java and hopefully convince more people to get out of the C-language application writing racket.

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 15:29 UTC (Wed) by qg6te2 (guest, #52587) [Link] (3 responses)

hopefully convince more people to get out of the C-language application writing racket

Come on drag. A little while ago you were exalting the C language. Python has its place, but it's never going to be a speed demon, nor is it going to be as efficient as C or C++. The lack of solid variable types in Python makes sure of that. While machines are getting faster, their power shouldn't be wasted through inefficient apps.

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 16:02 UTC (Wed) by kragil (guest, #34373) [Link]

Never say never. There are a lot of jit engines in the Python world and you never know how intelligent those will get.

And now it makes sense for a lot more apps to be developed in Python. If you need complex data processing you can still extend Unladen with C.

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 20:47 UTC (Wed) by drag (guest, #31333) [Link]

> Come on drag. A little while ago you were exalting the C language.

Well thanks for remembering. :) Ya sure. You can do anything you want in C, but because you can doesn't mean you should. :)

Like I said before the real advantage of not using C is the amount of time you save designing, writing, and debugging the code. In terms of programming time python is _FAST_. Very fast.

For example I wrote a program in python that processes a dump from a NAND flash on embedded Xscale system obtained from jtag. A portion of the flash uses a combination custom hardware driver combined with MSFLASH driver. There is no hardware balancing and the OS writes directly to the flash with fat file system.. so your dealing with combination of a custom load-balancing and block-to-mtd translation layer.

So the python program I wrote disassembled the entire sectioned dump into individual logical blocks in a array... then reassembled it in order, while filling in the blanks, to produce a drive image that was then mountable in Linux via loopback file system and the fat16 driver. From that then I could obtain the contents of the file system. I was happy it worked. I was able to write the program in one (long) evening and the total execution time of the script is less then 10-15 seconds to process a ~24MB file system.

The entire thing, after validating it on a few different drive images, was production-ready in a couple days after that. I even went through code review with a experienced programmer (which I am not) that never programmed in python before... but he was still able to follow the logic and understand the program.

---------------------------------------

The ability to quickly right, debug, and then have readable code, is a very valuable thing in a language.

It's beyond just trying to 'make programming easy' or 'approachable'. I am convinced that as the lines of code go up a program.. the utility of having the source code goes down. High LOC is a barrier to entry for people and defeats much of the purpose of having the project open source.

--------------------------------------

C is a very important thing and I don't hate it. It's just that it's very slow to program in.

So instead of using a shotgun approach with C and simply programming _everything_ in C you need to pick your battles. This means that you spend programmer time on C were it counts. You want C in areas of your programs and areas of your system that are very performance sensitive.

Having OpenGL libraries in C is a good thing. Having Widgets and application libraries in C are good things. This is because the time spent by programmers on producing optimized and debugged C code can be applied to the widest benefit to the system at large.

One of the very nice thing with Python vs something like Java is that it plays very well with others.

Like Java wants everything to be 'java'. Java-this, Java-that. Java-everything. So what ends up happening is that stuff gets slow.

However with Python you to your initial application in pure-python, but as the application matures and you identify areas were performance is poor, then you rewrite that using optimized C and import the code back in as modules.

Or like if I wanted to write a python-based web browser. I am not going to write the HTML renderer or the Javascript interpreter in Python.. that would be insane. Instead I am going to use webkit or gecko rendering engine and just make the UI in Python. When you click a button on a program it is not going to matter if it's C or if it's python or whatever... the button press will happen and the code backing it will happen faster then the next screen refresh so it's irrelevant.

However with the UI in python then the program UI would be quick and easy to write and I can spend my time refining and improving the UI interface rather then writing code... and other people (if I do my job properly) should be able to more easily extend it and modify it.

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 10:49 UTC (Thu) by NAR (subscriber, #1313) [Link]

The language itself doesn't make a program more efficient. Especially, badly written C/C++ applications have the tendency to use up much more memory than it's necessary due to memory leaks. In my experience the average developer lacks the skill to produce C/C++ code without memory leaks, buffer overflows, etc. That's where python, java, C#, etc. come into play, they eliminate a number of classes of bugs. I think there are a many people coding in C/C++ that should really not use such languages.

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 18:08 UTC (Wed) by iabervon (subscriber, #722) [Link] (17 responses)

Even if Python were as fast as C, Python is much harder to debug than most other languages, because just about any Python code could actually be correct, if some distant part of the program inserted variables in its scope or added attributes to its objects.

There are some applications Python is good for, and some of them care about performance, but there are others where it's much more useful to get compiler errors for large classes of common mistakes.

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 21:51 UTC (Wed) by dw (subscriber, #12017) [Link] (16 responses)

Relying on type checks from the C compiler hardly gets you much, it's a 1% improvement on ensuring the code is grammatically correct. I'm not sure which classes of common mistakes you're referring to, but preventing addition of integers and strings, and so on, is not my idea of a big win.

As for arbitrarily munging with module namespaces, and so on, well, you can also write inline asm from C or write jumps between function bodies. ;)

Unit/system tests that exercise code catch a hell of a lot more classes of bugs than C's idea of type checking ever could, which you should probably be doing for C projects too. The question is which language leaves you with more time to write such tests..

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 22:00 UTC (Wed) by clugstj (subscriber, #4020) [Link] (4 responses)

I'd have to say that it is MUCH MORE than 1% improvement. You can compile just about anything in Python. With Python you have to write a lot more unit tests to verify that the code is to the level of what a clean compile would tell you in a more strongly typed language.

Of course that flexibility can be useful, but it isn't free.

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 0:17 UTC (Thu) by drag (guest, #31333) [Link] (2 responses)

> I'd have to say that it is MUCH MORE than 1% improvement. You can compile just about anything in Python. With Python you have to write a lot more unit tests to verify that the code is to the level of what a clean compile would tell you in a more strongly typed language.

Python is a strongly typed language....

$ python
Python 2.5.4 (r254:67916, Feb 17 2009, 20:16:45)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> cow = "5"
>>> cow = 1 + cow
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'

-------------------

You can 'compile' it also:

$ cat > hello_world.py
print("hello world")

$ python
Python 2.5.4 (r254:67916, Feb 17 2009, 20:16:45)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import py_compile
>>> py_compile.compile("hello_world.py")
>>> exit()

$ rm hello_world.py ; chmod +x hello_world.pyc

$ ./hello_world.pyc
hello world

Well.. it's byte-compiled, not native machine code. Same difference, pretty much. What don't you get from executing a python program that you don't get from compiling C++?

There are plenty of tools for analyzing python code for correctness and whatnot....

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 0:22 UTC (Thu) by drag (guest, #31333) [Link] (1 responses)

Sorry, I know I am being a bit foolish.

I think you meant 'statically' typed, not 'strongly' typed.

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 4:12 UTC (Thu) by k8to (guest, #15413) [Link]

Python is definitely *not* statically typed.

>>> myvar = 4
>>> myvar = "hi"
>>>

Notice no error.

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 4:14 UTC (Thu) by k8to (guest, #15413) [Link]

With all languages, you have to write a lot more unit tests. It's just the statically typed crowd who operate under some fantasy that they don't.

That python forces you into this is a feature.

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 22:11 UTC (Wed) by iabervon (subscriber, #722) [Link]

The useful thing isn't the type checking but rather the fact that variables have to be declared in a lexically enclosing scope, which means that trivial typos are compile-time errors rather than runtime errors (when that line is actually reached). Also important are things like calling functions with the wrong number of arguments (swapping % and , in function calls). And it's hard to write unit tests for cases which shouldn't be reachable, but which are important for reporting when unexpected things happen.

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 10:54 UTC (Thu) by NAR (subscriber, #1313) [Link] (9 responses)

The type checks are really useful when the code is being refactored. The type is modified at one place, then the compiler shows all the other places where the code should be updated. In this case the unit tests might be not that useful, because e.g. it takes too long time to run the unit tests or the unit tests don't provide 100% coverage (happens in real world, especially with error cases).

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 10:56 UTC (Thu) by dw (subscriber, #12017) [Link] (8 responses)

This is not a feature of static languages. See for example Rope, http://rope.sourceforge.net/

Hmm... May be I misunderstood something

Posted Jul 17, 2009 4:41 UTC (Fri) by khim (subscriber, #9252) [Link] (7 responses)

How can the rope help in real usecase: change of the interface of in big codebase? I can explain how it's done with C:
1. Old version of feature (class, function, etc) is marked as __attribute__ ((deprecated)) (with comment about possible replacements).
2. After month or two feature is removed.

Note: the codebase is big and that means you can not get access to all users of the feature (they are in diffirent repositories, etc). How can this be done with rope?

Sure, the rope looks nice, but does it work for big projects? I was under impression that it can only do "IDE-style refactoring": where all project can be loaded in one tool. This is where Java IDEs are great, but this NOT the real advantage of static typing. The real advantage is simple fact that interface is checked at compile time and, the most iportant, changes to the interface are checked at compile time!

If you start adding unittests required to check the interface for all classes you are using you are losing the "number of lines" advantage Python offers over C/C++ really fast... Sure C/C++ are awful languages too, but it's related to stupid low-level C pointers, not to the static typing...

It's not rocket science..

Posted Jul 17, 2009 4:52 UTC (Fri) by dw (subscriber, #12017) [Link] (2 responses)

--- module.py.bak	2009-07-17 05:47:46.000000000 +0100
+++ module.py	2009-07-17 05:48:25.000000000 +0100
@@ -1,5 +1,11 @@
 
+import warnings
+
 def some_function():
+    warnings.warn('deprecated, use amazing_function() instead!')
+    return some_function_deprecated()
+
+def some_function_deprecated():
     do_old_fashioned_stuff()
     if True and 0x62:
         return

By some accounts, relying on GNU C features to 'increase' maintainability is on a road straight to hell. ;)

It's not rocket science..

Posted Jul 17, 2009 8:22 UTC (Fri) by amonnet (guest, #54852) [Link]

Sure,

this works well for commonly used codepath that will be covered by testings. For occasionnal codepath, i'm pretty sure no one will catch the warning until production.

Worse then useless

Posted Jul 17, 2009 15:32 UTC (Fri) by khim (subscriber, #9252) [Link]

Not only it fails to catch the errors at build time it gives end-user warnings about some program internals - the last thing user needs.

Sure, the Python is great. Sure, with right amount of testing you can make small or even medium programs work. Yet all this is breaking apart for a big programs.

This is not a disaster (one-size-fit-all is not good approach, really), hust something to keep in mind.

Hmm... May be I misunderstood something

Posted Jul 18, 2009 10:47 UTC (Sat) by bockman (guest, #3650) [Link] (3 responses)

Sure, compiler checks are useful. But are not enough, and this is a point that sometime is forgotten in comparing static and dynamic typing. An API can mantain its formal interface but change its meaning (e.g. the parameter 'length' was inches and now is centimeters ) and no compiler will help you with that.

And I don't think that in a dynamic language one has to do explicit unit test for wrong parameter types : misusage of a function is bound to trigger some exception or make anyway fail the test. The resulting error message might not be the most informative, but usually lead you to the original problem. What you have to do is ensure full code coverage, which is a good practice also in statically-typed languages.

Not that I would dislike having more compiler-time checks in python: there are tools that help with that, and maybe they will help more in the future making clever usage of python3 annotations.

Ciao
------
FB

Hmm... May be I misunderstood something

Posted Jul 18, 2009 19:34 UTC (Sat) by farnz (subscriber, #17727) [Link] (1 responses)

Actually, given an expressive type system, the formal interface would change for that change in meaning. For example, using a type system comparable to that of Haskell (but C-ish syntax), I could define a function Storeys buildingHeightInStoreys( Metres<Int> length );. When I change that function to work in inches, it becomes Storeys buildingHeightInStoreys( Inches<Int> length );.

Throw in type inference, and you get some of the benefits of dynamic languages like Python. Again, using C-ish syntax, I could declare a function USDollars estimatedRent( auto height );, and write suitable behaviour for each variation on height (perhaps by unifying on height in storeys, and estimating from there).

Having said all that, it's a tradeoff in terms of work - the question is not whether static types are enough to stop all bugs on their own, but whether the reduction in tests that they permit is outweighed by the cost of static types. I believe that in C, this isn't true. In Haskell, however, people claim that it is true; this is one that remains to be tested more formally.

Hmm... May be I misunderstood something

Posted Jul 18, 2009 19:45 UTC (Sat) by dw (subscriber, #12017) [Link]

In this particular case I think 'the pen is mightier than the sword'.. just have a paragraph in your style guide requiring the scale and unit be included in the variable name, or at least documented at the variable's declaration site.

I personally work on one project where units of time are stored in things like "lengthMs", "startDelayMs", etc. Seems to work well for me.

Compiler is not substitute for brain, sorry...

Posted Jul 19, 2009 6:32 UTC (Sun) by khim (subscriber, #9252) [Link]

Sure, compiler checks are useful. But are not enough, and this is a point that sometime is forgotten in comparing static and dynamic typing.

Sorry, but this is bull-shit.

An API can mantain its formal interface but change its meaning (e.g. the parameter 'length' was inches and now is centimeters ) and no compiler will help you with that.

Fire the person who commited such a change (or revoke his commit rights) - problem solved. It's resposibility of programmer to guarantee that formal interface changes if the semantic changes: new method should get some suffix prefix so it's impossible to mix old and new version, it's not a rocket science. Bad programmer can write awful unmaintable program in any language.

And I don't think that in a dynamic language one has to do explicit unit test for wrong parameter types : misusage of a function is bound to trigger some exception or make anyway fail the test.

Not at all. Take your own example: suppose you have an interface where you enter paraments of sattelite and it returns some stats. Suppose you have two implementations (different approximation methods or something). Unittest for interface will test that everything works for trivial case, unitests for implementations will check error margins and other such things. The exception will be triggered, alright: when orbiter will either try to orbit Mars below surface or "NoSuchMethod" exception will be thrown and the system will be shut down. By that time it's kinda too late...

What you have to do is ensure full code coverage, which is a good practice also in statically-typed languages.

Full code coverage is nice and all, but unfortunatelly it's not an option for a big system: you can only test one component a time, because the whole system is too big.

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 9:16 UTC (Thu) by suckit (guest, #59597) [Link] (3 responses)

Yeah, sounds great, but according to a quick test, currently it doesn't really offer any noticeable speed increase (if you compare 5x against 3-4 percents plus).
See:
http://suckit.blog.hu/2009/07/16/python_unladen_swallow_p...

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 15:04 UTC (Thu) by mepr (guest, #4819) [Link] (2 responses)

is this a network bound or cpu bound program

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 15:16 UTC (Thu) by suckit (guest, #59597) [Link]

In python, an authoritative DNS server is CPU bound. :)

Unladen Swallow 2009Q2 released

Posted Jul 17, 2009 14:38 UTC (Fri) by xoddam (subscriber, #2322) [Link]

Huh? I don't know that! Auuuuuuugh!

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 16:41 UTC (Wed) by rriggs (guest, #11598) [Link] (4 responses)

I'm really looking to the Q3 release where the intention is to remove the GIL. I have had many situations in the past decade where I needed speed, had many CPUs available and a problem that lent itself to Python & threading. But the GIL always got in the way in these situations. I could only choose Python OR threading.

When I (invariably) did choose to use Python, I usually would end up spending most of my time on multi-processing & IPC issues and not the main problem domain.

Some of the newer modules (e.g. process) do help, but they are not nearly as helpful as removing the GIL will be.

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 20:54 UTC (Wed) by drag (guest, #31333) [Link] (2 responses)

I like using forks and sockets for IPC.

Linux does 'COW' with memory for forks so that as the number of processing 'threads' go up the memory overhead does not match. The only unique code would be stuff that gets created after the fork, which is probably going to be unique to each process/thread anyways if you do your job right and load up the modules you need prior to the fork().

I don't know how helpful that approach would be. Probably not much, but it's what I like doing.

Unladen Swallow 2009Q2 released

Posted Jul 15, 2009 23:33 UTC (Wed) by rriggs (guest, #11598) [Link] (1 responses)

When you have 16 worker threads all consuming work from a producer thread, the amount of coding you have to do for sockets (which I use as well) is a bit cumbersome (serializing the data, select() or poll() loops, waitpid(), etc.) given the clean interface one has with Queue and Thread.

Sometimes serializing the data (pickle/unpickle or what ever makes sense) takes up too much overhead.

Memory usage is rarely an issue. I've got enough RAM, enough CPU; I just need my favorite language to make the use of the resources I have a bit simpler.

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 11:28 UTC (Thu) by pboddie (guest, #50784) [Link]

When you have 16 worker threads all consuming work from a producer thread, the amount of coding you have to do for sockets (which I use as well) is a bit cumbersome (serializing the data, select() or poll() loops, waitpid(), etc.) given the clean interface one has with Queue and Thread.

There's a list of probably appropriate solutions for you on the Python Wiki; my humble offering, pprocess, tries to hide the awkward socket management behind fairly straightforward abstractions, as (I believe) do other solutions including the 2.6/3.x standard library offering, multiprocessing.

There are also libraries like Kamaelia which take a more radical approach to doing concurrent programming, and these certainly shouldn't require you to manage the tedious details.

Unladen Swallow 2009Q2 released

Posted Jul 16, 2009 0:31 UTC (Thu) by sergey (guest, #31763) [Link]

I've just tried the multiprocessing module in Python 2.6 for this and was able to make a single-threaded program scale on multiple CPUs quite nicely. It was a custom database dump code, and it took less then hour to make necessary changes and debug multiprocessig-specific quirks. What's even better was to find out that it works well with py2exe, so I was able to package up the whole thing into a single executable file and give it to my teammates who don't has Python installed. I'm sure multiprocessing won't work for some "interesting" requirements, but it seems to be a very efficient and well-designed general purpose solution.