Nuitka 0.60 released

[Posted September 28, 2018 by corbet]

Nuitka is a compiler for the Python 2.7 and 3.7 languages; version 0.6.0 is now available. "This release adds massive improvements for optimization and a couple of bug fixes. It also indicates reaching the mile stone of doing actual type inference, even if only very limited." At this point, the claim is that all Python language features have been implemented, so the focus is shifting toward optimization.

I'm confused...

Posted Oct 1, 2018 5:40 UTC (Mon) by pr1268 (guest, #24648) [Link] (8 responses)

From the article:

Nuitka is a Python compiler. [...] You feed it your Python app, it does a lot of clever things, and spits out an executable or extension module.

And later (on the same page):

It translates the Python into a C program [...]

So which is it? A native code compiler or a C-language translator?

And assuming the latter, then why not go straight from Python to assembly?

A compiler that translated code straight from any of the contemporarily-popular languages to arch-specific native assembly would really be nice. Just my $0.02.

I'm confused...

Posted Oct 1, 2018 7:11 UTC (Mon) by jtaylor (subscriber, #91739) [Link] (2 responses)

It is likely translated to C so it can conveniently use the Python C-API and make use of the optimizations C compilers can already do.

I'm confused...

Posted Oct 1, 2018 14:39 UTC (Mon) by jamesh (guest, #1159) [Link] (1 responses)

And by generating code that uses the libpython C API, it means that it can trivially call into CPython extension modules.

I guess the main opportunities for optimisation would come from identifying occasions when it is safe _not_ to use the libpython API. For example, if you know the two values you are adding are integers then generate code that adds two ints rather than calling PyNumber_Add() on two PyObjects.

I'm confused...

Posted Oct 1, 2018 15:10 UTC (Mon) by kayhayen (guest, #127589) [Link]

That's exactly what it does. Initially it basically only meant to replace bytecode with Python C/API call, then avoid that as much as possible, and lately, with 0.6.0 first for bool, for 0.6.1 for void, and hopefully soon for int, to stay within C types as much as possible.

I call it graceful degradation. Code that can be fast should be. And if you go all in on special Python features it will become less optimized, but still faster.

Obviously a very hard project that I am in there, but from now on I am hoping to improve performance with every release for the 0.6.x series, whereas 0.5.x had 33 teleases preparing that in the background mostly.

Yours,
Kay (Author of Nuitka)

I'm confused...

Posted Oct 1, 2018 10:19 UTC (Mon) by wazoox (subscriber, #69624) [Link] (4 responses)

That's detailed in the developer manual: http://nuitka.net/doc/developer-manual.html
It generates C11 code with a fall-back on C++03 for the cases where a good C11 compiler isn't available (Windows).

I'm confused...

Posted Oct 1, 2018 15:13 UTC (Mon) by kayhayen (guest, #127589) [Link] (3 responses)

Actually there is MinGW64 which appears to generate substantially faster code than MSVC (64 bits Python2 pystone is what I looked at only), and it does C11. And there is also Clang, which upcoming versions will support as well via "clang-cl.exe", where I am hoping to benefit from the fact that Clang does C11. People are claiming that Clang is better than MSVC too. In my benchmark it fell into the middle of MSVC and MinGW64.

Yours,
Kay

I'm confused...

Posted Oct 1, 2018 23:17 UTC (Mon) by atai (subscriber, #10977) [Link] (2 responses)

MinGW64 actually means gcc

I'm confused...

Posted Oct 1, 2018 23:42 UTC (Mon) by k8to (guest, #15413) [Link]

Sure, though I think the runtime for the produced executables is sometimes a significant factor for this type of situation, so it's useful to provide that.

I'm confused...

Posted Oct 2, 2018 13:21 UTC (Tue) by kayhayen (guest, #127589) [Link]

Yes, it uses gcc code at the core, but e.g. MinGW is apparently less Windows compatible, doesn't offer all the tools, and doesn't work for Nuitka. The C runtime actually also matters a bit and I think that is where gcc ports differ. I for my part wished there was less variations for gcc on Windows.

Nuitka 0.60 released

Posted Oct 1, 2018 20:35 UTC (Mon) by mageta (subscriber, #89696) [Link] (2 responses)

Not sure whether its obvious, but how does this compare to Cython (http://cython.org/)? From what I remember Cython can do similar things. Compile python modules to C and then compile/link against libpython. This optimizes only the own module.

Is this similar? What are some differences, just for comparison sake.

Nuitka 0.60 released

Posted Oct 1, 2018 23:46 UTC (Mon) by k8to (guest, #15413) [Link] (1 responses)

You can find discussions of this already, some in passing is discussed here

http://nuitka.net/posts/static-compilation-that-is-the-po...

the article isn't written specifically to address Cython so Cython fans might object since this isn't really working hard to provide a lot of evidence for its viewpoints. That isn't the goal of the article though, so I think it's very understandible.

I'd love to hear more about the comparison than what I've been able to find.

Nuitka 0.60 released

Posted Oct 2, 2018 13:39 UTC (Tue) by kayhayen (guest, #127589) [Link]

I have promised people that more benchmarks will show up now that I am more actively working on the performance side of things. Follow me on Twitter or my TWN (This week in Nuitka) posts, I surely will point out stuff in the coming month. And there is http://speedcenter.nuitka.net but I consider it barely usable.

For Cython I think it would be unfair to publish them and call them numbers. After all you are expected to apply a bit of cdef magic to it, and not doing it, would be cheating.

And thanks for the link. I once wrote that indeed and every word of that is still true.

As for type inference from code annotations, still not there, but once substantial effects are achieved by saying it's an int, there will be attempts to show how Cython code annotations and Nuitka not using them, coming to what I hope similar results.

And Nuitka will still be full of performance bugs, where it doesn't even match CPython for some more time. I am fighting unicode in-place mysteries on Python3 recently, and pystone on Python3 is not nearly as much faster as pystone on Python2 when compiled, there will be bad things still, but the good news is, finally it is becoming the primary focus, and things will be doable.

Yours,
Kay