Pyjion

Dino Viehland and Brett Cannon of the Microsoft Azure data science tools group presented on Pyjion, a just-in-time (JIT) compiler for Python, at the 2016 Python Language Summit. They also discussed a JIT API they would like to be added to Python so that various JIT engines can be dropped into Python for testing and evaluation.

Cannon started by noting that Pyjion is the only Python JIT that targets Python 3 only, with Viehland adding that it is based on 3.5.1. There are three goals to the Pyjion project, which Cannon and Viehland have been working on in their spare time at Microsoft. The first is to create an API for CPython that will allow plugging in a JIT for the language.

Cannon said that a proposal for a frame evaluation API, which would allow a JIT engine to be called when a frame object is being executed, was posted to python-ideas in mid-May. A PEP was posted to the python-dev mailing list shortly after the summit. The idea behind it is "pretty simple", Viehland said.

A second goal was to produce a proof-of-concept JIT to help drive the API design, Cannon said. They used the CoreCLR JIT to start with, but the back-end might change at some point, Viehland said.

The third goal was to create a C++ framework for CPython JITs to build from. CoreCLR is written in C++; the framework may be useful to those trying to use other C++-based JITs. For example, the framework does some translation of bytecode to machine code and some type inference, Viehland said.

The reason they are trying this is fairly obvious—"faster is nicer"—Cannon said. Pyjion does its JIT at the code-object level. The Python bytecode is translated to the equivalent Microsoft intermediate language (MSIL) code. An abstract interpreter is used to gather details on the code, such as inferring types and recognizing when float and integer values can be handled without boxing.

A key piece of the puzzle is that Pyjion uses the CPython API to maintain compatibility, Cannon said. That means that extension libraries written in C (e.g. NumPy) will still run with Pyjion. In fact, the entire Python test suite is passing, with the exception of some variable-tracing tests.

All of that was done with just two changes to the CPython API. A function pointer was added to the interpreter state to call out whenever a frame gets evaluated (InterpreterState->eval_frame). In addition, some "scratch space" was added to Python code objects (PyCodeObject->co_extra). The scratch space is simply a Python object, so the memory management of it is straightforward (it goes away when the code object does).

The eval_frame hook is useful for things like debugging and tracing as well, Cannon said. It allows an injection point that did not exist before, which surprisingly opens a lot of doors. It is also simple to use.

The scratch space is used by Pyjion to track various attributes of the code: how many times it has been executed or whether JIT compilation has already been tried and failed, for example. It also contains a pointer to the JIT-compiled code and some other housekeeping information.

There were some "bumps in the road", especially with regard to the stack. Python has two stacks (one for execution and one for exception handling), while CoreCLR only has one. A bigger problem is that there are some bytecodes in CPython that leave items on the stack when the frame exits, which is forbidden in CoreCLR.

Nick Coghlan noted that 3.6 is the right time to fix any of these bytecode issues since the bytecode format will be changing. "People will hate us already, so do it now". Alex Gaynor said that PyPy has already fixed the problems with extra data left on the stack; CPython could just copy those changes.

Cannon then presented some preliminary benchmarks compared to stock Python 3.5.1. Several showed some improvement using Pyjion, though the string-heavy tests were much worse; there has been no effort to optimize those yet, he said. Out of the 41 benchmarks they ran, 14 are slower with Pyjion, 12 are the same, and 15 are faster. As Viehland noted, that was all accomplished as a part-time effort by the two of them.

There are plenty of opportunities for further optimization, Cannon said. The main thing they were striving for was compatibility with the C extensions, which they were able to achieve. Viehland added that inlining is not yet supported, which would provide a huge boost; there are also more types that could be optimized. Cannon said that they "want a JIT space race in Python", which is possible with only small additions to the API.

One attendee asked about supported platforms. Cannon said he had not built it for Linux, simply because he hadn't ported the build scripts yet. Viehland said that he expects that it will run well on Linux since "the whole point is to get ASP.NET on Linux", but they haven't done it yet.

Larry Hastings pointed out that code objects have always been immutable in Python, though he is not sure that it matters that the scratch space changes that. Cannon said that the scratch space is simply that, so it could be thrown away without causing any problems. If a code object were serialized, the scratch space could simply be ignored and the code object would still run.

The API does not provide support for tracing JITs such as PyPy, one attendee said. Cannon agreed and said that the API would have to change to support all possible JITs. Viehland said that it may make sense to add more to the API over time. The current proposal "just gets our foot in the door".

Beyond that, Pyjion is x86_64-only at this point; there is no support for ARM processors. Viehland said there has been thought about moving to the ChakraCore JIT, which does support ARM.