Jython Native Interface (JyNI)

In his session at the 2015 Python Language Summit, Stefan Richthofer described a project he has been working on for the last two years to enable Jython (Python on the Java virtual machine) to use Python's C extensions. It is called Jython Native Interface (JyNI) and it provides an adaption layer between Jython, which, like Java, uses the Java Native Interface (JNI) to access native code, and Python's C API.

Jython reimplements various built-in Python types (e.g. PyIntObject, PyStringObject, ...) in Java. The idea behind JyNI is to create a C-compatible view into these objects that stays in sync with them. The C API is "large and complex", which is one obstacle that needs to be surmounted. In addition, CPython uses macros in some places to access data objects; those macros cannot be switched out at runtime by dynamically loaded functions.

There are also a few minor obstacles to JyNI, including exception handling and the global interpreter lock (GIL), but the biggest problem area is garbage collection. Making it all work without leaking memory is challenging, he said.

There are three approaches that are being used to unify objects between Java and Python (i.e. to allow access as either a PyObject or a jobject). For Python object types that have Java versions available, a PyObject wrapper for the jobject is used. If there is no object of that type available in Java, the reverse is done (a jobject wraps a PyObject). The third option is used when macros are used to access the object in Python. It is a data structure that mirrors the object state between a PyObject and a jobject version. JyNI tries to avoid mirroring but it cannot be avoided for some kinds of objects.

CPython provides garbage collection for extensions, so some extensions may be relying on that for their memory management. Java does not provide any garbage collection to native code, so extensions must manage their own memory. Resolving that will, seemingly, be the biggest challenge for JyNI going forward.