PyMetabiosis

Romain Guillebert, who is one of the PyPy developers, introduced his PyMetabiosis project at the 2015 Python Language Summit. The idea behind it is to make it easier for people to use alternative Python implementations (such as PyPy) by making C extensions available to them.

The biggest barrier to using alternative implementations is the availability of the C extensions, he said. PyPy and others cannot just support the C API because it is too closely tied to the CPython implementation details. For example, it assumes reference counting for garbage collection, which is known for being inefficient and for making it difficult to get rid of the global interpreter lock (GIL). PyPy doesn't use reference counting; it can emulate it, but that is slow.

So he has come up with a "hackish way to make C extensions work" for PyPy. PyMetabiosis embeds CPython into PyPy, which allows importing C extensions using CPython. PyPy wraps a proxy around a CPython PyObject that implements the needed methods, which call out to an extension via the C API.

PyMetabiosis allows a lot of Python extensions to be accessed from PyPy. For example, it allows sharing NumPy arrays between PyPy and CPython without any copying. It also did not take much code to implement PyMetabiosis; Guillebert said he spent about two weeks getting it working.

It is, unfortunately, still slow. It also makes PyPy dependent on CPython, which is not ideal. It doesn't work on Windows either. PyMetabiosis uses an environment variable that must point to a CPython virtual environment. Another limitation is that PyPy objects that are passed to CPython and get altered (e.g. lists) do not have those changes reflected in PyPy.

Ultimately, though, PyMetabiosis is just another slow compatibility layer that has different properties from other choices, he said. He would like to see an effort to incrementally improve the C extension API. He suggested looking at the APIs provided by other languages, such as those for Java (using the Java Native Interface), V8 JavaScript, Lua, or Julia, which are "way more" implementation independent.

Someone asked about "going the other way around" and embedding PyPy into CPython for use by performance-sensitive code. Often the numerical crunching done by NumPy and SciPy needs all of the performance boost it can get, while the rest of the code could be terribly written, but it doesn't really matter performance-wise. Guillebert agreed that both ways would work.

The current situation, with both a NumPy and a NumPyPy for the two different Python implementations, is "not a great long-term solution", an attendee said. Users really just want one version, but they need some path to get at PyPy's just-in-time (JIT) compiler. If there is an effort to rewrite or enhance the C API, providing that path should be considered.

Nick Coghlan said that the C Foreign Function Interface (CFFI) should be refactored so that it can serve as an extension module generator. CFFI is based on Lua's C API. Glyph Lefkowitz suggested that extension developers are often ignoring CFFI; when they actually try to use it, CFFI generally solves their problems, he said. The key is to get people to stop hand coding their C extensions, Coghlan said.

Note: you can avoid this step in the future by logging into your LWN account.