Pyodide: Python for the browser

Python in the browser has long been an item on the wish list of many in the Python community. At this point, though, JavaScript has well-cemented its role as the language embedded into the web and its browsers. The Pyodide project provides a way to run Python in the browser by compiling the existing CPython interpreter to WebAssembly and running that binary within the browser's JavaScript environment. Pyodide came about as part of Mozilla's Iodide project, which has fallen by the wayside, but Pyodide is now being spun out as a community-driven project.

History

Iodide, introduced in 2019, was an effort to create an in-browser notebook for scientific exploration and visualization, akin to Jupyter and JupyterLab. In that introductory post, the mismatch between JavaScript and scientific computing was noted—most of the existing ecosystem is Python-based—which is where the idea for Pyodide came from:

When we started thinking about making the web better for scientists, we focused on ways that we could make working with Javascript better, like compiling existing scientific libraries to WebAssembly and wrapping them in easy to use JS APIs. When we proposed this to Mozilla’s WebAssembly wizards, they offered a more ambitious idea: if many scientists prefer Python, meet them where they are by compiling the Python science stack to run in WebAssembly.

Getting that working seemed like it would be a big project, but it took only two weeks for Mike Droettboom to get Python running in an Iodide notebook. Over the following months, he and others added support for the most popular Python scientific packages (many of which are implemented in C like the CPython interpreter), such as NumPy, Matplotlib, pandas, SciPy, and scikit-learn. There were concerns about the performance of Pyodide, but performance turned out not to be a barrier for the use case:

Running the Python interpreter inside a Javascript virtual machine adds a performance penalty, but that penalty turns out to be surprisingly small — in our benchmarks, around 1x-12x slower than native on Firefox and 1x-16x slower on Chrome. Experience shows that this is very usable for interactive exploration.

A month after the introduction of Iodide, Droettboom posted more details about Pyodide. It is built using Emscripten, which provides a way to compile C and C++ to WebAssembly, along with "a compatibility layer that makes the browser feel like a native computing environment". That layer is necessary for a tool like Python:

If you were to just take this WebAssembly and load it in the browser, things would look very different to the Python interpreter than they do when running directly on top of your operating system. For example, web browsers don’t have a file system (a place to load and save files). Fortunately, emscripten provides a virtual file system, written in JavaScript, that the Python interpreter can use. By default, these virtual “files” reside in volatile memory in the browser tab, and they disappear when you navigate away from the page. (emscripten also provides a way for the file system to store things in the browser’s persistent local storage, but Pyodide doesn’t use it.)

By emulating the file system and other features of a standard computing environment, emscripten makes moving existing projects to the web browser possible with surprisingly few changes.

In order to do useful work within the browser environment, programs need access to the Document Object Model (DOM) of the browser, which is provided by the JavaScript APIs available there. That means Python and JavaScript code need to work together in various ways. Most of the basic types (e.g. numbers, arrays, strings) can be relatively easily mapped between the languages, but Python treats the object and dict types as distinct, while JavaScript conflates them to a certain extent: all objects can be treated as dicts. To handle that difference, object types in both languages and dict types in Python are represented by proxies in the other language; that allows each language full access to the other's data types.

As an example of these proxies, the post talks about accessing the DOM object from Python as follows:

    from js import document
    x = document.getElementById("myElement")

All of this happens through proxies that look up what the document object can do on-the-fly. Pyodide doesn’t need to include a comprehensive list of all of the Web APIs the browser has.

Given the focus on large data sets in the kinds of processing often done by NumPy and the like, there is a need to provide an efficient multi-dimensional array type that can largely be shared between JavaScript and Python. Copying huge arrays between the two would be a major performance hit, but will also likely overtax the amount of memory available to the browser. So an array type that is stored on the heap shared between the two was created; the small, language-specific dab of metadata describing the array is all that needs to be copied.

Present

Fast-forward two years, and Pyodide has made a good deal of progress; it released version 0.17 on April 22. At the same time, it announced that the project is becoming independent, complete with a GitHub repository, a governance system that draws from that of CPython, and a code of conduct based on Rust's and incorporating pieces from others. Meanwhile, the Iodide project has been discontinued, but Pyodide is more than simply a part of Iodide at this point: "Pyodide has attracted a large amount of interest from the community, remains actively developed, and is used in many projects outside of Mozilla."

The 0.17 release has a number of interesting features. Support for Python asyncio has been added, so that Python coroutines can run in the browser event loop; a JavaScript Promise can be awaited in Python and vice versa with Python awaitables. Error handling has also been upgraded so that exceptions generated by Python can be caught in JavaScript; that can be done in the other direction, as well.

A new version of Emscripten has been adopted, which has helped shrink the size of the binaries needed ("the SciPy package shrank dramatically from 92 MB to 15 MB"). It has also helped on the performance side of things. Using the latest toolchain (including the LLVM backend) results in a 25-30% improvement in run times. Overall, performance since the 2019 announcement has improved greatly: "Performance ranges between near native to up to 3 to 5 times slower, depending on the benchmark."

As with most (all?) projects, Pyodide is looking for people interested in contributing in various ways. "There are many ways to contribute, including code contributions, documentation improvements, adding packages, and using Pyodide for your applications and providing feedback." There is a roadmap with some ideas of plans for the future.

As Mozilla's Dan Callahan said in a PyCon 2018 keynote, Python may be getting left behind because the platforms that people are using are changing. While laptops, desktops, and servers support Python just fine, many people are only using phones and tablets where Python is not really present. But there is a unifying platform across all of those devices (and, presumably, others that come down the road): the web. If Python wants to stay relevant, it needs a reasonable browser story; Pyodide may be part of the right path toward that end.