An introduction to Pluto

Pluto is a new computational notebook for the Julia programming language. Computational notebooks are a way to program inside of a web browser, storing code, annotations, and output, including graphics, in a single place. They became popular with the advent of the Jupyter notebook, which originally targeted Julia, Python, and R—the names got mashed together to make the word "Jupyter".

Pluto is similar in many ways to Jupyter, which I wrote about earlier. It uses the same basic mode of interaction that is based on input and output cells; both notebook formats are well-suited to exploration, sharing calculations, and education. In an earlier article, I reviewed progress in Julia since v. 1.0 was released two years ago. It also went into some detail about its special attractions of Julia in the area of scientific computing. Readers who are unfamiliar with Julia may want to review some of the earlier articles; here, I concentrate entirely on Pluto.

Like Julia, Pluto has an MIT license. It was created by Fons van der Plas and Mikołaj Bochenski, and has been developed on GitHub since March 2020. Despite its recent vintage, it's already mature enough for serious use; in fact, it's being used in an ongoing open MIT course. But users should keep in mind that the version number, as of this writing, is only 0.12.4; the program's behavior is certainly not set in stone. Pluto advises the user, upon startup, if a fresher version is available in the repository.

Readers who would like to try out Pluto right away and don't have Julia installed can use the Binder service to run a notebook with nothing but a web browser. When a user visits this page, it spins up a Julia instance on a server and opens a notebook interface to it that is ready for experimentation. Whether opened through Binder or running locally, the notebook offers the user an initial page with the choices to open a sample, existing, or new notebook. The series of sample notebooks offers an excellent hands-on introduction to the use of Pluto.

Why another notebook?

Jupyter has become successful because it offers a style of interaction and modes of sharing that are not available through other means. Although it is used most often with Python, the list of kernels—language back-ends that it works with—is astounding in both its size and in the obscurity of some of the languages that it includes.

Given all this, the reader can be forgiven for wondering why we need a completely new notebook project. One answer is that Jupyter presents two problems that work against its intended uses. Briefly, these are its global hidden state and its incompatibility with version control. I'll expand on these two issues below. Although they are widely seen as problems, there is, naturally, disagreement about how serious they are. This YouTube video is an entertaining demonstration of the issues that arise when using Jupyter, presented in the lion's den, the 2018 JupyterCon, by data scientist Joel Grus. Jeremy Howard responded [YouTube] at the 2020 JupyterCon, showing in detail some of the ways to overcome these problems.

At right is a screenshot from my laptop, showing Python 3 running in a Jupyter notebook; it illustrates the hidden state problem. The entire notebook is shown. Notice that a is defined in two contradictory ways, followed by a cell where we ask for the value of this variable. The response from the Python kernel is yet a third value, from an assignment that is nowhere to be found in the notebook because I deleted it.

The problem here is that the visible state of the notebook at any particular time may be inconsistent, because it depends on the order in which the cells were executed and possibly on cells that no longer exist. If the notebook is saved in an inconsistent state and shared, the recipient will open it and see calculations that may not be reproducible from the inputs as shown, or may depend on the cells being executed in some undocumented order.

But what if the notebook worked in a different way, that made these issues moot points? What if it were always consistent, and it didn't matter in what order the cells appeared?

Pluto is different

Pluto appears, at first glance, quite like Jupyter and similar notebooks, but the way it works is rather different.

In Jupyter, when a cell is executed, the result appears, and nothing else happens. Any global variables defined in the cell are available for subsequent execution of other cells. The variables can be redefined freely. The upshot is that the output of any particular cell execution depends on a hidden global state: which cells were previously executed and in what order.

Pluto, instead, analyzes the code in all of the cells and constructs a dependency graph, so that it knows the order in which the cells must be executed; this is based on which cells use which variables. Cells can be grabbed with the mouse and arranged in any order and this has no effect on the results. When the code in a cell is changed, the cell is run; all of the cells that depend on it, and only those cells, are also run, in dependency order. Therefore one is not allowed to define a global variable in more than one cell; an attempt to do so results in an error message.

The Pluto GitHub page makes this promise: "At any instant, the program state is completely described by the code you see." There is no hidden state, and nothing depends on the order in which cells were run.

Execution, as with Jupyter, can be performed with the shift-return key combination or by clicking a button. One difference from Jupyter that jumps out immediately is that results are shown above their input cells, a convention that seems counterintuitive to me. While a cell is executing, its left-hand border turns into an in-progress indicator. One can observe the status of a calculation through the dependency graph by watching as these indicators jump from cell to cell. Below each cell is printed, in unobtrusive grey type, the execution time taken when calculating that cell, which is a nice touch. The following figure illustrates these elements of the interface:

The Pluto creators describe their project as a "reactive notebook", borrowing this term from its emergence as a description of JavaScript libraries, such as React, that allow the creation of web apps where the state of the page automatically reacts to a change made by the user.

In Pluto this means that, when the user executes a change to the value of any global variable, the dependency cascade is triggered, and everything that depends on that variable is automatically updated.

An example calculation

To provide a feel for the use of Pluto notebooks, this section will go through a short but meaningful calculation. Our recent article introduced the use of the DifferentialEquations and Plots packages; here we will use these again.

To begin working with Pluto, one must start the Julia read-eval-print loop (REPL), add the Pluto package if it's not already installed, and then execute using Pluto followed by Pluto.run(port=1234), substituting a different port if required. Current versions will then automatically open a browser window pointing to localhost at this port. The machine running the Julia process and the one running the browser need not be the same; in that case an extra host argument to Pluto.run will allow it to access a remote instance.

Our calculation is the numerical solution of the differential equation for a damped, nonlinear oscillator.

The figure above shows the top of the notebook page and the first four cells. The first two perform the necessary imports. The next cell is a comment describing the equations to be solved, and the fourth cell defines ndo!(), the function that translates this math problem into a form that DifferentialEquations can digest. This function modifies its first argument, du, which holds the derivatives of x and v; the "!" sigil in the name is a Julia convention to indicate argument-modifying functions. Example 2 on this page of the package documentation explains what is being done here.

The crossed-out eye icon to the left of the third cell hides the input, which in this case is a bit of Markdown. It makes sense to keep the input of Markdown cells hidden, unless editing them, to make the notebook easier to read. Here is what this cell looks like with the input exposed:

As this shows, in addition to standard Markdown, Pluto can render LaTeX math. The Markdown can also interpolate the values of variables and calculations by using a dollar sign (e.g. $(2*var)). As with code cells, the Markdown gets automatically updated if any of the variables that it uses happen to change.

This system has three parameters that influence the nature of the solutions: k is the restoring force, or the "spring constant"; α is the nonlinearity, a second term in the restoring force; and d is the strength of the damping, or the amount of friction. The next three cells, as shown in the following figure, contain the time span for the solution (tspan), the initial conditions for the position (u0, which consists of x, and the velocity), and an array (p) holding the three parameters.

The next cell defines an iterator that goes from one to 50 in steps of one; as its name (αs) suggests, I plan to let α take on this list of values.

The final cell in this figure calculates the 50 solutions, one for each value of the nonlinearity parameter α, and plots them all together in a waterfall plot. I use the variable i to offset the plots; since it is initialized outside of the loop, but incremented in the loop, it must be declared global because all blocks in Julia create lexical scopes. The whole cell needs to be wrapped in a begin…end block because, otherwise, Pluto will complain that i is being defined twice. This is the only difference between the Pluto code and the way this would be written in an ordinary Julia file, where this block would be unnecessary.

The other functions in this cell are similar to those we used in the previous article to solve the logistic equation. The exclamation mark on the plot! command is so we can "mutate" the plot by adding a new line to it with each call. Inside the plot! function, the notation sol[1,:] is a convenience interface defined by the package that extracts the first of the two components of the solution, which in this case are x and v.

Here is the result:

I think the plot is rather interesting, as I've not seen the solution space of this equation displayed this way before. It shows how the wavelength varies with time and with the amount of nonlinearity. Julia and Pluto make it easy to play with things and experiment, which can lead to new insights.

This last figure also shows the bottom of the window, with a box for sending instant feedback to the developers, which is a nice idea. At the bottom left is a link that allows the user to opt-in to sending anonymous statistics about the use of Pluto to the developers to help them to improve the product. There is also the button for the "Live Docs" popup window, which is described below.

Integration

The reactive nature of Pluto notebooks can be taken further by using HTML "bindings". Using a simple macro, any variable can be bound to the output of any HTML input type. The following figure shows a screenshot from my laptop where I've bound three variables to three types of widgets.

The illustration on the right shows user interaction with the color picker.

In Jupyter, similar functionality can be had by importing a widget library. In Pluto, an alternative to writing HTML directly is offered by a Julia widget library. Using this library, one can, for example, replace the HTML code for a slider:

    @bind x html"<input type=range min=1 max=5>"

with:

    @bind x slider(1:5)

Those who want to go beyond standard HTML inputs can create any widget they wish using JavaScript because the output of a JavaScript function can be bound to a Julia variable.

Pluto notebooks integrate well into a more general coding environment. Code is not locked into the notebook, and code that exists outside of a notebook can be brought in. One can import (almost) any package into a notebook, just as in normal Julia code. The known exceptions are things that don't make sense in a browser-based notebook, such as REPL-specific packages, or some packages that use macros to radically redefine the language's syntax. Some parallel-processing packages also will not work, specifically those that use Distributed.

It is also possible to include local files, but a notebook that does this will no longer be portable, because those files may not exist on others' machines. So a truly portable notebook should only import from public repositories.

Pluto notebooks are stored as plain text files on disk. Jupyter's ipynb native notebook file format is also text, but there is a crucial difference. Jupyter files include not just the inputs and the code, but also whatever outputs happened to be displayed when the notebook was saved. This has three consequences. First, meaningful diffs between versions are usually impossible to get, which severely limits the usefulness of version control. Second, the person opening a shared notebook is likely to see whatever partially executed state the saved (or auto-saved) file was in before it was shared. Third, the files are not themselves legal code because of the mix of code and output data; execution can only take place through the notebook interface.

In contrast, Pluto notebooks include only the input cells; the order in which cells appear in the file is based on the dependency graph and does not follow the visual order of the cells in the notebook. This presentation order is preserved through comments in the file. Because of this design choice, Pluto notebooks can be usefully managed using version control. Notebooks can also be shared, and the recipient receives a predictable record of the code (and explanatory Markdown cells). Beyond that, notebook files are legal Julia code, so they able to be executed directly by the Julia compiler/interpreter. They can be imported or run as standalone programs, just as any file of Julia code, so they use the same .jl suffix.

This means that it's possible to freely go back and forth between editing code in an editor and in the notebook, creating files that can be imported as any other file in a project, without having to worry about stray bits of output left over in the notebook. One drawback of this design is that if, for example, a notebook contains some graphs or other output that take a long time to generate, it will need to be regenerated each time the notebook is freshly opened; and with Julia's notorious precompilation delays, this can mean waiting for several minutes before getting to work if a new package or a new version needs to be compiled. With Jupyter, in contrast, all of the output, including graphics, is embedded, so there is no delay.

Notebooks can be exported into static PDF or HTML forms with a click on the mysterious triangle-and-circle icon near the top of the page. The order in which the cells are arranged in the notebook is preserved in the exported versions of the notebook, so the visual representation is preserved, regardless of the dependency graph.

Much of Pluto's interactivity and visualization is based on Observable, an interactive notebook for JavaScript, which the Pluto authors credit for being their inspiration. Users of Observable notebooks will immediately recognize the controls for creating and manipulating cells in Pluto. Its reactivity is based on Preact, a JavaScript library that is a lightweight alternative to React. Math rendering is provided by MathJax, and code editing in cells uses CodeMirror.

Rough edges

Despite its polish, Pluto still has a few rough edges, which is not surprising in such a young project.

The current version as of this writing does not work with my daily browser, qutebrowser, but it did work for me in Chrome. This is disappointing, as some previous versions did not have this limitation. The project does claim, however, that Chrome or Firefox are needed, but it is less than ideal for programs to require specific browsers.

The print statement seems to do nothing—but the output is actually being printed in the REPL where Pluto was started. This will probably confuse some users. It certainly surprised me, and I had already read the instructions where this is explained—but I forgot. Some might find this useful, however, as a way to emit logging information and keep it out of the notebook.

A new notebook in Pluto opens in ~/.julia/pluto_notebooks/, with an arbitrary name, instead of the directory where the REPL is running. This seems like an odd behavior, but is an improvement over previous versions that opened new notebooks in /tmp, which could cause the user to lose work if they didn't notice what was going on. Neither of these facts is noted in the online guides to the program. It would seem to be a good idea to enter the desired location of the notebook in the dialog at the top before beginning work.

Certain invocations of parallel processing, such as attempts to run on a multicore graphics processor, which work fine from files or the REPL, will fail from the notebook. This is a problem that is being worked on, but for now there are some simple workarounds.

There is not a huge amount of documentation available, but what exists may be sufficient, because the use of Pluto is fairly intuitive, and the built-in example notebooks are well done.

The GitHub home page serves as an introductory tutorial; I especially recommend the short animation on the page. There is also a FAQ and a good introductory video from this year's JuliaCon.

A help window will pop up when "?" is typed in a cell, similar to the help mode in the REPL. The built-in documentation popup window can be made to persist by clicking on "Live Docs" at the bottom of the window. In this case it will continuously offer definitions for anything that it recognizes as the user types into a cell. Also, the notebook has tab completion which works well, as does the syntax-aware styling of code in cells.

Future plans

The developers are interested in being able to export notebooks that can be used in their fully interactive form without requiring the user to have Julia installed. This would seem to be an ambitious aspiration; the Jupyter developers have been talking about the equivalent for a while, but it still does not exist. However, as noted, the Binder service will host anyone's Pluto notebook, which is helpful for those in school or corporate environments where it may not be possible to install software. The developers plan to include a single button for deployment to Binder in the near future.

The concept of notebooks within notebooks is on the table, as is a modification of the notebook for teacher-oriented features like grading. There may be improved display, debugging, and logging on the way, such as the display of intermediate values during a calculation. There is also interest in making the front-end more responsive, as well as allowing user theming.

One interesting question is whether Pluto, like Jupyter, will ever allow the use of languages other than Julia. As there is nothing online mentioning this, I asked Van der Plas about it. His answer is that they have made a deliberate choice to keep Pluto Julia-only:

Julia-only allows us to do lots of exciting things.[…] I think that the traditional programming experience from the 70s - terminal input, terminal output - is something that we urgently need to leave behind us. […] to get more meaningful interactions with a language, I believe that the polyglot environment is something that you need to leave behind. While Pluto will not become a multiple language environment, we have been working on closer integration with JS, and a lot of powerful stuff is possible already. Cells can output HTML, and that HTML can contain script tags, and Pluto's frontend will execute those. We have a small API layer around vanilla JS to allow you to send values back to Julia (reactively, this is how @bind works), output to the DOM, persist DOM state between cell updates and even change Pluto's UI.

Above we made the case for Pluto by showing how it solves the hidden global state and version control incompatibility problems that occur with Jupyter (and similar notebooks). But this is a third reason for developing a new notebook project: by confining itself to one language, Pluto can focus on being the best possible interactive environment for that language, without making any compromises to allow for a variety of kernels.

Although the dominant method for programming is likely to remain in a text editor like Vim or Emacs, or an integrated development environment, computational notebooks offer an intriguing alternative. They not only offer a unique style of interactive exploration, but can be a good platform for creating documents as well. People have written books using these notebooks, and they may be a way to realize Donald Knuth's concept of literate programming. The great advantage of computational notebooks in writing about programming is that the code can be run and checked before the notebook is exported into the final HTML or PDF form; the author can guarantee that the code runs and does what the book or article says it does.

The computational notebook format began in the 1980s with the proprietary Mathematica system. It was an innovative feat of engineering, but never achieved the widespread adoption of the free, open source, and language-independent Jupyter. However the Mathematica notebook idea has been incarnated by now in several guises in addition to Jupyter. For those using Julia and interested in a notebook interface, Pluto, the latest of these options, is an obvious choice: it offers all of the features of Jupyter, but is a clear advance over the older system. It will be interesting to see how this young project evolves in the coming years.