Out of order execution [LWN.net]

Out of order execution

Posted Jun 28, 2023 19:04 UTC (Wed) by summentier (subscriber, #100638)
In reply to: Out of order execution by SLi
Parent article: JupyterLab 4.0: a development environment for education and research

You may want to take a look at Pluto, which comes fairly close to what you are describing, albeit only works with Julia. A Pluto notebook is presented to you as a sequence of cells, but internally it is a directed acyclic graph, where the cells are the vertices and dependencies are the edges. Changing one cell triggers recomputation of all dependent cells along the graph, regardless of the order.

Having seen my students flounder around with the "hidden state" and out-of-order execution problems this to me seemed like godsend: with Jupyter, we have to impress onto the students that if something looks weird, first try to restart the kernel and execute everything. Even nbgrader, the auto-grading engine we use to grade submissions, does this before handing in the notebook such that students don't submit non-working code by mistake.

However, having worked with Pluto for a while, I have to say that the beautiful concept does not translate that well to the real world: (1) I find I often have heavy calculation in one of the cells, and I don't always want to recompute even if I change one of it's dependencies; (2) one of the big advantages of out-of-order execution is that I can watch an iterative procedure converge by, e.g., repeatedly executing a sequence of cells; (3) reordering cells works reasonably well for functions, where you can "hide details" down the file, but I find it is a poor match for my mental model when you are describing a sequence of steps; (4) Pluto must be extremely restrictive with global state, which I find does get in the way of experimentation.

to post comments

Out of order execution

Posted Jun 29, 2023 1:13 UTC (Thu) by rsidd (subscriber, #2582) [Link]

I haven't even tried pluto precisely because of these worries. I use jupyter+julia heavily (I never "got" jupyterlab but maybe I will give 4.0 a try). Jupyter is both a testing ground for new module functions, and a place to import and run the module. My system is to load the module on the top of the notebook, do testing and development (currently it's a clustering program and I have multiple notebooks open doing different benchmarks), and as and when I write a new function that works and is needed in the module, I put it there. I can also edit existing functions in the module and Revise.jl gets the notebook to "do the right thing" magically.

Out of order execution

Posted Jun 29, 2023 14:03 UTC (Thu) by ballombe (subscriber, #9523) [Link]

Indeed, pluto system should use versioning of both states and results, so that you could browse the full history of what was computed and get back/forward as needed.
(see each cells as a file in a fit repository and do 'git commit' each time a result is computed)