IPython and its Notebook [LWN.net]

By Jake Edge
March 19, 2014

Many different languages have a read-eval-print loop (REPL) for interactive use, but almost inevitably the REPL implementation is too simple for some people. Features like history retrieval and editing, incorporating graphics for data visualization, adding annotation capabilities, and so on, are often things that users want and, thus, a part of any replacement. IPython, or "interactive Python", is a popular substitute for the simple, text-only REPL provided by the standard C Python interpreter. It can also be used in combination with the IPython Notebook to put IPython "documents" into interactive web pages.

The original target for IPython was data visualization, but it has been used for much more than that. It is one of the primary teaching tools used in various informal Python tutorials and explanations found on the web, and it is not uncommon to see conference talks that are accompanied by IPython notebooks. IPython is even included as an interactive shell on the newly redesigned Python home page (look for the ">_" icon).

IPython is similar, in some ways, to commercial numerical computing packages like Mathematica and MATLAB. It also takes on some of the roles of an integrated development environment (IDE) by providing syntax highlighting, tab completion, popping up function API references, and so on. In addition, it provides direct access to Python's "help" functionality with the "help()" (or help(object)) command. But that is just a tiny taste of what IPython does.

To start with, it provides extensive command-line editing features that work especially well with Python's idiosyncratic indentation rules. Pasting code into IPython works well, which is not something that can be said for the standard REPL. To use it, one must learn the difference between Shift-ENTER (execute the code being edited) and ENTER (add a newline at the cursor). That takes a little getting used to, but IPython makes editing Python much easier.

All of the input and output is saved and numbered, so that it can be recalled and manipulated in various ways. The "magic" commands (which start with "%", such as %hist for history) provide extra features outside of Python to do things like load and save files, create aliases and macros, time execution of Python code, call out to the system shell, set up graphics consoles, etc. The %quickref command is one of the more useful when getting started with IPython, though the "?" command that gives a good overview of the features of the program is indispensable in the early going.

The program can be invoked in a number of different ways. For a text-based console in the current terminal window, a simple ipython will do, but the Qt-based console (with menus and tabs for multiple workspaces) can be invoked with the --qt-console command-line switch. The workspaces can either be connected to the same running Python instance (or "kernel") or can have their own. There are also some command-line parameters to control the loading of commonly used libraries. The --matplotlib[=qt4] will preload the matplotlib library (with an optional Qt 4 backend), while the --pylab option loads numpy in addition to matplotlib. The --pylab option is semi-deprecated because it loads numpy into the root namespace (though it can still be useful for quick-and-dirty scratch sessions).

The preloading options clearly show the data visualization roots of IPython, but the program is not limited to numeric plotting by any means. There many kinds of output that it can handle, including image formats, SVG, HTML, audio, video, and LaTeX. While any of that can be done within the console, it is the notebooks that show off IPython output best.

For example, the link in the previous paragraph goes to a notebook created to describe IPython Notebook output types. It uses the IPython Notebook Viewer (often abbreviated as "nbviewer") to display the contents of the notebook, which were created using IPython. Notebooks can be viewed with nbviewer, but interacting with them is done using notebook mode (which we will get to momentarily) locally. There are also extensions for Chrome, Firefox, and Safari that will add an nbviewer icon to the toolbar for even faster access.

There are a number of example notebooks shown on the nbviewer page, ranging from IPython, IRuby, and IJulia introductions to full-on textbooks, as well as notebooks covering the analysis of mathematical functions, an introduction to pandas Python-based data analysis tool, and so on. There are even more notebooks listed in the gallery.

Notebook mode is invoked by running "ipython notebook", which starts a web server process running on the local machine and opens a browser tab to http://127.0.0.1:8888/ to access it. From there, one can create, edit, or interact with an IPython notebook. An existing notebook can be loaded from the local system or from the web, and edited using the same techniques (and keystrokes, commands, etc.) as the console version uses. The notebook can then be saved and shared with others.

As alluded to above, IPython is not restricted to the Python language. Though Python is the only "official" kernel, there are active projects for Julia, Haskell, Ruby, R, and others. The IPython kernel will support multiple languages, as well, so those languages can be mixed and matched as needed.

The stable version is 1.2.1 from February, but a 2.0 release is imminent. The first beta of 2.0 was released on March 7. Development is being funded in large part by a $1.15 million Sloan Foundation grant, though it should also be noted that Microsoft donated $100,000 toward IPython development in August 2013. There is a rather aggressive roadmap for future releases, with lots of new features.

A recent interview with IPython founder Fernando Perez sheds some more light on the plans for the future. The main focus of the 2.0 release is on integrating IPython's capabilities with JavaScript for web-based notebooks. For 3.0, the plans include multi-user servers so that notebooks can be shared in realtime. Perez is targeting mid-2014 for 3.0, which seems like an ambitious target. 4.0, building on the multi-user server is planned for early 2015.

We have only scratched the surface of what IPython is and can do in this look. It is an impressive tool that is worth a look for data visualization, data analysis, and a whole host of other applications. Moving to it permanently instead of the standard REPL seems like a no-brainer for anyone doing anything particularly complicated in that environment.

Scipy

Posted Mar 21, 2014 6:29 UTC (Fri) by marcel.oliver (subscriber, #5441) [Link] (2 responses)

Yes, it's truly a great tool. Maybe it should be emphasized that IPython is "just" the user-facing plumbing, but much of it's usefulness comes from a collection of semi-independent projects which together make up the Scientific Python stack. In terms of general usability and flexibility, it has far surpassed Matlab (the elephant in the room) and there are only few specific niches where one can say that Matlab is still in the lead.

I take some issue with disqualifying the --pylab option as useful only for "quick and dirty" work. Quick yes, but dirty no. Code written with core functionality in the root namespace is easier to read, especially when the emphasis is on a clear expression of the underlying algorithm. A code which is as clear as possible is easier to reuse, can be used in an expository setting, and is faster to write, debug, and modify. (This is not even speaking about the educational setting where the efficiency advantage of pylab is enormous.) So in my experience, the choice of importing pylab comes down to whether you write code which needs to interface with other code, thus needs clear namespace separation and management. But this comes at a cost, and for code which is self-contained (which is a different property than "quick and dirty"), I see little reason to bear this cost. In fact, it is a great advantage of Scipy/Python that it can be used both ways without getting in the way, and I really hope that the view that pylab is dirty and should really be depreciated is a cursory remark that makes sense in a particular context but does not define the project vision. I strongly believe that pylab deserves the same guarantees for long-time stability of imported interfaces as any other piece of the stack.

Scipy

Posted Mar 21, 2014 17:37 UTC (Fri) by andrel (guest, #5166) [Link] (1 responses)

(Hey, I wrote my thesis using your LaTeX style sheet!)

One area where Matlab is still in the lead is educational curriculum and outreach. I get about an invitation a month to free webinars on using Matlab, their marketing team come to town about twice a year to give seminars, and bookstores are full of textbooks with Matlab examples.

Also many labs are dependent on debugged legacy Matlab code; I know scientists who moved from Python to Matlab for this reason.

Does IPython have a Simulink equivalent?

Scipy

Posted Mar 25, 2014 15:58 UTC (Tue) by marcel.oliver (subscriber, #5441) [Link]

I have been using Scipy in the classroom for a number of years now and I am very satisfied with the experience. It is true that while many good textbooks are sprinkled with Matlab examples, the use of Scipy is still very rare in print. As an instructor, I am not so unhappy about this state of affairs because recoding an algorithm in Scipy is usually good exercise...

The state of documentation is not very uniform. I tried to do my small share by writing a brief introduction which is pretty much all that is needed for a lab course in numerical analysis/scientific computing (shame on me for advocating the use of pylab, but in the classroom setting it's very very helpful). The material that turns up with a Google search is also generally of good quality, so that I never really missed the absence of sponsored webinars, etc.

Simulink is the only substantial piece of the Matlab universe for which there is no Scipy equivalent; there a probably a few other niches under my radar as well.

From the educational perspective, I like very much that we can easily outfit the students' own laptops, for any of the major operating systems, by providing publicly available download links, and they can carry and use the software anytime anywhere. Although we have a campus license, Matlab installation on student machines is a hassle and working with Matlab requires a life network connection (it's many years though that I tried). Still we have not converted the whole curriculum from Matlab, first because Matlab is very entrenched in Electrical Engineering (Simulink...) and second because of the usual inertia converting teaching materials. But whatever I get my hands on I usually switch, and in my expericence it really pays off.

IPython and its Notebook

Posted Apr 5, 2014 5:47 UTC (Sat) by lipak (guest, #43911) [Link]

I was surprised to find no mention of Sage (sagemath.org) either in the article or in the comments! As a mathematician, I may be biased, but Sage takes iPython to a new level by gluing in numerous systems. Moreover, it has had a synergistic development with iPython---both tools have benefitted from each other. In particular, one nice idea in Sage is integration of cython.