New horizons for Julia

Julia, a free, general-purpose programming language aimed at science, engineering, and related arenas of technical computing, has steadily improved and widened its scope of application since its initial public release in 2012. As part of its 1.11 release from late 2024, Julia made several inroads into areas outside of its traditional focus, provided its users with advances in tooling, and has seen several improvements in performance and programmer convenience. These recent developments in and around Julia go a long way to answer several longstanding complaints from both new and experienced users. We last looked in on the language one year ago, for its previous major release, Julia 1.10.

Static binaries

It's simple to share Julia projects in environments where Julia is already installed: provide the text files containing the program and one additional text file, called Project.toml, which lists the direct dependencies of the code. This latter file is generated automatically by Julia. The recipient need merely execute the single command instantiate to have Julia install any needed packages to exactly reproduce the environment.

This works well with colleagues who happen to be Julia users; it is great for collaborative development and for reproducibility in scientific research. However, it's a dead end for colleagues who do not have Julia installed. Distributing the program in the form of an "app" would solve the problem, but a normal installation of Julia provides no obvious way to do this.

As Julia is a compiled language, one might wonder what the problem is. After all, you can compile your Fortran or C program into a compact, self-contained (perhaps, depending on shared libraries) binary, and even target a variety of architectures with cross-compilation. Anyone can run these binaries, without a compiler installed.

It has also been possible to compile a Julia program into a static binary for some time, using the PackageCompiler package. Experienced Julia hands use this package to create "system images" (or "sysimages") that contain the Julia runtime along with frozen versions of packages used in their projects. These sysimages load instantly and save time when one is not currently using the Julia read-eval-print loop (REPL).

The major improvements in precompilation and package-loading times that came with 1.10 largely obsoleted the use of PackageCompiler to save startup time. PackageCompiler might still be a solution to the problem of binary distribution except for one serious issue: the binaries that it generates are huge.

The sysimages created by PackageCompiler contain the Julia runtime, the entire standard library, parts of the LLVM compiler, and more. The result is that a "hello, world" program is burdened with about 150MB of stuff that it does not use. (The compiled "hello, world" program will, for example, contain the BLAS linear algebra routines, because, in its current state of development, PackageCompiler does not figure out that they are not needed.) This is a vast improvement over earlier incarnations of PackageCompiler, which turned "hello world" into a gargantuan 900MB, but does not compare well with, for example, the Fortran equivalent. I just used the gfortran compiler to process a Fortran 90 "hello world" program and found that it resulted in a tiny 17KB binary.

Efforts toward reducing the size of binaries have also involved another package for static compilation. This package is only useful for relatively small Julia programs, however, and is confined to a severely limited set of language features. Within these limitations, StaticCompiler can produce truly small binaries, similar in size to those produced by C and Fortran. Because of the limitations, however, interest remained focused on the evolution of PackageCompiler.

Why can't PackageCompiler just strip out the unneeded stuff and create binaries of a reasonable size? That's the obvious question, but finding a way to eliminate the unneeded baggage turned out to be a difficult problem. Julia's dynamic design and its focus on REPL-driven development mean that its runtime and "just-ahead-of-time" compiler were envisioned to be always available behind the scenes. They are used to compile new methods for newly-encountered mixtures of argument types and to keep the whole array of language features at the ready, including various types of introspection. This is the essential reason that binaries created by PackageCompiler are so large, and why so much work was required to allow the programmer nearly full use of all language features while not dragging the entire runtime along with each compilation target.

The goal of creating small binaries has occupied a handful of talented developers over the past few years, and they have recently achieved a new level of success. Development builds of Julia now include an experimental method of invoking the compiler called juliac. This compiler invocation script comes with a --trim flag that "trims" away whatever is not needed to run the program, while still allowing the use of nearly all normal Julia features. The idea is that the user will be able to type juliac at the terminal, instead of julia, to compile a binary rather than run a program or start the REPL. The result is an impressive 90% reduction in final binary size compared with the current version of PackageCompiler. The exact size of "hello world" is fluctuating some in the development builds as of this writing, but it is around 900KB.

The next major Julia release, 1.12, is likely to appear in mid-2025. It will finally include the ability to generate static binaries of a reasonable size, appropriate for distribution. This will be a refinement of the juliac mechanism that is already merged into the development branch.

Static compilation will be the definitive answer to the complaints and wishes of many Julia programmers who have wanted their favorite language to be able to do what the traditional workhorses for scientific computing have done for decades.

juliaup

Julia has always been easy to install and upgrade on operating systems such as Linux and BSD: simply download and expand the tarball and place a link to the julia binary in your path. Julia's package system also makes it easy to keep any number of Julia versions installed on the same machine, with dependencies correctly resolved for projects using different versions of the language. However, it's clear from questions on forums that there are many users of Julia who are less comfortable using command-line tools to navigate their filesystems. Installing, upgrading, or switching among versions often leads to confusion.

Now there is a utility called "juliaup" to make their lives simpler. As juliaup has become the recommended way to install and upgrade Julia, it's now widely used even by Linux adopters (although I am set in my ways and don't bother with it). juliaup does provide some useful abstractions. These are built around its concept of "channels". Some examples of juliaup channels are release, referring to the latest stable version of Julia; lts, referring to the "long term support" release; beta, pointing to the most recent beta; and nightly, pointing to, perhaps unsurprisingly, the nightly build. Thanks to this channel abstraction, one need never refer to Julia's version numbers.

To get the latest nightly, a user simply types:

    $ juliaup update nightly

juliaup also provides a new, flexible julia command for starting the language. To run the nightly build, the command is:

    $ julia +nightly

After executing the command:

    $ juliaup default release

the plain julia command will now use the stable release.

These are just a few examples of how juliaup works. It comes with a collection of other commands to do such things as list available channels, update all channels at once, or refer to specific Julia versions, as well.

Julia in web browsers

These days, fairly complex simulations written in Julia can run completely within the web browser. Alexander Barth has created a fun and addictive simulation of fluid flow that runs in the browser. The visitor can draw boundaries with the mouse and watch the pressure field adapt. The same author has also created a browser-based simulation of surface waves. Those with an interest in chaos theory can find an in-browser solution of the famous Lorenz system of ordinary differential equations. It's remarkably fast, and the demonstration comes with a tutorial explaining how it was done.

These three examples were written not in JavaScript, but in Julia, and run in the browser thanks to WebAssembly. Launched in 2015, WebAssembly (often abbreviated "Wasm") is a binary instruction format that has become widely deployed in modern web browsers. It potentially allows any language to run in a browser, provided someone has written a WebAssembly compiler for it.

The growing list of source languages with WebAssembly compilers in various stages of development is encouraging. We can already use Python, C, Go, Lua, Rust, and many others as replacements for JavaScript. However, despite the impressive examples above, Julia for WebAssembly is still in the early stages of development. Only a restricted subset of the language is supported (for example, no multidimensional arrays). However, the problems encountered in creating small binaries have some overlap with the problems of targeting WebAssembly, so recent progress in the former is likely to help speed along the latter.

Readers interested in getting their Julia programs running in the browser should start with the WebAssemblyCompiler package. The author warns that the code is experimental, but it comes with guided examples.

Features of 1.11

The latest release of Julia is 1.11.3, unveiled on January 21, 2025. The release notes for the 1.11 major version, which appeared on October 8, provide detailed information on the improvements to the language in that release. Here I'll summarize some of the most important and interesting language developments since 1.10.

MemoryRef datatype: many array operations had been implemented in C up to now, which created some overhead. These have now been moved to Julia code thanks to a new datatype called MemoryRef. The performance improvements coming from this change include a doubling in speed of the push!() function that inserts items onto the end of an array. The maintainer of the WebAssembly package remarked that the MemoryRef datatype will finally allow multidimensional arrays to be used in Julia programs destined for WebAssembly.

public keyword: as pointed out in our article about Julia's package system, names in a module can be explicitly exported, which allows their use without namespacing. For example, Base is a library that is automatically imported into every Julia environment. It provides hundreds of functions, from basic mathematical workhorses such as sin() to string manipulation routines such as split(). Julia programmers can use these functions by invoking their unadorned names because they are exported by the modules in which they are defined.

However, Base also contains functions useful in more unusual contexts that are not exported. One example is Base.OneTo(n), which returns something like the interval 1:n, but slightly different in the way that it's treated by the type system. Those who know that they need this function must namespace it by prefixing Base. Although not exported, Base.OneTo() is "safe" to use; it's intended for public consumption and will not break future code. This is not the case for every unexported function defined in every module. Up to now, there was no standard way to tell the difference between a safe unexported function and one that should be considered "private", in the sense that it may not be future-proof. Now Julia's public keyword fills this gap by marking names that are safe to use even if they are not exported. Of course Julia will not stop programmers from using names not so blessed, but now it will warn them.

stdlib excision: Julia 1.11 continues the process that has been underway for a couple of years to remove modules from the standard library and turn them into normal packages. This allows them to be developed independently from releases of the language, which generally allows a faster pace of improvements. It also allows Julia to be distributed with a smaller sysimage, purged of unnecessary libraries, that can start faster and consume less memory. In the latest release, modules from stdlib will all have their own version numbers, formalizing their independent existence.

@main: Julia 1.11 added a @main macro that marks which function in a Julia program file will be the entry point when the program is run (rather than imported). This lets programmers use the same file as a resource imported into other programs and as a file to be executed. For example, a module that consists of a list of function definitions can be imported into another program without any side effects. But if one of those functions is decorated with @main, then, if the same file is executed (for instance by typing "julia file.jl"), in addition to the functions being defined, the one so marked will be run. This is semantically identical to inserting a line at the bottom of the file invoking the name of the main function; thanks to @main, one version of the file can now be used both ways.

Conclusion

Due to the accumulation of improvements in the last several versions of Julia, startup times have become fast enough that I have begun to use the language for utility scripting, something that would have been impractical a few years ago. This trend continues in the most recent version, and the greater ease of compiling programs promised in the next version should make Julia even more suited for building system utilities. It's already become a serious contender to replace Bash and Python for my purposes. If WebAssembly targeting continues to make progress, which the improvements to compilation will also contribute to, then Julia will become a welcome replacement for (or addition to) JavaScript as well, further extending its scope of application.

A solution to the problem of small binaries, to be released as part of 1.12, should allay the misgivings of those for whom the difficulty of sharing compiled programs has kept them away from the language. For Julia's core original audience, who were accustomed to amortizing large compilation times over simulation times measured in days or weeks, the new language's awkwardness in some interactive contexts was an acceptable tradeoff to finally have a modern, expressive medium that did not entail compromises in performance. By successfully meeting the objections of Julia's critics in several key areas, its developers have ensured that their language is poised to widen its scope far beyond that original core user base.