The Ninja build tool

Ninja is a build tool, similar in spirit to make, that was born from the Chrome browser project in 2011. It is unique among build systems in that it was explicitly designed to be used as an "assembly language" by higher-level build tools. Beyond that, Ninja is fast. It is used by several large open-source projects and it has changed the way I approach build systems.

Ninja lacks many of the features that make users are familiar with: no conditionals, no string-manipulation functions, and no patterns or wildcards. Instead, you are supposed to put this logic into a separate program that generates Ninja build files; you aren't supposed to write Ninja build files by hand, though they are still quite readable. Some people write this generator program from scratch in a high-level language like Python, and they usually call it configure. Others use a build system such as CMake, GYP, or Meson, all of which can generate Ninja files based on rules written in their own custom language.

Despite its minimalist philosophy, Ninja does have a few features that make lacks: a progress report on its standard output, support for filenames with spaces, a "pools" mechanism to limit the parallelism of certain rules, and the ability to "include" other Ninja files without polluting the namespace of the including file. Ninja also has built-in support for some features that you could implement in a Makefile by using various techniques, but it's tedious to do so. I'll cover some of these in the article.

Rules and targets

A Ninja file to compile and link a C program from two source files might look like what's below. I have highlighted the targets (that is, the files that are created by the build process) in bold:

    rule cc
      command = gcc -c -o $out $in
      description = CC $out
    
    rule link
      command = gcc -o $out $in
      description = LINK $out
    
    build source1.o: cc source1.c
    build source2.o: cc source2.c
    build myprogram: link source1.o source2.o

The last three lines are build statements. For example myprogram is a target, source1.o and source2.o are its inputs or dependencies, and link is the rule (defined a few lines earlier) that will build the target from the inputs. In the rule's command line, $out is replaced by the target and $in is replaced by the list of inputs, with the appropriate shell quoting. Ninja also supports implicit dependencies, implicit outputs, and order-only dependencies. Providing a description for a rule turns on automake-style silent rule output.

Auto-generated dependencies

In Makefiles, the standard technique for detecting changes to implicit dependencies (such as C header files) is to generate a small Makefile snippet that specifies the dependencies and to include this generated file into your main Makefile.

The equivalent technique in a Ninja file looks like this:

    rule cc
      command = gcc -c -o $out $in -MMD -MF $out.d
      depfile = $out.d
      deps = gcc

Here, -MMD tells GCC to output the list of included files and -MF says where to write it. Normal compilation happens too; the dependency file is generated as a side-effect of the compilation process. The depfile statement tells Ninja where to read those additional dependencies from. The first time you run Ninja, it will build the source1.o file because it is out of date compared to its explicit source1.c dependency. The next time Ninja is run, it will check to see if source1.c has changed or if any of the header files listed in depfile have changed. To support this technique, Ninja understands just the subset of Makefile syntax that is actually generated by C pre-processors. Ninja also supports a similar feature of the Microsoft Visual C++ compiler, which prints specially-formatted lines to stderr when given the /showIncludes flag.

Keeping track of build command lines

Ninja remembers the command line that was used to build each target. If the command line changes (probably because the build.ninja file itself changed), then Ninja will rebuild the target, even if the dependencies didn't change. Ninja tracks this information across invocations by storing a hash of the build command for each target in a .ninja_log file in the top-level build directory.

You can do this with Makefiles too — the technique is described here and it's used by the Makefiles for both Git and Linux. But it's tedious, error-prone, and slow.

Generating (and re-generating) the build file

Ninja doesn't support loops, wildcards, or patterns; you're supposed to generate the Ninja build file from another program. A simple configure script written in Python might look like this:

    f = open("build.ninja", "w")
    sources = ["source1.c", "source2.c"]
    for source in sources:
        f.write("build {outputs}: {rule} {inputs}\n".format(
                        outputs=source.replace(".c", ".o"),
                        rule="cc",
                        inputs=source))
    # etc

If you want to support environment variables like $CFLAGS, it is best practice to read these variables in the configure script, and bake the values into the Ninja file. This makes it easier to maintain multiple build folders, such as a debug and a production build. The autotools behave this way.

Now if you edit your configure script to add another source file, source3.c, you'll want to ensure that build.ninja is re-generated. You can achieve this with another Ninja rule:

    rule configure
      command = ./configure
      generator = 1
    build build.ninja: configure

Thus, if build.ninja is out of date (older than configure), Ninja will run configure to re-create build.ninja, before it does anything else. The generator statement is necessary to exclude the target (the build.ninja file) from being removed by Ninja's auto-generated clean command. In practice, you would also want to remember any parameters originally given to configure (such as $CFLAGS), and bake them into the rule that re-runs configure.

If you're using a generator program like CMake, the principle is the same. The build.ninja file generated by CMake will arrange for itself to be re-generated if you edit CMake's build description file CMakeLists.txt.

Performance

Ninja's original motivation was speed. A no-op build of Chrome (where all the targets are already up to date) reportedly took 10 seconds with make, but less than a second with Ninja.

According to my own benchmarks, Ninja's speed difference is only really significant for very large projects (on Linux, at least). However I didn't try to implement, in make, my own version of Ninja's "rebuild if command-line changes", which would presumably slow the performance of make further.

Ninja generators and users

CMake is the most widely-used build system with Ninja support. CMake has always been a "meta build system" in that it generates build files for other build systems: various varieties of Makefiles, XCode project files for Mac, or Visual Studio project files for Windows. Since v2.8.8 in 2012, it can generate Ninja files as well.

GYP ("Generate Your Projects") is the build system used by Chromium and related projects such as the V8 JavaScript engine. As far as I know, it doesn't have much adoption elsewhere, and I don't know much about it.

Meson is a fairly recent build system that seems to be gaining traction. Unlike CMake, which seems to have come from the Windows world, Meson wants to provide first-class support for the Linux open-source ecosystem such as pkg-config, Gnome/GLib, etc., and the maintainers are happy to merge patches to support these types of projects. Maybe one day this will be able to replace autotools. (For many projects at least.) The GStreamer project recently merged "experimental" support for building with Meson — see this talk [video] at last month's GStreamer conference.

A few other Ninja generators are listed on the Ninja wiki, but it's hard to tell which of those are toy projects and which are suitable for large or complex projects.

Large projects that use Ninja include Chromium, of course; LLVM since 2012, via CMake; the Android Open Source Project, since late 2015, by parsing and translating GNU Makefiles; and GStreamer's experimental Meson-based build system mentioned above. Ninja is available in major Linux distributions — after all, it's needed to build Chromium. It's usually packaged as "ninja-build".

The Ninja community

Ninja was originally written by Evan Martin, who was working on the Chrome browser, in 2011. Martin handed over maintainership in April 2014 to Nico Weber (also on the Chrome team) because "I actually haven't myself used Ninja in something like two years", he said, having left the Chrome team. Even so, Martin is still active on the mailing list and in the Git logs.

In the last two years, the release cadence has slowed down to one release every six to ten months. The feature set is pretty stable; these days the releases contain mostly bug fixes, though some useful new features do occasionally make their way in.

The latest major release (1.7.1 in April 2016) had 160 commits by 28 different contributors. 57% of the commits were from Weber and Martin, 14% from other Google employees, 5% from Kitware (the company behind CMake), 7% from a handful of other companies (SAP, Bloomberg, SciTools), and the remaining 17% of commits from 16 contributors whose company affiliation isn't obvious from the Git logs. The mailing list is fairly quiet, with a handful of threads per month, but it does include a good amount of feature discussion, not just support queries.

Some fairly obvious bugs are still open after four years: for example, Ninja doesn't support sub-second timestamp resolution. Other convenient features never get implemented (such as deleting output files if the build failed like GNU Make's .DELETE_ON_ERROR) partly because it's easy to implement workarounds in your Ninja-file generator. Keeping the Ninja codebase small and focused seems to be the driving philosophy. All in all the project seems healthy and mature. Ninja is written in C++ (12,000 lines of it, of which 40% are tests). It is released under the Apache 2.0 license.

Ninja's big idea

For me, Ninja's biggest contribution is to popularize the concept of generating build files from a real programming language. Many projects will find CMake or Meson to be a good fit, but when the needs are more complex, it can be surprisingly simple and elegant to use a real programming language like Python atop a dumb build system like Ninja or even a subset of make.

At $DAY_JOB, we build what is essentially a custom Linux distribution for appliances, with services packaged in Docker images. The build system was getting hard to debug and we were losing confidence in the correctness of incremental builds. We decided to try Ninja. Step one was to get rid of patterns and conditionals in the Makefiles, and write a Python script to generate the Makefiles. Step two was to output Ninja instead of make format. Before even getting to step two, however, we had already gained significant improvements in understandability, traceability, and debuggability.

Of course generating Makefiles is not a new idea — the autotools and CMake have been doing it for decades. But Ninja has taught me just how easy and flexible this approach is. For more information, Ninja's manual is a short and pleasant read. The free book "The Performance of Open Source Applications" has a chapter on Ninja that covers the original motivations for Ninja and some implementation details.