The Ninja build tool

http://shakebuild.com/

One of the reasons I bring it up is because, Shake has support for reading and executing `.ninja` files! Originally, this feature was only used to benchmark Shake against Ninja to see how it faired (spoiler alert: it's pretty much just as fast). Shake also has a lot of other features, even when you only use it for Ninja; for example it can generate profiling reports of your build system, so you can see what objects/rules took the most time, etc. I actually use LLVM's CMake build system to generate .ninja files, then use Shake to run the actual build. It's useful sometimes when I occasionally want to see what takes up the most time while compiling[1]. Some people here might like that. I believe the 'lint' mode in Shake can also detect classes of errors inside Ninja files like dependency violations, so that's useful too.

The actual Shake build system itself, however, is almost an entirely different beast, mostly because it's more like a programming language library you create build systems from, rather than a DSL for a specific tool: more like e.g. Waf than CMake, so to speak. So on top of things like parallelism pools like Ninja, extending that even further beyond, to incorporate features like distributed object result caching (a la Bazel/Blaze inside Google) is quite feasible and doable. It also has extremely powerful dependency tracking features; e.g. I can have a config file of key-value pairs, and Shake tracks changes all the way down to individual variable assignments themselves, not the actual mtime or whatever of the file. You can express a dependency on the output of `cc --version`, so if the user does `export CC=clang-4.0; ./rebuild`, only rules that needed the C compiler get rerun, etc. I've been using lots of these features in a small Verilog processor I've been working on. I can just run the timing analysis tool on my design, it generates a resulting report, run a parser to parse the report inside the build system itself, and the build can fail if the constraints are violated, with a pretty error-report, breakdown, etc in the terminal window. If I extended it, I could even get the build to give me longest paths, etc out of the resulting report.

It's almost life-changing when your build system is this powerful -- things that you'd previously express as bizarre shell scripts or "shell out" to other programs to accomplish, you can just write directly in the build system itself. This, in effect, completely changes the dynamics around what your build system can even do and what its responsibilities are. I find it surprisingly simple and freeing when everything can be done "In one place", so to speak, and I'm not as worried about taking on complex features that will end in a million tears down the road.

That said, Shake is on the extreme end of "I need a really powerful build system". It's only going to pay off with serious investment and need for the features. We're going to use it in the next version of the Glasgow Haskell Compiler, but our build system is an insanely complex non-recursive Make fiasco with all kinds of impressive tricks inside of it that have destroyed its maintainability over time (in an ironic twist of fate -- since most of these tricks were intended to make the build system more reliable and less brittle, but only came at a large cost. Don't look at how the sausage is made, etc etc.)

If you can, these days I normally suggest people just use something like Make, or CMake+Ninja. There are some fundamental concepts they might lack direct analogs of in comparison to Shake or whatever, but they're pretty good and most software doesn't *really* need an exceptionally complex build system. Honestly, I would probably just like Make a lot more if the terse syntax didn't get utterly ridiculous in some cases like interpolating inside macros, escaping rules, etc, and I'd like CMake more if it WAS_NOT_SO_VERBOSE.

[1] related: LLVM really, really needs a way to leverage Ninja pools for its link rules, because if you have too many cores, you'll eat all your RAM from 10x concurrent `ld` processes. I really hate that, because Ninja loves to automatically use up every core I have by default, even if it's 48+ of them :)