Maintainer confidential: Opportunities and challenges of the ubiquitous but under-resourced Yocto Project (Linux.com)

Indeed while I'll probably get lynched for saying it: The truth is, well written autoconf build system is very easy to cross compile (while a badly written one, or worse a project that attempts to reinvent Make in a common interpreted language for it's build system, or a poorly written one in cmake, or probably any other build system) may fall flat on it's face, and go toward impossible to cross compile.

As people have pointed out: Some of the common stumbling blocks are: auto-detect, host info leakage, self-hosting (that is, the project needs to run some part of itself on the build machine to fully build it self, even for another target).

And while there are tool to make this easier: the fact remains, lots of notable projects don't use them. Or worse insist on using quasi-home rolled build systems that ignore every thing learned about software build systems in the last 40 years.

The problem with any sort of heuristic like, "this is a compile line so it can be run in parallel with this other compile invocation" is that builds just aren't that easy. A "real project" which has a build system that's as simple as "compile a bunch of files, maybe make some libraries, then link it" is vanishingly rare.

Builds want to generate output files (create headers or source files that contain Git SHA ids, etc.), they have all sorts of DSLs that they use to generate source code, they rely on other tools (yacc/lex are just the OG examples). And it's not even enough to say "do all the weird stuff that's not a compilation first, then do all the compilation" because sometimes the output generated requires some of the build to be completed first.

I think if you ever tried to actually create this "meta-build tool" you are imagining, you'll quickly run up against reality. IMO the only possible way to succeed is to create a "meta-build tool" that can use each project's own build environment, with a unified interface over it. This is basically what Arch does, and there are many other similar examples.

You reference compile_commands.json but that is manifestly insufficient as it captures ONLY compile commands and there is so, so much more to even a basic build system that is not represented by compile_commands.json.

The most important thing is to to capture the outputs of any build command. The inputs are relatively easy to derive - you can run the build without the inputs and see where it fails - or use a FUSE filesystem or ptrace to see which files the build attempts to open. Or (like ccache) go to great efforts to understand the build command line. Tools to do this exist.

As NYKevin mentions below: bazel provides the robustness. It enforces "hermetic" builds where the build will fail if you have not correctly described the dependencies. This is why such a tool would be generating bazel BUILD files (or some other similarly "hermetic" build system).

I agree that such a tool couldn't run without human supervision over all codebases out there, but I believe that such a tool could dramatically reduce the manpower requirements for porting a bunch of build systems. As I said in the conclusion of the blog post linked above - the problem is social because there is too much work for one person to do - but with the right tooling you might be able to dramatically reduce the number of people you'd need to be pulling in the same direction.

In terms of build power - I think a model like this could help because it's much more efficient. You only rebuild the files that have changed so builds should be faster and use fewer resources. Currently if the man page of libreoffice changes the whole package must be rebuilt. With a build system that understands the dependencies between the files in a more fine-grained manner this wouldn't be a problem.