|
|
Subscribe / Log in / New account

Barnes: Parallel ./configure

Tavian Barnes takes on the tedious process of waiting for configure scripts to run.

I paid good money for my 24 CPU cores, but ./configure can only manage to use 69% of one of them. As a result, this random project takes about 13.5× longer to configure the build than it does to actually do the build.

The purpose of a ./configure script is basically to run the compiler a bunch of times and check which runs succeeded. In this way it can test whether particular headers, functions, struct fields, etc. exist, which lets people write portable software. This is an embarrassingly parallel problem, but Autoconf can't parallelize it, and neither can CMake, neither can Meson, etc., etc.

(Thanks to Paul Wise).


to post comments

Interesting and I hope eventually useful

Posted Apr 29, 2025 14:49 UTC (Tue) by dskoll (subscriber, #1630) [Link] (5 responses)

This is interesting work and I hope it finds its way back into autotools (or whatever the configure machinery is called.) Some large projects run ./configure a bunch of times in different subdirectories and speeding that up would be a big win.

(I know there are people who hate autotools and think it should just go away, but that's not likely to happen any time soon, at least for C and C++ projects.)

Interesting and I hope eventually useful

Posted Apr 29, 2025 14:56 UTC (Tue) by bredelings (subscriber, #53082) [Link] (3 responses)

https://books.google.com/ngrams/graph?content=autoconf%2C...

This chart suggests that autoconf is waning, at least, compared to cmake.

What's really surprising is that prior to 1900, the usage of cmake was low, but substantially higher than auconf.

P.S. More seriously, I tried including "meson", but it was mostly finding physics references

Interesting and I hope eventually useful

Posted May 2, 2025 9:41 UTC (Fri) by agraven (subscriber, #159039) [Link] (2 responses)

I suspect that prior to 1900 the usage of cmake was in fact zero :p

Interesting and I hope eventually useful

Posted May 2, 2025 10:05 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (1 responses)

I agree about that :) . However, "cmake" is probably far more likely to show up as a typo (a transposition with "make" being a real word and "c" being an end-of-word letter often enough) than "autoconf".

Interesting and I hope eventually useful

Posted May 2, 2025 21:17 UTC (Fri) by excors (subscriber, #95769) [Link]

The Ngram page has helpful "search in Google Books" links, which show it's OCR errors rather than typos: the 19th century books are saying things like "at the point C make the angle ACD=ASB", or the 'c' is from a margin note or footnote marker etc. The old typesetters did their jobs properly, it's Google who's getting it wrong.

Interesting and I hope eventually useful

Posted Apr 29, 2025 15:22 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

> (I know there are people who hate autotools and think it should just go away, but that's not likely to happen any time soon, at least for C and C++ projects.)

TTBOMK, its plans for C++ modules are "none", so its future on that front is definitely questionable if you ask me (though it is really more of an `automake` thing than `autoconf`, but they're so intertwined…). There's one enterprising developer that's been looking at making it more "native" to GCC (but I haven't seen any news on that front for over a month), but this means (more or less) embedding a build system in the compiler which is unlikely to really be all that effective or standard between implementations.

dependencies between tests

Posted Apr 29, 2025 16:14 UTC (Tue) by tzafrir (subscriber, #11501) [Link] (4 responses)

Can you test anything that depends on a previous test? I guess you mostly work around it by having a separate stage for checking compiler flags.

BTW: for something far uglier I recently happened to post: https://github.com/openucx/xpmem/pull/80

dependencies between tests

Posted Apr 29, 2025 17:04 UTC (Tue) by jepler (subscriber, #105975) [Link]

probably yes but the Makefile might need the dependencies made explicit so it doesn't run the depending test before its dependent header has been generated.

dependencies between tests

Posted Apr 29, 2025 18:36 UTC (Tue) by ballombe (subscriber, #9523) [Link] (1 responses)

autoconf already has a cache feature , see --config-cache, --cache-file, etc.
One can also use ccache.

dependencies between tests

Posted May 7, 2025 2:14 UTC (Wed) by zougloub (guest, #46163) [Link]

Well, ccache skips caching of autoconf ("conftest") checks.

dependencies between tests

Posted Apr 29, 2025 20:21 UTC (Tue) by iabervon (subscriber, #722) [Link]

Make's core functionality is generating files that depend on other generated files, so that would just naturally work. However, it's a bit trickier to get it to figure out by itself that a test uses the compiler flags and therefore depends on checking the compiler flags (as opposed to the developer indicating the dependency).

configure once, compile many times

Posted Apr 29, 2025 16:35 UTC (Tue) by marcH (subscriber, #57642) [Link] (4 responses)

I think this common design "pattern" is based on the assumption that developers configure "once" and then compile many times? Obviously wrong for automation now.

Can't hurt for sure.

configure once, compile many times

Posted Apr 30, 2025 9:54 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (3 responses)

It doesn't help that configure is traditionally a shell script. The POSIX-compatible subset of the shell has no lighter-weight parallelism facility than spawning an entire process, with all of the usual problems that implies. Probably nobody thought it was worth it, before 24-core systems became a thing that normal people had on their desks.

configure once, compile many times

Posted Apr 30, 2025 10:26 UTC (Wed) by jengelh (guest, #33263) [Link] (2 responses)

When you have 24 threads, just run 24 configures (for different build types, different packages), problem solved ;-)

Also, with many platforms and (non-)standards having fallen out of fashion in the last two decades, a lot of tests are also more or less obsolete in the sense that one's configure.ac need not invoke them any more, like

$CC not supporting -o
ANSI C prototype support
Support for keywords "inline", "volatile", "const"
AC_SYS_LARGEFILE (just unconditionally -D_LARGEFILE_SOURCE?)

configure once, compile many times

Posted Apr 30, 2025 14:03 UTC (Wed) by abatters (✭ supporter ✭, #6932) [Link]

> When you have 24 threads, just run 24 configures (for different build types, different packages), problem solved ;-)

Indeed; that is what Yocto does.

configure once, compile many times

Posted Apr 30, 2025 14:49 UTC (Wed) by marcH (subscriber, #57642) [Link]

> When you have 24 threads, just run 24 configures (for different build types, different packages), problem solved ;-)

This is useful but it's a different type of inefficiency because all these threads repeat mostly the same things.

Some smart caching would help in both situations.

Parallelizing a shell script

Posted Apr 29, 2025 18:31 UTC (Tue) by fest3er (guest, #60379) [Link] (8 responses)

Generally speaking, are the configure tasks simply sequential? Are there (or can there be) dependencies?

Over the past 15 years, I've wondered if configure is a simplistic "look for this, look for that" script. If it is largely sequential, perhaps with a few dependencies, it should be relatively trivial to parallelize.

I wrote a shell script years ago to use all cores when processing thousands of images (e.g., gamma adjustment or watermark). Granted, this is a form of SIMD, but the principle should apply to running thousands of different tasks in parallel. Other than GCC and Linux, pkgs I build for Smoothwall Express spend most of their time running configure; binutils is especially slow. Some pkgs can spend 30 sec. running configure, and 500ms compiling and linking. How hard could it be to modify ./configure? :) :)

Low probability alternative concept: maintain a central per-system config-status.db. Pkgs should update the central DB when installed or uninstalled. If this is possible, then a pkg's autoconf would reference the central config-status.db, falling back to figuring it out itself if the answer isn't in the central DB. Should be a darn sight faster.

Parallelizing a shell script

Posted Apr 29, 2025 20:16 UTC (Tue) by pizza (subscriber, #46) [Link]

> Generally speaking, are the configure tasks simply sequential? Are there (or can there be) dependencies?

The individual statements may or may not have dependencies on prior statements, but the only way to ultimately know one way or another is to fully generate the configure shell script and then evaluate (or at least fully parse) it.

Parallelizing a shell script

Posted Apr 29, 2025 20:24 UTC (Tue) by ballombe (subscriber, #9523) [Link]

> Low probability alternative concept: maintain a central per-system config-status.db. Pkgs should update the central DB when installed or uninstalled. If this is possible, then a pkg's autoconf would reference the central config-status.db, falling back to figuring it out itself if the answer isn't in the central DB. Should be a darn sight faster.

It is already implemented by autoconf, see info autoconf, 7.4.

Parallelizing a shell script

Posted Apr 29, 2025 20:33 UTC (Tue) by wahern (subscriber, #37304) [Link] (3 responses)

> Generally speaking, are the configure tasks simply sequential? Are there (or can there be) dependencies?

Most could be done in parallel, but there are dependencies. For example, you can't run C tests without first discovering the correct compiler invocation. The problem is there's no way to declare dependencies in the tests; it's all implicitly based on sequential execution of the configure script. Make does that well, but autoconf itself just isn't designed to be used that way. Interestingly, GCC utilizes Make internally to manage parallel LTO.

> Low probability alternative concept: maintain a central per-system config-status.db.

The value in autoconf is, IME, dealing with edge cases, especially unforeseen edge cases (thus feature test, not version test), plus providing a consistent, comprehensive, well-known configuration process for cross-compiling and distro packagers (and especially distro packagers cross-compiling). Improving the experience for simple stuff might make it run faster in some common cases, but it will inevitably cause problems for the important edge cases. A shared local database wouldn't be very helpful, and would likely even cause confusion or erroneous results for, e.g., cross-compiling, or when faced with variances in feature flags like _GNU_SOURCE. Plus, it wouldn't help with CI, where you want to start from a clean slate everytime. And if you want to cache values between edit-compile-run cycles, autoconf already provides that.

That said, there is pkg-config, which already has integration with autoconf. I've never found it very useful, but perhaps it's useful for distro packagers.

I don't use autoconf much myself, though. IME regular Makefiles (with simple shell tests) and preprocessor tests suffice for most Unix-like builds, at least for the stuff I write. I do heavily use the HAVE_FOO pattern, though, so if something breaks it's easy to workaround with a -DHAVE_FOO=0 or -DHAVE_FOO=1, without having to hack the source beforehand. When I do use autoconf, I always modify the generation of config.h so HAVE_FOO macros can be modified from CPPFLAGS when invoking Make; in particular, so one can specify -DHAVE_FOO=0 (normally autoconf's HAVE_FOO defines are meant to be used with #ifdef, not #if, which makes it impossible to disable an erroneously enabled feature).

Parallelizing a shell script

Posted Apr 30, 2025 5:51 UTC (Wed) by tzafrir (subscriber, #11501) [Link]

So what if you have an autoconf script that needs to run over 800 tests that most of them are completely independent? Why not run them in parallel? The script I posted above is a replacement to such a parallel autoconf script. It starts sequential but at some point starts running separate jobs in subprocesses, each in its own temporary subdirectory. BTW: the temporary subdirectory and repeating tested file name means that ccache doesn't cache it well.

It was better than nothing, but still slow.
https://github.com/knneth/mlnx-ofa_kernel/tree/e9390164da...
See config/parallel-build.m4

And it's ugly. And still slow. Just not as slow as running all of them sequentially.

Parallelizing a shell script

Posted Apr 30, 2025 8:23 UTC (Wed) by smcv (subscriber, #53363) [Link] (1 responses)

> That said, there is pkg-config, which already has integration with autoconf. I've never found it very useful, but perhaps it's useful for distro packagers.

pkg-config is for a specific purpose: it's for linking to a dependency with a cooperative maintainer (they must provide .pc metadata describing their library), and is most useful if the dependency meets several assumptions:

* it's usually a C or C++ library (although pkg-config can also be used to check for "data" packages like wayland-protocols)
* there is only one implementation of the API, or maybe a small finite number (for example IJG libjpeg and libjpeg-turbo; but if each proprietary Unix had its own unique implementation then pkg-config is probably the wrong tool)
* the API/ABI is sufficiently carefully-managed that 99% of the time the only compile-time functionality checks you need are of the form "do we have version 1.2.3 or better?"
* the .pc file is correctly-written and describes all the dependencies

It's a replacement for every library having its own special "detect me" script like sdl2-config, or its own special Autoconf macro like AM_PATH_GLIB_2_0, or both, which scales poorly - especially when those scripts and macros tend to work in simple cases, but not work in more complex situations like cross-compiling. It is not a replacement for Autoconf checks in their full generality.

For libraries that meet those assumptions (for example libjpeg, libpng, GTK, SDL, Qt), yes it will make it much, much easier for distro packagers if you check for the library with pkg-config instead of open-coding your own dependency detection. It'll probably make your dependency detection easier and more reliable for you, too. It's also good for users who are installing your software in unusual environments with dependencies installed in non-standard places (it gives them a non-library-specific way they can override the automatic detection and force a particular result), or users or packagers who are cross-compiling.

pkg-config is not for detecting whether your software is being compiled on a platform where libc does or doesn't have a frobnicate() function, or whether the compiler supports the -Wno-complaining option, or whether C++17 is available. That's outside its scope. Those checks are likely to be the ones with biggest impact on how long it takes to run ./configure, just because if they are checked at all, they tend to be rather numerous (each one is small and quick, but the time taken adds up), so using pkg-config will not necessarily speed up a configure script significantly: the reasons to use it are non-speed-related.

pkg-config also isn't for detecting libraries that aren't cooperating with the pkg-config ecosystem: if the dependency library has no .pc file at all, or a .pc file that is well-intentioned but wrong, then pkg-config will not help you to detect it (but adding or fixing the .pc file so that the dependency can be detected more easily might be a valid feature request or bug report).

Parallelizing a shell script

Posted Apr 30, 2025 13:17 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

Note that .pc files tend to deal in raw flags, so they're not useful unless you're working with a GNU-like (or sufficiently GNU-like) compiler frontend. They also tend to prefer finding things with search paths, so convoluted setups like this don't work with the typical `-L/path -lfoo` pattern:

- /lib/libA.so <- desired
- /lib/libB.so
- /lib2/libA.so
- /lib2/libB.so <- desired

Whichever you put first, one of the `-L` flags will cause one of the desired libraries to be shadowed. CMake, at least, vastly prefers just passing libraries by absolute path rather than using `-L` search path semantics.

While search paths are probably broken here (e.g., runtime is still likely confused even if you link the intended libraries), flags like `-static` have contextual meaning that can cause an unintended library usage.

Parallelizing a shell script

Posted Apr 30, 2025 6:22 UTC (Wed) by mathstuf (subscriber, #69389) [Link] (1 responses)

There are lots of checks one can just defer to the preprocessor (e.g., all sizeof(builtin_type) have preprocessor bits), so those can just go away completely. Probing for APIs you can't actually work without (e.g., `stat`) is also silly and usually better relegated to "is Windows or not" checks. Others would be easier with some C++ to do TMP to get the required information (e.g., detecting argument counts and types) and generate headers directly that way instead of probing for known signatures.

There are sometimes dependencies (e.g., finding hdf5 before checking for its APIs).

Parallelizing a shell script

Posted Apr 30, 2025 8:43 UTC (Wed) by smcv (subscriber, #53363) [Link]

> Probing for APIs you can't actually work without (e.g., `stat`) is also silly and usually better relegated to "is Windows or not" checks

Yes, this! At a libc API level, actively-maintained POSIXy operating systems are more similar than they are different, and the fastest possible configure check is the one you don't do.

The first portability decision to make in a project is whether it aims to be POSIX-only, or whether it aims to be portable to Windows. POSIX and Windows are sufficiently different "shapes" that if both are supported, in practice there are going to be two code paths anyway; and it's likely that it will be easier to test "#ifdef _WIN32" or ask the build system "is this Windows?" rather than inventing configure checks that end up meaning the same thing.

For the POSIX code path, if you choose a baseline level of functionality and say that you require it, a lot of configure checks become unnecessary: there's no point in checking for "normal POSIX things" unless someone has gone to the effort of testing a code path for specific platforms where they're missing (and it's 2025, so 1980s proprietary Unix is probably out of scope for most projects now). Similarly if you can nominate some reasonable C standard like C99 or C11 and say "your compiler must be at least this modern", or the equivalent for C++, then there's no need to feature-test for individual functions from that standard.

Similarly, for the Windows code path if you have one, if you choose baseline Windows/SDK/compiler versions and assume them, then configure checks for anything provided by that baseline are just a waste of time.

Depending what the project does and how much OS integration it needs, some projects will also need a special code path for the macOS/iOS/etc. family, or for Android, or something like that. Again those are easily detected at the preprocessor or build system level.

*Then*, after distinguishing between those major code paths, it becomes useful to do feature-tests for whether particular corners of functionality are available or not - but every time there's a feature-test for whether there is a frobnicate() function, it should be because the project's source code is genuinely looking at whether HAVE_FROBNICATE is defined, and doing things differently in the case where it is, and the case where it isn't.

Parallelizing...

Posted Apr 29, 2025 20:41 UTC (Tue) by aleXXX (subscriber, #2742) [Link] (4 responses)

I agree that it's embarrassing that in 2025 the configure/cmake step runs sequentially.

But parallelzing it is not trivial in the general case.
In cmake, the logic can look like this:
...
check_include_file(zstd.h HAVE_ZSTD_H)

if(HAVE_ZSTD_H)
check_c_source_compiles("int main(){...some C code}" SOME_CHECK_SUCCESS)
endif()

if(SOME_CHECK_SUCCESS)
.. do other stuff
endif()

Things basically have to be done sequentially if scripted like this, since any check can influence (global) variables which are used by following checks.

Some time is also spent on disk IO, which is especially slow under Windows. Not sure how much that would benefit from parallelization.

Parallelizing...

Posted Apr 30, 2025 6:27 UTC (Wed) by mathstuf (subscriber, #69389) [Link] (3 responses)

The current plan is something like (all subject to bikeshedding) `try_compile(ASYNC token)`. This performs the trial on another thread. You then "realize" it with `cmake_await(TOKENS token)` which make the results of that trial present in the main execution. This way you can run a lot of trials concurrently without having to deal with thinking about threading or anything.

FD: I'm a CMake developer and have been involved in the design discussions for this.

Parallelizing...

Posted Apr 30, 2025 20:25 UTC (Wed) by aleXXX (subscriber, #2742) [Link] (2 responses)

I haven't heard of this (e.g. on the devel channel).
Is there a gitlab ticket for it ?

Parallelizing...

Posted May 1, 2025 7:44 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (1 responses)

I mentioned it in this issue[1] but couldn't find a public link there either (4 years ago). I *know* it has been discussed but I can't find records of it :( . I'll continue the search.

[1] https://gitlab.kitware.com/cmake/cmake/-/issues/16966#not...

Parallelizing...

Posted May 1, 2025 12:35 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

So my syntax discussions were on internal chats, but the main issue tracking it is here: https://gitlab.kitware.com/cmake/cmake/-/issues/18929

Other configure systems might be easier ...

Posted Apr 30, 2025 2:17 UTC (Wed) by scruffie (guest, #5704) [Link] (1 responses)

I noticed that SQLite recently moved to autosetup (from commit history, about November), which uses Tcl. It might be easier to add opt-in parallelism in something like that instead of the terrible languages of autoconf or CMake.

Other configure systems might be easier ...

Posted Apr 30, 2025 3:26 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

Wow. It's amazing that it actually can compile Tcl and run the build faster than ./configure finishes.

Ninja?

Posted Apr 30, 2025 9:00 UTC (Wed) by zoobab (guest, #9945) [Link] (3 responses)

Have you tried Ninja Build? https://ninja-build.org/ I think it has parrallelism and Cmake support, but not sure, it has been 9 years since I have not used it.

BTW the github repo https://github.com/tavianator/parconf should have more detailed instructions on how to use it.

Ninja?

Posted Apr 30, 2025 9:45 UTC (Wed) by rschroev (subscriber, #4164) [Link]

Ninja is a build tool, not a configure tool. It's an alternative to make, not to autoconf/CMake.

Ninja?

Posted Apr 30, 2025 14:38 UTC (Wed) by marcH (subscriber, #57642) [Link]

ninja is a "simplified" Make, reduced to its core. You can use some Make features and skip autotools for simple, less portable projects. Not so much with ninja which specifically recommends that ninja.build be generated by another tool in the second sentence at https://ninja-build.org/ (typically: generated by CMake or meson)

Ninja?

Posted Apr 30, 2025 14:47 UTC (Wed) by tavianator (guest, #103445) [Link]

> BTW the github repo https://github.com/tavianator/parconf should have more detailed instructions on how to use it.

You're right, I should add a readme. But in the meantime,

$ ./configure
$ make

will do it. And you can run the demo app with ./app * or something if you want.

But the point is not really to run it, the point is to read it and/or copy-paste it :)

There are more ways to skin a cat

Posted Apr 30, 2025 11:27 UTC (Wed) by jafd (subscriber, #129642) [Link] (1 responses)

One way to sidestep the issue for CI/CD environments is to cache the configure artifacts (like config.h), keyed by the checksum of the configure script itself + whatever script was used to prop up the environment and install build dependencies. Then inject the results from cache, and you don't need to run configure at all, until you tweak the configure or its surroundings. Look ma, I'm back to the flow where I configure once and build many times.

Obviously there are caveats, like the versions of stuff differing from run to run if you specify the packages fuzzily enough, but the world of C/C++ moves way slower than your friendly neighbourhood Javascript ecosystem, and this is solvable by bracketing your dependency versions in the script that's installing them. However, I believe it to be a case of "you can't win them all", and moving some complexity to a place where it's easier to fight it.

You can also try to slap a parallel configure on top of it to win build time when there is a cache miss. I assume most people wouldn't bother.

There are more ways to skin a cat

Posted Apr 30, 2025 12:28 UTC (Wed) by farnz (subscriber, #17727) [Link]

If I understand the intentions properly, this is what the config.cache file is meant to help with; it's just that autoconf lacks any notion of cache management, or indeed hooks that you could use to partially invalidate a cache on changes.

Make and subshell evaluation

Posted Apr 30, 2025 21:58 UTC (Wed) by jond (subscriber, #37669) [Link] (1 responses)

I wonder if (or can) the following pattern be parallelized. Does (GNU) make execute the three subshells serially?

FOOFLAGS := $(shell pkg-config —cflags foo)
BARFLAGS := $(shell pkg-config —cflags bar)
BAZFLAGS := $(shell pkg-config —cflags baz)

Default:
cc $(FOOFLAGS) $(BARFLAGS) $(BAZFLAGS)…

Make and subshell evaluation

Posted May 1, 2025 14:14 UTC (Thu) by madscientist (subscriber, #16861) [Link]

No. Only targets can be built in parallel.

it’s not embarrassingly parallel

Posted May 2, 2025 4:15 UTC (Fri) by mirabilos (subscriber, #84359) [Link]

It’s only parallelisable if you do it like autoconf does instead of with dependencies between the steps.

It’s also the step of which you want linear, analysable output if something does go wrong, and in most such situations all you have is a build log from some build dæmon. This is why I decided to output instead of writing to a configure.log-like file for mine.

Add layer below configuration tool?

Posted May 2, 2025 9:58 UTC (Fri) by cyperpunks (subscriber, #39406) [Link]

If there was a layer below configuration tool, cmake/autoconf could quickly perform queries to get correct information about a system.


Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds