Barnes: Parallel ./configure
I paid good money for my 24 CPU cores, but ./configure can only manage to use 69% of one of them. As a result, this random project takes about 13.5× longer to configure the build than it does to actually do the build.The purpose of a ./configure script is basically to run the compiler a bunch of times and check which runs succeeded. In this way it can test whether particular headers, functions, struct fields, etc. exist, which lets people write portable software. This is an embarrassingly parallel problem, but Autoconf can't parallelize it, and neither can CMake, neither can Meson, etc., etc.
(Thanks to Paul Wise).
Posted Apr 29, 2025 14:49 UTC (Tue)
by dskoll (subscriber, #1630)
[Link] (5 responses)
This is interesting work and I hope it finds its way back into autotools (or whatever the configure machinery is called.) Some large projects run ./configure a bunch of times in different subdirectories and speeding that up would be a big win.
(I know there are people who hate autotools and think it should just go away, but that's not likely to happen any time soon, at least for C and C++ projects.)
Posted Apr 29, 2025 14:56 UTC (Tue)
by bredelings (subscriber, #53082)
[Link] (3 responses)
This chart suggests that autoconf is waning, at least, compared to cmake.
What's really surprising is that prior to 1900, the usage of cmake was low, but substantially higher than auconf.
P.S. More seriously, I tried including "meson", but it was mostly finding physics references
Posted May 2, 2025 9:41 UTC (Fri)
by agraven (subscriber, #159039)
[Link] (2 responses)
Posted May 2, 2025 10:05 UTC (Fri)
by mathstuf (subscriber, #69389)
[Link] (1 responses)
Posted May 2, 2025 21:17 UTC (Fri)
by excors (subscriber, #95769)
[Link]
Posted Apr 29, 2025 15:22 UTC (Tue)
by mathstuf (subscriber, #69389)
[Link]
TTBOMK, its plans for C++ modules are "none", so its future on that front is definitely questionable if you ask me (though it is really more of an `automake` thing than `autoconf`, but they're so intertwined…). There's one enterprising developer that's been looking at making it more "native" to GCC (but I haven't seen any news on that front for over a month), but this means (more or less) embedding a build system in the compiler which is unlikely to really be all that effective or standard between implementations.
Posted Apr 29, 2025 16:14 UTC (Tue)
by tzafrir (subscriber, #11501)
[Link] (4 responses)
BTW: for something far uglier I recently happened to post: https://github.com/openucx/xpmem/pull/80
Posted Apr 29, 2025 17:04 UTC (Tue)
by jepler (subscriber, #105975)
[Link]
Posted Apr 29, 2025 18:36 UTC (Tue)
by ballombe (subscriber, #9523)
[Link] (1 responses)
Posted May 7, 2025 2:14 UTC (Wed)
by zougloub (guest, #46163)
[Link]
Posted Apr 29, 2025 20:21 UTC (Tue)
by iabervon (subscriber, #722)
[Link]
Posted Apr 29, 2025 16:35 UTC (Tue)
by marcH (subscriber, #57642)
[Link] (4 responses)
Can't hurt for sure.
Posted Apr 30, 2025 9:54 UTC (Wed)
by NYKevin (subscriber, #129325)
[Link] (3 responses)
Posted Apr 30, 2025 10:26 UTC (Wed)
by jengelh (guest, #33263)
[Link] (2 responses)
Also, with many platforms and (non-)standards having fallen out of fashion in the last two decades, a lot of tests are also more or less obsolete in the sense that one's configure.ac need not invoke them any more, like
$CC not supporting -o
Posted Apr 30, 2025 14:03 UTC (Wed)
by abatters (✭ supporter ✭, #6932)
[Link]
Indeed; that is what Yocto does.
Posted Apr 30, 2025 14:49 UTC (Wed)
by marcH (subscriber, #57642)
[Link]
This is useful but it's a different type of inefficiency because all these threads repeat mostly the same things.
Some smart caching would help in both situations.
Posted Apr 29, 2025 18:31 UTC (Tue)
by fest3er (guest, #60379)
[Link] (8 responses)
Over the past 15 years, I've wondered if configure is a simplistic "look for this, look for that" script. If it is largely sequential, perhaps with a few dependencies, it should be relatively trivial to parallelize.
I wrote a shell script years ago to use all cores when processing thousands of images (e.g., gamma adjustment or watermark). Granted, this is a form of SIMD, but the principle should apply to running thousands of different tasks in parallel. Other than GCC and Linux, pkgs I build for Smoothwall Express spend most of their time running configure; binutils is especially slow. Some pkgs can spend 30 sec. running configure, and 500ms compiling and linking. How hard could it be to modify ./configure? :) :)
Low probability alternative concept: maintain a central per-system config-status.db. Pkgs should update the central DB when installed or uninstalled. If this is possible, then a pkg's autoconf would reference the central config-status.db, falling back to figuring it out itself if the answer isn't in the central DB. Should be a darn sight faster.
Posted Apr 29, 2025 20:16 UTC (Tue)
by pizza (subscriber, #46)
[Link]
The individual statements may or may not have dependencies on prior statements, but the only way to ultimately know one way or another is to fully generate the configure shell script and then evaluate (or at least fully parse) it.
Posted Apr 29, 2025 20:24 UTC (Tue)
by ballombe (subscriber, #9523)
[Link]
It is already implemented by autoconf, see info autoconf, 7.4.
Posted Apr 29, 2025 20:33 UTC (Tue)
by wahern (subscriber, #37304)
[Link] (3 responses)
Most could be done in parallel, but there are dependencies. For example, you can't run C tests without first discovering the correct compiler invocation. The problem is there's no way to declare dependencies in the tests; it's all implicitly based on sequential execution of the configure script. Make does that well, but autoconf itself just isn't designed to be used that way. Interestingly, GCC utilizes Make internally to manage parallel LTO.
> Low probability alternative concept: maintain a central per-system config-status.db.
The value in autoconf is, IME, dealing with edge cases, especially unforeseen edge cases (thus feature test, not version test), plus providing a consistent, comprehensive, well-known configuration process for cross-compiling and distro packagers (and especially distro packagers cross-compiling). Improving the experience for simple stuff might make it run faster in some common cases, but it will inevitably cause problems for the important edge cases. A shared local database wouldn't be very helpful, and would likely even cause confusion or erroneous results for, e.g., cross-compiling, or when faced with variances in feature flags like _GNU_SOURCE. Plus, it wouldn't help with CI, where you want to start from a clean slate everytime. And if you want to cache values between edit-compile-run cycles, autoconf already provides that.
That said, there is pkg-config, which already has integration with autoconf. I've never found it very useful, but perhaps it's useful for distro packagers.
I don't use autoconf much myself, though. IME regular Makefiles (with simple shell tests) and preprocessor tests suffice for most Unix-like builds, at least for the stuff I write. I do heavily use the HAVE_FOO pattern, though, so if something breaks it's easy to workaround with a -DHAVE_FOO=0 or -DHAVE_FOO=1, without having to hack the source beforehand. When I do use autoconf, I always modify the generation of config.h so HAVE_FOO macros can be modified from CPPFLAGS when invoking Make; in particular, so one can specify -DHAVE_FOO=0 (normally autoconf's HAVE_FOO defines are meant to be used with #ifdef, not #if, which makes it impossible to disable an erroneously enabled feature).
Posted Apr 30, 2025 5:51 UTC (Wed)
by tzafrir (subscriber, #11501)
[Link]
It was better than nothing, but still slow.
And it's ugly. And still slow. Just not as slow as running all of them sequentially.
Posted Apr 30, 2025 8:23 UTC (Wed)
by smcv (subscriber, #53363)
[Link] (1 responses)
pkg-config is for a specific purpose: it's for linking to a dependency with a cooperative maintainer (they must provide .pc metadata describing their library), and is most useful if the dependency meets several assumptions:
* it's usually a C or C++ library (although pkg-config can also be used to check for "data" packages like wayland-protocols)
It's a replacement for every library having its own special "detect me" script like sdl2-config, or its own special Autoconf macro like AM_PATH_GLIB_2_0, or both, which scales poorly - especially when those scripts and macros tend to work in simple cases, but not work in more complex situations like cross-compiling. It is not a replacement for Autoconf checks in their full generality.
For libraries that meet those assumptions (for example libjpeg, libpng, GTK, SDL, Qt), yes it will make it much, much easier for distro packagers if you check for the library with pkg-config instead of open-coding your own dependency detection. It'll probably make your dependency detection easier and more reliable for you, too. It's also good for users who are installing your software in unusual environments with dependencies installed in non-standard places (it gives them a non-library-specific way they can override the automatic detection and force a particular result), or users or packagers who are cross-compiling.
pkg-config is not for detecting whether your software is being compiled on a platform where libc does or doesn't have a frobnicate() function, or whether the compiler supports the -Wno-complaining option, or whether C++17 is available. That's outside its scope. Those checks are likely to be the ones with biggest impact on how long it takes to run ./configure, just because if they are checked at all, they tend to be rather numerous (each one is small and quick, but the time taken adds up), so using pkg-config will not necessarily speed up a configure script significantly: the reasons to use it are non-speed-related.
pkg-config also isn't for detecting libraries that aren't cooperating with the pkg-config ecosystem: if the dependency library has no .pc file at all, or a .pc file that is well-intentioned but wrong, then pkg-config will not help you to detect it (but adding or fixing the .pc file so that the dependency can be detected more easily might be a valid feature request or bug report).
Posted Apr 30, 2025 13:17 UTC (Wed)
by mathstuf (subscriber, #69389)
[Link]
- /lib/libA.so <- desired
Whichever you put first, one of the `-L` flags will cause one of the desired libraries to be shadowed. CMake, at least, vastly prefers just passing libraries by absolute path rather than using `-L` search path semantics.
While search paths are probably broken here (e.g., runtime is still likely confused even if you link the intended libraries), flags like `-static` have contextual meaning that can cause an unintended library usage.
Posted Apr 30, 2025 6:22 UTC (Wed)
by mathstuf (subscriber, #69389)
[Link] (1 responses)
There are sometimes dependencies (e.g., finding hdf5 before checking for its APIs).
Posted Apr 30, 2025 8:43 UTC (Wed)
by smcv (subscriber, #53363)
[Link]
Yes, this! At a libc API level, actively-maintained POSIXy operating systems are more similar than they are different, and the fastest possible configure check is the one you don't do.
The first portability decision to make in a project is whether it aims to be POSIX-only, or whether it aims to be portable to Windows. POSIX and Windows are sufficiently different "shapes" that if both are supported, in practice there are going to be two code paths anyway; and it's likely that it will be easier to test "#ifdef _WIN32" or ask the build system "is this Windows?" rather than inventing configure checks that end up meaning the same thing.
For the POSIX code path, if you choose a baseline level of functionality and say that you require it, a lot of configure checks become unnecessary: there's no point in checking for "normal POSIX things" unless someone has gone to the effort of testing a code path for specific platforms where they're missing (and it's 2025, so 1980s proprietary Unix is probably out of scope for most projects now). Similarly if you can nominate some reasonable C standard like C99 or C11 and say "your compiler must be at least this modern", or the equivalent for C++, then there's no need to feature-test for individual functions from that standard.
Similarly, for the Windows code path if you have one, if you choose baseline Windows/SDK/compiler versions and assume them, then configure checks for anything provided by that baseline are just a waste of time.
Depending what the project does and how much OS integration it needs, some projects will also need a special code path for the macOS/iOS/etc. family, or for Android, or something like that. Again those are easily detected at the preprocessor or build system level.
*Then*, after distinguishing between those major code paths, it becomes useful to do feature-tests for whether particular corners of functionality are available or not - but every time there's a feature-test for whether there is a frobnicate() function, it should be because the project's source code is genuinely looking at whether HAVE_FROBNICATE is defined, and doing things differently in the case where it is, and the case where it isn't.
Posted Apr 29, 2025 20:41 UTC (Tue)
by aleXXX (subscriber, #2742)
[Link] (4 responses)
But parallelzing it is not trivial in the general case.
if(HAVE_ZSTD_H)
if(SOME_CHECK_SUCCESS)
Things basically have to be done sequentially if scripted like this, since any check can influence (global) variables which are used by following checks.
Some time is also spent on disk IO, which is especially slow under Windows. Not sure how much that would benefit from parallelization.
Posted Apr 30, 2025 6:27 UTC (Wed)
by mathstuf (subscriber, #69389)
[Link] (3 responses)
FD: I'm a CMake developer and have been involved in the design discussions for this.
Posted Apr 30, 2025 20:25 UTC (Wed)
by aleXXX (subscriber, #2742)
[Link] (2 responses)
Posted May 1, 2025 7:44 UTC (Thu)
by mathstuf (subscriber, #69389)
[Link] (1 responses)
[1] https://gitlab.kitware.com/cmake/cmake/-/issues/16966#not...
Posted May 1, 2025 12:35 UTC (Thu)
by mathstuf (subscriber, #69389)
[Link]
Posted Apr 30, 2025 2:17 UTC (Wed)
by scruffie (guest, #5704)
[Link] (1 responses)
Posted Apr 30, 2025 3:26 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Apr 30, 2025 9:00 UTC (Wed)
by zoobab (guest, #9945)
[Link] (3 responses)
BTW the github repo https://github.com/tavianator/parconf should have more detailed instructions on how to use it.
Posted Apr 30, 2025 9:45 UTC (Wed)
by rschroev (subscriber, #4164)
[Link]
Posted Apr 30, 2025 14:38 UTC (Wed)
by marcH (subscriber, #57642)
[Link]
Posted Apr 30, 2025 14:47 UTC (Wed)
by tavianator (guest, #103445)
[Link]
You're right, I should add a readme. But in the meantime,
$ ./configure
will do it. And you can run the demo app with ./app * or something if you want.
But the point is not really to run it, the point is to read it and/or copy-paste it :)
Posted Apr 30, 2025 11:27 UTC (Wed)
by jafd (subscriber, #129642)
[Link] (1 responses)
Obviously there are caveats, like the versions of stuff differing from run to run if you specify the packages fuzzily enough, but the world of C/C++ moves way slower than your friendly neighbourhood Javascript ecosystem, and this is solvable by bracketing your dependency versions in the script that's installing them. However, I believe it to be a case of "you can't win them all", and moving some complexity to a place where it's easier to fight it.
You can also try to slap a parallel configure on top of it to win build time when there is a cache miss. I assume most people wouldn't bother.
Posted Apr 30, 2025 12:28 UTC (Wed)
by farnz (subscriber, #17727)
[Link]
Posted Apr 30, 2025 21:58 UTC (Wed)
by jond (subscriber, #37669)
[Link] (1 responses)
FOOFLAGS := $(shell pkg-config —cflags foo)
Default:
Posted May 1, 2025 14:14 UTC (Thu)
by madscientist (subscriber, #16861)
[Link]
Posted May 2, 2025 4:15 UTC (Fri)
by mirabilos (subscriber, #84359)
[Link]
It’s also the step of which you want linear, analysable output if something does go wrong, and in most such situations all you have is a build log from some build dæmon. This is why I decided to output instead of writing to a configure.log-like file for mine.
Posted May 2, 2025 9:58 UTC (Fri)
by cyperpunks (subscriber, #39406)
[Link]
Interesting and I hope eventually useful
Interesting and I hope eventually useful
I suspect that prior to 1900 the usage of cmake was in fact zero :p
Interesting and I hope eventually useful
Interesting and I hope eventually useful
Interesting and I hope eventually useful
Interesting and I hope eventually useful
dependencies between tests
dependencies between tests
dependencies between tests
One can also use ccache.
dependencies between tests
dependencies between tests
configure once, compile many times
configure once, compile many times
configure once, compile many times
ANSI C prototype support
Support for keywords "inline", "volatile", "const"
AC_SYS_LARGEFILE (just unconditionally -D_LARGEFILE_SOURCE?)
configure once, compile many times
configure once, compile many times
Parallelizing a shell script
Parallelizing a shell script
Parallelizing a shell script
Parallelizing a shell script
Parallelizing a shell script
https://github.com/knneth/mlnx-ofa_kernel/tree/e9390164da...
See config/parallel-build.m4
Parallelizing a shell script
* there is only one implementation of the API, or maybe a small finite number (for example IJG libjpeg and libjpeg-turbo; but if each proprietary Unix had its own unique implementation then pkg-config is probably the wrong tool)
* the API/ABI is sufficiently carefully-managed that 99% of the time the only compile-time functionality checks you need are of the form "do we have version 1.2.3 or better?"
* the .pc file is correctly-written and describes all the dependencies
Parallelizing a shell script
- /lib/libB.so
- /lib2/libA.so
- /lib2/libB.so <- desired
Parallelizing a shell script
Parallelizing a shell script
Parallelizing...
In cmake, the logic can look like this:
...
check_include_file(zstd.h HAVE_ZSTD_H)
check_c_source_compiles("int main(){...some C code}" SOME_CHECK_SUCCESS)
endif()
.. do other stuff
endif()
Parallelizing...
Parallelizing...
Is there a gitlab ticket for it ?
Parallelizing...
Parallelizing...
I noticed that SQLite recently moved to autosetup (from commit history, about November), which uses Tcl. It might be easier to add opt-in parallelism in something like that instead of the terrible languages of autoconf or CMake.
Other configure systems might be easier ...
Other configure systems might be easier ...
Ninja?
Ninja?
Ninja?
Ninja?
$ make
There are more ways to skin a cat
If I understand the intentions properly, this is what the config.cache file is meant to help with; it's just that autoconf lacks any notion of cache management, or indeed hooks that you could use to partially invalidate a cache on changes.
There are more ways to skin a cat
Make and subshell evaluation
BARFLAGS := $(shell pkg-config —cflags bar)
BAZFLAGS := $(shell pkg-config —cflags baz)
cc $(FOOFLAGS) $(BARFLAGS) $(BAZFLAGS)…
Make and subshell evaluation
it’s not embarrassingly parallel
Add layer below configuration tool?
