Not logged in
Log in now
Create an account
Subscribe to LWN
LWN.net Weekly Edition for May 16, 2013
A look at the PyPy 2.0 release
PostgreSQL 9.3 beta: Federated databases and more
LWN.net Weekly Edition for May 9, 2013
(Nearly) full tickless operation in 3.10
The faster Automake dies the better off we'll all be. Anything is better than automake: scons, waf, cmake, etc.
Please, start to use them. Samba switched to waf for samba4 and results are great so far. And it's hard to find a body of code more complicated than samba4.
KS2011: Afternoon topics
Posted Oct 26, 2011 17:37 UTC (Wed) by nix (subscriber, #2304)
The list of things autoconf and automake do for you in a consistent fashion is endless, and scons does next to none of them. Neither do projects which use scons, and some of those facilities (notably CFLAGS overriding and install-time prefix or DESTDIR support) are essential for any project to be easily packaged. The best you'll get with scons is a means of doing this which is inconsistent for every single package: not very useful.
(cmake used to be bad in this regard too, but is much better these days, though it still supports no analogue of the useful site-config files, so you need wrapper scripts to simulate that. waf appears to provide some of them, but in a profoundly non-extensible fashion, apparently under the belief that if you want to configure CFLAGS the same way across many packages, you'll be happy to hack the waf scripts for every one of them to ensure they're not overriding it. Wrong.)
Posted Oct 26, 2011 19:03 UTC (Wed) by cjwatson (subscriber, #7322)
Posted Oct 26, 2011 19:46 UTC (Wed) by tetromino (subscriber, #33846)
Posted Oct 27, 2011 0:19 UTC (Thu) by nix (subscriber, #2304)
Posted Oct 27, 2011 20:04 UTC (Thu) by Cyberax (✭ supporter ✭, #52523)
And installing scons is just as simple as installing one package. So what's a big deal with it?
Posted Oct 27, 2011 22:15 UTC (Thu) by nix (subscriber, #2304)
(Sure, the problem goes away if it's a Python package... but if it's a Python package, why the heck aren't they using distutils, which supports prefix and destdir and everything necessary already, with zero effort needed from the packager?)
Posted Oct 28, 2011 10:28 UTC (Fri) by Rudd-O (subscriber, #61155)
You don't have to learn anything new to use waf. If you know how to use configure, you know how to use waf.
Posted Oct 28, 2011 16:50 UTC (Fri) by nix (subscriber, #2304)
Posted Oct 31, 2011 9:57 UTC (Mon) by pkern (subscriber, #32883)
Also wasn't it the insane thing which actually shipped a compressed tarball with Python libs in sources that use it? (The "waf" binary alongside "configure".)
I know Debian had to patch a bunch of packages because waf was stupid and the result didn't build on hppa. You couldn't just regenerate the output because you needed the right version of waf to do that, i.e. the shipped one. So that needed patching. And you couldn't just apply the same patch neither because the build system is in a compressed blob.
Oh my, please spare us of this ridiculous thing.
Posted Oct 31, 2011 13:43 UTC (Mon) by Cyberax (✭ supporter ✭, #52523)
Waf can produce 'wafscript' which is basically a 'configure' file. That's not a preferred form for modifications, like you wouldn't want to modify 'configure' script directly (and not 'configure.ac'). Why waf won't build on hppa - I have no idea, it's a pure Python app.
Their stance regarding system-wide installation is curious, but not a problem.
Posted Oct 26, 2011 20:05 UTC (Wed) by pj (subscriber, #4506)
Posted Oct 26, 2011 21:44 UTC (Wed) by mathstuf (subscriber, #69389)
Personally, I'm more partial to CMake than other build systems for C/C++. Some languages have it better: Python has setuptools or distribute, and Haskell has cabal. These are built with the language in mind and work fine until other things are needed (e.g., asciidoc/xmlto for man pages, doxygen for documentation, and non-standard testing suites (especially when language bindings are concerned and you can't build/use the bound language without hacks)). Others languages still have it hard such as Java with maven or whatever the XML horror du jour is.
What I *really* want is a build system that generates Makefiles (or Ninja files, or Visual Studio projects, or XCode projects, or what have you) (CMake), defaults to out of source builds (CMake and whatever autotools magic glibc has that no one bothers to copy), has a declarative syntax (cabal), and has no need to ship generated files (CMake, raw Makefiles).
I have hacks for CMake to handle LaTeX builds (including double and triple+bibtex passes) with out-of-source builds, symlinking my dotfiles, generating cross-referenced doxygen, and more, but I think a build system that supports something more akin to make's generator rules (something like Prolog/Mercury maybe?) would be nicer to work with (CMake's escaping and argument parsing is less than ideal to manage with complicated things). Implicit know-how of supporting system copies and bundled libraries with automatic switches (which can be disabled if there are patches which make in-the-wild copies not work) would be wonderful as well. CMake's external_project_add gets close, but still has some rough edges (such as needing manual switches for system copies support).
Posted Oct 27, 2011 19:52 UTC (Thu) by pbonzini (subscriber, #60935)
mkdir build; cd build; ../configure && make
Posted Oct 27, 2011 20:11 UTC (Thu) by mathstuf (subscriber, #69389)
Out-of-source *build* is probably bad wording. More precisely, .gitignore should be empty and git status (and the equivalent in the other VCS's) should also be empty starting with no generated files.
Posted Oct 29, 2011 19:05 UTC (Sat) by fuhchee (subscriber, #40059)
The cure to that is to mean to commit autoconf/automake-generated files.
Posted Oct 31, 2011 20:46 UTC (Mon) by mathstuf (subscriber, #69389)
Which, IMO, is worse than having them in the source tree after their generation. The autoconf/automake *generated* files belong in the build directory.
Posted Oct 31, 2011 20:55 UTC (Mon) by fuhchee (subscriber, #40059)
If you say so. :-)
Posted Nov 1, 2011 14:30 UTC (Tue) by nix (subscriber, #2304)
It's certainly not true if your source directory is a release tarball (or other release medium). Autoconf et al should have been run for you by that point, and the result tested. That way end users don't need anything but a shell and ordinary build tools to run the build. (This is one area where cmake falls down: all the builders need a copy of it.)
Posted Nov 1, 2011 14:41 UTC (Tue) by fuhchee (subscriber, #40059)
In practice, if people are pragmatic, it's fine.
Developers can regenerate the files at will with any version that works.
In the case of version control branch merge problems, regenerate them again.
Posted Oct 28, 2011 18:51 UTC (Fri) by anatolik (subscriber, #73797)
I really like the idea of Tup build system http://github.com/gittup that stores graph into local sqlite database and reparses/rebuilds graph only when files are changed - this makes iterative development much more pleasant. Another cool feature is dependencies autodiscovering - under the hood it uses fuse (and fuse-like) library for that (this works on linux, macosx and windows). And the third feature that I like is "monitor" - inotify based daemon that updates graph of dependencies in background while you change files in your editor.
I made some experiments with my project (100K) and found that null build takes ~1.6 sec without monitor and 0.09 sec with monitor. Null build for my gmake based build system on the same project takes 42 secs (it parses makefiles files, builds graph of dependencies, scans files for modification, but does not run any commands).
Posted Oct 28, 2011 20:13 UTC (Fri) by mathstuf (subscriber, #69389)
It doesn't support -B or -n make flags though. The percentage output is nice. However, it doesn't seem to support out-of-source builds (I sort of hacked bootstrap.sh to do the bootstrap successfully, but there's no further support to get a full build working). The code base looks clean, so maybe I can get a patch and convince the maintainer to accept out-of-source as an option.
Posted Oct 28, 2011 20:38 UTC (Fri) by anatolik (subscriber, #73797)
> make -n
> it doesn't seem to support out-of-source builds
AFAIK the tup author in favor of adding it. It is better to contact the maillist as I am not sure about his plans.
Oh, yeah, I remembered 4th thing that I like in tup - "buffered output". Output from commands is always printed atomically. No more interlaced output! The interlaced output is especially annoying if you have an error in one of widely used header files - this makes error messages really difficult to read.
Posted Oct 28, 2011 21:00 UTC (Fri) by mathstuf (subscriber, #69389)
So if I update a system library or header, it will relink the relevant parts? That's usually the corner case I've run into.
> AFAIK the tup author in favor of adding it. It is better to contact the maillist as I am not sure about his plans.
> Oh, yeah, I remembered 4th thing that I like in tup - "buffered output". Output from commands is always printed atomically. No more interlaced output! The interlaced output is especially annoying if you have an error in one of widely used header files - this makes error messages really difficult to read.
I usually do `make -kj8` followed by `make` to do this, so this should help that. I rarely do parallel builds from vim however (<F8> is bound to "build", autodetecting CMake, make, pdflatex, cabal, rpmbuild, and a few others based on the current file) so interlaced output never really bothered me there.
Posted Oct 27, 2011 0:06 UTC (Thu) by mhelsley (subscriber, #11324)
The alternative is probably something vaguely like static analysis of the code. Static analysis is notoriously complicated and often produces a flood of false-positives though.
So my guess is we'll still have humans involved in dependency generation and maintenance for quite some time -- even with use of tools like strace :).
Posted Oct 27, 2011 2:28 UTC (Thu) by nlhepler (subscriber, #54047)
As for standardizing on a build system, I have mixed feelings about using automake. It's a PITA, even the presenters admit this. All the problems with regenerating the configure, autoconf-ing, incompatibilities between versions, etc make it an absolute pain to use. Extending something like waf or fabricate to perform all the tests that are needed (is libA around? what about libB?, etc), and to build a monolithic C function to grab platform-specific information seems like a much less painful approach. Also, fabricate is a single python file that can easily be included with your package -- not the best approach, but it could give something like a make.py a fallback if it's not available system-wide.
Posted Oct 27, 2011 20:27 UTC (Thu) by Cyberax (✭ supporter ✭, #52523)
And automake sucks at that, by blindingly propagating flags even where they shouldn't be propagated. To get the same effect in SCons you just need to add ENV=os.environ to Environment constructor.
Scons supports cross-compilation just fine, toolchain files are much easier to use than tons of --with-yet-another-fscking-lib switches that automake requires (and then fails nonetheless). And if have a complex build that requires building an executable tool as a part of the process... Well, scons and cmake are much better in that regard.
Autotools might be doing things consistently, but it's doing them consistently fugly.
Posted Oct 27, 2011 22:23 UTC (Thu) by nix (subscriber, #2304)
Automake gets it *just* right: the CFLAGS, LDFLAGS and so on are propagated correctly, supplemented with makefile-specific flags for those targets where it is necessary, and possibly occasionally overridden for those very very few targets where that is necessary (though this elicits a warning). This is done *by default*: there is no need for the makefile author to hack about with anything or remember to do anything (which nearly all will of course forget).
Automake doesn't take any --with- switches at all: are you thinking of Autoconf? Even *that* only requires them if autodetection fails, in which case you are installing things in nonstandard prefixes and should expect to have to do things by hand.
Oh, and it doesn't require building any kind of 'executable tool' at all, only a portable shell script. The entire *reason* why configure scripts are a shell script is because the distributor / builder needs nothing but a shell and core Unix shell tools: they do *not* need Autoconf or Automake. For scons you need a whole flipping Python installation -- fairly common these days, you think? I wish you were right: Python is by no means ubiquitous. The shell is. Until recently Autoconf's shell requirements were so lax that it actually allowed Solaris 2.6-and-below /bin/sh, which doesn't support shell functions: so we're talking a 1980s shell here!
Posted Oct 27, 2011 22:35 UTC (Thu) by Cyberax (✭ supporter ✭, #52523)
So create a proper configuration with these flags instead of hacking them in using command line or system environment. That way they'll be cleanly visible in the build file instead of being tucked-in in a some obscure driver script.
>Automake doesn't take any --with- switches at all: are you thinking of Autoconf? Even *that* only requires them if autodetection fails, in which case you are installing things in nonstandard prefixes and should expect to have to do things by hand.
Yes, sorry. Of course I meant autoconf, and you have to do a lot of --with if you try to do cross-compilation with even remotely non-standard libdirs.
>Oh, and it doesn't require building any kind of 'executable tool' at all, only a portable shell script. The entire *reason* why configure scripts are a shell script is because the distributor / builder needs nothing but a shell and core Unix shell tools: they do *not* need Autoconf or Automake.
Portable? They don't work with MSVC which is probably the most popular C++ compiler. I can hardly call them 'portable'. Even between Unix platforms it's a hit-and-miss - most Linux software requires tweaking to build correctly on HP-UX.
Posted Oct 28, 2011 16:57 UTC (Fri) by nix (subscriber, #2304)
I suspect that you don't understand what I'm driving at. A build system which requires distributors to hack every build script in order to make simple adjustments to flags and paths will never be popular among distributors, and packages that use it will forever be discriminated against and make packagers and builders despair when they meet them.
Portable? They don't work with MSVC which is probably the most popular C++ compiler.
So, sorry, portability to MSVC is rarely worth considering when looking at Unix build systems.
Even for such programs, msys provides a platform specifically intended to allow configure scripts to run. (It's not heavyweight.)
Posted Oct 30, 2011 23:09 UTC (Sun) by cortana (subscriber, #24596)
Posted Nov 23, 2011 22:33 UTC (Wed) by oak (subscriber, #2786)
I think python-minimal is quite a bit smaller than POSIX base system (shell, perl, awk, GNU core utils etc). I don't understand why Autools generate scripts that are supposed to work with some foobar non-POSIX shells that don't support functions (which makes configure scripts pretty much unreadable), but then requires Perl etc.
Posted Nov 25, 2011 18:46 UTC (Fri) by nix (subscriber, #2304)
Autools generate scripts that are supposed to work with some foobar non-POSIX shells that don't support functions
Posted Oct 28, 2011 10:27 UTC (Fri) by Rudd-O (subscriber, #61155)
the only problem with waf is that the documentation is arcane and the code even more so.
but those are problems with the autocrap too, so nothing new there.
Posted Oct 28, 2011 11:33 UTC (Fri) by josh (subscriber, #17465)
Posted Oct 30, 2011 17:20 UTC (Sun) by foom (subscriber, #14868)
I'm sure a similar rule could be made for waf-using packages, with similarly problematic results.
Posted Oct 30, 2011 18:12 UTC (Sun) by josh (subscriber, #17465)
Posted Oct 30, 2011 21:31 UTC (Sun) by foom (subscriber, #14868)
Yet, it's still necessary for Debian to include automake 1.4, 1.7, 1.9, 1.10, and 1.11, and autoconf 2.13, 2.59, 2.64, and 2.67.
If it were that easy to upgrade, I'd expect that everything would've been converted to at least automake 1.10 by now (which came out in 2006).
Posted Oct 30, 2011 23:11 UTC (Sun) by cortana (subscriber, #24596)
Posted Nov 1, 2011 2:42 UTC (Tue) by foom (subscriber, #14868)
Yes. Some reference material.
Posted Nov 4, 2011 0:36 UTC (Fri) by geofft (subscriber, #59789)
Posted Nov 7, 2011 0:23 UTC (Mon) by foom (subscriber, #14868)
Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds