Kamp: A Generation Lost in the Bazaar [LWN.net]

Kamp: A Generation Lost in the Bazaar

Posted Aug 20, 2012 23:06 UTC (Mon) by alankila (subscriber, #47141)
In reply to: Kamp: A Generation Lost in the Bazaar by pboddie
Parent article: Kamp: A Generation Lost in the Bazaar

It's also possible to use, say, pkg-config. Which can spit out cflags and ldflags and compilation flags without any autotools voodoo. There is no need to hardcode library paths, that is simply a strawman.

Autotools brings a lot of complexity, and I would personally be far less opposed to it, if:

1. there were less code generation layers and fewer programming languages and fewer programs involved. Code-generation is imho amateur's tool. Very few systems that use this method stay maintainable, and it generates pain at every level when you hunt for the cause of error message in generated files that get increasingly bewildering the further down in the autogeneration chain the file is.

2. (related to 1.) the results were prettier. A hand-crafted makefile for a project can be a kilo or two. Autotools spits 50 kB behemots in every directory. Hand-rolled Makefiles are actually really pretty in comparison.

3. when it doesn't generate something right, there should be some way to reset the project tree to start over that isn't "rm -rf and fetch again from version control". There are so many caches and vague directories involved, and if you don't manage to hit every one of them, it won't generate things right and this makes everything related to fixing autotools builds horrible. In practice I've found the rm -rf solution to work best.

4. the build system is really unstable. It seems that every new version breaks a build that used to work before.

So yeah, before I start to rage incoherently, I'll just stop here.

(Log in to post comments)

Kamp: A Generation Lost in the Bazaar

Posted Aug 21, 2012 7:07 UTC (Tue) by cmccabe (guest, #60281) [Link]

I've done a few conversions to CMake, and none of them took longer than a weekend. The main issues were administrative (getting cmake installed on the build servers, making sure the continuous build continued to work.) The maintenance overhead of autotools is very, very, high, and you'll get back those two days and more in a few weeks. And you'll have a build system that you understand and have confidence in, rather than a black box.

I don't know much about QNX. I believe CMake supports it in some form, but they had trouble finding anyone who actually used QNX to do maintenance.

Kamp: A Generation Lost in the Bazaar

Posted Aug 22, 2012 8:14 UTC (Wed) by drag (subscriber, #31333) [Link]

> I don't know much about QNX.

Probably not much call for cross compiling suites for the software on gas pumps.

Kamp: A Generation Lost in the Bazaar

Posted Aug 23, 2012 20:44 UTC (Thu) by hummassa (subscriber, #307) [Link]

But some of us have to... (and voting machines, and ATMs...)

Kamp: A Generation Lost in the Bazaar

Posted Aug 21, 2012 10:18 UTC (Tue) by pboddie (subscriber, #50784) [Link]

It's also possible to use, say, pkg-config. Which can spit out cflags and ldflags and compilation flags without any autotools voodoo. There is no need to hardcode library paths, that is simply a strawman.

I wasn't defending autotools as such, but pointing out that the alternative in the wider world, beyond the group that the author berates and believes is responsible for the downfall of civilisation, is not another configuration tool but in fact nothing at all.

Of course there is no need to hard-code library paths, and of course you can use other tools instead, but the hard reality is that anything is a step up from the default, which is not to address deployment issues whatsoever. To say that such an observation is a "strawman" is laughable: I never claimed that there was a "need" for any such practice.

The fact is that the set of all people who can write code and the set of all people who know about software engineering practices to an acceptable degree do not fully intersect, and where they do intersect, people often choose to take an established tool instead of using another for a variety of reasons instead of making a choice purely on technical merit.

Kamp: A Generation Lost in the Bazaar

Posted Aug 22, 2012 0:16 UTC (Wed) by alankila (subscriber, #47141) [Link]

Okay. I believe your criticism of my criticism is fair. I withdraw it -- I misunderstood. I simply found the juxtaposition you made so gross that I objected instinctively.

If I am reading this communication correctly, you are stating that autotools is in fact *the only* option "in the wider world". If that is the case, I imagine it might be time to narrow some of that wider world so that the design could still be sane while working for almost everybody...

Kamp: A Generation Lost in the Bazaar

Posted Aug 22, 2012 13:51 UTC (Wed) by pboddie (subscriber, #50784) [Link]

I was just saying that the author thinks that people should just move up from autotools and use something better or write something cleaner, when in fact a lot of software developers aren't even aware of autotools or other ways of solving the problems of configuration and deployment. Indeed, they might not even be aware that those problems exist. For those developers "even" autotools is better than what they're doing.

In essence, the author is enjoying a luxury problem: he can complain about people who know what he is talking about while overlooking the hordes of software developers who hard-code everything, fire up Visual Studio, and pass round the resulting binary. It's like someone at a gentleman's club complaining about the manners of the doorman when just along the street people are swearing and fighting each other.

Kamp: A Generation Lost in the Bazaar

Posted Aug 21, 2012 23:43 UTC (Tue) by alankila (subscriber, #47141) [Link]

I said nothing about cross-compiling. You're somehow combining unrelated statements at different contexts. I've previously said that I'm pleased that Android is largely innoculated against autotools because it is so bad at cross-compiling, though. Of course, I was hastily corrected and am now aware that large part of the autotools macros do not actually need to run the resulting programs, but can do their work already through inspecting output of cpp (or something).

As to the quality of autotools and its approach to code generation, I personally believe it is a prime example of code generation done wrong. Regardless, I agree that code generation IS an essential tool for some other contexts (e.g. I have nothing against gcc transforming C to assembly), but when you write configure.ac to generate configure.in to generate configure to generate Makefile.in to generate Makefile to generate the final product ... At some point you have used too much of a good thing, damnit. And I think my objection is not really about the many layers involved, but about the quality of the implementation at each layer, which imho sucks. Hard.

"make distclean". I will try to remember that if it ever happens that I have to deal with autotools again.

Kamp: A Generation Lost in the Bazaar

Posted Aug 22, 2012 0:55 UTC (Wed) by cesarb (subscriber, #6266) [Link]

> but when you write configure.ac to generate configure.in to generate configure to generate Makefile.in to generate Makefile to generate the final product ...

AFAIK, configure.ac is just the modern name for configure.in. You will not have both at the same time, and both are used to generate configure.

Usually, you do not use configure to generate Makefile.in. Instead, you use automake to generate Makefile.in from Makefile.am, and configure to generate Makefile from Makefile.in.

So, what you have is that you write configure.ac to generate configure, Makefile.am to generate Makefile.in, and configure and Makefile.in together generate Makefile. But that is simplifying things, there is also config.h.in to generate config.h together with configure, and some stamp.h.in/stamp-h thing.

But wait, there is more! I forgot aclocal. So, you write configure.ac to generate aclocal.m4 to generate configure. And then there is libtool. Since this post is already too long, let me just link to a page of its manual with an easy-to-understand ASCII art graphic: https://www.gnu.org/software/libtool/manual/html_node/Int... (there is one for autoconf too, see https://www.gnu.org/savannah-checkouts/gnu/autoconf/manua...).

autotools is fun!

Kamp: A Generation Lost in the Bazaar

Posted Aug 27, 2012 11:35 UTC (Mon) by pboddie (subscriber, #50784) [Link]

I almost feel I could get some quick technical support here. People reading my other comments may get the impression that I really like autotools, but I spent the weekend updating OpenWrt packages that rely on the underlying build systems of projects that are mostly autotools systems, and there can be some weird things going on.

First of all, a lot of projects use automake which doesn't really lend itself to concise, easily diagnosable scripts - there's plenty of "where does the Makefile get *that* from?" and "do I really need a few hundred lines of settings for hello_world.c?" - and having written .in files by hand a while ago, I do get the feeling that people might just be taking something that works for someone else and stuffing it into a situation where it doesn't necessarily make sense. Here, I disagree with the author of the article: the "bazaar" may make such arguably inappropriate re-use easier since you can just search the Internet for something that works, but it isn't a precondition of "bazaar" development that you just copy stuff around.

Cross-compilation is always going to stress build systems, and I found cases of things wanting to run recently built or installed things to configure themselves (Gtk+, I think) as well as a particularly bizarre interaction with pkg-config producing build-time "staging" paths instead of "target" paths. I'm pretty sure that OpenWrt does some magic to make pkg-config do the right thing, but then some other inscrutable thing seems to be overriding it.

What I would probably conclude from much of this is that less is, more often than not, more: a more economically stated set of configuration resources would probably produce fewer surprises, especially outside the narrow mode of operation of the upstream developers.

Kamp: A Generation Lost in the Bazaar

Posted Aug 22, 2012 13:32 UTC (Wed) by nix (subscriber, #2304) [Link]

The code generation I was really alluding to was not GCC's C->assembly transformations, but the various tools that run during a GCC build to turn machine description files into (often immense and ugly) C files implementing some aspect of those md files in the finished compiler. You could interpret the md files at runtime to implement the same behaviour, but it would be intolerably slow.

As for autotools' use of code generation, that has one raison d'etre: to ensure that a configure script has no dependencies other than on core shell-script tools. In this it is quite different from cmake and most other build tools (you can't build a cmake project without cmake installed). This property is probably an essential one for anything involved in the bootstrapping core (GNU Make has a similar trick, with a shell script that will build an initial make for you on a system that has none). This property is perhaps less important these days, when everyone has a Linux box at hand, but in the old days of proprietary Unix when even compilers were hard to come by and barely worked this property was essential.

As for 'make distclean', this is not an autotools thing but rather a make target defined in the GNU coding standards, and thus implemented even by GNU projects that do not use Autoconf at all. Automake happens to generate this target because part of its purpose is to make it easier to conform to the GNU coding standards by generating most of its mandated targets automatically.