automake vs. GNU make [LWN.net]

automake vs. GNU make

Posted Feb 7, 2008 0:51 UTC (Thu) by stevenj (guest, #421) [Link]

Regarding recursive make considered harmful, his solution is to use one gigantic Makefile. Why not put all of your source code in one gigantic .c file while you're at it, and then you can skip "make" entirely? In any case, to each his own, but many many projects have voted with their feet on this one.

I've used libtool for years, and it works (on essentially every Unix-like system, and even on Windows with MinGW); implying it doesn't is baseless FUD. Yes, it has had bugs from time to time. Yes, it is slow as hell because it launches a shell process for each compile. Yes, it is hard to use properly if you don't use automake too. But what is the alternative if you want to build shared libraries portably?

automake vs. GNU make

Posted Feb 7, 2008 1:05 UTC (Thu) by vmole (guest, #111) [Link]

No, his solution is NOT put everything in one giant make file. You use "include" (yes, that makes it GNU make specific. See previous post). I've done this. It's a lot easier to maintain than recursive make files, because most of the sub-directory makefile fragments are just the list of source files.

My experience is that libtool does not work reliably on AIX, and it never gets fixed (or it gets fixed, then broken again). I found that simply writing a small (<10 lines) shell script for each implementation was far more reliable and easier to debug. Admittedly, this was 2001-2006. Maybe it's better now. But a 8500 line shellscript is inherently fragile.

automake vs. GNU make

Posted Feb 7, 2008 1:51 UTC (Thu) by stevenj (guest, #421) [Link]

With AIX there are difficulties with shared libraries because of the limitations of that operating system: it doesn't (or didn't, at least) allow inter-library dependencies between shared libraries. MinGW (Windows) is the same way. If you want to use libtool on these systems you have to pass the "-no-undefined" flag to tell it you have no shared-library dependencies, which libtool cannot assume for you in general.

Of course, this is documented in the manual.

Again, you're blaming the tool for the complexity of the underlying problem.

(And your suggestion of writing a script for every system that you want to support shared libraries on is basically saying that every programmer should re-implement libtool themselves, probably badly because they aren't aware of all the platform variations in shared-library semantics/syntax or even a large subset thereof.)

automake vs. GNU make

Posted Feb 7, 2008 3:59 UTC (Thu) by jamesh (subscriber, #1159) [Link]

I think you've got things backwards: the -no-undefined flag doesn't really make sense _unless_
shared library dependencies exist.

With the -no-undefined flag set, your library must directly link to all the libraries whose
symbols you used (leaving no "undefined" symbols).

Even on platforms where -no-undefined is not required, it is usually worth using since the
linker can see which library each referenced symbol comes from, making the job of runtime
linking easier.

automake vs. GNU make

Posted Feb 7, 2008 5:02 UTC (Thu) by stevenj (guest, #421) [Link]

Even if there are no library inter-dependencies in your library, you must specify -no-undefined or libtool will refuse to build it on AIX or MinGW, since libtool is apparently unable to detect automatically whether there are unresolved symbols at link time. But you're right that the flag can also be used if you specify library dependencies with -l flags explicitly at link time.

I've also heard that this is useful for prelinking as well, as you mention.

(Apparently, -no-undefined used to be the default, which seems sensible to me, but supposedly they got too many complaints.)

automake vs. GNU make

Posted Feb 7, 2008 16:59 UTC (Thu) by vmole (guest, #111) [Link]

No, my point is that problem is so complex that pretending you can hide it with libtool is misleading. The developers have know and care that AIX and MinGW are different, and need special variations, or the end-user, trying to build the package, has to track down the docs and figure out where to add the magic option.

You're right: I'm pretty ignorant about the internal details of autoconf et. al. But as an end user, who has 20+ years of experience building software on a wide variety of platforms, I've found that I spent a lot more time fighting autoconf/automake/libtool problems than I did for packages that asked me to uncomment the appropriate variable settings in the Makefile. Why? Because if the appropriate variable was available in the Makefile, then someone had actually, really, already built it on that platform. If not, then it was pretty easy to see what needed to be done.

I should clarify: by "autoconf problems" I don't necessarily mean problems with the autoconf system itself. It does what it's told. The problem is software developers who believe that using autoconf et. al. magically solves all their portability issues, and all they need to do is copy a few scripts/templates from some other project. This is the "autoconf culture" problem I mentioned way back in my first comment. It may not be what the autoconf developers intended or wanted, but it sure is what has happened.

That's all for me.

automake vs. GNU make

Posted Feb 7, 2008 21:36 UTC (Thu) by nix (subscriber, #2304) [Link]

No, my point is that problem is so complex that pretending you can hide it with libtool is misleading. The developers have know and care that AIX and MinGW are different, and need special variations, or the end-user, trying to build the package, has to track down the docs and figure out where to add the magic option.

Actually, if you use libtool and libltdl you *don't* need to care that Linux, Solaris, MacOS X, AIX, HP-UX 10 and Windows all use quite different methods to build shared libraries with different names and semantics; at least not unless you're trying to go beyond what static libraries allow and do symbol versioning or something like that.

The end-user need have no clue at all.

libtool has its problems (astonishing sloth being one of them: why anyone complains that configure takes too long to run for very small projects, when libtool slows down building drastically regardless of project size, I have no idea) but you don't seem to know what those problems are. At least you're focusing on men made of straw when there are perfectly visible giants of problems looming a few feet away.

automake vs. GNU make

Posted Feb 7, 2008 21:55 UTC (Thu) by vmole (guest, #111) [Link]

Yes, that's the promise. But in my experience (and in many others), as the end user of several libtool using projects, it's not reality. Is it that all these projects are setting things up wrong? Possibly. I don't know, and I don't really care, because if so, it's apparently as hard to get right as using the actual OS tools would be, which would at least be debuggable by normal human beings.

And I'm well aware and have suffered extensively at libtool's sloth. And its insanely long invocation lines, which make make logs painful to read and hide errors.

automake vs. GNU make

Posted Feb 7, 2008 23:18 UTC (Thu) by nix (subscriber, #2304) [Link]

Your reality differs from my reality. I've had some cosmetic problems on 
Solaris 2.4 and 2.5.1 (we're talking prehistoric here) and I needed to 
install GNU sed on HP-UX and AIX, but other than that, nothing really.

(Mind you, the need for GNU sed *was* unacceptable --- it was also a bug.)

automake vs. GNU make

Posted Feb 7, 2008 23:25 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

> [libtool is] apparently as hard to get right as using the actual OS
> tools would be, which would at least be debuggable by normal human
> beings.

"at least be debuggable" - very well put, I completely agree.
Why does automake actually need libtool at all ? I mean it generates the 
makefile code, it could as well just generate the code for calling the 
actual OS tools directly in the makefiles. This would remove this one 
layer of indirection.

Alex

automake vs. GNU make

Posted Feb 8, 2008 0:44 UTC (Fri) by nix (subscriber, #2304) [Link]

automake can't generate the commands libtool executes because automake 
runs on the distributor's machine, not the builder's, and doesn't have a 
clue what sort of system the build will take place on.

automake vs. GNU make

Posted Feb 8, 2008 7:20 UTC (Fri) by aleXXX (subscriber, #2742) [Link]

Ah, yes, indeed.
(I didn't work with autotools in the last years).

Then, couldn't the configure script handle that ? It runs on the build 
machine.

Alex

automake vs. GNU make

Posted Feb 8, 2008 21:28 UTC (Fri) by nix (subscriber, #2304) [Link]

Yes, it could: but in libtool 1.5 it doesn't :(

I never said libtool didn't suck. It's just better than anything else 
around right now for the job it does, and it has a *lot* of hard-won 
knowledge of shared library wierdness on manifold systems encoded into it 
(as autoconf does of other cross-system variation).

automake vs. GNU make

Posted Feb 7, 2008 23:54 UTC (Thu) by stevenj (guest, #421) [Link]

You didn't read the manual, and as a result it failed on AIX because you asked libtool to support semantics unavailable on that platform, and as a result you loudly complain here that libtool is broken and buggy. This is not a convincing critique of libtool.

libtool solves a big part of the problem: its manual specifies the lowest common denominator of shared library semantics, tells you how to indicate whether you obey those semantics, and builds the resulting library on every system that supports the semantics you request. The fact that you still have to know that there might be some differences between systems, and that you might have to read the manual to learn how to deal with these differences, is not a reason to throw it out and rewrite everything yourself from scratch (the only other "solution" you have suggested). No matter what portability tool you use, developers will still need to know something about the differences between platforms.

automake vs. GNU make

Posted Feb 8, 2008 0:23 UTC (Fri) by vmole (guest, #111) [Link]

You didn't read the manual, and as a result it failed on AIX because you asked libtool to support semantics unavailable on that platform

*I* didn't ask libtool to do anything. I'm the end-user. I'm not supposed to have know about libtool, or the variations in shared library implementations. Right? Isn't that the whole friggin point?

And if your response is "But there's too many differences, libtool can't hide everything", then we are in 100% agreement. Where we differ is whether or not it's worth the attempt, and whether libtool is the correct direction. So be it. But if you continue to ignore those of us who have problems, and blame the users, well, it will never get better.

automake vs. GNU make

Posted Feb 8, 2008 0:42 UTC (Fri) by stevenj (guest, #421) [Link]

You haven't suggested any way for things to get better, you've just flamed a tool because you've encountered a couple of buggy packages that misused it, and it didn't correct for all the deficiencies of the developers. Then you tried to debug the problem yourself and failed because you didn't bother to read the fine manual.

Your suggestions, as far as I can tell, have been either for every developer to reinvent the wheel by rolling their own platform-dependent scripts, or for the developers to push the whole problem onto the end-users by giving them a raw Makefile and telling them to fix the compiler options themselves. Neither of these seems like an improvement.

automake vs. GNU make

Posted Feb 8, 2008 5:42 UTC (Fri) by lovelace (guest, #278) [Link]

Again, you're blaming the tool for the complexity of the underlying problem.

But, isn't this tool supposed to take care of the underlying complexity? So, haven't you completely invalidated your argument for it? It's supposed to make the task of creating libraries similar across dissimilar systems yet by your own description it does not.

I don't remember all the details now since it's been a while, but back when we were trying to port KDE 3.x to native Qt on the Mac libtool was a constant source of problems.

automake vs. GNU make

Posted Feb 8, 2008 7:19 UTC (Fri) by aleXXX (subscriber, #2742) [Link]

libtool on the Mac (a native binary for dealing with libraries on the 
Mac) is something different than the autotools libtool (portable shell 
script which should deal with shared libs on all systems).

Alex

automake vs. GNU make

Posted Feb 8, 2008 16:05 UTC (Fri) by lovelace (guest, #278) [Link]

Hi Alex!

Yep, I'm aware of that, but that's not what I was referring to.  I was referring to the
autotools libtool.  Unfortunately, like I said, that was about 5 years ago and I cannot
remember the specifics.  So, rather than make more unsubstantiated accusations, I'll just
leave it at that.  I will mention, though, that that episode in particular cemented my general
dislike of the autotools libtool that still stands to this day.

automake vs. GNU make

Posted Feb 8, 2008 21:26 UTC (Fri) by nix (subscriber, #2304) [Link]

It reduces the complexity, but fundamentally these systems have different 
*runtime* semantics, which can't be entirely hidden. For simple uses 
(DT_NEEDED-style simple symbol lookup of libraries with no undefined 
symbols, and dlopen()/dlsym()/dlclose()-style dynamic loading), libtool 
does a good job. Anything more complex will hit trouble on one system or 
another.

automake vs. GNU make

Posted Feb 8, 2008 21:46 UTC (Fri) by lovelace (guest, #278) [Link]

Ah, now I'm remembering more about what the problems where.

1. Shared libraries and modules are the same on Linux (and lots of other Unices) but are
different on the Mac.  Libtool had a difficult time understanding this.
2. Until fairly recently (Tiger, iirc) shared libraries couldn't be easily dlopened on the
mac, only modules could.

Since KDE makes extensive use of dlopen-ing modules to accomplish things this made things
quite tricky and libtool wasn't really that much help.

So, yeah, quite a different runtime system.  Newer versions of OS X have gotten quite a bit
better on the dlopen-ing front, but they are still fundamentally different.  And, I wouldn't
even try to use libtool to create OS X frameworks....

automake vs. GNU make

Posted Feb 8, 2008 23:18 UTC (Fri) by nix (subscriber, #2304) [Link]

I'd expect MacOS X support for modules to have been OK since

2003-03-20  Peter O'Gorman  <peter@pogma.com>

        * ltmain.in: Always use $echo not echo for consistency.
        Changes for darwin building. Warn if linking against libs linked
        with -module. Use module_cmds if available and building a module,
        move convenience double lib check,

What else is wrong?

And, yes, your point 2 is hard for libltdl to overcome: if you build the 
library as a lib, not a module, you'd have been stuck whatever libtool 
did.

automake vs. GNU make

Posted Feb 7, 2008 1:53 UTC (Thu) by sward (subscriber, #6416) [Link]

I've also used his include-based Makefile structure on projects that were originally written
using recursive make.  The Makefile's got smaller, simpler, faster, and more reliable.
Hierarchy still takes *some* thought, but it really wasn't difficult.

automake vs. GNU make

Posted Feb 7, 2008 3:04 UTC (Thu) by stevenj (guest, #421) [Link]

I have no doubt about that, if you compare hand-written recursive make with GNU make+include. I'm more doubtful if you compare to automake input files, which are pretty compact (and support a portable include directive if you have definitions you want to share between directories etc.).

Of course, it's possible that Tom Tromey or someone may come up with some very clever include file that we can all use in our GNU Makefiles that gives us 95% of automake's functionality in a cleaner and faster way (at the expense of requiring GNU make to build, which in this day and age isn't too much of a burden). And then we'll just use 'include' for subdirectories. But until such a thing gets released, the alternative is to write the include files yourself, support all of the standard targets yourself, be careful to use only portable sh constructs yourself, and so on; in the vast majority of cases, automake seems a simpler and more robust solution at present.

A technical question about using include instead of recursive make via automake (recursive make by hand is crazy): how do you separate the namespaces for the subdirectories? For example, if you have a source file with the same name in two subdirectories (e.g. main.c), do you have to jump through hoops to prevent it from getting confused? Or what if you want a 'make clean' target in two subdirectories? I didn't see any obvious way to handle such separation cleanly in the GNU make manual.

automake vs. GNU make

Posted Feb 7, 2008 4:25 UTC (Thu) by masuel (guest, #28661) [Link]

>For example, if you have a source file with the same name in two 
> subdirectories (e.g. main.c), do you have to jump through hoops to 
> prevent it from getting confused?

The way I do it is always reference the tree from the build root.

so a/main.c is never looks like b/main.c

and yes I can start a build from the subdirs etc.

non-recursive is the way to go. 

Just require gnu make don't try supporting version 8.79 (rh9) clients just 
make people upgrade....

automake vs. GNU make

Posted Feb 7, 2008 4:41 UTC (Thu) by sward (subscriber, #6416) [Link]

Files (source or derived) in separate directories are *always* in separate namespaces, in
Miller's approach.  The 'make' is run once, in the root directory, and all sources and targets
are described from that point of view.  So you don't build 'main.c', you build 'foo/main.c'.
You use target-specific overrides if you need to customize things, e.g.:

   # in foo/include.mk:
   $(FOO_OBJ): CPPFLAGS += -Ifoo

Now, if you wanted a selective clean (for one subdir), I agree there isn't a good way to do
this in the include-based structure (other than calling it something else, e.g. 'foo/clean').
Of course, you could also include a local Makefile in the subdir just to make things more
familiar:

   # in foo/Makefile:
   clean: 
       cd ..; $(MAKE) foo/clean

Did that help, or would a more complete example be clearer?

automake vs. GNU make

Posted Feb 7, 2008 4:55 UTC (Thu) by vmole (guest, #111) [Link]

Even neater: Stick the following as GNUMakefile in each of your subdirs:

CURMOD:=$(shell pwd | sed -e 's,.*/src/\(.*\),\1,')
all :
        @cd $${PWD%$(CURMOD)} && $(MAKE) --no-print-directory $(CURMOD)

% :: FORCE
        @cd $${PWD%$(CURMOD)} && $(MAKE) --no-print-directory $(CURMOD)/$@

FORCE:

which automatically transfers any make command up to the top level and re-invokes with the same arguments. (You'll need to tweak the sed command to match your layout.)

automake vs. GNU make

Posted Feb 7, 2008 5:18 UTC (Thu) by stevenj (guest, #421) [Link]

That's clear enough. Yes, using foo/main.c and bar/main.c in the Makefile rules will clearly work (unless you have one of the rare compilers that doesn't support -c -o, although you can hack around that with enough effort I suppose).

It seems a bit ugly to me, though, that your subdirectory Makefiles (to be included) must be written differently depending on the name of the directory they are in. The analogous thing in a programming language would be, if you had a subroutine foo(), to require that all local variables be named with a "foo_" prefix.

automake vs. GNU make

Posted Feb 7, 2008 5:54 UTC (Thu) by sward (subscriber, #6416) [Link]

That isn't really necessary for variable names (that you don't need to reference later), only
the source and targets need to be qualified.  A fairly typical example:

# Define sources for this component:
MY_SRC = $(wildcard foo/*.c)
MY_OBJ = $(MY_SRC:.c=.o)
MY_TGT = foo/libfoo.a

# Additional flags and dependencies for building this component:
$(MY_OBJ): CPPFLAGS += -I foo
$(MY_TGT): $(MY_OBJ)

# Add these to the master Makefile's lists:
SOURCE += $(MY_SRC)
TARGET += $(MY_TGT)
CLEAN  += $(MY_OBJ)

The next component can reuse MY_SRC etc.  You wouldn't even need these variables, if you had
no need for target-specific variables or dependencies.  (But of course, you usually do).  If
you wanted to keep a reference to 'foo/libfoo.a', for use in other Makefiles, you might
replace MY_TGT with LIBFOO.

In short, you only need unique names for global variables; local ones can be overwritten.

You do need the component path ('foo' in this example) in a few spots, but it is trivial to
change that if the directory structure changes.

automake vs. GNU make

Posted Feb 7, 2008 7:41 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

> But what is the alternative if you want to build shared libraries
> portably?

I don't know about Scons, but you can use CMake, and it will just work:

add_library(foo SHARED foo.c bar.c)

This will give you depending what you need:
-UNIX Makefiles (not quite sure if it requires GNU make, I don't think 
so)
-MS nmake makefiles
-Borland Makefiles
-Watcom Makefiles
-cygwin Makefiles
-XCode projects
-Visual Studio >= 6 projects.

I wondered for years why suddenly the comopiler can't be called directly 
anymore, to find out that it's just libtool and you can use the compiler 
very well directly (which removes one layer of indirection -> makes it 
easier to understand).

Alex

automake vs. GNU make

Posted Feb 7, 2008 17:11 UTC (Thu) by stevenj (guest, #421) [Link]

UNIX Makefiles (not quite sure if it requires GNU make, I don't think so)

You are sorely mistaken if you think that there is one way to build shared libraries that works on all flavors of Unix. If you only care about Windows, GNU/Linux and similar systems with gcc, and MacOS X, then yes, generating three makefiles will work, but that's not solving anything like the whole problem.

automake vs. GNU make

Posted Feb 7, 2008 23:17 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

CMake knows how to build shared libs on all supported platforms (which 
support shared libs) with all supported toolchains (GNU, IBM, Sun, 
Borland, MS, Portland, Intel, HP and more). For several 
platforms/toolchains this is tested every night (unfortunately there 
aren't nightly tests for all supported combinations).
http://www.cmake.org/Testing/Dashboard/20080206-0100-Nigh...

Alex

automake vs. GNU make

Posted Feb 8, 2008 5:50 UTC (Fri) by lovelace (guest, #278) [Link]

The makefiles CMake generates are not portable and can only be used on the system that they're generated on. If you move to a different system, you use CMake to generate makefiles for that system. That's how they can reliably create shared libraries on multiple systems using makefiles.

automake vs. GNU make

Posted Feb 7, 2008 7:15 UTC (Thu) by madscientist (subscriber, #16861) [Link]

libtool isn't so bad except for one thing, which makes it unusable.  No support for
cross-compilation.  It's so utterly lame that every part of the GNU build environment
(binutils, gcc, autoconf, automake) is very cognizant of cross-compilation and works very,
very hard to make setting it up and using it as simple as possible... except libtool.

Whenever I see a package come through that uses libtool I absolutely cringe because I know it
will be a huge PITA.

automake vs. GNU make

Posted Feb 7, 2008 17:17 UTC (Thu) by stevenj (guest, #421) [Link]

That's totally untrue; automake+libtool supports cross-compiling just fine (note that you can hardly use automake for libraries without using libtool, so saying that automake supports cross-compiling but libtool doesn't is a little strange). I use libtool for cross-compiling all the time. (e.g. on one of my projects we cross-compile shared libraries for Windows from GNU/Linux using mingw).

automake vs. GNU make

Posted Feb 7, 2008 22:57 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

I don't know much about the cross compiling support of autotools.
Do autotools support cross compiling for systems e.g. with 8 bit 
microcontrollers, which maybe don't have an operating system at all ?

Alex

automake vs. GNU make

Posted Feb 7, 2008 23:23 UTC (Thu) by nix (subscriber, #2304) [Link]

8-bit, I'm not sure. Things like arm-elf (without an OS), sure, and has 
for next to forever.

automake vs. GNU make

Posted Feb 11, 2008 1:43 UTC (Mon) by GreyWizard (subscriber, #1026) [Link]

Autotools support cross compilation in the abstract: to the extent that they work for one
cross compiling target they should work for any that the underlying compiler can support.  As
for that, GCC does support at least some 8-bit microcontrollers, as attested here:

    http://gcc.gnu.org/ml/gcc/2002-08/msg01809.html

automake vs. GNU make

Posted Feb 11, 2008 8:37 UTC (Mon) by aleXXX (subscriber, #2742) [Link]

What about compilers != gcc, e.g. sdcc ?

Alex

automake vs. GNU make

Posted Feb 11, 2008 20:01 UTC (Mon) by nix (subscriber, #2304) [Link]

Other compilers are generally supported. autoconf supports anything which 
runs as $CC and supports -g, -o, and -c with the customary meanings; 
automake supports anything with the caveat that dependency analysis may be 
inaccurate without compiler support (said support being a few lines of 
shell script in the depcomp script: if you need it for your compiler, 
please submit a patch!)

libtool's support is necessarily compiler-by-compiler and 
platform-by-platform, and so cannot cover everything (its entire purpose 
being to smooth out variation in the way different compilers and platforms 
create shared objects): nonetheless, on Linux, 1.5.26 supports GCC, KCC, 
Intel C++, the Portland Group's compiler, Compaq C++, and even Sun C, 
although who'd be using that on a Linux box I have no clue. (For a 
complete list of compiler * platform combinations, look in libtool.m4.)

(libtool obviously doesn't support sdcc, since as far as I can tell sdcc 
can't generate PIC code at all :) )