automake vs. GNU make [LWN.net]

automake vs. GNU make

Posted Feb 7, 2008 1:51 UTC (Thu) by stevenj (subscriber, #421)
In reply to: automake vs. GNU make by vmole
Parent article: LCA: Disintermediating distributions

With AIX there are difficulties with shared libraries because of the limitations of that operating system: it doesn't (or didn't, at least) allow inter-library dependencies between shared libraries. MinGW (Windows) is the same way. If you want to use libtool on these systems you have to pass the "-no-undefined" flag to tell it you have no shared-library dependencies, which libtool cannot assume for you in general.

Of course, this is documented in the manual.

Again, you're blaming the tool for the complexity of the underlying problem.

(And your suggestion of writing a script for every system that you want to support shared libraries on is basically saying that every programmer should re-implement libtool themselves, probably badly because they aren't aware of all the platform variations in shared-library semantics/syntax or even a large subset thereof.)

(Log in to post comments)

automake vs. GNU make

Posted Feb 7, 2008 3:59 UTC (Thu) by jamesh (subscriber, #1159) [Link]

I think you've got things backwards: the -no-undefined flag doesn't really make sense _unless_
shared library dependencies exist.

With the -no-undefined flag set, your library must directly link to all the libraries whose
symbols you used (leaving no "undefined" symbols).

Even on platforms where -no-undefined is not required, it is usually worth using since the
linker can see which library each referenced symbol comes from, making the job of runtime
linking easier.

automake vs. GNU make

Posted Feb 7, 2008 5:02 UTC (Thu) by stevenj (subscriber, #421) [Link]

Even if there are no library inter-dependencies in your library, you must specify -no-undefined or libtool will refuse to build it on AIX or MinGW, since libtool is apparently unable to detect automatically whether there are unresolved symbols at link time. But you're right that the flag can also be used if you specify library dependencies with -l flags explicitly at link time.

I've also heard that this is useful for prelinking as well, as you mention.

(Apparently, -no-undefined used to be the default, which seems sensible to me, but supposedly they got too many complaints.)

automake vs. GNU make

Posted Feb 7, 2008 16:59 UTC (Thu) by vmole (subscriber, #111) [Link]

No, my point is that problem is so complex that pretending you can hide it with libtool is misleading. The developers have know and care that AIX and MinGW are different, and need special variations, or the end-user, trying to build the package, has to track down the docs and figure out where to add the magic option.

You're right: I'm pretty ignorant about the internal details of autoconf et. al. But as an end user, who has 20+ years of experience building software on a wide variety of platforms, I've found that I spent a lot more time fighting autoconf/automake/libtool problems than I did for packages that asked me to uncomment the appropriate variable settings in the Makefile. Why? Because if the appropriate variable was available in the Makefile, then someone had actually, really, already built it on that platform. If not, then it was pretty easy to see what needed to be done.

I should clarify: by "autoconf problems" I don't necessarily mean problems with the autoconf system itself. It does what it's told. The problem is software developers who believe that using autoconf et. al. magically solves all their portability issues, and all they need to do is copy a few scripts/templates from some other project. This is the "autoconf culture" problem I mentioned way back in my first comment. It may not be what the autoconf developers intended or wanted, but it sure is what has happened.

That's all for me.

automake vs. GNU make

Posted Feb 7, 2008 21:36 UTC (Thu) by nix (subscriber, #2304) [Link]

No, my point is that problem is so complex that pretending you can hide it with libtool is misleading. The developers have know and care that AIX and MinGW are different, and need special variations, or the end-user, trying to build the package, has to track down the docs and figure out where to add the magic option.

Actually, if you use libtool and libltdl you *don't* need to care that Linux, Solaris, MacOS X, AIX, HP-UX 10 and Windows all use quite different methods to build shared libraries with different names and semantics; at least not unless you're trying to go beyond what static libraries allow and do symbol versioning or something like that.

The end-user need have no clue at all.

libtool has its problems (astonishing sloth being one of them: why anyone complains that configure takes too long to run for very small projects, when libtool slows down building drastically regardless of project size, I have no idea) but you don't seem to know what those problems are. At least you're focusing on men made of straw when there are perfectly visible giants of problems looming a few feet away.

automake vs. GNU make

Posted Feb 7, 2008 21:55 UTC (Thu) by vmole (subscriber, #111) [Link]

Yes, that's the promise. But in my experience (and in many others), as the end user of several libtool using projects, it's not reality. Is it that all these projects are setting things up wrong? Possibly. I don't know, and I don't really care, because if so, it's apparently as hard to get right as using the actual OS tools would be, which would at least be debuggable by normal human beings.

And I'm well aware and have suffered extensively at libtool's sloth. And its insanely long invocation lines, which make make logs painful to read and hide errors.

automake vs. GNU make

Posted Feb 7, 2008 23:18 UTC (Thu) by nix (subscriber, #2304) [Link]

Your reality differs from my reality. I've had some cosmetic problems on 
Solaris 2.4 and 2.5.1 (we're talking prehistoric here) and I needed to 
install GNU sed on HP-UX and AIX, but other than that, nothing really.

(Mind you, the need for GNU sed *was* unacceptable --- it was also a bug.)

automake vs. GNU make

Posted Feb 7, 2008 23:25 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

> [libtool is] apparently as hard to get right as using the actual OS
> tools would be, which would at least be debuggable by normal human
> beings.

"at least be debuggable" - very well put, I completely agree.
Why does automake actually need libtool at all ? I mean it generates the 
makefile code, it could as well just generate the code for calling the 
actual OS tools directly in the makefiles. This would remove this one 
layer of indirection.

Alex

automake vs. GNU make

Posted Feb 8, 2008 0:44 UTC (Fri) by nix (subscriber, #2304) [Link]

automake can't generate the commands libtool executes because automake 
runs on the distributor's machine, not the builder's, and doesn't have a 
clue what sort of system the build will take place on.

automake vs. GNU make

Posted Feb 8, 2008 7:20 UTC (Fri) by aleXXX (subscriber, #2742) [Link]

Ah, yes, indeed.
(I didn't work with autotools in the last years).

Then, couldn't the configure script handle that ? It runs on the build 
machine.

Alex

automake vs. GNU make

Posted Feb 8, 2008 21:28 UTC (Fri) by nix (subscriber, #2304) [Link]

Yes, it could: but in libtool 1.5 it doesn't :(

I never said libtool didn't suck. It's just better than anything else 
around right now for the job it does, and it has a *lot* of hard-won 
knowledge of shared library wierdness on manifold systems encoded into it 
(as autoconf does of other cross-system variation).

automake vs. GNU make

Posted Feb 7, 2008 23:54 UTC (Thu) by stevenj (subscriber, #421) [Link]

You didn't read the manual, and as a result it failed on AIX because you asked libtool to support semantics unavailable on that platform, and as a result you loudly complain here that libtool is broken and buggy. This is not a convincing critique of libtool.

libtool solves a big part of the problem: its manual specifies the lowest common denominator of shared library semantics, tells you how to indicate whether you obey those semantics, and builds the resulting library on every system that supports the semantics you request. The fact that you still have to know that there might be some differences between systems, and that you might have to read the manual to learn how to deal with these differences, is not a reason to throw it out and rewrite everything yourself from scratch (the only other "solution" you have suggested). No matter what portability tool you use, developers will still need to know something about the differences between platforms.

automake vs. GNU make

Posted Feb 8, 2008 0:23 UTC (Fri) by vmole (subscriber, #111) [Link]

You didn't read the manual, and as a result it failed on AIX because you asked libtool to support semantics unavailable on that platform

*I* didn't ask libtool to do anything. I'm the end-user. I'm not supposed to have know about libtool, or the variations in shared library implementations. Right? Isn't that the whole friggin point?

And if your response is "But there's too many differences, libtool can't hide everything", then we are in 100% agreement. Where we differ is whether or not it's worth the attempt, and whether libtool is the correct direction. So be it. But if you continue to ignore those of us who have problems, and blame the users, well, it will never get better.

automake vs. GNU make

Posted Feb 8, 2008 0:42 UTC (Fri) by stevenj (subscriber, #421) [Link]

You haven't suggested any way for things to get better, you've just flamed a tool because you've encountered a couple of buggy packages that misused it, and it didn't correct for all the deficiencies of the developers. Then you tried to debug the problem yourself and failed because you didn't bother to read the fine manual.

Your suggestions, as far as I can tell, have been either for every developer to reinvent the wheel by rolling their own platform-dependent scripts, or for the developers to push the whole problem onto the end-users by giving them a raw Makefile and telling them to fix the compiler options themselves. Neither of these seems like an improvement.

automake vs. GNU make

Posted Feb 8, 2008 5:42 UTC (Fri) by lovelace (subscriber, #278) [Link]

Again, you're blaming the tool for the complexity of the underlying problem.

But, isn't this tool supposed to take care of the underlying complexity? So, haven't you completely invalidated your argument for it? It's supposed to make the task of creating libraries similar across dissimilar systems yet by your own description it does not.

I don't remember all the details now since it's been a while, but back when we were trying to port KDE 3.x to native Qt on the Mac libtool was a constant source of problems.

automake vs. GNU make

Posted Feb 8, 2008 7:19 UTC (Fri) by aleXXX (subscriber, #2742) [Link]

libtool on the Mac (a native binary for dealing with libraries on the 
Mac) is something different than the autotools libtool (portable shell 
script which should deal with shared libs on all systems).

Alex

automake vs. GNU make

Posted Feb 8, 2008 16:05 UTC (Fri) by lovelace (subscriber, #278) [Link]

Hi Alex!

Yep, I'm aware of that, but that's not what I was referring to.  I was referring to the
autotools libtool.  Unfortunately, like I said, that was about 5 years ago and I cannot
remember the specifics.  So, rather than make more unsubstantiated accusations, I'll just
leave it at that.  I will mention, though, that that episode in particular cemented my general
dislike of the autotools libtool that still stands to this day.

automake vs. GNU make

Posted Feb 8, 2008 21:26 UTC (Fri) by nix (subscriber, #2304) [Link]

It reduces the complexity, but fundamentally these systems have different 
*runtime* semantics, which can't be entirely hidden. For simple uses 
(DT_NEEDED-style simple symbol lookup of libraries with no undefined 
symbols, and dlopen()/dlsym()/dlclose()-style dynamic loading), libtool 
does a good job. Anything more complex will hit trouble on one system or 
another.

automake vs. GNU make

Posted Feb 8, 2008 21:46 UTC (Fri) by lovelace (subscriber, #278) [Link]

Ah, now I'm remembering more about what the problems where.

1. Shared libraries and modules are the same on Linux (and lots of other Unices) but are
different on the Mac.  Libtool had a difficult time understanding this.
2. Until fairly recently (Tiger, iirc) shared libraries couldn't be easily dlopened on the
mac, only modules could.

Since KDE makes extensive use of dlopen-ing modules to accomplish things this made things
quite tricky and libtool wasn't really that much help.

So, yeah, quite a different runtime system.  Newer versions of OS X have gotten quite a bit
better on the dlopen-ing front, but they are still fundamentally different.  And, I wouldn't
even try to use libtool to create OS X frameworks....

automake vs. GNU make

Posted Feb 8, 2008 23:18 UTC (Fri) by nix (subscriber, #2304) [Link]

I'd expect MacOS X support for modules to have been OK since

2003-03-20  Peter O'Gorman  <peter@pogma.com>

        * ltmain.in: Always use $echo not echo for consistency.
        Changes for darwin building. Warn if linking against libs linked
        with -module. Use module_cmds if available and building a module,
        move convenience double lib check,

What else is wrong?

And, yes, your point 2 is hard for libltdl to overcome: if you build the 
library as a lib, not a module, you'd have been stuck whatever libtool 
did.