LCA: Disintermediating distributions [LWN.net]

LCA: Disintermediating distributions

Posted Feb 6, 2008 20:50 UTC (Wed) by asamardzic (guest, #27161) [Link]

Beside portability that there are enough other reasons to switch from autotools to something easier to use. I know there are gurus which understand autotools, but my impression is that the big majority of Linux developers doesn't really understand it.

I can only second that. I consider myself rather experienced, developing mostly for Unix, both my open-source projects and at my work, for number of years and heavily using autotools during the first time; still, I was never able to confidently accomplish anything more complicated with it, for example to create my own set of macros to find given library or package. Discovering CMake, while it certainly has its own set of quirks, was like nirvana for me - I was finally able to properly understand my build system, I was able to read generated makefiles, and - yes, I was able to write modules (in CMake terminology) for finding other stuff on the system without any problem.

So, while I couldn't care less about Windows portability (well, at least for my open-source work), I really couldn't find call for some kind of uniting around autotools (if I understood that part of the arcticle properly) compelling.

Thumbs up on CMake

Posted Feb 7, 2008 6:48 UTC (Thu) by robla (subscriber, #424) [Link]

We're in the process of switching over to CMake for the Second Life viewer, and from what
we've seen so far, it looks great.  When you have a lot of Windows and Mac programmers on your
team, it's a huge win for those people to be able to use the dev tools they are used to (e.g.
Visual Studio and XCode). The fact that CMake generates build files for those and a lot more
is a pretty unique characteristic among build systems.

LCA: Disintermediating distributions

Posted Feb 6, 2008 22:56 UTC (Wed) by stevenj (guest, #421) [Link]

It's also hard to convince e.g. students who have a hard time learning C or C++ that they additionally have to learn autoconf, automake, libtool, shell, m4 and make syntax. It just doesn't make sense if the build tool is more complicated than the programming language itself.

Skilled programmers don't have a hard time learning a new syntax. What makes a tool like autoconf difficult is that the underlying portability problem it solves is quite hard and intrinsically complicated.

In my experience, non-trivial projects that avoid autoconf tend to start re-inventing it, badly, as soon as they move beyond compiling just on GNU/Linux systems and interacting in non-trivial ways with the build environment. There's a reason why most major free projects use it (except for Python and Perl programs, which live in their own self-contained little build universes, although once they start depending on other external non-Python/Perl libraries it can quickly get messy again).

Similar things happen with automake, although admittedly some of automake's complexity comes from having to deal with the underlying deficiencies of 'make'. But there is also a fair amount of intrinsic complexity from the fact that you want to support a lot of make targets (make dist, make clean, make distclean, make check, make install, make uninstall, etc. etc.), VPATH builds, source files that are built by scripts or programs in some projects, multiple languages and compilers, different compile/link flags for different programs in the same projects, building shared libraries on multiple platforms with very different shared-library build procedures and semantics, etcetera etcetera.

By the time you understand the underlying problem, using the actual tool is pretty straightforward and well documented (see e.g. the "autobook").

LCA: Disintermediating distributions

Posted Feb 6, 2008 23:26 UTC (Wed) by vmole (guest, #111) [Link]

What makes a tool like autoconf difficult is that the underlying portability problem it solves is quite hard and intrinsically complicated.

Actually, the problems that autoconf "solves" are the *easy* parts of portability. Actually dealing with the semantic differences between OS's is the hard part. The worst part is that autoconf encourages a really bad programming style of ifdefs and the use of the 6 slightly OS-specific versions of foofunc() rather than just following the dang standard. Henry Spencer told us why that was a bad idea [warning: PDF] 15 years ago, but the GNU people failed to heed him.

LCA: Disintermediating distributions

Posted Feb 6, 2008 23:42 UTC (Wed) by vapier (subscriber, #15768) [Link]

the void you describe is largely addressed by things like the gnulib project.  it takes care
of all the OS-specific issues by checking to see if the function in question is usable on the
host system.  if it isnt, the gnulib version is provided.  thus the code that *you* write is
able to freely assume that function in question is available.

as for encouraging ifdef's, there are simply some things you cannot solve otherwise.  how do
you propose people integrate optional support for addon libraries ?  perhaps you can support a
wide range of graphic formats, but only if the external library is available ... yes, you can
write autotool code to largely avoid this (make most pieces standalone files that are
optionally compiled), but it doesnt make the issue magically go away due to the nature of the
code (assuming C/C++ here).

LCA: Disintermediating distributions

Posted Feb 7, 2008 0:01 UTC (Thu) by vmole (guest, #111) [Link]

No, you can't get rid of them entirely, and Mr. Spencer doesn't claim you can. They were added to the language for a reason, after all. But all too much autoconf code has stuff like this:

#ifdef HAVE_INDEX
p = index(s, c)
#elif HAVE_STRCHR
p = strchr(s, c)
#endif

Yeah, that's a really old example, but while the names of the functions change, the style doesn't.

And while I'm at it, why do I still have to sit through messages like

Checking for C89...okay
Checking for stdio.h...found
Checking for strcpy()...found
...

Hey, if it's C89, then ALL THOSE FUNCTIONS AND HEADER FILES ARE THERE. If not, the implementation is broken WAY beyond what autoconf can solve.

And, not to go off on a rant (too late!), why do I have to sit through 3 minutes of autoconf masturbation just to go to a 3 second compile of 300 lines of standard C?

Yes, I know that isn't really autoconf's "fault". It can be used in reasonable ways. But the autoconf culture encourages such bad choices, because 95% of the users don't understand it and just copy other peoples' bad choices.

Feh.

LCA: Disintermediating distributions

Posted Feb 7, 2008 0:28 UTC (Thu) by stevenj (guest, #421) [Link]

In my experience with developing many actual programs using autoconf, the checks in the configure script are a direct consequence of porting to some platform or another. ("Crap, MinGW doesn't support gettimeofday, we'll need to check for it and whatever alternative is available.")

Most new code these days assumes C89 support at least, and doesn't check any more for functions like "strcpy". On the other hand, as recently as a few years ago the default Solaris compiler didn't support "const" by default unless you jumped through hoops, and hence a lot of programs had checks for things like that.

It's true that, at some point, older programs might be able to remove checks for things that only break on ancient platforms, but it can be really hard to decide exactly when to remove such checks. It's safer to leave them in and wait the extra seconds than to break the build on an extant architecture.

Regarding the time for configure to run, its true that the configure script is often slower these days than compiling the program. For large programs, parallel make speeds up things a lot, while parallelizing configure tests is extremely tricky (although it has been discussed a lot by the autoconf developers). For small projects, the configure script may call the compiler more times than the Makefile itself, but long experience has shown that actually trying to compile something in the configure script is by far the most reliable way of performing a feature test. In any case, the number of times that one has to run "configure" is small (most people don't run it at all anymore, since prepackaged binaries from distros are ubiquitous), and it's better to sacrifice a little build time than to sacrifice robustness.

LCA: Disintermediating distributions

Posted Feb 7, 2008 0:52 UTC (Thu) by vmole (guest, #111) [Link]

In my experience with developing many actual programs using autoconf, the checks in the configure script are a direct consequence of porting to some platform or another.

My experience says otherwise. Things like checking C++ related stuff (and even Fortran!) in projects that are pure C. And the sucessful checks for C89 followed by checks for individual C89 standardized functions is widespread, and almost certainly because there's some autoconf macro that does all of it. It's just stupid and annoying.

LCA: Disintermediating distributions

Posted Feb 7, 2008 1:40 UTC (Thu) by stevenj (guest, #421) [Link]

The checks for C++ and Fortran in C-only programs was due to a libtool 1.5 bug that has since been fixed, IIRC.

And it's true that AC_PROG_CC (the check for a C compiler) automatically calls a check to make sure the compiler is in C89 (ISO C90) mode, and if not it tries to find an option to put the compiler in C89 mode. Reliably checking whether the compiler is in C89 mode involves, among other things, checks for stdio.h. These checks were still required fairly recently — e.g. on AIX circa 2003 you had to use "-qlanglvl=extc89" or it didn't handle macro parameters in completely ANSI fashion, and on HPUX the compiler was non-ANSI by default until at least the late 90s. And many of these systems were still running long, long after their release dates (e.g. I heard from Solaris users with a compiler that defaulted non-ANSI as recently as a few years ago).

When autoconf has a default check, there's usually a good reason for it; most developers don't have experience on a wide enough variety of platforms to appreciate this. Try to get a patch accepted into autoconf sometime and you'll see what they have to deal with and why so many complaints about autoconf are founded in ignorance.

(And if your configure time is dominated by the default check for an ANSI compiler, your project is pretty small indeed. As I said, in my experience most configure scripts times for projects of decent size are dominated by checks that the programmers explicitly included in response to portability problems --- or by checks that are invoked by those checks like the case above, which the autoconf developers put in there for good reason. And, really, it's not like this is a huge problem—how often do you run configure scripts, and how many seconds would you be willing to shave off at the risk of losing portability to some machine?)

LCA: Disintermediating distributions

Posted Feb 7, 2008 7:51 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

> In my experience with developing many actual programs using autoconf,
> the checks in the configure script are a direct consequence of porting
> to some platform or another. ("Crap, MinGW doesn't support
> gettimeofday, we'll need to check for it and whatever alternative is
> available.")

Yes, basically the configure checks are necessary.
Autoconf itself wasn't so bad. But then additionally you have to 
understand how it works together with automake and libtool. This is what 
made it really hard for me.

And I'm sure this is also the reason why many programs run much more 
configure checks than they actually need. They take the autotools stuff 
from some existing project and just change it so that it can build their 
program. With this approach you end up with all the configure checks the 
original script had, and maybe later on you have to add some more. IMO it 
is simply too hard to build a simple program using autotools. Who wants 
to learn that just to build hello world ?

Using other build systems, as e.g. cmake, makes this trivial:

add_program(helloworld hello.c)

That's all, cmake will do a few checks to find the compiler, check that 
it actually works and figure out if it's a 32 or 64 bit compiler, and 
that's all.
I guess for Scons it's similar.

Also autotools don't doesn't push you to use some modular style, at least 
in KDE we ended up with a few huge scripts which did a lot of magic. E.g. 
CMake strongly encourages a modular system (while it is of course still 
possible to throw everything into one file, but then you intentionally 
work around its features).

Alex

LCA: Disintermediating distributions

Posted Feb 7, 2008 20:04 UTC (Thu) by vapier (subscriber, #15768) [Link]

the issues you raise i dont really see being "solved" or even really addressed any differently
with other build systems (cmake/scons/whatever).  a build system cannot really compensate for
a coder's inability to write clean code.  what do you see in other build systems that
encourages different style of coding ?

the example you cite could easily be relegated to a header file without making the real code
messy.  while it is an older example, there are many similar situations that could generally
be solved the same way: take care of OS-differences in one location (sep source or header
file) and keep everything else clean.

LCA: Disintermediating distributions

Posted Feb 7, 2008 21:22 UTC (Thu) by vmole (guest, #111) [Link]

the example you cite could easily be relegated to a header file without making the real code messy.

Bingo. But they're not. Why not? Because people just copy the way previous programmers did things. Maybe the autotools problem is that there are just way too many bad examples out there.

As for other build systems, all I can do is quote the comment immediately above yours, from Alex, apparently a KDE developer: "CMake strongly encourages a modular system (while it is of corse still possible to throw everything into one file, but then you intentionally work around its features)." Autoconf does makes bad usage patterns as easy (or even easier, on first glance) as good ones.

LCA: Disintermediating distributions

Posted Feb 6, 2008 23:55 UTC (Wed) by stevenj (guest, #421) [Link]

There are lots of cases where just following the dang standard is not practical, or not sufficient. For one thing, not all platforms implement the dang standard, and if you don't want to fail completely when this happens you need some workaround. For another thing, in some applications it's extremely useful to support functionality that may not be available on all platforms—for example, SSE instructions or high-resolution timers.

Also, a lot of what autoconf deals with is checking for things in the build environment which essential but not standardized, such as how to link shared libraries. e.g. POSIX threads and OpenMP are two examples of formally standardized libraries that you can depend on, but each compiler and OS has its own command to link with them (see here and here). Or suppose you want to use features from the 1999 ANSI C standard, which has been out for 9 years now but compilers (including gcc) still make you jump through hoops to enable support for it, and of course each compiler has its own hoop (which autoconf will detect).

Also, free-software projects often build upon other projects so as to avoid re-inventing the wheel, and there are lots of extremely useful libraries (from GNU readline to HDF5 to LAPACK to Expat to Boost to...I'm just picking things at random) that are not standardized by any standards body. Part of autoconf's job is to help you detect whether such a library is present and contains the function you want (it may not, e.g. if it is the wrong version).

And heaven help you if you want to link together multiple languages, e.g. you have a C++ program and you want to link Fortran numerical libraries (e.g. LAPACK), without autoconf to help you detect how to do it with your compiler (each one has a different incantation).

Also...well, just look at the autoconf documentation for the variety of kinds of things one has to check for. As I said, there's a reason for its popularity, which extends far beyond "the GNU people"...it fills a real need. People who don't understand what it does are doomed to reinvent it badly.

LCA: Disintermediating distributions

Posted Feb 7, 2008 0:12 UTC (Thu) by vmole (guest, #111) [Link]

I agree that those are all problems that autoconf/libtool/etc. claim to solve. My experience (which is extensive) is that they don't reliably work, and I spend a *LOT* more time figuring out the problem and fixing it than I did with packages that simply ask me to set a few variables in the beginning of the Makefile.

I'd guess that if all you ever work with is Linux, BSD, and possibly Solaris, these tools do work, mostly. OTOH, those are the really easy ones.

LCA: Disintermediating distributions

Posted Feb 7, 2008 0:39 UTC (Thu) by stevenj (guest, #421) [Link]

The autoconf developers go to great lengths to support systems beyond Linux and BSD, and it's simply untrue that the tools break on other systems. I personally work on software that has run for years on everything from HPUX to Tru64 to UNICOS to AIX to MinGW using the autotools.

It's true that many people don't know what to do when configure fails. The usual mistake is to start poring over the configure script (which is essentially object code) rather than RTFM. configure --help gives a clue: most problems can be solved by setting an environment variable on the command line with configure LDFLAGS=... or whatever. The most common problems in my experience are due to libraries installed in nonstandard locations, and in this case there's simply no way around requiring the user to tell you where things are (which in autoconf is done by setting LDFLAGS and CPPFLAGS).

It's also true that some programmers misuse autoconf. e.g. even though the autoconf manual strenuously recommends doing feature tests by actually compiling and linking things, autoconf also provides a macro to get a canonical target name (e.g. i386-linux-gnu) and some programmers take the shortcut of explicitly testing this when they shouldn't. The difficulty is that if you are testing for a feature that does not have a built-in autoconf test, writing a portable feature test is hard, especially if you don't have many platforms to test on—but again, I think this is somewhat intrinsic to the problem and is not really autoconf's fault.

LCA: Disintermediating distributions

Posted Feb 6, 2008 23:35 UTC (Wed) by DonDiego (subscriber, #24141) [Link]

What 'underlying deficiencies of make' are you talking about?  I don't see any such thing.

LCA: Disintermediating distributions

Posted Feb 6, 2008 23:45 UTC (Wed) by vapier (subscriber, #15768) [Link]

autotools allows you to create a build system that will work on any system that has a shell
and make.  that make may not be GNU make.  it may have bugs.  its feature set may be severely
limited compared to what you're used to.  many projects out there who set out to write their
own build system do so using GNU make which means they write code that Works For Them.
non-portable GNU-isms creep in which means the build system is no longer portable to many
targets which sort of defeats the original intent: making it portable.

LCA: Disintermediating distributions

Posted Feb 7, 2008 0:17 UTC (Thu) by vmole (guest, #111) [Link]

"Don't write portable Makefiles, use a portable make." I don't remember the source of that quote, but I do know that trying to deal with each broken vendor version of make leads to nothing pain and tears before bedtime. If you're building software, just install gmake and be done with it.

LCA: Disintermediating distributions

Posted Feb 7, 2008 0:38 UTC (Thu) by nix (subscriber, #2304) [Link]

Well, I know Paul D. Smith (the GNU make author) has said that in the 
past, and I agree with him (for what very little it's worth).

LCA: Disintermediating distributions

Posted Feb 7, 2008 1:11 UTC (Thu) by stevenj (guest, #421) [Link]

Note that GNU make itself is built using automake, so it seems the GNU make developers have voted with their feet on this one.

LCA: Disintermediating distributions

Posted Feb 7, 2008 1:41 UTC (Thu) by stevenj (guest, #421) [Link]

(Although, to be fair, it's not like they could require GNU make to build.  =)

LCA: Disintermediating distributions

Posted Feb 7, 2008 7:05 UTC (Thu) by madscientist (subscriber, #16861) [Link]

Although GNU make does come with an automake environment, it also provides a shell script that
can be used to build make.  Obviously this will recompile and relink everything every time you
run it, but GNU make is not such a huge program that this is a problem.  And once you've got
it built once, you can use that make for subsequent builds.  Having this avoids the catch-22
of needing some make to build make.

That said, automake is awesome especially if you're developing highly portable tools, which
most of the GNU tools are.  For GNU make I don't so much care about the portability aspects,
although that's nice too (but the shell script above would be enough of an "out" for GNU make
itself).  The great thing about automake is all the default rules it provides, including
things like distcheck for building new packages, etc.  These rules save huge amounts of time
and effort for package developers/maintainers.

LCA: Disintermediating distributions

Posted Feb 15, 2008 11:51 UTC (Fri) by ekj (guest, #1524) [Link]

The catch-22 of needing a make-program to compile gnu make isn't that much of a problem
really.

You need a C compiler to compile GCC too. If you want it self-compiled you need to compile it
twice:

First use whatever C-compiler you happen to have lying around to compile GCC. 

Then use your fresh gcc to compile gcc.

LCA: Disintermediating distributions

Posted Feb 15, 2008 22:00 UTC (Fri) by nix (subscriber, #2304) [Link]

With recent versions, thanks to the magic of top-level bootstrap, `make' 
should give you a compiler and libraries byte-for-byte identical to what 
you'd have got if you did the recompile-it dance. (Older versions wouldn't 
have recompiled libiberty with the new compiler before linking that 
compiler with it; top-level bootstrap has fixed that.)

LCA: Disintermediating distributions

Posted Feb 7, 2008 20:11 UTC (Thu) by vapier (subscriber, #15768) [Link]

it's really a mindset.  do you tell every user their system sucks and they should install
Linux and up-to-date utilities ?  or do you use automake and everyone out there can build your
project without a problem ?  some people will choose the former while others will pick the
latter.  in the latter case, automake is the only realistic solution.

LCA: Disintermediating distributions

Posted Feb 7, 2008 0:15 UTC (Thu) by stevenj (guest, #421) [Link]

Try using raw 'make' in a project with subdirectories sometime.

Another obstacle is that 'make' relies on 'sh' (a fairly primitive language) if you want to do anything nontrivial, and portable shell programming requires extreme care. (This is also a source of unfortunate complexity in autoconf and automake, but they provide tools to lessen your dependence on sh, at least.)

LCA: Disintermediating distributions

Posted Feb 7, 2008 4:39 UTC (Thu) by roelofs (guest, #2599) [Link]

Try using raw 'make' in a project with subdirectories sometime.

Have:

        $(MAKE) -C subdir

        cd subdir && $(MAKE)

I believe even MS nmake circa 1990 supported that much, and it's not even close to a standard make. Every Unix make I've used (couple dozen) was better than that.

But it has been quite a few years since I messed with any of this stuff, so craniorectal impaction is always a possibility.

Greg

LCA: Disintermediating distributions

Posted Feb 7, 2008 10:05 UTC (Thu) by epa (subscriber, #39769) [Link]

Have a look at the article linked from the parent post.  Calling one 'make' from another
makefile tends to lead to all sorts of crufty problems.  The paper gives examples, and my own
experience certainly confirms it.  That said, you can use make in a large project, just not by
recursive invocation.

LCA: Disintermediating distributions

Posted Feb 7, 2008 11:52 UTC (Thu) by cortana (subscriber, #24596) [Link]

> Try using raw 'make' in a project with subdirectories sometime. 

Then you have already failed. Why on earth would anyone want to call make recursively? The
only reason I have ever heard is "because I don't understand what I'm doing, and I cargo
culted my build system from someone else, who also did not understand what he was doing".

LCA: Disintermediating distributions

Posted Feb 6, 2008 23:46 UTC (Wed) by wingo (guest, #26929) [Link]

Side note: I'm enjoying this discussion, really nice comments.

I agree with you, stevenj, except that I think that GNU make is now sufficient: there is no
need for automake. The creator of automake thinks so, too: http://tromey.com/blog/?p=394

automake vs. GNU make

Posted Feb 7, 2008 0:05 UTC (Thu) by stevenj (guest, #421) [Link]

I'm not sure what the context of that post is, but I suspect he's talking about a rather specialized case. Realize what automake gives you. With two lines:

bin_PROGRAMS = hello
hello_SOURCES = hello.c hello.h

automake will generate a Makefile that will support all the GNU standard targets (make install, uninstall, clean, distclean, clean, dist, ...), support VPATH builds properly, integrate with autoconf's detection of the compiler and compiler flags, automatically do dependency tracking (e.g. it will figure out that hello.c depends on hello.h, assuming it does), and so on.

GNU make does none of that for you. The advantages of automake over raw make, even GNU make, are even more dramatic for projects with subdirectories (recursive "make" is notoriously difficult to get right), or for projects that have to build shared libraries (the only reasonable way to do this portably is with GNU libtool, but calling libtool by hand is a PITA).

automake vs. GNU make

Posted Feb 7, 2008 0:33 UTC (Thu) by vmole (guest, #111) [Link]

recursive "make" is notoriously difficult to get right

And there's no good reason for it, except habit. Peter Miller has a nice paper of why recursive make is bad, and a solution. And yes, I converted a fairly large build system from recursive make to his approach, without moving any code. The no-act case went, where everything is up-to-date, went from over a minute to about 5 seconds. More importantly, builds got much more reliable and much more efficient, re-building everything that needed to be rebuilt, and nothing that didn't. Lovely.

the only reasonable way to do this portably is with GNU libtool

HAHAHAHAHAHA. You funny. Libtool is the devil's spawn. If it worked, it would worth it. Maybe.

automake vs. GNU make

Posted Feb 7, 2008 0:51 UTC (Thu) by stevenj (guest, #421) [Link]

Regarding recursive make considered harmful, his solution is to use one gigantic Makefile. Why not put all of your source code in one gigantic .c file while you're at it, and then you can skip "make" entirely? In any case, to each his own, but many many projects have voted with their feet on this one.

I've used libtool for years, and it works (on essentially every Unix-like system, and even on Windows with MinGW); implying it doesn't is baseless FUD. Yes, it has had bugs from time to time. Yes, it is slow as hell because it launches a shell process for each compile. Yes, it is hard to use properly if you don't use automake too. But what is the alternative if you want to build shared libraries portably?

automake vs. GNU make

Posted Feb 7, 2008 1:05 UTC (Thu) by vmole (guest, #111) [Link]

No, his solution is NOT put everything in one giant make file. You use "include" (yes, that makes it GNU make specific. See previous post). I've done this. It's a lot easier to maintain than recursive make files, because most of the sub-directory makefile fragments are just the list of source files.

My experience is that libtool does not work reliably on AIX, and it never gets fixed (or it gets fixed, then broken again). I found that simply writing a small (<10 lines) shell script for each implementation was far more reliable and easier to debug. Admittedly, this was 2001-2006. Maybe it's better now. But a 8500 line shellscript is inherently fragile.

automake vs. GNU make

Posted Feb 7, 2008 1:51 UTC (Thu) by stevenj (guest, #421) [Link]

With AIX there are difficulties with shared libraries because of the limitations of that operating system: it doesn't (or didn't, at least) allow inter-library dependencies between shared libraries. MinGW (Windows) is the same way. If you want to use libtool on these systems you have to pass the "-no-undefined" flag to tell it you have no shared-library dependencies, which libtool cannot assume for you in general.

Of course, this is documented in the manual.

Again, you're blaming the tool for the complexity of the underlying problem.

(And your suggestion of writing a script for every system that you want to support shared libraries on is basically saying that every programmer should re-implement libtool themselves, probably badly because they aren't aware of all the platform variations in shared-library semantics/syntax or even a large subset thereof.)

automake vs. GNU make

Posted Feb 7, 2008 3:59 UTC (Thu) by jamesh (guest, #1159) [Link]

I think you've got things backwards: the -no-undefined flag doesn't really make sense _unless_
shared library dependencies exist.

With the -no-undefined flag set, your library must directly link to all the libraries whose
symbols you used (leaving no "undefined" symbols).

Even on platforms where -no-undefined is not required, it is usually worth using since the
linker can see which library each referenced symbol comes from, making the job of runtime
linking easier.

automake vs. GNU make

Posted Feb 7, 2008 5:02 UTC (Thu) by stevenj (guest, #421) [Link]

Even if there are no library inter-dependencies in your library, you must specify -no-undefined or libtool will refuse to build it on AIX or MinGW, since libtool is apparently unable to detect automatically whether there are unresolved symbols at link time. But you're right that the flag can also be used if you specify library dependencies with -l flags explicitly at link time.

I've also heard that this is useful for prelinking as well, as you mention.

(Apparently, -no-undefined used to be the default, which seems sensible to me, but supposedly they got too many complaints.)

automake vs. GNU make

Posted Feb 7, 2008 16:59 UTC (Thu) by vmole (guest, #111) [Link]

No, my point is that problem is so complex that pretending you can hide it with libtool is misleading. The developers have know and care that AIX and MinGW are different, and need special variations, or the end-user, trying to build the package, has to track down the docs and figure out where to add the magic option.

You're right: I'm pretty ignorant about the internal details of autoconf et. al. But as an end user, who has 20+ years of experience building software on a wide variety of platforms, I've found that I spent a lot more time fighting autoconf/automake/libtool problems than I did for packages that asked me to uncomment the appropriate variable settings in the Makefile. Why? Because if the appropriate variable was available in the Makefile, then someone had actually, really, already built it on that platform. If not, then it was pretty easy to see what needed to be done.

I should clarify: by "autoconf problems" I don't necessarily mean problems with the autoconf system itself. It does what it's told. The problem is software developers who believe that using autoconf et. al. magically solves all their portability issues, and all they need to do is copy a few scripts/templates from some other project. This is the "autoconf culture" problem I mentioned way back in my first comment. It may not be what the autoconf developers intended or wanted, but it sure is what has happened.

That's all for me.

automake vs. GNU make

Posted Feb 7, 2008 21:36 UTC (Thu) by nix (subscriber, #2304) [Link]

No, my point is that problem is so complex that pretending you can hide it with libtool is misleading. The developers have know and care that AIX and MinGW are different, and need special variations, or the end-user, trying to build the package, has to track down the docs and figure out where to add the magic option.

Actually, if you use libtool and libltdl you *don't* need to care that Linux, Solaris, MacOS X, AIX, HP-UX 10 and Windows all use quite different methods to build shared libraries with different names and semantics; at least not unless you're trying to go beyond what static libraries allow and do symbol versioning or something like that.

The end-user need have no clue at all.

libtool has its problems (astonishing sloth being one of them: why anyone complains that configure takes too long to run for very small projects, when libtool slows down building drastically regardless of project size, I have no idea) but you don't seem to know what those problems are. At least you're focusing on men made of straw when there are perfectly visible giants of problems looming a few feet away.

automake vs. GNU make

Posted Feb 7, 2008 21:55 UTC (Thu) by vmole (guest, #111) [Link]

Yes, that's the promise. But in my experience (and in many others), as the end user of several libtool using projects, it's not reality. Is it that all these projects are setting things up wrong? Possibly. I don't know, and I don't really care, because if so, it's apparently as hard to get right as using the actual OS tools would be, which would at least be debuggable by normal human beings.

And I'm well aware and have suffered extensively at libtool's sloth. And its insanely long invocation lines, which make make logs painful to read and hide errors.

automake vs. GNU make

Posted Feb 7, 2008 23:18 UTC (Thu) by nix (subscriber, #2304) [Link]

Your reality differs from my reality. I've had some cosmetic problems on 
Solaris 2.4 and 2.5.1 (we're talking prehistoric here) and I needed to 
install GNU sed on HP-UX and AIX, but other than that, nothing really.

(Mind you, the need for GNU sed *was* unacceptable --- it was also a bug.)

automake vs. GNU make

Posted Feb 7, 2008 23:25 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

> [libtool is] apparently as hard to get right as using the actual OS
> tools would be, which would at least be debuggable by normal human
> beings.

"at least be debuggable" - very well put, I completely agree.
Why does automake actually need libtool at all ? I mean it generates the 
makefile code, it could as well just generate the code for calling the 
actual OS tools directly in the makefiles. This would remove this one 
layer of indirection.

Alex

automake vs. GNU make

Posted Feb 8, 2008 0:44 UTC (Fri) by nix (subscriber, #2304) [Link]

automake can't generate the commands libtool executes because automake 
runs on the distributor's machine, not the builder's, and doesn't have a 
clue what sort of system the build will take place on.

automake vs. GNU make

Posted Feb 8, 2008 7:20 UTC (Fri) by aleXXX (subscriber, #2742) [Link]

Ah, yes, indeed.
(I didn't work with autotools in the last years).

Then, couldn't the configure script handle that ? It runs on the build 
machine.

Alex

automake vs. GNU make

Posted Feb 8, 2008 21:28 UTC (Fri) by nix (subscriber, #2304) [Link]

Yes, it could: but in libtool 1.5 it doesn't :(

I never said libtool didn't suck. It's just better than anything else 
around right now for the job it does, and it has a *lot* of hard-won 
knowledge of shared library wierdness on manifold systems encoded into it 
(as autoconf does of other cross-system variation).

automake vs. GNU make

Posted Feb 7, 2008 23:54 UTC (Thu) by stevenj (guest, #421) [Link]

You didn't read the manual, and as a result it failed on AIX because you asked libtool to support semantics unavailable on that platform, and as a result you loudly complain here that libtool is broken and buggy. This is not a convincing critique of libtool.

libtool solves a big part of the problem: its manual specifies the lowest common denominator of shared library semantics, tells you how to indicate whether you obey those semantics, and builds the resulting library on every system that supports the semantics you request. The fact that you still have to know that there might be some differences between systems, and that you might have to read the manual to learn how to deal with these differences, is not a reason to throw it out and rewrite everything yourself from scratch (the only other "solution" you have suggested). No matter what portability tool you use, developers will still need to know something about the differences between platforms.

automake vs. GNU make

Posted Feb 8, 2008 0:23 UTC (Fri) by vmole (guest, #111) [Link]

You didn't read the manual, and as a result it failed on AIX because you asked libtool to support semantics unavailable on that platform

*I* didn't ask libtool to do anything. I'm the end-user. I'm not supposed to have know about libtool, or the variations in shared library implementations. Right? Isn't that the whole friggin point?

And if your response is "But there's too many differences, libtool can't hide everything", then we are in 100% agreement. Where we differ is whether or not it's worth the attempt, and whether libtool is the correct direction. So be it. But if you continue to ignore those of us who have problems, and blame the users, well, it will never get better.

automake vs. GNU make

Posted Feb 8, 2008 0:42 UTC (Fri) by stevenj (guest, #421) [Link]

You haven't suggested any way for things to get better, you've just flamed a tool because you've encountered a couple of buggy packages that misused it, and it didn't correct for all the deficiencies of the developers. Then you tried to debug the problem yourself and failed because you didn't bother to read the fine manual.

Your suggestions, as far as I can tell, have been either for every developer to reinvent the wheel by rolling their own platform-dependent scripts, or for the developers to push the whole problem onto the end-users by giving them a raw Makefile and telling them to fix the compiler options themselves. Neither of these seems like an improvement.

automake vs. GNU make

Posted Feb 8, 2008 5:42 UTC (Fri) by lovelace (guest, #278) [Link]

Again, you're blaming the tool for the complexity of the underlying problem.

But, isn't this tool supposed to take care of the underlying complexity? So, haven't you completely invalidated your argument for it? It's supposed to make the task of creating libraries similar across dissimilar systems yet by your own description it does not.

I don't remember all the details now since it's been a while, but back when we were trying to port KDE 3.x to native Qt on the Mac libtool was a constant source of problems.

automake vs. GNU make

Posted Feb 8, 2008 7:19 UTC (Fri) by aleXXX (subscriber, #2742) [Link]

libtool on the Mac (a native binary for dealing with libraries on the 
Mac) is something different than the autotools libtool (portable shell 
script which should deal with shared libs on all systems).

Alex

automake vs. GNU make

Posted Feb 8, 2008 16:05 UTC (Fri) by lovelace (guest, #278) [Link]

Hi Alex!

Yep, I'm aware of that, but that's not what I was referring to.  I was referring to the
autotools libtool.  Unfortunately, like I said, that was about 5 years ago and I cannot
remember the specifics.  So, rather than make more unsubstantiated accusations, I'll just
leave it at that.  I will mention, though, that that episode in particular cemented my general
dislike of the autotools libtool that still stands to this day.

automake vs. GNU make

Posted Feb 8, 2008 21:26 UTC (Fri) by nix (subscriber, #2304) [Link]

It reduces the complexity, but fundamentally these systems have different 
*runtime* semantics, which can't be entirely hidden. For simple uses 
(DT_NEEDED-style simple symbol lookup of libraries with no undefined 
symbols, and dlopen()/dlsym()/dlclose()-style dynamic loading), libtool 
does a good job. Anything more complex will hit trouble on one system or 
another.

automake vs. GNU make

Posted Feb 8, 2008 21:46 UTC (Fri) by lovelace (guest, #278) [Link]

Ah, now I'm remembering more about what the problems where.

1. Shared libraries and modules are the same on Linux (and lots of other Unices) but are
different on the Mac.  Libtool had a difficult time understanding this.
2. Until fairly recently (Tiger, iirc) shared libraries couldn't be easily dlopened on the
mac, only modules could.

Since KDE makes extensive use of dlopen-ing modules to accomplish things this made things
quite tricky and libtool wasn't really that much help.

So, yeah, quite a different runtime system.  Newer versions of OS X have gotten quite a bit
better on the dlopen-ing front, but they are still fundamentally different.  And, I wouldn't
even try to use libtool to create OS X frameworks....

automake vs. GNU make

Posted Feb 8, 2008 23:18 UTC (Fri) by nix (subscriber, #2304) [Link]

I'd expect MacOS X support for modules to have been OK since

2003-03-20  Peter O'Gorman  <peter@pogma.com>

        * ltmain.in: Always use $echo not echo for consistency.
        Changes for darwin building. Warn if linking against libs linked
        with -module. Use module_cmds if available and building a module,
        move convenience double lib check,

What else is wrong?

And, yes, your point 2 is hard for libltdl to overcome: if you build the 
library as a lib, not a module, you'd have been stuck whatever libtool 
did.

automake vs. GNU make

Posted Feb 7, 2008 1:53 UTC (Thu) by sward (subscriber, #6416) [Link]

I've also used his include-based Makefile structure on projects that were originally written
using recursive make.  The Makefile's got smaller, simpler, faster, and more reliable.
Hierarchy still takes *some* thought, but it really wasn't difficult.

automake vs. GNU make

Posted Feb 7, 2008 3:04 UTC (Thu) by stevenj (guest, #421) [Link]

I have no doubt about that, if you compare hand-written recursive make with GNU make+include. I'm more doubtful if you compare to automake input files, which are pretty compact (and support a portable include directive if you have definitions you want to share between directories etc.).

Of course, it's possible that Tom Tromey or someone may come up with some very clever include file that we can all use in our GNU Makefiles that gives us 95% of automake's functionality in a cleaner and faster way (at the expense of requiring GNU make to build, which in this day and age isn't too much of a burden). And then we'll just use 'include' for subdirectories. But until such a thing gets released, the alternative is to write the include files yourself, support all of the standard targets yourself, be careful to use only portable sh constructs yourself, and so on; in the vast majority of cases, automake seems a simpler and more robust solution at present.

A technical question about using include instead of recursive make via automake (recursive make by hand is crazy): how do you separate the namespaces for the subdirectories? For example, if you have a source file with the same name in two subdirectories (e.g. main.c), do you have to jump through hoops to prevent it from getting confused? Or what if you want a 'make clean' target in two subdirectories? I didn't see any obvious way to handle such separation cleanly in the GNU make manual.

automake vs. GNU make

Posted Feb 7, 2008 4:25 UTC (Thu) by masuel (guest, #28661) [Link]

>For example, if you have a source file with the same name in two 
> subdirectories (e.g. main.c), do you have to jump through hoops to 
> prevent it from getting confused?

The way I do it is always reference the tree from the build root.

so a/main.c is never looks like b/main.c

and yes I can start a build from the subdirs etc.

non-recursive is the way to go. 

Just require gnu make don't try supporting version 8.79 (rh9) clients just 
make people upgrade....

automake vs. GNU make

Posted Feb 7, 2008 4:41 UTC (Thu) by sward (subscriber, #6416) [Link]

Files (source or derived) in separate directories are *always* in separate namespaces, in
Miller's approach.  The 'make' is run once, in the root directory, and all sources and targets
are described from that point of view.  So you don't build 'main.c', you build 'foo/main.c'.
You use target-specific overrides if you need to customize things, e.g.:

   # in foo/include.mk:
   $(FOO_OBJ): CPPFLAGS += -Ifoo

Now, if you wanted a selective clean (for one subdir), I agree there isn't a good way to do
this in the include-based structure (other than calling it something else, e.g. 'foo/clean').
Of course, you could also include a local Makefile in the subdir just to make things more
familiar:

   # in foo/Makefile:
   clean: 
       cd ..; $(MAKE) foo/clean

Did that help, or would a more complete example be clearer?

automake vs. GNU make

Posted Feb 7, 2008 4:55 UTC (Thu) by vmole (guest, #111) [Link]

Even neater: Stick the following as GNUMakefile in each of your subdirs:

CURMOD:=$(shell pwd | sed -e 's,.*/src/\(.*\),\1,')
all :
        @cd $${PWD%$(CURMOD)} && $(MAKE) --no-print-directory $(CURMOD)

% :: FORCE
        @cd $${PWD%$(CURMOD)} && $(MAKE) --no-print-directory $(CURMOD)/$@

FORCE:

which automatically transfers any make command up to the top level and re-invokes with the same arguments. (You'll need to tweak the sed command to match your layout.)

automake vs. GNU make

Posted Feb 7, 2008 5:18 UTC (Thu) by stevenj (guest, #421) [Link]

That's clear enough. Yes, using foo/main.c and bar/main.c in the Makefile rules will clearly work (unless you have one of the rare compilers that doesn't support -c -o, although you can hack around that with enough effort I suppose).

It seems a bit ugly to me, though, that your subdirectory Makefiles (to be included) must be written differently depending on the name of the directory they are in. The analogous thing in a programming language would be, if you had a subroutine foo(), to require that all local variables be named with a "foo_" prefix.

automake vs. GNU make

Posted Feb 7, 2008 5:54 UTC (Thu) by sward (subscriber, #6416) [Link]

That isn't really necessary for variable names (that you don't need to reference later), only
the source and targets need to be qualified.  A fairly typical example:

# Define sources for this component:
MY_SRC = $(wildcard foo/*.c)
MY_OBJ = $(MY_SRC:.c=.o)
MY_TGT = foo/libfoo.a

# Additional flags and dependencies for building this component:
$(MY_OBJ): CPPFLAGS += -I foo
$(MY_TGT): $(MY_OBJ)

# Add these to the master Makefile's lists:
SOURCE += $(MY_SRC)
TARGET += $(MY_TGT)
CLEAN  += $(MY_OBJ)

The next component can reuse MY_SRC etc.  You wouldn't even need these variables, if you had
no need for target-specific variables or dependencies.  (But of course, you usually do).  If
you wanted to keep a reference to 'foo/libfoo.a', for use in other Makefiles, you might
replace MY_TGT with LIBFOO.

In short, you only need unique names for global variables; local ones can be overwritten.

You do need the component path ('foo' in this example) in a few spots, but it is trivial to
change that if the directory structure changes.

automake vs. GNU make

Posted Feb 7, 2008 7:41 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

> But what is the alternative if you want to build shared libraries
> portably?

I don't know about Scons, but you can use CMake, and it will just work:

add_library(foo SHARED foo.c bar.c)

This will give you depending what you need:
-UNIX Makefiles (not quite sure if it requires GNU make, I don't think 
so)
-MS nmake makefiles
-Borland Makefiles
-Watcom Makefiles
-cygwin Makefiles
-XCode projects
-Visual Studio >= 6 projects.

I wondered for years why suddenly the comopiler can't be called directly 
anymore, to find out that it's just libtool and you can use the compiler 
very well directly (which removes one layer of indirection -> makes it 
easier to understand).

Alex

automake vs. GNU make

Posted Feb 7, 2008 17:11 UTC (Thu) by stevenj (guest, #421) [Link]

UNIX Makefiles (not quite sure if it requires GNU make, I don't think so)

You are sorely mistaken if you think that there is one way to build shared libraries that works on all flavors of Unix. If you only care about Windows, GNU/Linux and similar systems with gcc, and MacOS X, then yes, generating three makefiles will work, but that's not solving anything like the whole problem.

automake vs. GNU make

Posted Feb 7, 2008 23:17 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

CMake knows how to build shared libs on all supported platforms (which 
support shared libs) with all supported toolchains (GNU, IBM, Sun, 
Borland, MS, Portland, Intel, HP and more). For several 
platforms/toolchains this is tested every night (unfortunately there 
aren't nightly tests for all supported combinations).
http://www.cmake.org/Testing/Dashboard/20080206-0100-Nigh...

Alex

automake vs. GNU make

Posted Feb 8, 2008 5:50 UTC (Fri) by lovelace (guest, #278) [Link]

The makefiles CMake generates are not portable and can only be used on the system that they're generated on. If you move to a different system, you use CMake to generate makefiles for that system. That's how they can reliably create shared libraries on multiple systems using makefiles.

automake vs. GNU make

Posted Feb 7, 2008 7:15 UTC (Thu) by madscientist (subscriber, #16861) [Link]

libtool isn't so bad except for one thing, which makes it unusable.  No support for
cross-compilation.  It's so utterly lame that every part of the GNU build environment
(binutils, gcc, autoconf, automake) is very cognizant of cross-compilation and works very,
very hard to make setting it up and using it as simple as possible... except libtool.

Whenever I see a package come through that uses libtool I absolutely cringe because I know it
will be a huge PITA.

automake vs. GNU make

Posted Feb 7, 2008 17:17 UTC (Thu) by stevenj (guest, #421) [Link]

That's totally untrue; automake+libtool supports cross-compiling just fine (note that you can hardly use automake for libraries without using libtool, so saying that automake supports cross-compiling but libtool doesn't is a little strange). I use libtool for cross-compiling all the time. (e.g. on one of my projects we cross-compile shared libraries for Windows from GNU/Linux using mingw).

automake vs. GNU make

Posted Feb 7, 2008 22:57 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

I don't know much about the cross compiling support of autotools.
Do autotools support cross compiling for systems e.g. with 8 bit 
microcontrollers, which maybe don't have an operating system at all ?

Alex

automake vs. GNU make

Posted Feb 7, 2008 23:23 UTC (Thu) by nix (subscriber, #2304) [Link]

8-bit, I'm not sure. Things like arm-elf (without an OS), sure, and has 
for next to forever.

automake vs. GNU make

Posted Feb 11, 2008 1:43 UTC (Mon) by GreyWizard (guest, #1026) [Link]

Autotools support cross compilation in the abstract: to the extent that they work for one
cross compiling target they should work for any that the underlying compiler can support.  As
for that, GCC does support at least some 8-bit microcontrollers, as attested here:

    http://gcc.gnu.org/ml/gcc/2002-08/msg01809.html

automake vs. GNU make

Posted Feb 11, 2008 8:37 UTC (Mon) by aleXXX (subscriber, #2742) [Link]

What about compilers != gcc, e.g. sdcc ?

Alex

automake vs. GNU make

Posted Feb 11, 2008 20:01 UTC (Mon) by nix (subscriber, #2304) [Link]

Other compilers are generally supported. autoconf supports anything which 
runs as $CC and supports -g, -o, and -c with the customary meanings; 
automake supports anything with the caveat that dependency analysis may be 
inaccurate without compiler support (said support being a few lines of 
shell script in the depcomp script: if you need it for your compiler, 
please submit a patch!)

libtool's support is necessarily compiler-by-compiler and 
platform-by-platform, and so cannot cover everything (its entire purpose 
being to smooth out variation in the way different compilers and platforms 
create shared objects): nonetheless, on Linux, 1.5.26 supports GCC, KCC, 
Intel C++, the Portland Group's compiler, Compaq C++, and even Sun C, 
although who'd be using that on a Linux box I have no clue. (For a 
complete list of compiler * platform combinations, look in libtool.m4.)

(libtool obviously doesn't support sdcc, since as far as I can tell sdcc 
can't generate PIC code at all :) )

automake vs. GNU make

Posted Feb 7, 2008 0:37 UTC (Thu) by nix (subscriber, #2304) [Link]

The intent is to do it all for you with a suitably-smart include file, 
using nothing but GNU make language and portable shell within it (perhaps 
with some extensions on the make front).

I've done similar things in the past (replacing horrible proprietary build 
systems). It's doable.

LCA: Disintermediating distributions

Posted Feb 7, 2008 1:21 UTC (Thu) by stevenj (guest, #421) [Link]

By the way, don't get me wrong, the autotools are far from perfect. Life would be much much simpler for the autotools developers and users alike if they could rely on more than a small portable subset of sh and make being available on the build hosts. For one thing, it means that they have to use use three languages (sh, m4, and make) instead of one. For another thing, portable sh doesn't have functions so the generated configure scripts are huge and slow (launching lots of processes). For another thing, make is not integrated with the configure tests, so you can't parallelize the tests or run only those tests that are needed when something changes, and so on.

The problem is that valid, constructive criticisms of the autotools largely get lost in the noise of people who argue from ignorance of the real complexity of the problems that these tools have to address, and which aren't addressed by most of the suggested alternatives.

LCA: Disintermediating distributions

Posted Feb 7, 2008 7:57 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

> who argue from ignorance of the real complexity of the problems that
> these tools have to address, and which aren't addressed by most of the
> suggested alternatives. 

As maintainer of the KDE4 build system, which is able to build KDE4 on
-Linux
-FreeBSD
-Solaris
-Mac OSX (makefiles or XCode)
-Windows (mingw + GNU make, cl + nmake or Visual Studio, not cygwin)

I am well aware of the complexities and cmake handles them all.

Alex

LCA: Disintermediating distributions

Posted Feb 7, 2008 17:21 UTC (Thu) by stevenj (guest, #421) [Link]

Libtool handles many, many more Unix flavors than that. Yes, if you are willing to restrict yourself to recent versions of five platforms, it simplifies life a lot.

LCA: Disintermediating distributions

Posted Feb 7, 2008 19:09 UTC (Thu) by bronson (subscriber, #4806) [Link]

In theory.

In practice, I've spent many awful hours chasing libtool breakage on even the most common
platforms.

libtool adds extreme complexity to packages in an attempt to make them build everywhere.  More
software means more bugs.  The end result is preductable: libtool packages often fail to build
on even the most common platforms.

First you need to track down and install the exact version of automake used by the packager,
twiddle the configure file to remove syntax errors, try to figure out the magic combination of
undocumented --enable and --disable switches that will actually compile, and so on...  This
whole process is more abstract and opaque than editing Makefiles directly, but is it better?
Does it actually save time?

libtool is a bad abstraction.  Like CORBA, it tries to gloss over horrifyingly complex
problems but, instead of solving them, it seems to just push them around.  Perhaps I'm getting
philisophical, but I think they too hard to insulate the user from the real world.  D-Bus
doesn't try as hard as CORBA and, as a result, it's much more usable and reliable.  And a heck
of a lot smaller.  I feel like libtool could learn the same lesson.

I worked as a release engineer, so I admittedly had to wrestle with libtool a lot more than
the average person.  The missing ' in configure was the winner; finding that that took 7 hours
of binary searching an unreadable 350K shell script.  For comparison, it takes 3-4 hours to
learn how to use cmake from scratch!  libtool solves some problems, creates others, and the
end result seems to be a wash.

Anyhow, that's just my experience.  Not good.  Allow me to cast my vote for a simpler libtool
in the future.

LCA: Disintermediating distributions

Posted Feb 7, 2008 22:52 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

The platforms I listed are the platforms where I know that KDE4 
successfully builds.

I don't know if anybody already tried to build KDE4 on NetBSD, AIX, QNX, 
HP-UX, IRIX, BeOS, DragonFly, Hurd or any of the other platforms CMake 
supports. I'm quite sure the bigger problems in getting KDE4 work there 
will be to get all the required libraries working on these platforms 
(dbus, some multimedia lib like xine, a compatible X, etc.) and to get 
the compiler actually compile the C++ code without errors (some compilers 
don't accept everything what gcc accepts, some because they are older, 
some because they are more picky).

Alex

LCA: Disintermediating distributions

Posted Feb 10, 2008 20:40 UTC (Sun) by oak (subscriber, #2786) [Link]

My experience (mostly from about 10 years ago) has also been that 
Autotools (not just how they are used) are much larger problem for 
portability than the software they try to build.  Autotools scripts work 
only on platforms that Autotools officially support.  If you have just 
(GNU) make and shell like claimed above, Autotools falls down on its face, 
soils itself and cries for Mama.

It was much easier just to build GNU make and re-write the build scripts 
as cleaner Makefiles instead of trying to port first the huge mass of 
Autotools dependencies (Perl being one of the first/larger roadblocks) and 
then debug what other software the scripts need (after a long build fails) 
+ iterate that.

For me the solution seems obvious.  Solve the problem instead of kludging 
around it.

It seems insane/impossible to try to make the scripts autotools generate 
to projects portable to "everything".  Why not instead reduce Autotools 
script dependencies and make sure that those dependencies do everything 
needed and are portable to/buildable everywhere in addition to being 
standards compliant, small and of exemplary clean design?

The very small downside would be that then those couple of binaries need 
to be built for given platform before the much cleaned up, debuggable, 
faster, saner and otherwise better new autotools scripts can be run.  But 
as those programs are small & very portable, that should be trivial and 
including their binaries to CYGWIN and GNU coreutils would solve this for 
>90% of the developers.

As a result, all world's embedded developers would thank you!

LCA: Disintermediating distributions

Posted Feb 7, 2008 8:49 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

> Skilled programmers don't have a hard time learning a new syntax. 

Yes, true (still why do I need multiple languages just to build 
software ?).
Anyway, but I was talking about students starting to learn C or C++. They 
have enough problems with C/C++ itself. Under Linux you can 

-require them to learn basic Makefile syntax: ok as long as the software 
is very simple (write makefile, then run make)

-require them to learn autoconf+automake+shell+Makefile: this is 
unrealistic (write Makefile.am, then write configure.in, then run 
automake, then run autoconf, then run configure, then run make)

-require them to learn e.g. cmake: ok, also for slightly more complicated 
things, as e.g. building a shared library and installing it (write 
CMakeLists.txt, run cmake, then run make or hit F8 in kdevelop)

Alex

LCA: Disintermediating distributions

Posted Feb 7, 2008 17:28 UTC (Thu) by stevenj (guest, #421) [Link]

Students just learning C and C++ for the first time are not going to be able to write non-trivial software that builds on a dozen different systems without modification. Or perhaps I should say, they are probably not going to be able to write non-trivial software on their own. (Or complex projects with more than a handful of source files, or shared libraries, or mix languages, etcetera.) No matter what build system you use, they are going to need help with packaging, porting, and distribution, not to mention many other aspects of the software development.

In short, the case of students just learning how to program is simply not relevant for the autotools, because the autotools address problems that those students aren't even close to dealing with. Yes, such students should be perfectly fine with raw 'make', and shouldn't try for configure scripts.

LCA: Disintermediating distributions

Posted Feb 7, 2008 22:38 UTC (Thu) by aleXXX (subscriber, #2742) [Link]

I agree, but not completely.

Having students write makefiles like:

foo: main.c foo.c
    gcc -Wall -O2 main.c foo.c -o foo

is ok, splitting this into separate rules for compiling and linking may 
still be ok. Actually I think it is a good thing if they learn how 
makefiles work.
But this is only acceptable for very simple projects (maybe I have become 
lazy). The next step from that to autotools is huge IMO. The step to 
cmake is small:

add_library(foolib SHARED foo.c bar.c)
add_executable(hello main.c)
target_link_libraries(hello foolib)

install(TARGETS hello DESTINATION bin)
install(TARGETS foolib DESTINATION lib)
install(FILES foo.h DESTINATION include)

I really think this is doable. I'd say that this is understandable even 
without reading documentation. 
It works on all platforms, including the shared lib. It creates MSVC 
projects if you want it, or Eclipse projects if you want that. It builds 
also out-of-source. It gives you complete targets: all, foolib, hello, 
install, clean, help (!), foo.o, foo.s, foo.E (!).
The good thing is basic things are very simple with cmake, and building 
on that you can add functionality/required configure checks one by one 
(not students, but if somebody actually wants to have the software 
portable).

Alex

LCA: Disintermediating distributions

Posted Feb 6, 2008 23:38 UTC (Wed) by vapier (subscriber, #15768) [Link]

your statement on the availability of POSIX shells, while true, is irrelevant.  autotools does
not depend on a POSIX shell.  it is written (by design) in portable shell, not POSIX shell.
the difference is that portable shell works on all of those fun ancient systems that lack POSIX
features and/or have broken aspects to them.  that is why the generated code is so large.  so
that it works *everywhere* (barring the systems which lack shells).

scons in such a case is a pipe dream.  your system is too old to have a stable POSIX shell but
new enough to have python usable for scons ?  doubtful.

LCA: Disintermediating distributions

Posted Feb 7, 2008 0:36 UTC (Thu) by vmole (guest, #111) [Link]

Solaris /bin/sh is one of those ancient shells. Python works fine on Solaris. Or you can edit everything to look for /usr/xpg4/bin/ksh, if it's there. Have fun.

LCA: Disintermediating distributions

Posted Feb 7, 2008 20:30 UTC (Thu) by vapier (subscriber, #15768) [Link]

i dont need to edit anything when using autotools.  it's already aware of the Solaris
idiosyncrasy to store up-to-date utilities in /usr/xpg4/bin/.  with literally no effort on my
part or anyone building one of my projects on Solaris, autotools will locate that directory
and use those utilities.  fantastic ! :)

LCA: Disintermediating distributions

Posted Feb 7, 2008 11:27 UTC (Thu) by mdz@debian.org (subscriber, #14112) [Link]

> While Windows is the only major platform which doesn't have a POSIX 
> shell, it is on the other hand the platform which has like 80 or 90% 
> market share. So one could also say "only 10 to 20% of installed systems 
> have a POSIX shell".

On the third hand, there is a POSIX shell, and build tools based on it, available for
approximately 100% of these.  They aren't necessarily installed by default (many Linux systems
don't have a C compiler by default either these days), but developers are more than capable of
installing a software package for this purpose.