LWN: Comments on "LCA: Disintermediating distributions"

apt-get build-dep: better than sliced bread?

goaty — 2008-02-22T07:24:36+00:00

Have you ever tried apt-get build-dep on a fresh install that doesn't have any build tools
installed? It will pull in the entire build system: gcc, header files, everything, with one
command! With a fast disk and a fast network it is insanely pleasurable.

Kids growing up in the Age of APT may never realise that building stuff used to be *hard*.
It's lucky there are other operating systems around for them to use and understand what pain
means.

Partially distributed distributions

muwlgr — 2008-02-17T21:48:48+00:00

Of course you could build your own kernel, pack it into proper .deb file and install it by
dpkg. Just run "aptitude install kernel-package", then "man make-kpkg".

LCA: Disintermediating distributions

nix — 2008-02-15T22:00:47+00:00

With recent versions, thanks to the magic of top-level bootstrap, `make' 
should give you a compiler and libraries byte-for-byte identical to what 
you'd have got if you did the recompile-it dance. (Older versions wouldn't 
have recompiled libiberty with the new compiler before linking that 
compiler with it; top-level bootstrap has fixed that.)

LCA: Disintermediating distributions

ekj — 2008-02-15T11:51:11+00:00

The catch-22 of needing a make-program to compile gnu make isn't that much of a problem
really.

You need a C compiler to compile GCC too. If you want it self-compiled you need to compile it
twice:

First use whatever C-compiler you happen to have lying around to compile GCC. 

Then use your fresh gcc to compile gcc.

Bug fix cycle for most users is 100 x slower than it need be

dkite — 2008-02-15T02:28:39+00:00

I have always graduated towards rolling release distros, or in the case 
of KDE, rolling cvs/svn builds.

Derek

automake vs. GNU make

nix — 2008-02-11T20:01:37+00:00

Other compilers are generally supported. autoconf supports anything which 
runs as $CC and supports -g, -o, and -c with the customary meanings; 
automake supports anything with the caveat that dependency analysis may be 
inaccurate without compiler support (said support being a few lines of 
shell script in the depcomp script: if you need it for your compiler, 
please submit a patch!)

libtool's support is necessarily compiler-by-compiler and 
platform-by-platform, and so cannot cover everything (its entire purpose 
being to smooth out variation in the way different compilers and platforms 
create shared objects): nonetheless, on Linux, 1.5.26 supports GCC, KCC, 
Intel C++, the Portland Group's compiler, Compaq C++, and even Sun C, 
although who'd be using that on a Linux box I have no clue. (For a 
complete list of compiler * platform combinations, look in libtool.m4.)

(libtool obviously doesn't support sdcc, since as far as I can tell sdcc 
can't generate PIC code at all :) )

automake vs. GNU make

aleXXX — 2008-02-11T08:37:49+00:00

What about compilers != gcc, e.g. sdcc ?

Alex

automake vs. GNU make

GreyWizard — 2008-02-11T01:43:23+00:00

Autotools support cross compilation in the abstract: to the extent that they work for one
cross compiling target they should work for any that the underlying compiler can support.  As
for that, GCC does support at least some 8-bit microcontrollers, as attested here:

    http://gcc.gnu.org/ml/gcc/2002-08/msg01809.html

LCA: Disintermediating distributions

oak — 2008-02-10T20:40:06+00:00

My experience (mostly from about 10 years ago) has also been that 
Autotools (not just how they are used) are much larger problem for 
portability than the software they try to build.  Autotools scripts work 
only on platforms that Autotools officially support.  If you have just 
(GNU) make and shell like claimed above, Autotools falls down on its face, 
soils itself and cries for Mama.

It was much easier just to build GNU make and re-write the build scripts 
as cleaner Makefiles instead of trying to port first the huge mass of 
Autotools dependencies (Perl being one of the first/larger roadblocks) and 
then debug what other software the scripts need (after a long build fails) 
+ iterate that.

For me the solution seems obvious.  Solve the problem instead of kludging 
around it.

It seems insane/impossible to try to make the scripts autotools generate 
to projects portable to "everything".  Why not instead reduce Autotools 
script dependencies and make sure that those dependencies do everything 
needed and are portable to/buildable everywhere in addition to being 
standards compliant, small and of exemplary clean design?

The very small downside would be that then those couple of binaries need 
to be built for given platform before the much cleaned up, debuggable, 
faster, saner and otherwise better new autotools scripts can be run.  But 
as those programs are small & very portable, that should be trivial and 
including their binaries to CYGWIN and GNU coreutils would solve this for 
>90% of the developers.

As a result, all world's embedded developers would thank you!

Partially distributed distributions

Lennie — 2008-02-10T16:03:54+00:00

And I forgot to mention the apt-get build-dep command.

Which will install the build-dependencies of the package.

Partially distributed distributions

Lennie — 2008-02-10T15:55:28+00:00

Also you can use apt-pinning to get a newer version for Debian (Ubuntu ?) testing or even
experimental.

If you add the other 'version' of the distribution (like Debian testing) to your
/etc/apt/sources.list and specify which package you want to upgrade in /etc/apt/preferences
(that's the pinning-part).

It will automatically upgrade any dependencies as well.

This way you can test a newer version and even go back to the distrbution original version
from for example Debian-stable very easily.

If you created a patch for a package (if you did the apt-get source, dpkg-source -x;
dpkg-buildpackage), don't forget to change the version-number in the debian/control file, tag
your initial or organisation at the end of the existing version number with a 1 or 2, etc. so
you can upgrade the package later.

And other way to upgrade a package is to add only the source lines to your
/etc/apt/sources.list and do a apt-get source, dpkg-source, dpkg-buildpackage. You might need
to change the debian/control-file to specify a different version number as dependency or
install that first.

On upgrade (apt-get -u dist-upgrade) or even update of your distribution it will automatically
upgrade your packages, etc. if needed.

Partially distributed distributions

alextingle — 2008-02-09T14:49:18+00:00

$ apt-get source PACKAGE
$ sudo apt-get build-dep PACKAGE

Then, once you've made your change / applied the patch / whatever...

$ dpkg-buildpackage -rfakeroot -b

...and hey-presto, your very own fixed package.

Going upstream

JoeBuck — 2008-02-09T02:00:18+00:00

For relatively simple, self-contained programs that you invoke from a terminal, it's not hard to experiment with upstream versions. The key point is that the familiar (at least to developers) configure, make, make install sequence installs programs into /usr/local by default, which is an area that distros don't touch. Alternatively, you can built it in a special place, to make it easy to blow away whenever your distro fixes the problem.

So, if you want the upstream vim, download the source tarball, and do

tar jxf vim-7.1.tar.bz2
cd vim71
./configure --prefix=/usr/local/vim71
make
sudo make install

You now have built the upstream vim and installed it in /usr/local/vim71/bin, without disturbing your distro's vim.

(You will of course need to have the appropriate development tools installed).

Partially distributed distributions

tzafrir — 2008-02-09T01:59:55+00:00

'apt-get source' has been here for a while.

Debian also has now 'debcheckout', but this is more for the package maintainer's version
control system.

You can certainly build your own package and install it. apt considers also locally-installed
packages.

LCA: Disintermediating distributions

cortana — 2008-02-09T00:53:53+00:00

> All you Launchpad users, how can you bring yourselves to rely on a system 
> that can be taken away at any time?

Unfortunately, if I want to report bugs in Ubuntu, I have no other choice, since the Ubuntu
folks don't seem to give a damn about the (far superior) reportbug utility.

automake vs. GNU make

nix — 2008-02-08T23:18:38+00:00

I'd expect MacOS X support for modules to have been OK since

2003-03-20  Peter O'Gorman  <peter@pogma.com>

        * ltmain.in: Always use $echo not echo for consistency.
        Changes for darwin building. Warn if linking against libs linked
        with -module. Use module_cmds if available and building a module,
        move convenience double lib check,

What else is wrong?

And, yes, your point 2 is hard for libltdl to overcome: if you build the 
library as a lib, not a module, you'd have been stuck whatever libtool 
did.

Partially distributed distributions

magnus — 2008-02-08T22:03:30+00:00

I wish there was a "get source" command in the package manager, that would download the source
for a package configured exactly the same way as the installed package was configured. The
package would be flagged as "customized" in the package database. You could then experiment
with different patches, code changes etc and when you're done either revert back to the
original package or keep your custom changes. It could also support automatically generating
patches to send to the distributor or upstream (in case you managed to fix a bug), creating
custom binary packages etc. 

I think something like that would help a lot to reduce the barrier between users and
developers. Does any distribution already have such a system?

automake vs. GNU make

lovelace — 2008-02-08T21:46:05+00:00

Ah, now I'm remembering more about what the problems where.

1. Shared libraries and modules are the same on Linux (and lots of other Unices) but are
different on the Mac.  Libtool had a difficult time understanding this.
2. Until fairly recently (Tiger, iirc) shared libraries couldn't be easily dlopened on the
mac, only modules could.

Since KDE makes extensive use of dlopen-ing modules to accomplish things this made things
quite tricky and libtool wasn't really that much help.

So, yeah, quite a different runtime system.  Newer versions of OS X have gotten quite a bit
better on the dlopen-ing front, but they are still fundamentally different.  And, I wouldn't
even try to use libtool to create OS X frameworks....

automake vs. GNU make

nix — 2008-02-08T21:28:02+00:00

Yes, it could: but in libtool 1.5 it doesn't :(

I never said libtool didn't suck. It's just better than anything else 
around right now for the job it does, and it has a *lot* of hard-won 
knowledge of shared library wierdness on manifold systems encoded into it 
(as autoconf does of other cross-system variation).

automake vs. GNU make

nix — 2008-02-08T21:26:03+00:00

It reduces the complexity, but fundamentally these systems have different 
*runtime* semantics, which can't be entirely hidden. For simple uses 
(DT_NEEDED-style simple symbol lookup of libraries with no undefined 
symbols, and dlopen()/dlsym()/dlclose()-style dynamic loading), libtool 
does a good job. Anything more complex will hit trouble on one system or 
another.

Partially distributed distributions

JLCdjinn — 2008-02-08T18:44:11+00:00

Over the past two weeks, I have periodically encountered a crashing bug in VIM. I am running Kubuntu 7.10, and I run the update manager whenever it indicates that there is a pending update. vim --version tells me that the current binary includes patches 1-56. As of this writing, however, there are a total of 244 patches for VIM 7.1, and a bit of investigation indicates that a number of the remaining 188 patches fix crashing bugs; one of them could fix my bug. Naturally, I could compile, run, and test an upstream version, but, if it fixes the problem, I want to inject my upstream version into my system until I "return control" to the package manager. I would like to tell Kubuntu that I want to take full control of the VIM package, that I will be responsible for it, but I don't know how to do that.

This is a specific case of a general problem. There are many situations in which I would like to manually maintain the artifacts of a package on my system. I might want to do this in order to fine tune a package, such as the Linux kernel; when I want to test a bug fix, such as with KDE as described by Richard_J_Neill above; or when I want to actively develop a particular piece of software in the context of my overall system. Certainly, these cases are far from mutually exclusive. In order to do this cleanly, I need support from the distribution. I need a way to flip a switch in the package management system, which would tell the distribution that I will provide the dependencies filled by a particular package. I need the distribution to provide me with sufficient information that I can faithfully fill those dependencies. And then I need a way to relinquish control to the distribution, when I no longer need to closely control a particular package. It would be great if I could link any findings to both a distribution issue tracking system as well as any upstream issue tracking system.

To what degree do distributions provide this functionality now? How much documentation about this functionality exists? For example, on Debian- or Ubuntu-based distributions, how could I install a custom Linux kernel without getting in the way of the package management system? Having this ability might help us move in the direction suggested by Mr. Waugh, if I understand his intent.

automake vs. GNU make

lovelace — 2008-02-08T16:05:41+00:00

Hi Alex!

Yep, I'm aware of that, but that's not what I was referring to.  I was referring to the
autotools libtool.  Unfortunately, like I said, that was about 5 years ago and I cannot
remember the specifics.  So, rather than make more unsubstantiated accusations, I'll just
leave it at that.  I will mention, though, that that episode in particular cemented my general
dislike of the autotools libtool that still stands to this day.

LCA: Disintermediating distributions

holstein — 2008-02-08T14:23:35+00:00

What's even worse with Launchpad is how they keep it close source. 

What I understand of the move is the desire to bring launchpad.net as the center-point of
focus for the project they host. I suppose they probably fear seeing various project popping
up separate installation of launchpad (I recall reading something stating about that but I
can't find it back ATM).

We looked at "source-forge-like" tools at work some times ago ; most of them were not in a
shape that fit with what we wanted. But something like Launchpad would have been really great;
all the tools we needed, built-in with the concept of fetching upstream releases. But it's
closed. So, no bug fix from us...

Bug fix cycle for most users is 100 x slower than it need be

tzafrir — 2008-02-08T10:10:33+00:00

You can use Ubuntu Hardy right now. Or Debian Unstable. Or Fedora RawHide. OR whatever. But
most users are not able to cope with the nightly upgrades and weekly breakages.

It takes some debugging to get a stable distibution that will "just work" for a user.

Also: what do you mean by "must be built against the stable release"? A KDE 4 program should
be built against the KDE4 libraries in the stable version or from the nightly build? The libc
version from the stable or from the nightly? Xorg from the stable or from the nightly?

Bug fix cycle for most users is 100 x slower than it need be

Richard_J_Neill — 2008-02-08T07:48:34+00:00

There is one important thing that hasn't been mentioned so far, namely the speed of the
bug-fix cycle. Consider the majority of LWN readers; technically literate, able to file good
bug reports, and probably write some software of our own. Yet, we don't want to delve into
every single program. 

For example, I run Ubuntu, and I update every 6 months. I can't deal with all the
instabilities of running a devel version on my primary system. Let's say I find a bug in KDE,
which particularly annoys me.

(1) The KDE folks will quite reasonably expect me to test with the latest release before
filing the bug report. But I'm running a version somewhere between 3-9 months old, even though
I track the latest Ubuntu release. It's a major effort to re-build KDE, just to verify a
bug-report.

(2) If the bug gets fixed, (or has already been fixed), it will take an average of 3 months
before I get to evaluate, comment on, or benefit from the fix.

As a result:
  * The vast majority of technical users produce bug reports of limited value to upstream;
this is a huge waste of potential talent.
  * Motivation is limited, since most people don't get the benefit of the fixes for the bugs
that they report, in a timely manner.
  * Multiple people hit the same bug, even if it's fixed, because it doesn't get pushed
downstream. Again, a huge waste of time.
  * The oops-report-fix-enjoy cycle takes up to 9 months, instead of 3 days.

My suggestion is that distros should automatically backport every package on a nightly basis,
and provide binaries of the latest CVS, as well as the latest stable-release. These packages
must be built against the stable distro, and should be installable via the package manager.
(they should not be brought in by default though).

I also suggest that most end-users should file most of their bug reports upstream.
Distro-developers only have the resources to be dealing with bug-fixes that they can push out
to all users, eg security and application-crash bugs.

automake vs. GNU make

aleXXX — 2008-02-08T07:20:45+00:00

Ah, yes, indeed.
(I didn't work with autotools in the last years).

Then, couldn't the configure script handle that ? It runs on the build 
machine.

Alex

automake vs. GNU make

aleXXX — 2008-02-08T07:19:14+00:00

libtool on the Mac (a native binary for dealing with libraries on the 
Mac) is something different than the autotools libtool (portable shell 
script which should deal with shared libs on all systems).

Alex

automake vs. GNU make

lovelace — 2008-02-08T05:50:45+00:00

The makefiles CMake generates are not portable and can only be used on the system that they're generated on. If you move to a different system, you use CMake to generate makefiles for that system. That's how they can reliably create shared libraries on multiple systems using makefiles.

automake vs. GNU make

lovelace — 2008-02-08T05:42:42+00:00

Again, you're blaming the tool for the complexity of the underlying problem.

But, isn't this tool supposed to take care of the underlying complexity? So, haven't you completely invalidated your argument for it? It's supposed to make the task of creating libraries similar across dissimilar systems yet by your own description it does not.

I don't remember all the details now since it's been a while, but back when we were trying to port KDE 3.x to native Qt on the Mac libtool was a constant source of problems.

LCA: Disintermediating distributions

omez — 2008-02-08T02:09:17+00:00

I have 683 packages installed on my machine. If a distributor is not going to provide
dependency resolution information, I might spend three hours resoling the dependencies
manually for each package update. Optionally, I have (potentially) 683 different people who
don't communicate with each other providing incongruent dependency information. How is this a
good thing?

automake vs. GNU make

nix — 2008-02-08T00:44:53+00:00

automake can't generate the commands libtool executes because automake 
runs on the distributor's machine, not the builder's, and doesn't have a 
clue what sort of system the build will take place on.

automake vs. GNU make

stevenj — 2008-02-08T00:42:51+00:00

You haven't suggested any way for things to get better, you've just flamed a tool because you've encountered a couple of buggy packages that misused it, and it didn't correct for all the deficiencies of the developers. Then you tried to debug the problem yourself and failed because you didn't bother to read the fine manual.

Your suggestions, as far as I can tell, have been either for every developer to reinvent the wheel by rolling their own platform-dependent scripts, or for the developers to push the whole problem onto the end-users by giving them a raw Makefile and telling them to fix the compiler options themselves. Neither of these seems like an improvement.

automake vs. GNU make

vmole — 2008-02-08T00:23:23+00:00

You didn't read the manual, and as a result it failed on AIX because you asked libtool to support semantics unavailable on that platform

*I* didn't ask libtool to do anything. I'm the end-user. I'm not supposed to have know about libtool, or the variations in shared library implementations. Right? Isn't that the whole friggin point?

And if your response is "But there's too many differences, libtool can't hide everything", then we are in 100% agreement. Where we differ is whether or not it's worth the attempt, and whether libtool is the correct direction. So be it. But if you continue to ignore those of us who have problems, and blame the users, well, it will never get better.

automake vs. GNU make

stevenj — 2008-02-07T23:54:03+00:00

You didn't read the manual, and as a result it failed on AIX because you asked libtool to support semantics unavailable on that platform, and as a result you loudly complain here that libtool is broken and buggy. This is not a convincing critique of libtool.

libtool solves a big part of the problem: its manual specifies the lowest common denominator of shared library semantics, tells you how to indicate whether you obey those semantics, and builds the resulting library on every system that supports the semantics you request. The fact that you still have to know that there might be some differences between systems, and that you might have to read the manual to learn how to deal with these differences, is not a reason to throw it out and rewrite everything yourself from scratch (the only other "solution" you have suggested). No matter what portability tool you use, developers will still need to know something about the differences between platforms.

automake vs. GNU make

aleXXX — 2008-02-07T23:25:17+00:00

> [libtool is] apparently as hard to get right as using the actual OS
> tools would be, which would at least be debuggable by normal human
> beings.

"at least be debuggable" - very well put, I completely agree.
Why does automake actually need libtool at all ? I mean it generates the 
makefile code, it could as well just generate the code for calling the 
actual OS tools directly in the makefiles. This would remove this one 
layer of indirection.

Alex

automake vs. GNU make

nix — 2008-02-07T23:23:46+00:00

8-bit, I'm not sure. Things like arm-elf (without an OS), sure, and has 
for next to forever.

LCA: Disintermediating distributions

nix — 2008-02-07T23:22:32+00:00

autoconf-generated configure scripts also support the godsends which are 
site-config files. Nobody else seems to remember about that, which means 
you need to wrap another build system around your build system just to get 
your CFLAGS et al consistent.

LCA: Disintermediating distributions

nix — 2008-02-07T23:19:16+00:00

Yeah, like I said, it was really stupid of me not to upgrade. In fact I've 
*got* a more recent version installed: it's just this bloody old version 
in /usr/local/bin was hiding it... *sigh* chkdupexe time, I think.

automake vs. GNU make

nix — 2008-02-07T23:18:13+00:00

Your reality differs from my reality. I've had some cosmetic problems on 
Solaris 2.4 and 2.5.1 (we're talking prehistoric here) and I needed to 
install GNU sed on HP-UX and AIX, but other than that, nothing really.

(Mind you, the need for GNU sed *was* unacceptable --- it was also a bug.)

automake vs. GNU make

aleXXX — 2008-02-07T23:17:05+00:00

CMake knows how to build shared libs on all supported platforms (which 
support shared libs) with all supported toolchains (GNU, IBM, Sun, 
Borland, MS, Portland, Intel, HP and more). For several 
platforms/toolchains this is tested every night (unfortunately there 
aren't nightly tests for all supported combinations).
http://www.cmake.org/Testing/Dashboard/20080206-0100-Nigh...

Alex