LCA: Disintermediating distributions

It all started, says Jeff, some years ago, when he ran into Mark Shuttleworth fresh from a visit to Antarctica. Mark's pitch, says Jeff, "sounded like crack" at the time. By 2003 or so, it just didn't seem like there was a whole lot of room for a new distribution. But Mark had some interesting ideas, and Jeff signed on; the result, of course, was Ubuntu.

Ubuntu has clearly had some success, but, in some important ways, it has failed to work out - at least for Jeff. He found himself distracted by Ubuntu's lack of participation in Debian, from which it derived its product. There was a real tension between tracking Debian and tracking upstream projects more directly. Despite Jeff's insistence that Ubuntu should be tracking (and pushing updates into) Debian's unstable distribution, Ubuntu often chose to go with upstream, resulting in what is, in effect, a fork of the Debian distribution - in terms of both the technology and the community.

What Ubuntu was doing was taking upstream packages, modifying them, bringing in shiny new features, and generally looking for ways to differentiate itself from the other distributors. So, for example, the first Ubuntu release contained a great deal of Project Utopia work (aimed at making hardware "just work" with Linux) which had been done by developers from other distributions; Ubuntu shipped it first, though, and got a lot of credit for it. Novell's behind-closed-doors development of Xgl was motivated primarily by the wish to keep Ubuntu from shipping it first. Meanwhile, Red Hat had slowly learned that trying to differentiate itself by diverging from upstream was a path to pain. So Red Hat's developers created AIGLX, in an open, community oriented manner; the result is that AIGLX has proved to be the winning technology.

Events like these led Jeff to wonder about just where the integration of packages should be done - upstream or downstream? From Jeff's (GNOME-based) upstream point of view, he wonders why he doesn't have a direct relationship with his users. While most projects deliver their code through middlemen (distributors), there is an example of a project which has managed to maintain a much more direct relationship: Firefox. Most Firefox users are direct clients of the project - though most of them are Windows users. The Firefox trademark has been used to ensure that, even when distributors are involved, the upstream developers get a say in what is delivered to users.

So, what happens if you take out the middleman? It's instructive to look back at what life was like before there were distributors. It was, Jeff says, much like pigs playing in mud; perhaps they enjoyed it, but it was messy. There are, in fact, a lot of good things that distributors have done for us. You can get a fully integrated stack of software from one source, and the distributor acts, in a way, as the user's advocate toward the upstream project. We don't want to lose out on all that.

But, if one were to look at facilitating a more direct relationship between development project and their users, one would want to take advantage of a number of maturing technologies. These include:

• OpenID. Any process of distributing distributions must look at distributed identity, and OpenID is the way to do it.

• DOAP. "Sounds terrible" but it's a useful way of describing a project with XML. With a DOAP description, a user can find a project's mailing lists, bug tracker, source repository, etc.

• Atom. This is how projects can distribute information about what they are doing.

• XMPP. This is a Jabber-based message queueing and presence protocol. It can be used to more active publishing of information than Atom can do.

• Distributed revision control. Lots of functionality for integration between projects, and between upstream and downstream. Jeff sees git as a step backward, though; some of the other offerings, he thinks, have much better user interfaces.

Also important are the packaging efforts which are underway in a number of places. These include Fedora, which is "becoming competitive with Debian" as a community project. OpenSUSE has put together a build system which can create packages for a number of distributions. Debian has had a community build system for years; there is interest in Debian in going the next step, though - ideas like building packages directly from a distributed version control system. Ubuntu's Launchpad was "a spectacular vision," though the reality is "a bit of a snore"; it didn't achieve its goal of helping upstream and downstream work together.

Then there's Bugzilla, which is the "bug filing gauntlet" between projects and their users. The Debian bug tracking system has done a better job of facilitating bug reports by allowing them to be submitted by email. But most big projects are using Bugzilla. It would be much improved by using OpenID (so that users would not have to register to file bugs) and some sort of Atom-based feed which would make querying bugs easy.

If you take out the distribution, what do you replace it with? How do we achieve consistency? We need to create standards for how we interact with each other. And we can, in fact, be very good at consistency and standards when the need is clear. Good release management is a step toward that goal. GNOME once had very bad release management, but has pulled it together. Doing time-based releases was a hard sell, but few developers would want anything else now. Now GNOME release management just works.

Consistency in source management is needed. Once upon a time that was done through CVS, but CVS is no longer up to the job, and now every project is using a different distributed version control system. But, sooner or later, one of the competing projects will win out and "hopefully we'll have clarity again." Autotools and pkgconfig can also go a long way toward creating consistency between projects.

So, if we can push the available tools up into the upstream projects, those projects can get better at producing packages for distributions themselves. Once the tools (like bug trackers) can talk to each other, people will start making more use of them and network effects will take over. But, at the moment, the knowledge about integration remains at the distribution level.

Debian, Jeff thinks, is well placed to take on a project like this and push its integration knowledge upstream. While Debian has typically been ten years ahead of everybody else in its packaging and integration abilities, it currently has a "relevancy problem." Finding ways to help upstream projects support their users more directly while maintaining overall integration and consistency would be a perfect way for Debian to maintain its leadership in this area. That could change the game for everybody, bringing projects closer to their users and making us all "happy as pigs in mud."

> Once upon a time that was done through CVS, but CVS is no longer up to 
> the job, and now every project is using a different distributed version 
> control system. But, sooner or later, one of the competing projects 
> will win out and "hopefully we'll have clarity again." Autotools and
> pkgconfig can also go a long way toward creating consistency between
> projects.

He is right for cvs.
IMO obviously he isn't right for autotools.
Since some time a lot of projects are switching away from it, mainly to 
CMake (and Scons).
Autotools (beside from being really hard to learn for developers) have 
one real problem: they require a shell. Free software is becoming more 
and more portable, also to Windows. Typical Windows users/developers 
don't have a UNIX shell, so autotools don't work there (yes, there is 
cygwin and different levels of mingw with shell, but IMO these are all 
work-arounds). Look e.g. at Python, it has an autotools based build 
system and additionally to that it has MSVC project files. I don't know 
how e.g. Mozilla or Gimp or OpenOffice handle this, but having to 
maintain multiple independent buildsystems in parallel is obviously no 
good solution.

And, honestly, pkgconfig has a big design flaw. It stores information 
about how a library is to be used. But it stores it in a format which can 
be used just for one purpose: executing it and feeding the results 
directly as arguments into gcc. If the information would be available in 
some structured way, it could be actually processed and also used for 
other things (e.g. linking a library with pkgconfig information with 
MSVC, or getting the include dirs and definitions for use in an IDE so it 
can do proper autocompletion).

Alex

Pretty much. 

There are only two major platforms that realy matter nowadays.. and that's Windows and Linux. 

Every major Unix platform now has Linux compatability as a high priority and most Linux
developers consider cross-platform important, so it works out that OS X, FreeBSD, Solaris, and
friends can be essentially treated as 'Linux' with a little effort. Most of these sorts of
differences are abstracted away from common app developers by the GNU stuff.

The temptation is that by supporting Windows as a primary OS open source projects can
dramaticly increase their profile. They can attract media attention, get financing, and
hopefully attract developers. These are all good things.

The bad things about supporting, as I see it, Windows is that your going to massively increase
your support costs and make a number of very-easy-to-do things in Linux very hard because now
you have to do it the Windows way also.

There is one major project that I know of that has a Windows port, more or less, but refuses
to release it. This is Ardour, a significant audio production application for Linux. They
don't release it because they feel the costs of supporting Windows users with common audio
problems is to much for them to handle. (something like that, ardour folks forgive me if I am
wrong) If somebody that is a expert in Windows and is willing to spend all their time
supporting Ardour on it then they'd probably release it.

Other effects of Windows on Linux stuff I see is things like in popular high-level languages
they tend to shun forking and using sockets for IPC and concentrate on making dozens of
different high-level schemes for doing  shared memory IPC and multi-threading stuff. Most of
which has litle significant advantage over traditional 'posix-style' methods in a Linux-only
world, but matter dramaticly in the Windows one. (At least that was my impression.)

> Windows is now the only major platform that doesn't have a POSIXish
> shell; I'm somewhat suspicious of claims that the free software
> community should change its entire build toolchain because of one
> proprietary Microsoft operating system.

There are several points here.
While Windows is the only major platform which doesn't have a POSIX 
shell, it is on the other hand the platform which has like 80 or 90% 
market share. So one could also say "only 10 to 20% of installed systems 
have a POSIX shell".

I also have the impression that open source projects which are 
portable between UNIX and Windows are the most successful projects: 
Firefox, Thunderbird, OOo, gcc, Python, Ruby, Apache, MySQL, php, ...
Also if you are in a setting where you have Windows and UNIX users, 
software which runs on both systems has a significant advantage. So IMO 
it makes sense to support also Windows if you want to spread the use of 
free software and give people power over their data again.

Beside portability that there are enough other reasons to switch from 
autotools to something easier to use. I know there are gurus which 
understand autotools, but my impression is that the big majority of Linux 
developers doesn't really understand it. That's my personal experience in 
KDE and with other developers I know from real life. It's also hard to 
convince e.g. students who have a hard time learning C or C++ that they 
additionally have to learn autoconf, automake, libtool, shell, m4 and 
make syntax. It just doesn't make sense if the build tool is more 
complicated than the programming language itself.

About cygwin: I really like it, but I come from Linux background. While 
it doesn't take too long to install it, you end up with a UNIXish 
environment, which takes maybe around 100 MB or more on the harddisk and 
which feels really alien to Windows-only users/developers. And all that 
just to get a tool to generate build files ?

One of the nice things about cmake is that doesn't have any additional 
requirements like some libraries or scripting languages, you just install 
it and will work together with your native build tool (make, Xcode, 
Visual Studio).

Alex

We're in the process of switching over to CMake for the Second Life viewer, and from what
we've seen so far, it looks great.  When you have a lot of Windows and Mac programmers on your
team, it's a huge win for those people to be able to use the dev tools they are used to (e.g.
Visual Studio and XCode). The fact that CMake generates build files for those and a lot more
is a pretty unique characteristic among build systems.

the void you describe is largely addressed by things like the gnulib project.  it takes care
of all the OS-specific issues by checking to see if the function in question is usable on the
host system.  if it isnt, the gnulib version is provided.  thus the code that *you* write is
able to freely assume that function in question is available.

as for encouraging ifdef's, there are simply some things you cannot solve otherwise.  how do
you propose people integrate optional support for addon libraries ?  perhaps you can support a
wide range of graphic formats, but only if the external library is available ... yes, you can
write autotool code to largely avoid this (make most pieces standalone files that are
optionally compiled), but it doesnt make the issue magically go away due to the nature of the
code (assuming C/C++ here).

#ifdef HAVE_INDEX
p = index(s, c)
#elif HAVE_STRCHR
p = strchr(s, c)
#endif

Checking for C89...okay
Checking for stdio.h...found
Checking for strcpy()...found
...

> In my experience with developing many actual programs using autoconf,
> the checks in the configure script are a direct consequence of porting
> to some platform or another. ("Crap, MinGW doesn't support
> gettimeofday, we'll need to check for it and whatever alternative is
> available.")

Yes, basically the configure checks are necessary.
Autoconf itself wasn't so bad. But then additionally you have to 
understand how it works together with automake and libtool. This is what 
made it really hard for me.

And I'm sure this is also the reason why many programs run much more 
configure checks than they actually need. They take the autotools stuff 
from some existing project and just change it so that it can build their 
program. With this approach you end up with all the configure checks the 
original script had, and maybe later on you have to add some more. IMO it 
is simply too hard to build a simple program using autotools. Who wants 
to learn that just to build hello world ?

Using other build systems, as e.g. cmake, makes this trivial:

add_program(helloworld hello.c)

That's all, cmake will do a few checks to find the compiler, check that 
it actually works and figure out if it's a 32 or 64 bit compiler, and 
that's all.
I guess for Scons it's similar.

Also autotools don't doesn't push you to use some modular style, at least 
in KDE we ended up with a few huge scripts which did a lot of magic. E.g. 
CMake strongly encourages a modular system (while it is of course still 
possible to throw everything into one file, but then you intentionally 
work around its features).

Alex

the issues you raise i dont really see being "solved" or even really addressed any differently
with other build systems (cmake/scons/whatever).  a build system cannot really compensate for
a coder's inability to write clean code.  what do you see in other build systems that
encourages different style of coding ?

the example you cite could easily be relegated to a header file without making the real code
messy.  while it is an older example, there are many similar situations that could generally
be solved the same way: take care of OS-differences in one location (sep source or header
file) and keep everything else clean.

What 'underlying deficiencies of make' are you talking about?  I don't see any such thing.

autotools allows you to create a build system that will work on any system that has a shell
and make.  that make may not be GNU make.  it may have bugs.  its feature set may be severely
limited compared to what you're used to.  many projects out there who set out to write their
own build system do so using GNU make which means they write code that Works For Them.
non-portable GNU-isms creep in which means the build system is no longer portable to many
targets which sort of defeats the original intent: making it portable.

Well, I know Paul D. Smith (the GNU make author) has said that in the 
past, and I agree with him (for what very little it's worth).

(Although, to be fair, it's not like they could require GNU make to build.  =)

Although GNU make does come with an automake environment, it also provides a shell script that
can be used to build make.  Obviously this will recompile and relink everything every time you
run it, but GNU make is not such a huge program that this is a problem.  And once you've got
it built once, you can use that make for subsequent builds.  Having this avoids the catch-22
of needing some make to build make.

That said, automake is awesome especially if you're developing highly portable tools, which
most of the GNU tools are.  For GNU make I don't so much care about the portability aspects,
although that's nice too (but the shell script above would be enough of an "out" for GNU make
itself).  The great thing about automake is all the default rules it provides, including
things like distcheck for building new packages, etc.  These rules save huge amounts of time
and effort for package developers/maintainers.

The catch-22 of needing a make-program to compile gnu make isn't that much of a problem
really.

You need a C compiler to compile GCC too. If you want it self-compiled you need to compile it
twice:

First use whatever C-compiler you happen to have lying around to compile GCC. 

Then use your fresh gcc to compile gcc. 

With recent versions, thanks to the magic of top-level bootstrap, `make' 
should give you a compiler and libraries byte-for-byte identical to what 
you'd have got if you did the recompile-it dance. (Older versions wouldn't 
have recompiled libiberty with the new compiler before linking that 
compiler with it; top-level bootstrap has fixed that.)

it's really a mindset.  do you tell every user their system sucks and they should install
Linux and up-to-date utilities ?  or do you use automake and everyone out there can build your
project without a problem ?  some people will choose the former while others will pick the
latter.  in the latter case, automake is the only realistic solution.

        $(MAKE) -C subdir

        cd subdir && $(MAKE)

Have a look at the article linked from the parent post.  Calling one 'make' from another
makefile tends to lead to all sorts of crufty problems.  The paper gives examples, and my own
experience certainly confirms it.  That said, you can use make in a large project, just not by
recursive invocation.

> Try using raw 'make' in a project with subdirectories sometime. 

Then you have already failed. Why on earth would anyone want to call make recursively? The
only reason I have ever heard is "because I don't understand what I'm doing, and I cargo
culted my build system from someone else, who also did not understand what he was doing".

Side note: I'm enjoying this discussion, really nice comments.

I agree with you, stevenj, except that I think that GNU make is now sufficient: there is no
need for automake. The creator of automake thinks so, too: http://tromey.com/blog/?p=394

bin_PROGRAMS = hello
hello_SOURCES = hello.c hello.h

I think you've got things backwards: the -no-undefined flag doesn't really make sense _unless_
shared library dependencies exist.

With the -no-undefined flag set, your library must directly link to all the libraries whose
symbols you used (leaving no "undefined" symbols).

Even on platforms where -no-undefined is not required, it is usually worth using since the
linker can see which library each referenced symbol comes from, making the job of runtime
linking easier.

Your reality differs from my reality. I've had some cosmetic problems on 
Solaris 2.4 and 2.5.1 (we're talking prehistoric here) and I needed to 
install GNU sed on HP-UX and AIX, but other than that, nothing really.

(Mind you, the need for GNU sed *was* unacceptable --- it was also a bug.)

> [libtool is] apparently as hard to get right as using the actual OS
> tools would be, which would at least be debuggable by normal human
> beings.

"at least be debuggable" - very well put, I completely agree.
Why does automake actually need libtool at all ? I mean it generates the 
makefile code, it could as well just generate the code for calling the 
actual OS tools directly in the makefiles. This would remove this one 
layer of indirection.

Alex

automake can't generate the commands libtool executes because automake 
runs on the distributor's machine, not the builder's, and doesn't have a 
clue what sort of system the build will take place on.

Ah, yes, indeed.
(I didn't work with autotools in the last years).

Then, couldn't the configure script handle that ? It runs on the build 
machine.

Alex

Yes, it could: but in libtool 1.5 it doesn't :(

I never said libtool didn't suck. It's just better than anything else 
around right now for the job it does, and it has a *lot* of hard-won 
knowledge of shared library wierdness on manifold systems encoded into it 
(as autoconf does of other cross-system variation).

libtool on the Mac (a native binary for dealing with libraries on the 
Mac) is something different than the autotools libtool (portable shell 
script which should deal with shared libs on all systems).

Alex

Hi Alex!

Yep, I'm aware of that, but that's not what I was referring to.  I was referring to the
autotools libtool.  Unfortunately, like I said, that was about 5 years ago and I cannot
remember the specifics.  So, rather than make more unsubstantiated accusations, I'll just
leave it at that.  I will mention, though, that that episode in particular cemented my general
dislike of the autotools libtool that still stands to this day.

It reduces the complexity, but fundamentally these systems have different 
*runtime* semantics, which can't be entirely hidden. For simple uses 
(DT_NEEDED-style simple symbol lookup of libraries with no undefined 
symbols, and dlopen()/dlsym()/dlclose()-style dynamic loading), libtool 
does a good job. Anything more complex will hit trouble on one system or 
another.

Ah, now I'm remembering more about what the problems where.

1. Shared libraries and modules are the same on Linux (and lots of other Unices) but are
different on the Mac.  Libtool had a difficult time understanding this.
2. Until fairly recently (Tiger, iirc) shared libraries couldn't be easily dlopened on the
mac, only modules could.

Since KDE makes extensive use of dlopen-ing modules to accomplish things this made things
quite tricky and libtool wasn't really that much help.

So, yeah, quite a different runtime system.  Newer versions of OS X have gotten quite a bit
better on the dlopen-ing front, but they are still fundamentally different.  And, I wouldn't
even try to use libtool to create OS X frameworks....

I'd expect MacOS X support for modules to have been OK since

2003-03-20  Peter O'Gorman  <peter@pogma.com>

        * ltmain.in: Always use $echo not echo for consistency.
        Changes for darwin building. Warn if linking against libs linked
        with -module. Use module_cmds if available and building a module,
        move convenience double lib check,

What else is wrong?

And, yes, your point 2 is hard for libltdl to overcome: if you build the 
library as a lib, not a module, you'd have been stuck whatever libtool 
did.

I've also used his include-based Makefile structure on projects that were originally written
using recursive make.  The Makefile's got smaller, simpler, faster, and more reliable.
Hierarchy still takes *some* thought, but it really wasn't difficult.

>For example, if you have a source file with the same name in two 
> subdirectories (e.g. main.c), do you have to jump through hoops to 
> prevent it from getting confused?

The way I do it is always reference the tree from the build root.

so a/main.c is never looks like b/main.c

and yes I can start a build from the subdirs etc.

non-recursive is the way to go. 

Just require gnu make don't try supporting version 8.79 (rh9) clients just 
make people upgrade....

Files (source or derived) in separate directories are *always* in separate namespaces, in
Miller's approach.  The 'make' is run once, in the root directory, and all sources and targets
are described from that point of view.  So you don't build 'main.c', you build 'foo/main.c'.
You use target-specific overrides if you need to customize things, e.g.:

   # in foo/include.mk:
   $(FOO_OBJ): CPPFLAGS += -Ifoo

Now, if you wanted a selective clean (for one subdir), I agree there isn't a good way to do
this in the include-based structure (other than calling it something else, e.g. 'foo/clean').
Of course, you could also include a local Makefile in the subdir just to make things more
familiar:

   # in foo/Makefile:
   clean: 
       cd ..; $(MAKE) foo/clean

Did that help, or would a more complete example be clearer?

CURMOD:=$(shell pwd | sed -e 's,.*/src/\(.*\),\1,')
all :
        @cd $${PWD%$(CURMOD)} && $(MAKE) --no-print-directory $(CURMOD)

% :: FORCE
        @cd $${PWD%$(CURMOD)} && $(MAKE) --no-print-directory $(CURMOD)/$@

FORCE:

That isn't really necessary for variable names (that you don't need to reference later), only
the source and targets need to be qualified.  A fairly typical example:

# Define sources for this component:
MY_SRC = $(wildcard foo/*.c)
MY_OBJ = $(MY_SRC:.c=.o)
MY_TGT = foo/libfoo.a

# Additional flags and dependencies for building this component:
$(MY_OBJ): CPPFLAGS += -I foo
$(MY_TGT): $(MY_OBJ)

# Add these to the master Makefile's lists:
SOURCE += $(MY_SRC)
TARGET += $(MY_TGT)
CLEAN  += $(MY_OBJ)

The next component can reuse MY_SRC etc.  You wouldn't even need these variables, if you had
no need for target-specific variables or dependencies.  (But of course, you usually do).  If
you wanted to keep a reference to 'foo/libfoo.a', for use in other Makefiles, you might
replace MY_TGT with LIBFOO.

In short, you only need unique names for global variables; local ones can be overwritten.

You do need the component path ('foo' in this example) in a few spots, but it is trivial to
change that if the directory structure changes.

> But what is the alternative if you want to build shared libraries
> portably?

I don't know about Scons, but you can use CMake, and it will just work:

add_library(foo SHARED foo.c bar.c)

This will give you depending what you need:
-UNIX Makefiles (not quite sure if it requires GNU make, I don't think 
so)
-MS nmake makefiles
-Borland Makefiles
-Watcom Makefiles
-cygwin Makefiles
-XCode projects
-Visual Studio >= 6 projects.

I wondered for years why suddenly the comopiler can't be called directly 
anymore, to find out that it's just libtool and you can use the compiler 
very well directly (which removes one layer of indirection -> makes it 
easier to understand).

Alex

CMake knows how to build shared libs on all supported platforms (which 
support shared libs) with all supported toolchains (GNU, IBM, Sun, 
Borland, MS, Portland, Intel, HP and more). For several 
platforms/toolchains this is tested every night (unfortunately there 
aren't nightly tests for all supported combinations).
http://www.cmake.org/Testing/Dashboard/20080206-0100-Nigh...

Alex

libtool isn't so bad except for one thing, which makes it unusable.  No support for
cross-compilation.  It's so utterly lame that every part of the GNU build environment
(binutils, gcc, autoconf, automake) is very cognizant of cross-compilation and works very,
very hard to make setting it up and using it as simple as possible... except libtool.

Whenever I see a package come through that uses libtool I absolutely cringe because I know it
will be a huge PITA.

I don't know much about the cross compiling support of autotools.
Do autotools support cross compiling for systems e.g. with 8 bit 
microcontrollers, which maybe don't have an operating system at all ?

Alex

8-bit, I'm not sure. Things like arm-elf (without an OS), sure, and has 
for next to forever.

Autotools support cross compilation in the abstract: to the extent that they work for one
cross compiling target they should work for any that the underlying compiler can support.  As
for that, GCC does support at least some 8-bit microcontrollers, as attested here:

    http://gcc.gnu.org/ml/gcc/2002-08/msg01809.html

What about compilers != gcc, e.g. sdcc ?

Alex

Other compilers are generally supported. autoconf supports anything which 
runs as $CC and supports -g, -o, and -c with the customary meanings; 
automake supports anything with the caveat that dependency analysis may be 
inaccurate without compiler support (said support being a few lines of 
shell script in the depcomp script: if you need it for your compiler, 
please submit a patch!)

libtool's support is necessarily compiler-by-compiler and 
platform-by-platform, and so cannot cover everything (its entire purpose 
being to smooth out variation in the way different compilers and platforms 
create shared objects): nonetheless, on Linux, 1.5.26 supports GCC, KCC, 
Intel C++, the Portland Group's compiler, Compaq C++, and even Sun C, 
although who'd be using that on a Linux box I have no clue. (For a 
complete list of compiler * platform combinations, look in libtool.m4.)

(libtool obviously doesn't support sdcc, since as far as I can tell sdcc 
can't generate PIC code at all :) )

The intent is to do it all for you with a suitably-smart include file, 
using nothing but GNU make language and portable shell within it (perhaps 
with some extensions on the make front).

I've done similar things in the past (replacing horrible proprietary build 
systems). It's doable.

> who argue from ignorance of the real complexity of the problems that
> these tools have to address, and which aren't addressed by most of the
> suggested alternatives. 

As maintainer of the KDE4 build system, which is able to build KDE4 on
-Linux
-FreeBSD
-Solaris
-Mac OSX (makefiles or XCode)
-Windows (mingw + GNU make, cl + nmake or Visual Studio, not cygwin)

I am well aware of the complexities and cmake handles them all.

Alex

In theory.

In practice, I've spent many awful hours chasing libtool breakage on even the most common
platforms.

libtool adds extreme complexity to packages in an attempt to make them build everywhere.  More
software means more bugs.  The end result is preductable: libtool packages often fail to build
on even the most common platforms.

First you need to track down and install the exact version of automake used by the packager,
twiddle the configure file to remove syntax errors, try to figure out the magic combination of
undocumented --enable and --disable switches that will actually compile, and so on...  This
whole process is more abstract and opaque than editing Makefiles directly, but is it better?
Does it actually save time?

libtool is a bad abstraction.  Like CORBA, it tries to gloss over horrifyingly complex
problems but, instead of solving them, it seems to just push them around.  Perhaps I'm getting
philisophical, but I think they too hard to insulate the user from the real world.  D-Bus
doesn't try as hard as CORBA and, as a result, it's much more usable and reliable.  And a heck
of a lot smaller.  I feel like libtool could learn the same lesson.

I worked as a release engineer, so I admittedly had to wrestle with libtool a lot more than
the average person.  The missing ' in configure was the winner; finding that that took 7 hours
of binary searching an unreadable 350K shell script.  For comparison, it takes 3-4 hours to
learn how to use cmake from scratch!  libtool solves some problems, creates others, and the
end result seems to be a wash.

Anyhow, that's just my experience.  Not good.  Allow me to cast my vote for a simpler libtool
in the future.

The platforms I listed are the platforms where I know that KDE4 
successfully builds.

I don't know if anybody already tried to build KDE4 on NetBSD, AIX, QNX, 
HP-UX, IRIX, BeOS, DragonFly, Hurd or any of the other platforms CMake 
supports. I'm quite sure the bigger problems in getting KDE4 work there 
will be to get all the required libraries working on these platforms 
(dbus, some multimedia lib like xine, a compatible X, etc.) and to get 
the compiler actually compile the C++ code without errors (some compilers 
don't accept everything what gcc accepts, some because they are older, 
some because they are more picky).

Alex

My experience (mostly from about 10 years ago) has also been that 
Autotools (not just how they are used) are much larger problem for 
portability than the software they try to build.  Autotools scripts work 
only on platforms that Autotools officially support.  If you have just 
(GNU) make and shell like claimed above, Autotools falls down on its face, 
soils itself and cries for Mama.

It was much easier just to build GNU make and re-write the build scripts 
as cleaner Makefiles instead of trying to port first the huge mass of 
Autotools dependencies (Perl being one of the first/larger roadblocks) and 
then debug what other software the scripts need (after a long build fails) 
+ iterate that.

For me the solution seems obvious.  Solve the problem instead of kludging 
around it.

It seems insane/impossible to try to make the scripts autotools generate 
to projects portable to "everything".  Why not instead reduce Autotools 
script dependencies and make sure that those dependencies do everything 
needed and are portable to/buildable everywhere in addition to being 
standards compliant, small and of exemplary clean design?

The very small downside would be that then those couple of binaries need 
to be built for given platform before the much cleaned up, debuggable, 
faster, saner and otherwise better new autotools scripts can be run.  But 
as those programs are small & very portable, that should be trivial and 
including their binaries to CYGWIN and GNU coreutils would solve this for 
>90% of the developers.

As a result, all world's embedded developers would thank you!

> Skilled programmers don't have a hard time learning a new syntax. 

Yes, true (still why do I need multiple languages just to build 
software ?).
Anyway, but I was talking about students starting to learn C or C++. They 
have enough problems with C/C++ itself. Under Linux you can 

-require them to learn basic Makefile syntax: ok as long as the software 
is very simple (write makefile, then run make)

-require them to learn autoconf+automake+shell+Makefile: this is 
unrealistic (write Makefile.am, then write configure.in, then run 
automake, then run autoconf, then run configure, then run make)

-require them to learn e.g. cmake: ok, also for slightly more complicated 
things, as e.g. building a shared library and installing it (write 
CMakeLists.txt, run cmake, then run make or hit F8 in kdevelop)

Alex

I agree, but not completely.

Having students write makefiles like:

foo: main.c foo.c
    gcc -Wall -O2 main.c foo.c -o foo

is ok, splitting this into separate rules for compiling and linking may 
still be ok. Actually I think it is a good thing if they learn how 
makefiles work.
But this is only acceptable for very simple projects (maybe I have become 
lazy). The next step from that to autotools is huge IMO. The step to 
cmake is small:

add_library(foolib SHARED foo.c bar.c)
add_executable(hello main.c)
target_link_libraries(hello foolib)

install(TARGETS hello DESTINATION bin)
install(TARGETS foolib DESTINATION lib)
install(FILES foo.h DESTINATION include)

I really think this is doable. I'd say that this is understandable even 
without reading documentation. 
It works on all platforms, including the shared lib. It creates MSVC 
projects if you want it, or Eclipse projects if you want that. It builds 
also out-of-source. It gives you complete targets: all, foolib, hello, 
install, clean, help (!), foo.o, foo.s, foo.E (!).
The good thing is basic things are very simple with cmake, and building 
on that you can add functionality/required configure checks one by one 
(not students, but if somebody actually wants to have the software 
portable).

Alex

your statement on the availability of POSIX shells, while true, is irrelevant.  autotools does
not depend on a POSIX shell.  it is written (by design) in portable shell, not POSIX shell.
the difference is that portable shell works on all of those fun ancient systems that lack POSIX
features and/or have broken aspects to them.  that is why the generated code is so large.  so
that it works *everywhere* (barring the systems which lack shells).

scons in such a case is a pipe dream.  your system is too old to have a stable POSIX shell but
new enough to have python usable for scons ?  doubtful.

i dont need to edit anything when using autotools.  it's already aware of the Solaris
idiosyncrasy to store up-to-date utilities in /usr/xpg4/bin/.  with literally no effort on my
part or anyone building one of my projects on Solaris, autotools will locate that directory
and use those utilities.  fantastic ! :)

> While Windows is the only major platform which doesn't have a POSIX 
> shell, it is on the other hand the platform which has like 80 or 90% 
> market share. So one could also say "only 10 to 20% of installed systems 
> have a POSIX shell".

On the third hand, there is a POSIX shell, and build tools based on it, available for
approximately 100% of these.  They aren't necessarily installed by default (many Linux systems
don't have a C compiler by default either these days), but developers are more than capable of
installing a software package for this purpose.

The fun thing of autotools is that you don't need perl, python, cmake, scons or any additional
package (ignoring the Microsoft mess for a moment), but only a handful of sh tools which most
unix-style OSes have.

Yes, you just need bash...  a bunch of core utils...  and one heck of a lot of time and
specialized knowledge.

Autotools is a nightmare for any non-trivial task (speaking as an ex-release-engineer).  How
many different autoconfs do they ship with Debian now?  Three?  Four?  How much time has been
lost to otherwise rational people being forced to wade through impenetrable shell scripts?

I really look forward to the day that standardization comes to packaging.  I doubt that it
will involve autotools though!

hardly correct.  nothing in autotools depends on bash, and the "core utils" you refer to is a
pretty small list that any and all *nix system provides (awk/grep/sed/etc...).

anyone doing serious work does not care about older versions of autotools (there are actually
only 2 major versions of autoconf), which means you really have to work with the latest
release.

autotools does provide a consistent interface for distribution maintainers to utilize and
package up things.  in fact, most autotool based packages can trivially be handled with the
same template build code.

> anyone doing serious work does not care about older versions of autotools

You ever try to compile an older package with a newer version of autotools?  Failure city.
There's a reason Debian is forced to include so many version of autotools....  and, I assure
you, they are doing serious work.

The information *is* available in a structured way: search PKG_CONFIG_PATH 
and open the file named ${package}.pc therein (or in /usr/lib/pkgconfig if 
PKG_CONFIG_PATH is not set) and just parse the file directly. Its format 
is *really* not that complicated. I've parsed it in a few dozen lines of 
shell...

too bad cmake and scons continue to re-solve the same problems autotools solved eons ago
(cross-compiling / tool discovery / custom flags).  scons in particular is just terrible and
not in any sort of usable shape for serious projects.

So why are so many projects, which already have "working" autoconf build 
systems, doing all that "useless work" to switch to CMake? The next thing 
that happens is most of them seem to express great satisfaction and 
relief with the result. Is this just self-delusion to justify the work?

Sometimes. 

Like the migration lots of projects have done from going from CVS to SVN to Git or whatever
else.

When mucking around with a program have you ever noticed that once you program something out,
but find some reason to rewrite it, that it turns out to be better then your original version?
Ya sure you could of spent that time bugfixing the old code or hacking new features into it,
but your going to be almost certain that the new code is going to make your job of maintaining
and improving it just that much better.

I have a feeling that if many of those projects just stripped out all the stuff they used make
(or autotools or whatever) for and then reimplimented it from scratch then they would of been
nearly as happy with it.

Also projects that are ho-hum about converting to cmake are not all of a sudden going to turn
around and broadcast to the world that they spent a great deal of their time on something that
ended up not mattering a whole lot. It's not like they are going to end up being examples
while other projects just love it.

Sure, a simple project works great with any of several build strategies, 
so of course not everyone will be thrilled by switching. But if CMake 
only lacks the "I got burned by #!@%*& CMake" contingent, it's already 
ahead of the competition.

My *eyes* got burned by cmake's language.

Haven't they learned that capital letters make things *harder* to read? Have we learned
nothing since the days when Lisp was written in capitals?

Since CMake 2.4.3, released July 2006 (or around that version) the 
commands can be written lowercase, with the coming version 2.6 this is 
even the preferred style (i.e. which is used in the documentation).

Alex

YES! (Time to upgrade. Is my cmake really that old?

... 2.4.1. dammit.)

2.4.1 was a beta version, a lots of bugs were fixed for 2.4.3. Version 
2.4.8 has been released a few weeks ago, I recommend you use this. If 
there is no package for your distro, just download the binary package 
from www.cmake.org and just unpack it in some place you like, it will 
work.

Alex

Yeah, like I said, it was really stupid of me not to upgrade. In fact I've 
*got* a more recent version installed: it's just this bloody old version 
in /usr/local/bin was hiding it... *sigh* chkdupexe time, I think.

i didnt say it wasnt working *for them*.  they probably wouldnt have made the build system
change if it wasnt working *for them*.  the trouble is when *anyone else* tries to build the
package.  you want to cross-compile it ?  build it on a different platform ?  build with your
own compiler/flags ?  sorry, but the $flavor-of-the-month build system never thought of that.
time to go re-implement the wheel even though autotools already had it solved.

The worst I've found for this is Boost.Jam. It should *not* take 7Kb of 
diffs and a kilobyte of build-system shell scripting to do the equivalent 
of setting --prefix and CXXFLAGS when building Boost!

I'm not sure I get your point here, so I just state what cmake offers 
here:

CMake cvs (2.6.0 will be released soon) supports cross compiling (without 
scratchbox or any other emulators, but of course it can also be used 
inside scratchbox).

If you want to use your own CFLAGS/CXXFLAGS with CMake, you have at least 
two ways to do it:
set CFLAGS/CXXFLAGS when you run cmake, cmake will use them.
Or, later on, run "make edit_cache" and edit the 
CMAKE_C_FLAGS/CMAKE_CXX_FLAGS directly to what you want.

If you build the software on some system where it has never been built 
before it may very well be necessary that you have to do something on the 
buildsystem, add some more checks, add some other locations, other names 
for libraries (e.g. z lib has a lot of different names on Windows). I 
guess this is true for any buildsystem.

Alex

your comment backs up my point.  all of these alternative build systems consistently re-solve
the exact same problems that autotools solved eons ago.

cmake *just* added support for cross-compiling (and it isnt even in any released version) ?
without even looking at anything else about cmake, that tells me the project is useless to me.
i'm not saying my needs are the same as everyone out there, just that you cant champion a
replacement for autotools if it isnt a replacement.  i'm glad *you've* found it useful, but if
your target compiling audience is more than just you, then i feel sorry for those poor chaps
(where chaps != you).

the things i cite are just common examples ive come across quite frequently when dealing with
non-autotooled packages as a distribution maintainer (whether it be cmake or scons or hand
rolled or whatever).  they certainly a complete list.  i imagine there are numerous other
portability fixes autotools has which these "replacements" lack.

autoconf-generated configure scripts also support the godsends which are 
site-config files. Nobody else seems to remember about that, which means 
you need to wrap another build system around your build system just to get 
your CFLAGS et al consistent.

>Doing time-based releases was a hard sell, but few developers would want anything else now.
Now GNOME release management just works. 

So, maybe the pain isn't in the ad-hoc vs. scheduled releases, but rather the transition
between management styles.

So the Mozilla foundation, OpenOffice.org and the Mono project should handle all the
integration issues on their own?

The Mozilla distribution is a really good (bad?) example that shows how many wheels a vendor
needs to reinvent:

* A build far for many platforms (and they're always too far behind on Linux)
* Independent bug tracking system
* flawed integration (because the project needs to be self contained)

We will end up with a system made of klik packages.

A few other good programs whose author wanted to have the final say on their usage is qmail
and the rest of the djb programs. Great programs. But never got properly integrated.

No, absolutely not -- experiences such as these were merely jumping-off points for the
philosophical challenge (particularly for someone such as myself, who deeply values the
differentiation that the existence of distributions provides us against our proprietary
competitors).

Note that the thought experiment was not about "making packages for distributions", but "what
would happen if we take distributions out of the picture?". I think that misunderstanding is
the basis for your response.

If your conclusion is that distributions can't be taken out of the picture, then you're not
taking part in the thought experiment. That's okay, but I'm more interested in the thought
experiment than defending the status quo (which I can do very well for myself, but find it
unchallenging and lacking in philosophical value). :-)

The "experiment" is ongoing. You can always install packages directly and not through a
distribution. 

The fact is that most of us use a proper distribution and don't build Linux from scratch. 

Binary packages provided by third parties to Linux are known to be of lower quality (to say
the least). And if you want to completely eliminate distributions, you won't really have
reference platforms, and binary packages may be even more of a pain.

... but the key to this is figuring out what could be better, rather than actively choosing
the worst outcome. :-)

> Once the tools (like bug trackers) can talk to each other...

Amen brother!  I keep hoping that Trac will sprout this feature.  Why can't I run my own
personal Trac which just keeps track of all the bugs I've filed upstream?  Alas, Trac seems to
be stumbling over its 0.11 release at the moment.  Just not enough contributors I guess.

Just think if Canonical had decided to modularize and help Trac instead of putting untold
man-years into creating their do-everything cathedral named Launchpad?

Sigh.  Oh well.  Guess I'll keep hoping!

What's even worse with Launchpad is how they keep it close source. 

What I understand of the move is the desire to bring launchpad.net as the center-point of
focus for the project they host. I suppose they probably fear seeing various project popping
up separate installation of launchpad (I recall reading something stating about that but I
can't find it back ATM).

We looked at "source-forge-like" tools at work some times ago ; most of them were not in a
shape that fit with what we wanted. But something like Launchpad would have been really great;
all the tools we needed, built-in with the concept of fetching upstream releases. But it's
closed. So, no bug fix from us...

it's good to see that distributions are slowly awakening to the idea that there are others out
there.  i've always found it backward to have distributions championing this "open source"
stuff while turning around and adding features, fixing bugs, etc... and keeping all these
changes local.  why does an upstream maintainer need to search through all the random places
that package their code in case someone has changed something ?  distributions should be
taking more responsibility for working with upstream developers.  what is great for
$random-distribution is usually great for everyone.  redhat was horrendous in this respect,
but hopefully the fedora project will help them mend their ways.

may also be worthwhile to mention that this isnt just something to look at for linux
distributions as the bsd projects are largely the same thing.  they bring to the table the
core pieces (libc/kernel), but beyond that they are often just using the same packages.  also
good to point out that they are terrible at pushing code back to upstream.  once code gets
imported into their system, it's like a black hole where it never comes out again :(.

getting interconnectivity integrated in bugzilla would have a lovely trickle down affect i
would think.  we can talk about how people should be doing XYZ, but by simply making XYZ
readily available, it would more come into its own.

Here are a couple of distributed bug tracking projects:

http://www.distract.wellquite.org/

http://www.ditrack.org/

I like your idea about running your own Trac instance which tracs bugs you've reported.

Launchpad is working okay for me, so far.

Launchpad is still not free software and I have little hope it ever will be.

All these free software fanatics, all those volumes of email slagging Linus for putting vital
information irretrievably into a closed system, where are you now?? Why aren't you keeping
Launchpad honest?   At least BK left your info on your hard drive, it just kept it in a binary
format.  Launchpad swallows it completely and only returns it on its terms!  It's a FAR worse
scenario than BK, yet nobody seems to care.  It's weird.

All you Launchpad users, how can you bring yourselves to rely on a system that can be taken
away at any time?

Why is Canonical so serious about solving distributed problems with Bazaar and yet so ignorant
about them with Launchpad?  They should have named it "Cathedral".

So I gather you also disapprove of the fact that some projects are hosted in SourceForge,
Berlios.de and Google-Code, right?

As for Bazzar: It is a distributed version control system. Hence if you happen to have an
up-to-date copy of the repository of project FooBar that is hosted on EvilBazzarServer and a
moment later the owner of EvilBazzarServer decides to shut it down, you still have a copy of
FooBar. Complete with the history. And you can start your own repository.

(And for the record: I personally avoid using LaunchPad until they keep to their words of
releasing it)

Erm, SourceForge and BerliOS are open source.  Google code, maybe...  But Google doesn't
present itself as a paragon of openness the way Canonical does, so I don't really find
Google's behavior surprising.

As for Bazaar, that's exactly my point!  Why is Canonical so aware of the benefits of
distributing source code, yet so ignorant about the benefits distributing bugs?  It's truly
baffling.

SourceForge used to be free software. GForge (e.g: GNU Savannah) is based on that. Berlios.de
is based on an older version of that, but IIRC they don't publish its source with the various
adjustments they made there.

As for distributed bug-tracking system: I'm not sure this can actually work well. How do you
merge the comment I have added to the bug and the comment you have added to the bug? And if I
also closed that bug?

"How do you merge the comment I have added to the bug and the comment you have added to the
bug? And if I also closed that bug?"

Isn't this the same problem as decentralized revision control?

:-)

Threading of bug report comment threads, with individual thread termini 
having their own closedness state, branchpoints inheriting the closedness 
state of their downstream termini, and the bug as a whole marked as closed 
when all termini are closed or when the owner closes it...

... well, I can dream, can't I? Anyway, it's only a bit more than what 
threaded newsreaders were doing in 1988.

Apart from the coolness factor, can you give any practical advantages?

What I read from your description is that when you submit a bug report you really can't be
sure about its state. Until it got merged by "upstream". 

So reporting bugs becomes more time consuming.

Yeah. I initially meant to support storing and associating bugs and test-cases /
test-run-states inside monotone databases, transferring them around with revs etc; the crypto
stuff in there was to support independent quality auditing. I think it's viable, it just needs
time and energy.

Never got around to it. Getting the software itself right was hard enough!

> All you Launchpad users, how can you bring yourselves to rely on a system 
> that can be taken away at any time?

Unfortunately, if I want to report bugs in Ubuntu, I have no other choice, since the Ubuntu
folks don't seem to give a damn about the (far superior) reportbug utility.

Debian's bug tracking system supports linking Debian bugs to upstream bugs:

http://bts-link.alioth.debian.org/

I switched my home machines to Debian in 2001 mainly because I found building from upstream so
difficult. I'm happy that I can build from source, but installing a Debian package meant that
the software worked (was tested and supported) and was easily managed within my system.  In
2001 software for Linux often didn't "just work".  Much of my time was spent chasing
dependencies (development libraries or binary dependencies) and once I was able to compile
something I'd generally leave it alone.

My experience has been that distros have made life better for users by packaging software.
Logically this seems counterintuitive as you do have a "middleman" but prior to the Debian
software model you had chaos.  "Pigs playing in mud" wasn't a fun or effective use of my time.

I have 683 packages installed on my machine. If a distributor is not going to provide
dependency resolution information, I might spend three hours resoling the dependencies
manually for each package update. Optionally, I have (potentially) 683 different people who
don't communicate with each other providing incongruent dependency information. How is this a
good thing?

There is one important thing that hasn't been mentioned so far, namely the speed of the
bug-fix cycle. Consider the majority of LWN readers; technically literate, able to file good
bug reports, and probably write some software of our own. Yet, we don't want to delve into
every single program. 

For example, I run Ubuntu, and I update every 6 months. I can't deal with all the
instabilities of running a devel version on my primary system. Let's say I find a bug in KDE,
which particularly annoys me.

(1) The KDE folks will quite reasonably expect me to test with the latest release before
filing the bug report. But I'm running a version somewhere between 3-9 months old, even though
I track the latest Ubuntu release. It's a major effort to re-build KDE, just to verify a
bug-report.

(2) If the bug gets fixed, (or has already been fixed), it will take an average of 3 months
before I get to evaluate, comment on, or benefit from the fix.

As a result:
  * The vast majority of technical users produce bug reports of limited value to upstream;
this is a huge waste of potential talent.
  * Motivation is limited, since most people don't get the benefit of the fixes for the bugs
that they report, in a timely manner.
  * Multiple people hit the same bug, even if it's fixed, because it doesn't get pushed
downstream. Again, a huge waste of time.
  * The oops-report-fix-enjoy cycle takes up to 9 months, instead of 3 days.

My suggestion is that distros should automatically backport every package on a nightly basis,
and provide binaries of the latest CVS, as well as the latest stable-release. These packages
must be built against the stable distro, and should be installable via the package manager.
(they should not be brought in by default though).

I also suggest that most end-users should file most of their bug reports upstream.
Distro-developers only have the resources to be dealing with bug-fixes that they can push out
to all users, eg security and application-crash bugs. 

You can use Ubuntu Hardy right now. Or Debian Unstable. Or Fedora RawHide. OR whatever. But
most users are not able to cope with the nightly upgrades and weekly breakages.

It takes some debugging to get a stable distibution that will "just work" for a user.

Also: what do you mean by "must be built against the stable release"? A KDE 4 program should
be built against the KDE4 libraries in the stable version or from the nightly build? The libc
version from the stable or from the nightly? Xorg from the stable or from the nightly?

I have always graduated towards rolling release distros, or in the case 
of KDE, rolling cvs/svn builds.

Derek

I wish there was a "get source" command in the package manager, that would download the source
for a package configured exactly the same way as the installed package was configured. The
package would be flagged as "customized" in the package database. You could then experiment
with different patches, code changes etc and when you're done either revert back to the
original package or keep your custom changes. It could also support automatically generating
patches to send to the distributor or upstream (in case you managed to fix a bug), creating
custom binary packages etc. 

I think something like that would help a lot to reduce the barrier between users and
developers. Does any distribution already have such a system? 

'apt-get source' has been here for a while.

Debian also has now 'debcheckout', but this is more for the package maintainer's version
control system.

You can certainly build your own package and install it. apt considers also locally-installed
packages.

$ apt-get source PACKAGE
$ sudo apt-get build-dep PACKAGE

$ dpkg-buildpackage -rfakeroot -b

Also you can use apt-pinning to get a newer version for Debian (Ubuntu ?) testing or even
experimental.

If you add the other 'version' of the distribution (like Debian testing) to your
/etc/apt/sources.list and specify which package you want to upgrade in /etc/apt/preferences
(that's the pinning-part).

It will automatically upgrade any dependencies as well.

This way you can test a newer version and even go back to the distrbution original version
from for example Debian-stable very easily.

If you created a patch for a package (if you did the apt-get source, dpkg-source -x;
dpkg-buildpackage), don't forget to change the version-number in the debian/control file, tag
your initial or organisation at the end of the existing version number with a 1 or 2, etc. so
you can upgrade the package later.

And other way to upgrade a package is to add only the source lines to your
/etc/apt/sources.list and do a apt-get source, dpkg-source, dpkg-buildpackage. You might need
to change the debian/control-file to specify a different version number as dependency or
install that first.

On upgrade (apt-get -u dist-upgrade) or even update of your distribution it will automatically
upgrade your packages, etc. if needed.

And I forgot to mention the apt-get build-dep command.

Which will install the build-dependencies of the package.

Have you ever tried apt-get build-dep on a fresh install that doesn't have any build tools
installed? It will pull in the entire build system: gcc, header files, everything, with one
command! With a fast disk and a fast network it is insanely pleasurable.

Kids growing up in the Age of APT may never realise that building stuff used to be *hard*.
It's lucky there are other operating systems around for them to use and understand what pain
means.

tar jxf vim-7.1.tar.bz2
cd vim71
./configure --prefix=/usr/local/vim71
make
sudo make install

Of course you could build your own kernel, pack it into proper .deb file and install it by
dpkg. Just run "aptitude install kernel-package", then "man make-kpkg".