Washington Post article
is one of many expressing disappointment with
Microsoft's Vista release, which is famously late and which has failed to
live up to Microsoft's early promises. The article claims that the
problems are not specific to Microsoft:
The sad truth is that Microsoft's woes aren't unusual in this
industry. Large-scale software projects are perennially beset by
dashed hopes and bedeviling delays. They are as much a tar pit
today as they were 30 years ago, when a former IBM program manager
named Frederick P. Brooks Jr. applied that image to them in his
classic diagnosis of the programming field's troubles, "The
In this context, it behooves us to ask: is there a free software tar pit in
our future? What can we do to avoid a grim future where we bog down, our
software collapsing under its own weight?
Looking at the state of the free software community now, it is tempting to
say that, so far, we have nicely avoided the tar pit. But have we? Here
are a few dates from the past which may be of interest:
- The 2.2.0 kernel was released on January 26, 1999.
- 2.4.0 came out on January 4, 2001.
- 2.5.1 - the beginning of the next development series - was released on
December 16, 2001
The 2.5 development series was stalled for almost one full year while 2.4
reached a state which actually approached stable. Overall, the process
from 2.2.0 to a stable 2.4 took almost three years; the kernel was in a
"feature freeze" state for about two of those years. This was a time when
quite a few people - many of them kernel developers - felt let down by the
development process. This, your editor would attest, was a tar pit era.
One might well argue that the kernel has not yet escaped that tar pit. Like
Vista, we lack a shiny new next-generation filesystem; the only credible
attempt at such a filesystem (reiser4) remains in a stalled,
feature-reduced state. It seems likely, however, that most observers would
agree that the tar pit has been left far behind. The kernel development
process has been humming along at a high pace, delivering interesting new
releases every few months. The same story can be seen in many other parts
of the free software community.
If we accept that things have gotten better, it can be interesting to look
at why. One hint can be found in the same article:
Without that discipline, too often, software teams get lost in what
are known in the field as "boil-the-ocean" projects -- vast schemes
to improve everything at once. That can be inspiring, but in the
end we might prefer that they hunker down and make incremental
improvements to rescue us from bugs and viruses and make our
computers easier to use. Idealistic software developers love to
dream about world-changing innovations; meanwhile, we wait and wait
for all the potholes to be fixed.
Any successful free software project must get good at fixing potholes; in
the worst case, users (and distributors) will do the job for themselves.
In a well-managed project, the people who are trying to improve the whole
world will not get in the way of the pothole fixers. There is no single
team, charged with all the development on a project, which can get bogged
down in that way.
A "well-managed project" must find a way to keep whole-world improvements
from stopping everything else, however. The older, multi-year kernel
process did not always succeed on that front; the attempt to improve the
entire kernel ended up bogging down the entire process. The new kernel
with its short release cycles, has caused some developers to complain that
it is no longer possible to make major changes that require a long time to
settle down. To the extent that this complaint is true, it should maybe be
seen as a good thing. By only merging changes which can be brought to a
releasable state within a month or two, the new process sidesteps the
tar pit and keeps the development machine running.
One of the key suggestions in The Mythical Man Month is the
formation of "surgical teams" to support the lead programmer(s). Some of
the team members - such as the clerk who "keys in" the code - seem a little
quaint now. But the idea that the people running the project (or parts of
it) need lieutenants, documentation writers, tool makers, etc. still makes
a lot of sense. Once upon a time, the kernel lacked much of that
structure, with everything concentrating on a single developer - Linus
Torvalds. Now there is a vast network of lieutenants. Quite a few
developers focus their effort not on the kernel, but on the tools used by
kernel developers. All that's missing are the clerks - and, perhaps,
the documentation writers.
One of the biggest anti-tar pit technologies used by the free software
community would have been hard for Mr. Brooks to imagine back in 1972:
multiple, independent development teams. Any project of any size has a
wide range of independent, sometimes conflicting development efforts
happening at the same time. If one group bogs down, the others continue
unhindered. The process may seem inefficient, given that a significant
portion of the work which is done may never survive to a stable release.
Throwing away code can be painful, but it is far less so than throwing away
the entire project.
Peer review is also missing from the Brooks landscape. But peer review
helps to ensure one of the things he thought was vital for a successful
project: a clear conceptual architecture for the project. That
architecture may take a surprising form: few free software projects have the
sort of extensive design documentation that he probably had in mind. But a
crowd of reviewers can help to ensure that new code is consistent with the
principles behind a project - and that it is maintainable into the future.
In this context, it is notable (and worrisome) that an increasing number of
proposed kernel features are finding themselves stalled by a lack of
Finally, one should note that free software projects have mostly learned a
sure-fire way to avoid a failure to live up to their promises: they don't
make any. Vaporware tends to be scarce in this community; either the code
exists or it does not. Very few projects are truly controlled by one
corporation, so companies are also restrained in the promises they make
about future releases; they are in no position to ensure that those
promises are fulfilled. The relative freedom from marketing-driven
promises helps free software projects avoid disappointments - but it also
helps them to focus effort on objectives with a reasonable chance of
To argue that the free software community is immune to the problems of
large-scale software development would be foolish. For all their growth,
many or most components of a system like Linux are still a fraction of the
size of their equivalents on certain proprietary systems. As our code base
grows, there will undoubtedly be new challenges for those who would
continue to develop it. But the free systems we have today must certainly
far exceed the size of System/360 when Mr. Brooks was managing it, and we
would appear to be going strong. With widespread community participation,
improving tools, and the willingness to change our development models in
response to real-world problems, we should be about to stay out of that
tar pit for some time yet.
to post comments)