the Linux process for generating many rare flaws

The problem is the bugs with security implications that nobody spots *but* 
the blackhats :( they're hardly going to tell anyone about it (other than 
other blackhats), but they'll use the knowledge, oh yes.

I think the point is that an ad hoc testing process can only go so far.  Without a more
directed testing process that explicitly targets corner conditions, you can only say you've
found and fixed the common cases, for an ever expanding definition of "common."

If you have a system with very tight reliability requirements (think, for instance, medical
equipment, ABS brake controllers, life support systems), such a model is inappropriate.

I do disagree with the notion that the number of bugs above a certain level of rarity
monotonically increases with time.  If that were true, in the limit you'll end up with a
system that fails every time, in a completely different way every time it's used.  No
repeatable bugs, but rather a custom bug every time.  That defies belief.

Of course that's a ridiculous endpoint, but we have occasionally seen the precursors to that
in localized areas that start gaining a reputation of being pervasively flaky.  (Linux's VM
some years ago, for example.)

When a subsystem starts developing behavior that's too subtle and too quirky to adequately
analyze and diagnose, it gets reworked.  Rather than fix the bugs directly, they get replaced
wholesale.  We've seen that a number of times with varying levels of success.  The main point
is that any quirks that were accumulating there now have been flushed away.

Hello njs:

I see that I didn't explain my thinking very clearly.

I think that there are a few ways that you could have a development process that produces
fewer rare, subtle bugs.  They probably come at the cost of also producing good code slower.

One of these is relying more on code inspection and less on experiment, for your
bug-detection.  Then there is "writing more carefully" in the first place, which is I suppose
just like code inspection done by the original author.  That one presumably means you get less
written per day.

Another is emphasizing automated tests vs. manual tests.  I really like the way that in the
Twisted Python project, it is pretty hard to get the developers to fix a bug without
submitting a unit test that deterministically demonstrates that bug.  The Twisted project, as
a policy (more or less) doesn't apply patches to code that isn't tested, and doesn't add
features or fix bugs unless the feature or bug is exercised by a unit test.

Another is "designing for testability".  I work with an excellent programmer named Brian
Warner and I sometimes see him do something like "Hm, doing it this way would have plenty of
good properties, but I can't figure out how to write a unit test that would exercise that all
of that code in a deterministic way.  I think I'll do it a different way.".

Some of these may be inappropriate for kernel code (Twisted Python, and the project that Brian
Warner and I work on, are both free of the obligation to deal with hardware, for example).  Or
some of them may be appropriate for kernel code in general, but would be inappropriate for
Linux's goals of rapid evolution and high performance.  (For example, OpenBSD apparently does
extensive code review.)  But I hope that these give you some ideas about what I am thinking of
as possible alternatives.

In general, I have the suspicion that any development process that emphasizes producing code
quickly in Step 1, and then QA'ing it later in Step 2, is likely to add subtle rare bugs, as
contrasted with more "integrated" development process in which specific techniques intended to
prevent subtle rare bugs are part of Step 1.  This is doubly true when Step 2 is largely ad
hoc, i.e. it is not systematic or automated.

This general strategy -- rely on lots of users and downstream distributors and the like to do
lots of manual testing for you -- seems to be a core part of the Linux development process
culture.  (You can always tell that an idea has a profound effect on a culture when the
members of that culture think that there is no possible alternative. :-))

I do wish that there were a keen technical and cultural observer like Jon Corbet who wrote
detailed analyses of the development processes of other operating systems, starting with
OpenBSD and Solaris.