Re: Reporting bugs and bisection

On Mon, 14 Apr 2008 01:04:12 -0700
> 
> The steps to be taken are:
> 
> a) agree that we have a problem
> 


I for one do not agree that we have a problem.

Based on actual data on oopses (which very clearly excludes other kinds of
bugs, so I know I only see part of the story) we are doing reasonably
well. Lets look at the 2.6.25 cycle.  We got a total of roughly 2700
reports of oopses/warn_ons from users. (This may sound high to those of you
only reading lkml, but this includes automatically collected oopses from
Fedora 9 beta testers).  Out of these 2700, the top 20 issues account for
75% of the total reports.

Out of these 20 issues, 9 were from still out of tree drivers (wireless.git
and drm.git included in F9). These were caught before they even got close
to mainline.  The remaining 11 issues can be split in
1) The ones we caught and fixed
2) TCP/IP warnings that DaveM and co are chasing down hard (but have
   trouble finding reproducers) 
3) An EXT3 bug that in theory can cause data corruption, but in practice
   seems to happen after you yank out a USB stick with an EXT3 filesystem on
   (so it can't corrupt the disk data). Ted is working on this 
4) A bug (double free) that hits in the skb layer, probably caused by a bug
   in the ipv4 code (a first analysis + potential patch was mailed to netdev this weekend)
5) sysfs "existing file added" warning, mostly in the USB stack
   (gregkh claims he fixed this recently, I'm not entirely sure he got all cases)

And when I look beyond the first 20, the same pattern arises, we fixed the
majority of the issues before -rc9.  At position 25 we have less than 20
reports per bug. At position 35 we have less than 10 reports per bug.  At
position 50 we have less than 5 reports per bug. Conclusion there: the bugs
people actually hit fall of dramatically; there's a core set of issues that
gets hit a lot, the rest quickly gets reduced to noise levels.


To me this does not sound like we have a huge quality problem because
1) The distribution of the bugs is such that there is a relatively small
   set of core issues that are widely hit, and then there's a near
   exponential drop after that 
2) We are fixing the important bugs by and large before they hit a release
   (important as defined by the number of people actually hitting the bug)


 
I'll be writing a report with more details about this soon with more
analysis and statistics (I'll be looking at more detail around the top 25
issues, when they got introduced, when they got fixed etc)


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org