User: Password:
|
|
Subscribe / Log in / New account

Coverity's kernel code quality study

Linux Kernel Software Quality and Security Better than Most Proprietary
Enterprise Software, 4-Year Coverity Analysis Finds

Stanford computer science researchers analyze 5.7 million lines of software,
identify 985 software bugs   most already fixed by open source community

SAN FRANCISCO, December 14, 2004   Coverity, a software engineering company
focused on developing a better way to build software, today announced results
on Linux security compiled over four years of source code analysis of the
Linux kernel.  The recent 2.6 Linux production kernel now shipping in
operating system products from Novell and other major Linux software
companies contains 985 bugs in 5.7 million lines of code, well below the
industry average for commercial enterprise software.

Commercial software typically has 20 to 30 bugs for every thousand lines of
code, according to Carnegie Mellon University's CyLab Sustainable Computing
Consortium. This is equivalent to 114,000 to 171,000 bugs in 5.7 million
lines of code. 

The former director of cybersecurity for the U.S. Department of Homeland
Security, Amit Yoran, this month told a Washington, D.C. conference on
Homeland Security and Information Assurance that automatic code debuggers are
required to make software secure.

The Linux source code analysis project started in 2000 at the Stanford
University Computer Science Research Center as part of a massive research
initiative to improve core software engineering processes in the software
industry. The initiative continues on at Coverity, a commercial software
company started by five of the lead Stanford researchers.  Coverity customers
include the top vendors in networking, electronic design automation and
storage, among others. 

As a public service, Coverity will start providing bug analysis reports on a
regular basis and make a summary of the results freely available to the Linux
development community.

"This is a benefit to the Linux development community and we appreciate
Coverity's efforts to help us improve the security and stability of Linux,"
said Andrew Morton, lead Linux kernel maintainer.  "We've already addressed
the top priority bugs that Coverity has uncovered.  It's a very useful system
for high quality code." 

"Key Linux developers can now use the same tools that many of the world's
largest commercial IT vendors have integrated into their software development
process," said Seth Hallem, CEO of Coverity.  "Our findings show that Linux
contains 0.17 bugs per thousand lines of code, which is an extremely low
defect rate and is evidence of the strong security of Linux.  Many security
holes in software are the result of software bugs that can be eliminated with
good programming processes."  

Coverity found Linux bugs in five areas:
 	crash causing defects,
 	incorrect program behavior,
 	performance degradation,
 	Improper use of APIs,
 	security flaws 

Of the 985 bugs, 627 are in critical parts of the kernel and are broken down
as follows:
 	Crash causing: 569 
 	Buffer overruns: 25
 	Performance degradation (resource leaks): 33
 	Security: 100

A summary of the bugs is available at http://linuxbugs.coverity.com

Active members of the Linux kernel development community can obtain detailed
bug reports by contacting Coverity.


About Coverity's Products
SWAT's core technology is unique amongst source code analysis solutions in
both its precision and scalability. Unlike many competing technologies, SWAT
simulates the effects that the operations in the source code might have in
the runtime environment, rather than searching the source code for known,
dangerous coding patterns or potentially sloppy coding constructs. The result
is that the defects detected by SWAT's analysis platform are potentially
disastrous runtime errors that must be fixed in the source code. In addition,
SWAT is designed to integrate easily into existing software development
practices without any changes to existing build systems or existing
development tools.


About Coverity
Coverity, Inc. (www.coverity.com) is a software engineering company focused
on developing a better way to build software. While hardware design has
always been considered a difficult task that merits significant investments
in automation and verification, the notion that building software is just as
difficult has only recently gained credibility in the marketplace. Coverity
was founded to meet that insight with a solution: analyze source code with
sophisticated, automatic tools that allow software developers to identify
defects that could cause catastrophic failures or security breaches without
imposing any additional burden on the development cycle.






(Log in to post comments)

Coverity's kernel code quality study

Posted Dec 14, 2004 18:14 UTC (Tue) by southey (subscriber, #9466) [Link]

Note that 54% are in device drivers and 41% are due to NULL pointer dereferences. So Linux is probably has an even better quality than it first appears.

Coverity's kernel code quality study

Posted Dec 14, 2004 18:21 UTC (Tue) by MathFox (guest, #6104) [Link]

My take on the story is:
After four years of pestering the Linux coders with "Stanford code checker" reports, we hardly find any bugs with our code checker (which is the rebadged Stanford checker).
The number Coverty presents can not be compared with the number of bugs found in other projects before Coverty fixes.

Bugs per kLOC

Posted Dec 14, 2004 19:47 UTC (Tue) by man_ls (guest, #15091) [Link]

The number Coverty presents can not be compared with the number of bugs found in other projects before Coverty fixes.
Well said. Bugs per thousand lines of code (kLOC) can only be evaluated as a relative number, since we cannot know:
  • if blank lines of code are counted,
  • if comments are counted either,
  • if coding style matters (lone '{'s or '}'s)...
Otherwise, we can only suppose it is non-blank, non-comment lines of code what we are counting (the usual industry standard); and play with broad estimates, which I will presently do for the fun of it.

The figure given by Carnegie Mellon University, 20 or 30 bugs per kLOC, is definitely not for released software, but probably for written software before any testing happens. After release, the number would rather be 1 to 5 bugs per kLOC in commercial software. For mission-critical code, the count can be as low as 0.1 bugs per kLOC (as in Shuttle software), depending on cricicity and budget. Project size is also a factor.

Of course the rate in Linux is lower than in "commercial enterprise software"; an operating system kernel arguably is mission-critical software. 0.17 bugs per kLOC looks like a lot, even if those bugs are in device drivers, or especially then since they can take down the whole system, corrupt data, etc. (I remember estimates for w2k were 2 bugs per kLOC after release, but that includes the whole operating system, not just the kernel.)

But there is more. Nobody would expect that, after fixing the 985 bugs, Linux would magically become error-free. So 0.17 bugs per kLOC must be a lowest-bound estimate; the real figure will be higher.

All in all, a poor press release with not much real value, but great promotion for the Stanford Code Checker.

Bugs per kLOC

Posted Dec 14, 2004 23:02 UTC (Tue) by hppnq (guest, #14462) [Link]

I assume the Coverity program has a proper parser that allows for at least a proper comparison of the actual number of lines of code it has inspected. It must be much harder, for instance, to compare the complexity (that is inevitably related to the number of bugs found, I would say) of two programs.

But I fully agree with you that the press release reads more like a promotional flyer, which is a bit strange considering these people must know the tricks of the scientific trade.

Coverity's kernel code quality study

Posted Dec 14, 2004 23:58 UTC (Tue) by brouhaha (subscriber, #1698) [Link]

The number Coverty presents can not be compared with the number of bugs found in other projects before Coverty fixes.
Certainly it CAN be compared. If you were going to choose an OS today, would you decide to ignore the higher bug count of <some proprietary os> just because the owner hasn't had those Coverity fixes but might get them (or the equivalent) in the future?

Coverity's kernel code quality study

Posted Dec 15, 2004 0:42 UTC (Wed) by MathFox (guest, #6104) [Link]

You should realise that any (automated or manual) bug-checking process only finds a subset of the bugs that exist in a program. There ain't no silver bullet! The Stanford/Coverity bug checker will only find some of the bugs and be blind for the others.
Running an automated checker on a program will find a lot of bugs in the first run... but it will not find some bugs that hide in the bling spot of the checker. You can fix the bugs that the scanner finds and rerun it; but it can not help you with the bugs that it is blind for. So there will allways be an unknown(!) number of bugs left after a perfect scan.

It makes more sense to compare the numbers of bugs that were found on the first run of the scanner between different projects, or the total number of bugs spotted by the scanner instead of the number of <residual> bugs that the scanner finds after several iterations of bug fixing.

The holy grail in software testing is knowing the exact number of bugs in the product. :)

Coverity's kernel code quality study

Posted Dec 15, 2004 0:48 UTC (Wed) by emkey (guest, #144) [Link]

Shouldn't the holy grail be knowing exactly where all those bugs are? :-)

OpenBSD

Posted Dec 14, 2004 18:36 UTC (Tue) by freethinker (guest, #4397) [Link]

I wonder what a similar study would find in OpenBSD. I found some references to 2000-odd bugs found in Linux and OpenBSD at some time in the past, but no breakdown.

OpenBSD

Posted Dec 14, 2004 20:34 UTC (Tue) by error27 (subscriber, #8346) [Link]

Someone did a comparison and BSD was far worse.

But most of the bugs were in one small area of the code that wasn't used much... Compatability for something.

OpenBSD

Posted Dec 14, 2004 22:07 UTC (Tue) by JoeBuck (subscriber, #2330) [Link]

There's a problem with comparisons, though. Coverity started off as the "Stanford Checker", and those guys have been working with the Linux kernel for years, feeding back bugs as they find them. Not as much work has been done on the BSDs.

*BSD, so what?

Posted Dec 15, 2004 0:15 UTC (Wed) by gvy (guest, #11981) [Link]

May I reiterate that it's BSDs' problem an not anyone else's, deterring people off development and not providing incentive to cooperate?

I bet someone of that crowd whined in Ladislav's inbox to make his banner on distrowatch.com include "BSD" which is totally irrelevant there IMHO.

And I've personally taken enough whining instead of people doing something real to promote their favours to have this approach to that crowd, sorry.

Niche software doesn't deserve broad attention. Especially when providing nothing new to this world.

*BSD, so what?

Posted Dec 15, 2004 13:17 UTC (Wed) by nix (subscriber, #2304) [Link]

Niche software doesn't deserve broad attention.
What a parochial point of view. You realise the same `reasoning' could be applied by Windows people to Linux and the MacOS?

Everything deserves attention, if just to find good ideas in it and reuse them elsewhere.

*BSD, so what?

Posted Dec 16, 2004 0:46 UTC (Thu) by freethinker (guest, #4397) [Link]

Don't feed the trolls :)

Coverity's kernel code quality study

Posted Dec 15, 2004 8:26 UTC (Wed) by alexs (guest, #13637) [Link]

aside to the promotion
this press release shows
how a code project can improve
over some period of time
when a well tuned toolset is applied.

i suppose that a lot of kernel improvements
of the near and far future will be the result
of improved gnu compilers and possible assisted
by the occasional usage of the intel compilers.
stanford checker might decrease in his contribution
to the fixing rate as things tend to get tougher
in a phase where the overall amount of bugs in
the software is magnitudes below the normal rate.

lets give you some "troll bait": Linux is mature now. ;-)


Copyright © 2004, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds