Linux Kernel Software Quality and Security Better than Most Proprietary Enterprise Software, 4-Year Coverity Analysis Finds Stanford computer science researchers analyze 5.7 million lines of software, identify 985 software bugs most already fixed by open source community SAN FRANCISCO, December 14, 2004 Coverity, a software engineering company focused on developing a better way to build software, today announced results on Linux security compiled over four years of source code analysis of the Linux kernel. The recent 2.6 Linux production kernel now shipping in operating system products from Novell and other major Linux software companies contains 985 bugs in 5.7 million lines of code, well below the industry average for commercial enterprise software. Commercial software typically has 20 to 30 bugs for every thousand lines of code, according to Carnegie Mellon University's CyLab Sustainable Computing Consortium. This is equivalent to 114,000 to 171,000 bugs in 5.7 million lines of code. The former director of cybersecurity for the U.S. Department of Homeland Security, Amit Yoran, this month told a Washington, D.C. conference on Homeland Security and Information Assurance that automatic code debuggers are required to make software secure. The Linux source code analysis project started in 2000 at the Stanford University Computer Science Research Center as part of a massive research initiative to improve core software engineering processes in the software industry. The initiative continues on at Coverity, a commercial software company started by five of the lead Stanford researchers. Coverity customers include the top vendors in networking, electronic design automation and storage, among others. As a public service, Coverity will start providing bug analysis reports on a regular basis and make a summary of the results freely available to the Linux development community. "This is a benefit to the Linux development community and we appreciate Coverity's efforts to help us improve the security and stability of Linux," said Andrew Morton, lead Linux kernel maintainer. "We've already addressed the top priority bugs that Coverity has uncovered. It's a very useful system for high quality code." "Key Linux developers can now use the same tools that many of the world's largest commercial IT vendors have integrated into their software development process," said Seth Hallem, CEO of Coverity. "Our findings show that Linux contains 0.17 bugs per thousand lines of code, which is an extremely low defect rate and is evidence of the strong security of Linux. Many security holes in software are the result of software bugs that can be eliminated with good programming processes." Coverity found Linux bugs in five areas: crash causing defects, incorrect program behavior, performance degradation, Improper use of APIs, security flaws Of the 985 bugs, 627 are in critical parts of the kernel and are broken down as follows: Crash causing: 569 Buffer overruns: 25 Performance degradation (resource leaks): 33 Security: 100 A summary of the bugs is available at http://linuxbugs.coverity.com Active members of the Linux kernel development community can obtain detailed bug reports by contacting Coverity. About Coverity's Products SWAT's core technology is unique amongst source code analysis solutions in both its precision and scalability. Unlike many competing technologies, SWAT simulates the effects that the operations in the source code might have in the runtime environment, rather than searching the source code for known, dangerous coding patterns or potentially sloppy coding constructs. The result is that the defects detected by SWAT's analysis platform are potentially disastrous runtime errors that must be fixed in the source code. In addition, SWAT is designed to integrate easily into existing software development practices without any changes to existing build systems or existing development tools. About Coverity Coverity, Inc. (www.coverity.com) is a software engineering company focused on developing a better way to build software. While hardware design has always been considered a difficult task that merits significant investments in automation and verification, the notion that building software is just as difficult has only recently gained credibility in the marketplace. Coverity was founded to meet that insight with a solution: analyze source code with sophisticated, automatic tools that allow software developers to identify defects that could cause catastrophic failures or security breaches without imposing any additional burden on the development cycle.
Coverity's kernel code quality study
Posted Dec 14, 2004 18:14 UTC (Tue) by southey (subscriber, #9466) [Link]
Note that 54% are in device drivers and 41% are due to NULL pointer dereferences. So Linux is probably has an even better quality than it first appears.
Coverity's kernel code quality study
Posted Dec 14, 2004 18:21 UTC (Tue) by MathFox (guest, #6104) [Link]
My take on the story is:After four years of pestering the Linux coders with "Stanford code checker" reports, we hardly find any bugs with our code checker (which is the rebadged Stanford checker).The number Coverty presents can not be compared with the number of bugs found in other projects before Coverty fixes.
Bugs per kLOC
Posted Dec 14, 2004 19:47 UTC (Tue) by man_ls (guest, #15091) [Link]
The number Coverty presents can not be compared with the number of bugs found in other projects before Coverty fixes.Well said. Bugs per thousand lines of code (kLOC) can only be evaluated as a relative number, since we cannot know:
The figure given by Carnegie Mellon University, 20 or 30 bugs per kLOC, is definitely not for released software, but probably for written software before any testing happens. After release, the number would rather be 1 to 5 bugs per kLOC in commercial software. For mission-critical code, the count can be as low as 0.1 bugs per kLOC (as in Shuttle software), depending on cricicity and budget. Project size is also a factor.
Of course the rate in Linux is lower than in "commercial enterprise software"; an operating system kernel arguably is mission-critical software. 0.17 bugs per kLOC looks like a lot, even if those bugs are in device drivers, or especially then since they can take down the whole system, corrupt data, etc. (I remember estimates for w2k were 2 bugs per kLOC after release, but that includes the whole operating system, not just the kernel.)
But there is more. Nobody would expect that, after fixing the 985 bugs, Linux would magically become error-free. So 0.17 bugs per kLOC must be a lowest-bound estimate; the real figure will be higher.
All in all, a poor press release with not much real value, but great promotion for the Stanford Code Checker.
Bugs per kLOC
Posted Dec 14, 2004 23:02 UTC (Tue) by hppnq (guest, #14462) [Link]
I assume the Coverity program has a proper parser that allows for at least a proper comparison of the actual number of lines of code it has inspected. It must be much harder, for instance, to compare the complexity (that is inevitably related to the number of bugs found, I would say) of two programs.But I fully agree with you that the press release reads more like a promotional flyer, which is a bit strange considering these people must know the tricks of the scientific trade.
Coverity's kernel code quality study
Posted Dec 14, 2004 23:58 UTC (Tue) by brouhaha (subscriber, #1698) [Link]
The number Coverty presents can not be compared with the number of bugs found in other projects before Coverty fixes.Certainly it CAN be compared. If you were going to choose an OS today, would you decide to ignore the higher bug count of <some proprietary os> just because the owner hasn't had those Coverity fixes but might get them (or the equivalent) in the future?
Coverity's kernel code quality study
Posted Dec 15, 2004 0:42 UTC (Wed) by MathFox (guest, #6104) [Link]
You should realise that any (automated or manual) bug-checking process only finds a subset of the bugs that exist in a program. There ain't no silver bullet! The Stanford/Coverity bug checker will only find some of the bugs and be blind for the others.It makes more sense to compare the numbers of bugs that were found on the first run of the scanner between different projects, or the total number of bugs spotted by the scanner instead of the number of <residual> bugs that the scanner finds after several iterations of bug fixing.
The holy grail in software testing is knowing the exact number of bugs in the product. :)
Coverity's kernel code quality study
Posted Dec 15, 2004 0:48 UTC (Wed) by emkey (guest, #144) [Link]
Shouldn't the holy grail be knowing exactly where all those bugs are? :-)
OpenBSD
Posted Dec 14, 2004 18:36 UTC (Tue) by freethinker (guest, #4397) [Link]
I wonder what a similar study would find in OpenBSD. I found some references to 2000-odd bugs found in Linux and OpenBSD at some time in the past, but no breakdown.
OpenBSD
Posted Dec 14, 2004 20:34 UTC (Tue) by error27 (subscriber, #8346) [Link]
Someone did a comparison and BSD was far worse.
But most of the bugs were in one small area of the code that wasn't used much... Compatability for something.
OpenBSD
Posted Dec 14, 2004 22:07 UTC (Tue) by JoeBuck (subscriber, #2330) [Link]
There's a problem with comparisons, though. Coverity started off as the "Stanford Checker", and those guys have been working with the Linux kernel for years, feeding back bugs as they find them. Not as much work has been done on the BSDs.
*BSD, so what?
Posted Dec 15, 2004 0:15 UTC (Wed) by gvy (guest, #11981) [Link]
May I reiterate that it's BSDs' problem an not anyone else's, deterring people off development and not providing incentive to cooperate?
I bet someone of that crowd whined in Ladislav's inbox to make his banner on distrowatch.com include "BSD" which is totally irrelevant there IMHO.
And I've personally taken enough whining instead of people doing something real to promote their favours to have this approach to that crowd, sorry.
Niche software doesn't deserve broad attention. Especially when providing nothing new to this world.
*BSD, so what?
Posted Dec 15, 2004 13:17 UTC (Wed) by nix (subscriber, #2304) [Link]
Niche software doesn't deserve broad attention.What a parochial point of view. You realise the same `reasoning' could be applied by Windows people to Linux and the MacOS?
Everything deserves attention, if just to find good ideas in it and reuse them elsewhere.
Coverity's kernel code quality study
Posted Dec 15, 2004 8:26 UTC (Wed) by alexs (guest, #13637) [Link]
aside to the promotion
i suppose that a lot of kernel improvements
of the near and far future will be the result
of improved gnu compilers and possible assisted
by the occasional usage of the intel compilers.
stanford checker might decrease in his contribution
to the fixing rate as things tend to get tougher
in a phase where the overall amount of bugs in
the software is magnitudes below the normal rate.
lets give you some "troll bait": Linux is mature now. ;-)
Copyright © 2004, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds