Needed: code auditors [LWN.net]

Free software is said to be more secure than the proprietary alternatives for a number of reasons. Near the top of most peoples' lists is the openness of the code: with all those eyeballs on the code, security problems are found and fixed quickly. Over the years, however, we have seen numerous signs that fewer people are actually looking at code than many of us would like to believe. Too many vulnerabilities remain in our programs for years for us to have any real confidence that comprehensive auditing is going on.

There have been attempts to encourage developers to audit code. Almost exactly two years ago, the announcement went out for the Sardonix project. Sardonix was started after Crispin Cowan noted that the Linux Security Audit Project appeared to have stalled. Since auditing was not happening by itself, Sardonix sought to provide some motivation in the form of infrastructure and credit for auditors. With these incentives, it was hoped, some large-scale code auditing would start to happen.

Thus, with a little help from a DARPA grant, the Sardonix portal was launched. The portal would track the audit state of various free software programs and would give credit to those who did the auditing work. Sufficiently skilled and productive auditors would be able to accumulate a large "audit karma" to show their friends - and help improve the security of free software at the same time.

Two years later, the only auditing work which has been done on Sardonix was a small set of projects assigned by a university professor to his students. The last posting to the Sardonix mailing list was sent in November, 2003. The DARPA money has run out. Sardonix, it seems, is a project which has failed.

One can attribute this failure to a number of reasons. Certainly Sardonix was never promoted very well; almost nothing was heard from the project after the initial announcement. With an effort to jump-start the process and a set of vulnerabilities found by Sardonix auditors posted on Bugtraq, the project might just have achieved critical mass. As it is, Sardonix vanished into obscurity shortly after its launch, and few people ever heard of it again.

The sad fact remains, however, that, with or without Sardonix, very little auditing is getting done. The continuing stream of vulnerability reports, many for problems which have lurked undetected for years, make this clear. Auditing code is difficult, tedious, and error-prone work. It also tends to be thankless; strangely enough, many developers do not welcome news of vulnerabilities in their work (though most do respond and fix the problems). New vulnerability information requires careful handling; a sustained effort may be required to get the developer to take the problem seriously, but widespread disclosure of the problem must be avoided until developers and distributors have had a chance to react. To top it off, those who do seek out vulnerabilities in software are often seen as promoting their own agendas and making the problem worse. It is not surprising that few people are stepping up and taking on this work.

The free software community has a lot of work to do if it wants to live up to its promise of greater security. This battle must be fought on many fronts: safer programming techniques, containment strategies, detection and response, etc. But we also, somehow, have to find a way to get more critical eyeballs looking at our code. As recent events have shown, the black hats are already doing this work for their own purposes. If free software wants to live up to its pretensions of being a more secure alternative, it needs more developers reviewing the code.

(Log in to post comments)

Needed: code auditors

Posted Feb 5, 2004 5:49 UTC (Thu) by jamesm (guest, #2273) [Link]

While I agree that more needs to be done, I think this article misses good audting that occurs at several points:

1) When patches are submitted.

2) When other developers need to modify code and walk the codepaths with a fresh/different viewpoint.

3) When new developers start poking around to figure out how things work e.g. the kernelnewbies project has been the source of many, many kernel bugfix patches due to this.

4) When code review happens privately (or semi privately) before upstream submission.

5) Research and commercial projects which methodically look for classes of bugs (e.g. Stanford checker).

Code audit is more than planned, specific, high publicity projects (not to say that these are not useful).

It might be useful to find out more about why Sardonix failed. I'd guess that it was on a social level, rather than a technical one.

Look at OpenBSD

Posted Feb 5, 2004 12:03 UTC (Thu) by slowjoe (guest, #18834) [Link]

Yes, OpenBSD lead the field regarding code auditing.

Unfortunately, they don't seem to document the classes of bugs that they search for, and at trade shows, I've asked them about this...they aren't helpful in generalising. I just remember that Theo had apparently moved from worrying about printf bugs to bad handling of file descriptors.

Has anyone documented the OpenBSD bug classes?

Look at OpenBSD

Posted Feb 5, 2004 16:25 UTC (Thu) by ncm (subscriber, #165) [Link]

They very deliberately refuse to classify bugs according to whether they affect security. That is one of their better insights, because any bug may be a security hole, and it is very difficult, in general, to distinguish. Put another way, any such classification will have a very large fraction of erroneous assignments.

That's not to say they couldn't classify bugs along other axes. However, taking all bugs seriously, and minimizing the overhead involved in reporting and fixing them, are key enablers of their practice. Demanding classifications and other statistically useful overhead might backfire.

Look at OpenBSD

Posted Feb 5, 2004 17:30 UTC (Thu) by thoffman (subscriber, #3063) [Link]

It should be possible (and educational) to figure out how they do it just by looking at how their CVS tree has evolved over time.

And reading their mailing lists, of course. But even if they discuss security issues on closed lists, the code is open and someone who took the time could look at all the patches they make, classify them into bug fixes and features, and then look at all the bug fixes and start creating a "taxonomy" of bug fixes.

A well documented collection like that would be a very useful tool to teach other developers to audit code. Maybe some university prof out there will have her students put together documentation like that, and then audit some Linux code for the same sorts of problems?

Look at OpenBSD

Posted Feb 6, 2004 8:13 UTC (Fri) by Cato (subscriber, #7643) [Link]

I was talking about the culture of security auditing, not how they classify bugs etc - reproducing this will mean talking to people not just looking at CVS and email logs.

It's a problem of incentives

Posted Feb 5, 2004 17:59 UTC (Thu) by mrfredsmoothie (subscriber, #3100) [Link]

The article, hits on 3 points which serve well to define the real issue here:

"[In the Sardonix project, s]ufficiently skilled and productive auditors would be able to accumulate a large 'audit karma' to show their friends."
"with all those eyeballs on the code, security problems are found and fixed quickly ... [but]fewer people are actually looking at code than many of us would like to believe"
"the black hats are already doing this work for their own purposes"

I look at a lot of open source code. But, as ESR has said, we do OSS for a variety of reasons; scratching an itch, "establishing a rep," etc. I generally only look at OSS code because a) I'm using it (or want to), and b) it isn't working/won't work on my hardware/I want to extend it, and need to understand it.

I do OSS in my spare time. So, for me "'audit karma' to show [my] friends" is not significant enough incentive to do security auditing, which, as the author acknowledges, is thankless, difficult, error-prone and time consuming.

For "black hats," while this type of code inspection may be difficult, error-prone and difficult, it's probably not "thankless;" they will get root on your box/steal your credit card #, etc. They have a concrete incentive.

Fix the incentives (i.e., Sardonix could have used a little bit of that DARPA money, or OSDL could set up a fund, etc. to actually PAY users to do this type of work), and you'd be much more likely to see active participation. After all,isn't this just the kind of "services on top of OSS" new business model we're always hearing promoted?

Needed: code auditors

Posted Feb 9, 2004 15:01 UTC (Mon) by skx (subscriber, #14652) [Link]

Ironically I've been thinking for a while that I should be more organised with the auditing, and setup a couple of databases of programs etc.

If I do that, and then allow others to help then I'll essentially be duplicating the work of the Sardonix project.

More volunteers are good, and if people spend their time looking over code then we'll all benefit; but without some kind of rating of the competence of the auditors it's hard to have a "community" site.

I'd love to offer cash, but I can't afford it and would have no wish to manage it even if I could.

Steve
--
www.steve.org.uk

Needed: code auditors

Posted Feb 5, 2004 18:37 UTC (Thu) by kfiles (subscriber, #11628) [Link]

It seems that distributed proofreaders has filled a very similar need in the Gutenberg project. The problem that was faced there was that most people were daunted by the task of reviewing entire books or even chapters, and therefore the progress of Gutenberg was relatively slow.

DP (http://www.pgdp.net) makes it easy for individuals to read over a page of scenned text at a teim, whenever they have a free moment. The result has been a record amount of free text added to the project over the last 1-2 years.

Perhaps a Distributed Kernel Auditors project could do the same for C code: provide small snippets (with linked context if needed) for C coders with spare time to pore over, looking for buffer exploits, pointer dereferences, file descriptor misuse, etc. Like DP, it could provide guidlines on patterns of dangerous coding, multiple levels of proofreading, etc.

Providing the right tool for the job is a key to the success of any such effort.

--kirby

Needed: code auditors

Posted Feb 5, 2004 19:12 UTC (Thu) by iabervon (subscriber, #722) [Link]

I'm a bit mystified about using the stream of vulnerability reports as
evidence that auditing is not happening; the majority of vulnerability
reports are the result of auditing which found something. The stream of
vulnerability reports, in fact, demonstrates that auditing is an ongoing
process that turns up issues gradually over time, rather than an event
which finds all of the problems when it happens.

I'm not personally convinced that something like Sardonix is actually
helpful. It might be useful, however, to have software that tracks what
you've found clean, what you've patched, and the notes you've made on why
things work or don't work; it would then compare new versions against the
version you've looked at and look for patches which modify regions you've
looked at, so that you can check whether things are better or worse.
Communicating this information to other people is not necessarily useful,
however, since they may or may not trust your analysis.

systrace and the Stanford Checker (and Rational tools)

Posted Feb 5, 2004 23:40 UTC (Thu) by AnswerGuy (subscriber, #1256) [Link]

There are several things that could help with this auditing effort.

One is Niels Provos'systrace which is built-into recent versions of NetBSD and OpenBSD and is available as a kernel patch and attendent user-space utility/suite for Linux.

Systrace is a system call reference monitor which allows us to create a policy to intercept, allow and deny access to selected system calls (including options to match filename or other system call arguments with glob patterns and to allow specific system calls to be run with modified UID:GID -- user and group --- privileges). Any program that violates the policy is automatically killed and the violation is logged.

Of course this isn't a proactive as a code audit. However, it can help considerably. On a practical level it helps protect the system from many bugs as well as alerting the administrators to the presence of the bug. Naturally we will have to spend time building and refining the policies --- some programs might legitimately be executing extraordinary code (system calls) in rare cases (exception handling and error recovery).

Another tool that has been discussed by the LWN team occasionally over the years has been the Meta Level Compilation research project at Stanford, headed up by Dawson Engler. This has been used to detect numerous bugs in the Linux kernel (as test and research cases and for demonstration and proof-of-concept purposes).

Basically they use a modified compiler and a set of code extensions which, if I understand it correctly, allow them to apply a set of compilation assertions about the code paths and "machine state" leading into and upon return from functions defined in the code. This is similar to the concept of "programming by contract" (Eiffel, et al.) where pre-conditions, post-conditions and invariants can be defined for each code block and the compiler can warn of failures to conform to these policies.

Unfortunately I haven't heard of any publication or releases from the Stanford team in a couple of years and they've ignore e-mails from me requesting further information. I've inquired about the possibility of interviewing anyone on their team about the state of the project a few times.

Fortunately their project isn't the only one in the field. One that was mentioned in the LWN discussion thread over a year ago was splint (Secure Programming lint). Another is David Wheeler's flawfinder. His site also has links to several other tools in the same genre. I even seem to reading that Linus was working on some tools for this sort of work.

Certainly no tool or suite of tools will find every possible software flaw. However, the fact that code auditing is tedious, laborious, and error prone work screams for more diligent and widespread use of any tools that are available as well as for the development of more sophisticated code auditing tools. "Tedium" and labor and errors are precisely what we build computers to avoid!

Needed: code auditors

Posted Feb 9, 2004 9:54 UTC (Mon) by ekj (subscriber, #1524) [Link]

Why should anyone choose to work through Sardonix ? I'm not being flippant, I seriously wonder what they imagine they add to the process.

This year I've discovered around 5 bugs in free software. Two of them I sent patches for (both are now fixed, one through my patch, one was fixed in a different way), the 3 others I reported in the apropriate bugzilla-system, and 1 of those are also now fixed. The two remaining open ones are minor cosmetic bugs in kmail. (failing to redraw certain elements in certain cases)

But I never used Sardonix for anything. Why should I ? Why should anyone ? What do we need them for ?

I don't think I'm alone in contributing to fixing bugs in Free Software, yet in all honesty unable to see the point of projects like Sardonix.