Exploiting races in system call wrappers

A technique that is often used by security software, and has historically been a source of security holes, has once again been shown to be exploitable on many systems. Research recently presented by Robert N.M. Watson at the USENIX Workshop on Offensive Technologies (WOOT07) demonstrates race conditions in software that uses "system call wrapping" (or "hooking"). The race conditions can be exploited to circumvent the protections that the software is supposed to provide. Well behaved Linux software is not vulnerable, but other free operating systems do allow, and even encourage, the practice.

There are several different ways to implement wrappers, but at the core, they are kernel code that intercepts system calls from all applications, running their own code before and after the real system call. The wrapper code can see and modify all of the arguments being passed to and from the system call. This technique can be used to enforce various policies on the use of the system calls, denying or sharply restricting access. Logging, for audit trail purposes, all system call activity is another way the wrappers could be used.

Anti-virus or intrusion detection and prevention are the kinds of applications that use system call wrapping. Intercepting all calls to open(), for example, checking the file for viruses or illegal access and if so, returning an error, are the kinds of tasks that system call wrappers are used for. Notable users of system call wrappers are the OpenBSD and NetBSD Systrace facility, the Generic Software Wrappers Toolkit and the CerbNG firewall for FreeBSD.

Thus, intercepting system calls is a technique that is useful, but not without hazards. These recent vulnerabilities are endemic to the technique, not tied to a specific implementation. They exploit that bugaboo of system programmers everywhere: the race condition. Specifically, they are time-of-check-to-time-of-use (TOCTTOU) or other, similar, bugs.

A TOCTTOU exploit abuses the gap in time between the test for a condition and the use of an object that passes the test. If the object is changed in that gap, the restrictions that were supposed to be enforced by the test can be bypassed. The classic example is a setuid() program that tests a file for legal access by the real user before opening it. If the user replaces the file with a symlink to a file they can't legally access after the test, but before the open(), they have circumvented the security check.

Two similar race conditions have been identified for applications using system call wrappers: time-of-audit-to-time-of-use (TOATTOU) and time-of-replacement-to-time-of-use (TORTTOU). In both cases, the data that gets passed to the system call is manipulated. For TOATTOU, it is done to obscure the data from any auditing or logging that might be done, covering the tracks of an exploit from an intrusion detection application for example. In the TORTTOU case, if the data passed into the system call is changed by the wrapper, to implement "jail" functionality for instance, the exploit changes it back before the system call is made.

In his paper, "Exploiting Concurrency Vulnerabilities in System Call Wrappers" (PDF), Watson shows techniques to reliably exploit the race conditions in a variety of packages that use system call wrappers. On both single and multi-processor systems, mechanisms were found to exploit the time gap – because system calls, especially with wrappers, are not atomic operations.

For single processor systems, one of his examples used data that had its last byte on a swapped-out page. While the kernel is sleeping, awaiting the page to be swapped in, another process can change the data that has already been read. For multiprocessor systems, the windows are typically smaller, but it is not necessary to arrange for the kernel to sleep, a thread on a different processor can be used to alter the data. The main problem in that case is synchronizing with the kernel process so that the exploit knows when to change the data. Watson found several synchronization methods, one very simple one just spins waiting for the data to change and changes it back, effecting a TORTTOU exploit.

For these and other reasons, Linux does not export its system call table and actively discourages programmers from taking this approach. There are no real solutions to the problems Watson has identified unless the system call wrapping technique is abandoned. The two solutions he has suggested are either moving to a "message passing" architecture for system calls or to integrate the security checks into the kernel itself. He specifically mentions the Linux Security Modules approach as one that alleviates the system call wrapper race.

It is unfortunate that there are still many uses of system call wrapping in today's free operating systems. While the specific problems that Watson describes may not have been known, wrappers as a source of security bugs certainly have been. It is a seductive technique, one that seems simple to implement and foolproof, but it is clearly fraught with peril. The BSD family needs to find other ways to implement their security applications as do any Linux vendors who have ignored the kernel developers and continued to use the wrapping technique.