RLIMIT_NPROC and setuid()

The setuid() system call has always been something of a security problem for Linux (and other Unix systems). It interacts oddly with security and other kernel features (e.g. the unfortunately named "sendmail-capabilities bug") and is often used incorrectly in programs. But, it is part of the Unix legacy, and one that will be with us at least until the 2038 bug puts Unix systems out of their misery. A recent patch from Vasiliy Kulikov arguably shows these kinds of problems in action: weird interactions with resource limits coupled with misuse of the setuid() call.

There is a fair amount of history behind the problem that Kulikov is trying to solve. Back in 2003, programs that used setuid() to switch to a non-root user could be used to evade the limit on the number of processes that an administrator had established for that user (i.e. RLIMIT_NPROC). But that was fixed with a patch from Neil Brown that would cause the setuid() call to fail if the new user was at or above their process limit.

Unfortunately, many programs do not check the return value from calls to setuid() that are meant to reduce their privileges. That, in fact, was exactly the hole that sendmail fell into when Linux capabilities were introduced, as it did not check to see that the change to a new UID actually succeeded. Buggy programs that don't check that return value can cause fairly serious security problems because they assume their actions are limited by the reduced privileges of the switched-to user, but are actually still operating with the increased privileges (often root) that they started with. In effect, the 2003 change made it easier for attackers to cause setuid() to fail when RLIMIT_NPROC was being used.

Kulikov described the problem back in June, noting that it was not a bug in Linux, but allowed buggy privileged programs to wreak havoc:

In the posting, he suggested two possible solutions to the problem. The first is to move the check against RLIMIT_NPROC from set_user() (a setuid() helper function) to execve() as most programs will check the status of that call (and can't really cause any harm if they don't). The other suggestion is one that was proposed by Alexander Peslyak (aka Solar Designer) in 2006 to cause a failed setuid() call to send a SIGSEGV to the process, which would presumably terminate those misbehaving programs.

The first solution is not complete because it would still allow users to violate their process limit by using programs that do a setuid() that is not followed by an execve(), but that is a sufficiently rare case that it isn't considered to be a serious problem. Peslyak's solution was seen as too big of a hammer when it was proposed, especially for programs that do check the status of setuid(), and might have proper error handling for that case.

There were no responses to his initial posting, but when he brought it back up on July 6, he was pleasantly surprised to get a positive response from Linus Torvalds:

That led to the patch, which changed do_execve_common() to return an error (EAGAIN) if the user was over their process limit and removed the check from set_user(). The patch was generally well-received, though several commenters were not convinced that it should go into the -rc for 3.0 as Torvalds had suggested. In fact, as Brown dug into the patch, he saw a problem that might need addressing:

Basically, the problem is that switching the process to a new user could now exceed the process limit, but that limit wouldn't actually be enforced until an execve() was done (the failure of which would presumably cause the process to exit). In the interim, any execve() from another of the user's processes would fail. It's not clear how big of a problem that is, though it could certainly lead to unexpected behavior. Brown offered up a patch that would address the problem by adding a process flag (PF_NPROC_EXCEEDED) that would be set if a setuid() caused the process to exceed RLIMIT_NPROC and would then be checked in do_execve_common(). Thus, only the execve() in the offending process would fail.

Kulikov and Peslyak liked the approach, though Peslyak was not convinced it added any real advantages over Kulikov's original patch. He also pointed out that there could be a indeterminate amount of time between the setuid() and execve(), so the RLIMIT_NPROC test should be repeated when execve() is called: "It would be surprising to see a process fail on execve() because of RLIMIT_NPROC when that limit had been reached, say, days ago and is no longer reached at the time of execve()."

So far, Brown has not respun the patch to add that test. There is also the question of whether the problem that Brown is concerned about needs to be addressed at all, and whether it is worth using up another process flag bit (there are currently only three left) to do so. In the end, some kind of fix is likely to go in for 3.1 given Torvalds's interest in seeing this problem with buggy programs disarmed. It's unclear which approach will win out, but either way, setuid() will not fail due to exceeding the allowable number of processes.

As Kulikov and others noted, it is definitely not a bug in the kernel that is being fixed here. But, it is a common enough error in user-space programs—often with dire consequences—which makes it worthwhile to fix as a pro-active security measure. Peslyak listed several recent security problems that arose from programs that do not check the return value from setuid(). He also noted that the problem is not limited to setuid-root programs, as other programs that try to switch to a lesser—differently—privileged user can also cause problems when using setuid() incorrectly.

The impact of this fix is quite small, and badly written user-space programs—even those meant to run with privileges—abound, which makes this change more palatable than some other pro-active fixes. As we have seen before, setuid() is subtle and quick to anger; it can have surprising interactions with other seemingly straightforward security measures. Closing a hole with setuid(), even if the problem lives in user space, will definitely improve overall Linux security.