User namespaces + overlayfs = root privileges

The user namespaces feature is conceptually fairly straightforward—allow users to run as root in their own space, while limiting their privileges on the system outside that space—but the implementation has, perhaps unsurprisingly, proven to be quite tricky. There are some assumptions about user IDs and how they operate that are deeply wired into the kernel in various subsystems; shaking those out has taken some time, which led to some hesitation about enabling the feature in distribution kernels. But that reluctance has largely passed at this point, which makes the recent discovery of a root-privilege escalation using user namespaces and the overlay filesystem (overlayfs) that much more dangerous.

The basic idea, as described by "halfdog" in a blog post, is that a regular user can create new mount and user namespaces, mount an overlayfs inside them, and exploit a hole in the overlayfs implementation to create a setuid-root binary that can be run from outside the namespace. Effectively, a regular user can create a root-privileged binary and do whatever they want with it—thus it is a complete system compromise.

The exploit uses another property of namespaces that has always seemed like something of a bug: the /proc filesystem provides a route for processes outside of a namespace to "see" inside it. In this case, the overlayfs mounted inside the namespace can be accessed from outside of it by using information on the mounts inside the namespace via /proc/PID/cwd. As halfdog put it:

The exploit [C program] works like this:

• New mount and user namespaces are created for the process.
• That process then mounts an overlayfs atop /bin using temporary directories for the overlayfs "upperdir" and "workdir" directories. A writable overlayfs must have both of these directories; upperdir holds the files/directories that have been changed, while workdir is used as a work space to enable atomic overlayfs operations.
• The process inside the namespaces changes its working directory to the overlayfs, thus making it visible outside of the namespaces by way of /proc/PID/cwd.
• The process changes the su binary (in /bin) to be world-writable, but does not change the owner. That results in a new file being created in the upper overlay directory.
• A process outside of the namespaces writes anything it wants to that file without changing the setuid bit (more on that coming).
• The outer process then runs that su with root privileges.

That seems reasonably straightforward, but there is one difficulty: writing to a root-owned, setuid-enabled file from a non-root process will remove the setuid bit, which defeats the whole thing. So a variant of another exploit (SetgidDirectoryPrivilegeEscalation) described by halfdog is used to trick the setuid-root mount program into writing an ELF binary to the file. Since mount is owned by root, the write doesn't remove setuid, resulting in an setuid-root program with contents controlled by a regular user (attacker).

Because the process inside the namespaces made the file world-writable, a regular user outside the namespaces can run mount with its stderr hooked to the file of interest (which, crucially, only requires an open() that doesn't revoke setuid). Then, when mount writes to the file, it is doing so as root, so it doesn't trigger the setuid revocation either. The file cannot be written inside the namespaces, which would be easier since it does not require the /proc/PID/cwd dance, as it would be done as root inside the user namespace. Since that is not the same as root outside that namespace, the setuid revocation would still occur.

Perhaps a final entry could be made to the list above: "Profit!".

The fix is fairly simple; it was committed by Miklos Szeredi in early December and was merged into 4.4-rc4 (without any mention of the security implications). According to Al Viro's commit message, overlayfs was "too enthusiastic about optimizing ->setattr() away". It combined two operations that should have been done separately, which led to the creation of the setuid-root file in the upper overlay filesystem. After the fix it won't be possible to change su to be world-writable while still retaining root ownership and the setuid bit in the overlay.

The second exploit that was used to write the file appears to not yet have been fixed. Halfdog's description points to some email messages discussing the problem and possible solutions with several kernel developers, but no real resolution is evident. Obviously, if there was no way to write the overlay file, this particular avenue for exploiting the overlayfs bug would not have worked, but it is unknown if there are more ways to skin that particular cat.

All of the myriad interactions between various kernel subsystems (capabilities, namespaces, security modules, filesystems, /proc, and so on), especially given the "no regressions for user space" policy, make these kinds of bugs pretty much inevitable. One could imagine tossing out a bunch of currently expected behavior to simplify the complexity of those interactions, but that is not going to happen, so these types of problems will crop up from time to time.

This episode is also a reminder of the ingenuity that can go into an exploit of this kind. Other user namespace exploits and, indeed, exploits of bugs in other programs and kernel features have shown similar levels of cleverness. Often stringing together a few seemingly low-severity vulnerabilities results in something that almost appears as if it has exceeded the sum of its parts. Of course, white hats are not the only ones with the required level of skill, which makes efforts like the Kernel self-protection project, as well as other analysis and hardening projects, that much more important.