Privilege escalation via eBPF in Linux 4.9 and beyond [LWN.net]

Jann Horn has reported eight bugs in the eBPF verifier, one for the 4.9 kernel and seven introduced in 4.14, to the oss-security mailing list. Some of these bugs result in eBPF programs being able to read and write arbitrary kernel memory, thus can be used for a variety of ill effects, including privilege escalation. As Ben Hutchings notes, one mitigation would be to disable unprivileged access to BPF using the following sysctl: kernel.unprivileged_bpf_disabled=1. More information can also be found in this Project Zero bug entry. The fixes are not yet in the mainline tree, but are in the netdev tree. Hutchings goes on to say: "There is a public exploit that uses several of these bugs to get root privileges. It doesn't work as-is on stretch [Debian 9] with the Linux 4.9 kernel, but is easy to adapt. I recommend applying the above mitigation as soon as possible to all systems running Linux 4.4 or later."

Privilege escalation via eBPF in Linux 4.9 and beyond

Posted Dec 23, 2017 8:56 UTC (Sat) by bof (subscriber, #110741) [Link]

Oh well... Yet another case where, when baking kernels for production, "oh, a new feature, I want to eventually play with it, let's compile it in", backfires...

Fortunately I've got a somewhat comfortable full host+vm coverage sysctl updater. Festivities saved...

Privilege escalation via eBPF in Linux 4.9 and beyond

Posted Dec 23, 2017 10:34 UTC (Sat) by oliwarner (guest, #81320) [Link] (1 responses)

A sentence, even just a subclause explaining what eBPF is wouldn't hurt this article.

Privilege escalation via eBPF in Linux 4.9 and beyond

Posted Dec 24, 2017 11:40 UTC (Sun) by pclouds (guest, #76590) [Link]

This may help https://lwn.net/Articles/740157/

Privilege escalation via eBPF in Linux 4.9 and beyond

Posted Dec 23, 2017 23:33 UTC (Sat) by bluss (guest, #47454) [Link] (2 responses)

That is quite disheartening.

It would not bring that much assurance of quality, but I assume the verifier is a great target for fuzz testing a la libfuzzer, and it could help finding some bugs.

Privilege escalation via eBPF in Linux 4.9 and beyond

Posted Dec 24, 2017 22:02 UTC (Sun) by nix (subscriber, #2304) [Link]

The ludicrous complexity of perf_event_open() makes comprehensive fuzzing... hard. (Though the same criticism can be levelled at all competing kernel-level technologies, too. perf_event_open() is ludicrously complex because this is a ludicrously complex problem.)

Privilege escalation via eBPF in Linux 4.9 and beyond

Posted Dec 27, 2017 19:23 UTC (Wed) by iabervon (subscriber, #722) [Link]

This is a mismatch between what the verifier thinks the interpreter will do, and what the interpreter will actually do. If you generate random programs, they'll practically never pass verification unless you intentionally generate them to pass verification, in which case either you're generating programs that should pass (and won't hit any of these bugs) or you've worked out the rules the verifier is actually applying, in which case you can just report differences from the actual interpreter without running your fuzzer.

It would probably be more productive to set up a test harness where the verifier exports all of its beliefs at each point in the program, and run random (generally unsafe) programs through both the verifier and the actual interpreter, and check that all the verifier's beliefs are accurate. In the actual kernel, your random programs would probably all be safely rejected, but running them anyway would expose confusion as to what's going on with them.

For example, one of the bugs is that, in some cases, the verifier thinks an instruction will put a certain constant in a register, when it will actually put a different constant in the register. You're probably not going to generate a random program where that instruction is used, plus the program would be safe if the verifier's constant were put in the register, plus the program crashes the kernel with the actual constant put in the register. Eliminating the last two conditions as requirements for detecting a problem would make it plausible to find the issue.

How big is the verifier?

Posted Dec 26, 2017 14:53 UTC (Tue) by welinder (guest, #4699) [Link] (1 responses)

Is it small enough that it can be completely proven to work right?

How big is the verifier?

Posted Jan 5, 2018 14:02 UTC (Fri) by ecree (guest, #95790) [Link]

Almost all of it lives in kernel/bpf/verifier.c, which is 4,684 lines in -rc6 and 5,556 lines in net-next. (The increase mostly comes from Alexei's recent work to add intra-program call insns.) There are a few adjuncts, like the 180-line kernel/bpf/tnum.c, and various header files declaring some of its data structures.
It definitely can't be "completely proven" to be anything, but we do have a few hundred regression tests in tools/testing/bpf/selftests/, some of which (in test_align.c) examine the detailed register state output to ensure the verifier believes only what it's supposed to. However, the majority of the regression tests (chiefly test_verifier.c) just check whether the program is accepted or rejected, and in the latter case, whether the expected error message is produced. It would be nice to have more thorough coverage with test_align-style tests, but they're rather labour-intensive to write and to keep up-to-date when the verifier is changed.
There has been some disagreement lately (not all of it on public lists, but see https://www.spinics.net/lists/netdev/msg474278.html for some of it) about how to better secure eBPF; one suggestion, of limiting more of the functionality to root-only (or even _restoring_ root-only limits that used to exist and were removed by accident), was NACKed by maintainers on the grounds that developers would become sloppy because "it's root-only, no need to be careful". Further, while all of us agree that "simpler code is more secure", we have differing definitions of "simple".
All of which is to say there's a bit of a logjam right now and we don't seem to be able to agree on anything beyond fixing the known bugs; any (brave!) volunteers would be welcome imho.

Privilege escalation via eBPF in Linux 4.9 and beyond

Posted Dec 27, 2017 16:37 UTC (Wed) by eru (subscriber, #2753) [Link] (5 responses)

Can't help thinking this is a story we have heard many times before: an extension language is added to a piece of software to make it more flexible. Encouraged by early success, it is used more and more, and made ever more elaborate, until security problems result.

Privilege escalation via eBPF in Linux 4.9 and beyond

Posted Dec 29, 2017 8:52 UTC (Fri) by flussence (guest, #85566) [Link] (4 responses)

Would even Lua fit that description? From what I understand it's actually pretty secure, but maybe things were different in its early years.

Privilege escalation via eBPF in Linux 4.9 and beyond

Posted Dec 29, 2017 10:13 UTC (Fri) by eru (subscriber, #2753) [Link]

It would depend on how you use Lua. In most cases it is embedded in your application, so that it calls Lua code, and vice versa, and insecurities could result from this interaction, in various ways. I do use Lua in one piece of software I develop, but fortunately it is not security critical in any way. But I think it would be foolish to try to make sshd extensible with Lua, for example.

Privilege escalation via eBPF in Linux 4.9 and beyond

Posted Dec 30, 2017 0:46 UTC (Sat) by wahern (subscriber, #37304) [Link] (2 responses)

It's notable that Lua 5.1 had a bytecode verifier. After a year or so of one community member productively exploring ways to bypass the verifier, they decided to rip it out for the Lua 5.2 release. Their reasoning was that 1) the verifier created a false sense of security, and 2) by removing it they reduced the complexity of the codebase, permitting more investment (from themselves, from the community) in improving correctness of the most critical code, in particular the VM's core interpreter loop. Instead the interfaces for loading code gained a new parameter which controlled whether the loader accepted text chunks, bytecode chunks, or both.

These days the *official* Lua line is that maliciously crafted bytecode is simply unsafe, period. I don't know if they removed all the sanity checks from the interpreter loop or whether they moved in the direction of making the bytecode model intrinsically safer (i.e. removing exploitable semantics). In general, though, the emphasis is usually on simplicity. Notably, IIRC the compiler implements almost no optimizations, translating to bytecode in a single pass.[1] The compiler has rarely, if ever, had exploitable bugs; and it was always a better filter for malicious code than the bytecode verifier. The removal of the bytecode verifier simply recognized the reality of the situation. And at least in the context of Lua, the compiler is the better constraint checker even in the abstract.

FWIW, there are occasional bugs found in the Lua VM that can be tickled by innocuous-seeming code constructs. See https://www.lua.org/bugs.html IMO PUC Lua is a relatively safe implementation, especially compared to the standard engines for Python, Perl, Ruby, etc, but I would never run untrusted code from the wild without also isolating the process in a very strict seccomp, pledge, or VM sandbox. As a practical matter I think Lua's sandboxing capabilities are more oriented toward, e.g., isolating plugins. Lua's language semantics are first-class--lexical closures, stackful coroutines, guaranteed tail-call optimization, ephemerons--and non-trivial. Lua emphasizes simplicity but it doesn't handicap language constructs in the face of implementation shortcomings; they'll refactor the implementation. (The Lua implementation is small, but it's *astoundingly* small for a language with more powerful constructs than JavaScript, Tcl, or arguably even Perl 6; for one which also maintains near perfect feature symmetry through it's C API; and for one written in strict ISO C.) Lua's facilities like instruction hooks, environment tables, etc are really powerful sandboxing tools, but they don't save you from many of the bugs in the underlying implementation.

[1] Most of the performance advantage of LuaJIT comes from the smarter compiler and VM (the bytecode interpreter loop is carefully handwritten in assembly for each architecture), not from actual JIT'ing. Not coincidentally LuaJIT is also notorious (in some circles) for subtle and difficult bugs.

Privilege escalation via eBPF in Linux 4.9 and beyond

Posted Jan 1, 2018 20:21 UTC (Mon) by eru (subscriber, #2753) [Link] (1 responses)

In my Lua-extended application I did not even bother with bytecode (probably precompiled scripts would work, but I never advertised this to users, or even tested it myself). It would just have complicated the usage of the program needlessly, since from the user's point of view, loading the Lua source text is instantaneous anyway on the kinds of computers it is used on.

JavaScript, supplied as source (even if often obfuscated) seems to cause less security problems in web pages than byte-compiled Java applets, which are supposed to be verified when loading. Perhaps trying to run untrusted bytecodes safely is a bad idea, and it is better to stay at the source level?

Privilege escalation via eBPF in Linux 4.9 and beyond

Posted Jan 3, 2018 10:20 UTC (Wed) by intgr (subscriber, #39733) [Link]

But merging a source-to-eBPF compiler into the kernel doesn't seem like it would make us more secure. And there are already multiple languages that can be compiled into eBPF.

> seems to cause less security problems in web pages than byte-compiled Java applets

I think the majority of security issues came from the way Java attempted to sandbox applets from privileged calls (the Security Manager, basically a blacklist of methods) and vulnerabilities in exposed methods. These would be exploitable regardless of whether it's bytecode or compiled from source.