Impedance matching for BPF and LSM

The "kernel runtime security instrumentation" (KRSI) patch set has been making the rounds over the past few months; the idea is to use the Linux security module (LSM) hooks as a way to detect, and potentially deflect, active attacks against a running system. It does so by allowing BPF programs to be attached to the LSM hooks. That has caused some concern in the past about exposing the security hooks as external kernel APIs, which makes them potentially subject to the "don't break user space" edict. But there has been no real objection to the goals of KRSI. The fourth version of the patch set was posted by KP Singh on February 20; the concerns raised this time are about its impact on the LSM infrastructure.

The main change Singh made from the previous version effectively removed KRSI from the standard LSM calling mechanisms by using BPF "fexit" (for function exit) trampolines on the LSM hooks. That trampoline can efficiently call any BPF programs that have been placed on the hook without the overhead associated with the normal LSM path; in particular, it avoids the cost of the retpoline mitigation for the Spectre hardware vulnerability. The KRSI hooks are enabled by static keys, which means they have zero cost when they are not being used. But it does mean that KRSI looks less like a normal LSM as Singh acknowledged: "Since the code does not deal with security_hook_heads anymore, it goes from 'being a BPF LSM' to 'BPF program attachment to LSM hooks'."

Casey Schaufler, who has done a lot of work on the LSM infrastructure over the last few years, objected to making KRSI a special case, however: "You aren't doing anything so special that the general mechanism won't work." Singh agreed that the standard LSM approach would work for KRSI, "but the cost of an indirect call for all the hooks, even those that are completely unused is not really acceptable for KRSI’s use cases".

Kees Cook focused on the performance issue, noting that making calls for each of the hooks, even when there is nothing to do, is common for LSMs. Of the 230 hooks defined in the LSM interface, only SELinux uses more than half (202), Smack is next (108), and the rest use less than a hundred—several use four or less. He would like to see some numbers on the performance gain from using static keys to disable hooks that are not required. It might make sense to use that mechanism for all of LSM, he said. Singh agreed that it would be useful to have some performance numbers; "I will do some analysis and come back with the numbers." It is, after all, a bit difficult to have a discussion about improving performance without some data to guide the decision-making.

There are several intertwined pieces to the disagreement. The LSM infrastructure has generally not been seen as a performance bottleneck, at least until the advent of KRSI; instead, over recent years, the focus has been on generalizing that infrastructure to support arbitrary stacking of multiple LSMs in a running system. That has improved the performance of handling calls into multiple hooks (when more than one LSM defines one for a given operation) over the previous mechanism, but it was not geared to the high-performance requirements that KRSI is trying to bring to the LSM world.

In addition, the stacking work has made it so that LSMs can be stacked in any order; each defined hook for a given operation is called in order, the first that denies the operation "wins" and the operation fails without calling any others. That infrastructure is in place, but KRSI upends it to a certain extent. KRSI comes from the BPF world, so the list traversal and indirect calls used by the LSM infrastructure are seen as performance bottlenecks. KRSI always places itself (conceptually) last on the list and uses the BPF trampoline to avoid that overhead. That makes it a special case, unlike the other "normal" stackable LSMs, but that may be a case of a premature optimization, as Cook noted.

BPF developer Alexei Starovoitov does not see it as "premature" at all, however. "I'm convinced that avoiding the cost of retpoline in critical path is a requirement for any new infrastructure in the kernel." He thought that the LSM infrastructure should consider using static keys to enable its hooks, and that the mechanism employed by KRSI should be used there:

But the insistence on treating KRSI differently than the other LSMs means that perhaps KRSI should go its own way—or work on improving the LSM infrastructure as a whole. Schaufler said that he had not "gotten that memo" on avoiding retpolines and that the LSM infrastructure is not new. He is interested in looking at using static keys, but is concerned that the mechanism is too specific to use in the general case, where multiple LSMs can register hooks to be called:

The new piece is not KRSI per se, Singh said, but the ability to attach BPF programs to the security hooks is new. There are techniques available to make that have zero cost, so it makes sense to use them:

if (trace_foo_enabled) // <-- Overhead here, solved with static key
   trace_foo(a);  // <-- retpoline overhead, solved with fexit trampolines

The difficulty is that the LSM interface came about under a different set of constraints, Schaufler said. Those constraints have changed over time and the infrastructure is being worked on to improve its performance, but it still needs to work with the existing LSMs:

The goal of eliminating the retpoline overhead is reasonable, Cook said, but the LSM world has not yet needed to do so. "I think it's a desirable goal, to be sure, but this does appear to be an early optimization." He noted there is something of an impedance mismatch; the BPF developers do not want to see any performance hit associated with BPF, but the LSM developers "do not want any new special cases in LSM stacking". So he suggested adding a "slow" KRSI that used the LSM stacking infrastructure as it is today, followed by work to optimize that calling path.

But Starovoitov thought that KRSI should perhaps just go its own way. He proposed changing the BPF program type from BPF_PROG_TYPE_LSM to BPF_PROG_TYPE_OVERRIDE_RETURN and moving KRSI completely out of the LSM world: "I don't see anything in LSM infra that KRSI can reuse." He does not see a slow KRSI as an option and suggested that perhaps the LSM interface and the new BPF program type should be made mutually exclusive at kernel build time:

There are, of course, lots of users of the LSM interface, including most distributions, so it might difficult to go that route, Schaufler said. But Singh noted that the users of a BPF_PROG_TYPE_OVERRIDE_RETURN feature may be highly performance-sensitive such that they already disable LSMs "because of the current performance characteristics". But Singh did think that BPF_PROG_TYPE_OVERRIDE_RETURN might be useful on its own, separate from the KRSI work; he plans to split that out into its own patch set. He agreed with Cook's approach, as well, and plans to re-spin the KRSI patches to use the standard LSM approach as a starting point; "we can follow-up on performance".

The clash here was the classic speed versus generality tradeoff that pops up in the kernel (and elsewhere) with some frequency. The BPF developers are laser-focused on their "baby"—and squeezing every last drop of performance out of it—but there is a wider world in kernel-land, some parts of which have different requirements. It would seem that a reasonable compromise has been found here. Preserving the generality of the LSM approach while gaining the performance improvements that the BPF developers have been working on would be a win for both, really—taking the kernel and its users along for the ride.