LWN: Comments on "Managing sysctl knobs with BPF"

Managing sysctl knobs with BPF

yunwei37 — Wed, 15 Feb 2023 17:06:51 +0000

We have created a prototype runtime and toolchain: wasm-bpf, which runs Wasm in user space and eBPF in kernel space, and I think maybe just doing things like this can combine the advantages from both?

Wasm-bpf is a WebAssembly eBPF library, toolchain and runtime powered by CO-RE (Compile Once – Run Everywhere) libbpf and WAMR. It can help you build almost every eBPF programs or use cases to Wasm module.

https://github.com/eunomia-bpf/wasm-bpf

Examples of eBPF programs written in C, Rust and compiled to Wasm are provided, covering the use cases from tracing, networking to security.

Wasm-bpf offers several advantages for eBPF:

- Secure and reliable execution environment with Wasm's isolation in user space.
- Easy distribution and management with the Wasm container ecosystem and toolchain, for example, Wasm OCI images or docker for Wasm.
- Cross-language support for over 30 programming languages for eBPF user space programs
- Agile development with the ability to dynamically load and unload eBPF plugins when running.
- Lightweight and efficient, WebAssembly applications consume significantly less resources compared to Linux container applications.

reference:
- docker for wasm: https://docs.docker.com/desktop/wasm/
- awesome-wasm-langs: https://github.com/appcypher/awesome-wasm-langs
- WAMR: https://github.com/bytecodealliance/wasm-micro-runtime

Re: More anti-BPF FUD from Cyberax

togga — Tue, 16 Apr 2019 18:58:56 +0000

> "It's an option" is not an excuse for a broken feature.
I just don't see how this feature is broken, it opens up lots of features/possibilities. Hopefully there's good filtering and sense when reviewing/accepting new features and requires enough thought by users so it doesn't derail (look at Python for an example of that).

Re: More anti-BPF FUD from Cyberax

togga — Tue, 16 Apr 2019 18:47:37 +0000

> You look at the logs and see which rule caused the request to fail.
Yes. This is like looking at BPF printk's and perf events to user-space right? At assembly-level AppArmor is probably worse?

> BPF will always be hard to debug, it's almost a general-purpose VM and you'll have to debug it at basically the assembly level.
Correct. At least the assembly level is much easier here than for the AppArmor case since it's higher level. Complexity with the JIT could be an issue however. Another issue with BPF could be the usual effects of "no taste/craftsmanship" when given flexibility (over-engineering at one end and the tunnel-visioned mess on the other).

> Right now there are no tools for simulation. There are no tools to create BPF filters as well (nothing like "learning mode" in AppArmor and SELinux).
Lots of opportunities here :-)

Re: More anti-BPF FUD from Cyberax

Cyberax — Mon, 15 Apr 2019 21:17:03 +0000

> BPF adds features here, it doesn't cause regressions to anything. Besides, there is an BPF flag when you build your kernel you can say no to.
"It's an option" is not an excuse for a broken feature.

Re: More anti-BPF FUD from Cyberax

togga — Mon, 15 Apr 2019 21:00:10 +0000

> It's not acceptable to cause regressions in practical usability.

BPF adds features here, it doesn't cause regressions to anything. Besides, there is an BPF flag when you build your kernel you can say no to.

Re: More anti-BPF FUD from Cyberax

Cyberax — Mon, 15 Apr 2019 20:58:17 +0000

> How do you debug AppArmor? Is that fun?
You look at the logs and see which rule caused the request to fail.

> BPF might be hard to debug today, but the alternative is extending features to a number of DSL:s and specialized modules for a subset of possibilities.
BPF will always be hard to debug, it's almost a general-purpose VM and you'll have to debug it at basically the assembly level.

> BPF should at least be straightforward to simulate (or run as is) in user-space (think regression-tests) and once you crack this debug-nut, you'd be set for the BPF invasion.
Right now there are no tools for simulation. There are no tools to create BPF filters as well (nothing like "learning mode" in AppArmor and SELinux).

Re: More anti-BPF FUD from Cyberax

togga — Mon, 15 Apr 2019 20:50:36 +0000

> There is no source code for BPF. How are you going to debug it?

How do you debug AppArmor? Is that fun?

BPF might be hard to debug today, but the alternative is extending features to a number of DSL:s and specialized modules for a subset of possibilities.

BPF should at least be straightforward to simulate (or run as is) in user-space (think regression-tests) and once you crack this debug-nut, you'd be set for the BPF invasion.

Managing sysctl knobs with BPF

Cyberax — Mon, 15 Apr 2019 16:07:19 +0000

> Like systemd, BPF is well designed, well documented, rapidly developed and easier to use than some of the mechanisms it is displacing.
Except it's not. Systemd had well-designed tools for debugging and diagnostics from the beginning. BPF really has nothing so far.

Managing sysctl knobs with BPF

alison — Mon, 15 Apr 2019 13:42:55 +0000

>BPF is already running amok and it's basically impossible to debug when it inevitably goes wrong.

One does get the feeling that BPF is reiterating the history of systemd, in that it is absorbing and subsuming a variety of apparently unrelated functionality and extending into all parts of Linux. Like systemd, BPF is well designed, well documented, rapidly developed and easier to use than some of the mechanisms it is displacing. As this discussion points out, that doesn't mean that the current rapid expansion of its capabilities and our reliance on them is necessarily a good idea.

Managing sysctl knobs with BPF

lambda — Sat, 13 Apr 2019 00:51:28 +0000

Besides the running time, the lack of Turing-completeness is used to be able to trace all possible execution paths to prove static guarantees about the program behavior, such as doing static bounds checking. This allows the eBPF process to be able to access packet data without concern about accessing arbitrary data.

To do this for a system that allows for loops and/or recursion, you'd have to use more sophisticated analysis such as abstract interpretation, or something like bounded model checking tied to bounds on the execution. This seems like a much more heavyweight analysis to put in the kernel.

As mentioned, I can see some value in focusing on a single toolchain for various forms of safe, verified JIT compilation, but I don't see a simple approach to "just use WASM" to replace the whole classic BPF and eBPF use cases.

Managing sysctl knobs with BPF

Cyberax — Fri, 12 Apr 2019 20:54:44 +0000

> For instance, WASM is designed to be Turing-complete, while BPF is explicitly designed to not allow looping and thus be guaranteed to terminate in an amount of time predictable from the size of the program.
In practice, BPF can run for a fair amount of time because of function calls. And you also can limit the duration of WASM programs by only giving them a certain time budget and terminating them once they go over it.

Managing sysctl knobs with BPF

lambda — Fri, 12 Apr 2019 20:26:09 +0000

WASM and BPF have some overlap, but also some different requirements.

For instance, WASM is designed to be Turing-complete, while BPF is explicitly designed to not allow looping and thus be guaranteed to terminate in an amount of time predictable from the size of the program.

And of course, BPF long predates WASM, first being published in 1992, and eBPF started as an internal representation for JITing BPF before it was exposed as a compilation target on its own. eBPF appears to have been initially announced in 2014, while WASM wasn't announced until 2015.

While it might be an interesting project to figure out commonalities between WASM and eBPF and potentially have a limited WASM mode with the same kind of control flow restrictions as eBPF, that seems like a decently sized research and specification project, and if you wanted to replace the eBPF engine in the kernel with that, you'd have to write a verifier and JIT for that format that could be merged into the kernel, so likely couldn't reuse a lot of the existing WASM ecosystem which is mostly written in C++ and Rust. And of course for compatibility, you'd need to support frontends for classic BPF and eBPF.

While it might be nice to have more resources focused on a single safe, verifiable, embeddable virtual machine that could scale to these different use cases, I don't see an easy way to just drop eBPF and stick WASM into the kernel instead.

Re: More anti-BPF FUD from Cyberax

rahulsundaram — Fri, 12 Apr 2019 01:13:20 +0000

> It has to be reasonable. The mainline kernel is not a dumping ground for random pet features.

In many ways, that's precisely how it is.

There is no overarching kernel design. It is mostly a collection of random pet features. If the pet features happened to be usable and useful in ways you don't like, then tough luck.

Re: More anti-BPF FUD from Cyberax

Cyberax — Thu, 11 Apr 2019 23:03:19 +0000

> Large organisations are poor design, just as large programs are.
Sure. But you can't avoid them and they do a lot of things well.

> If the documentation is inadequate or unclear, pester the BPF core developers to improve it. On that, I'll side with you — I had to badger them repeatedly to write a spec for BTF.
I'm doing this.

> Ok, so you're just interested in spreading FUD, then. Your problem isn't with BPF as a design or a concept.
In the end everything is Turing machine approximation. So yes, my problems are with this particular implementation. It's not acceptable to cause regressions in practical usability.

> Now you're being silly. It might be a pre-requisite for it being a good idea to use it in your organisation; but an opt-in feature doesn't have to be perfect to be merged
It has to be reasonable. The mainline kernel is not a dumping ground for random pet features.

Re: More anti-BPF FUD from Cyberax

ecree — Thu, 11 Apr 2019 22:51:35 +0000

> As most large organizations...

Large organisations are poor design, just as large programs are.

> How can I do this? Is there a tutorial for it?

If the documentation is inadequate or unclear, pester the BPF core developers to improve it. On that, I'll side with you — I had to badger them repeatedly to write a spec for BTF.

> I'm not interested in principles.

Ok, so you're just interested in spreading FUD, then. Your problem isn't with BPF as a design or a concept.

> If not, then it needs to be a pre-requisite for the kernel merge.

Now you're being silly. It might be a pre-requisite for it being a good idea to use it in your organisation; but an opt-in feature doesn't have to be perfect to be merged, it just has to have a stable ABI. Which BPF does; adding verdict tracing wouldn't require changes to the existing ABI.

Re: More anti-BPF FUD from Cyberax

Cyberax — Thu, 11 Apr 2019 22:27:05 +0000

> That sounds like "your organisation is dysfunctional", not "BPF is bad".
As most large organizations...

> You can inspect the disassembly of the BPF program (in future including, as I said, BTF debuginfos); you can debug them with a kernel debugger if you really need to (directly analogous to an LSM; both run in kernel space);
How can I do this? Is there a tutorial for it?

> So, no, then. All AppArmor can do is tell you what syscalls etc. it denied; a BPF security program can in principle do the same thing
I'm not interested in principles. Is there such an infrastructure right now? If not, then it needs to be a pre-requisite for the kernel merge.

> I'd be interested to hear what you think should replace XDP, then, as doubtless would Cloudflare and Facebook.
I wouldn't mind if it just stays there. I don't care about it being used outside of the network stack.

Re: More anti-BPF FUD from Cyberax

ecree — Thu, 11 Apr 2019 22:22:28 +0000

> I'm debugging my software not working on a particular version of the fleet that is controlled by another team.

That sounds like "your organisation is dysfunctional", not "BPF is bad".

> I can inspect binaries with GDB, I can see their activity with strace(), I can look for source code in repositories, etc.
>
> None of this is possible with BPF.

You can inspect the disassembly of the BPF program (in future including, as I said, BTF debuginfos); you can debug them with a kernel debugger if you really need to (directly analogous to an LSM; both run in kernel space); you can look for the BPF program's source code in repositories too (e.g. it might be https://github.com/cilium/cilium that's running).

> Yes. Not quite single step, but explain failures

So, no, then. All AppArmor can do is tell you what syscalls etc. it denied; a BPF security program can in principle do the same thing (there may not be built-in mechanisms to do that right now, but there's nothing intrinsic about the BPF model that prevents them, and in the meantime there's always bpf_trace_printk()).

> BPF should have never been implemented at all.

I'd be interested to hear what you think should replace XDP, then, as doubtless would Cloudflare and Facebook.

Brendan Gregg might also have a thing or two to say about tracing. (You're the one who was saying debuggability was so important!)

Re: More anti-BPF FUD from Cyberax

Cyberax — Thu, 11 Apr 2019 21:34:48 +0000

> With BTF, soon BPF programs will come with complete source-level debuginfo. Also, if you're going around installing BPF programs without having the corresponding source, you're naturally going to have all the same problems as if you drop an object file in /bin and lose the source; how is that BPF's fault?
I'm not installing anything. I'm debugging my software not working on a particular version of the fleet that is controlled by another team. Or I'm debugging a server after its administrator had died from a heroin overdose (real example).

> drop an object file in /bin and lose the source; how is that BPF's fault?
I can inspect binaries with GDB, I can see their activity with strace(), I can look for source code in repositories, etc.

None of this is possible with BPF.

> Can you single-step an LSM implementation?
Yes. Not quite single step, but explain failures: https://wiki.debian.org/AppArmor/Debug

> I was merely saying that AppArmor is not a replacement for cgroup-bpf either, because if it was, cgroup-bpf wouldn't have been invented.
BPF should have never been implemented at all.

Re: More anti-BPF FUD from Cyberax

ecree — Thu, 11 Apr 2019 20:51:14 +0000

> There is no source code for BPF. How are you going to debug it?
With BTF, soon BPF programs will come with complete source-level debuginfo. Also, if you're going around installing BPF programs without having the corresponding source, you're naturally going to have all the same problems as if you drop an object file in /bin and lose the source; how is that BPF's fault?

> Can you single-step BPFs attached to /proc files?
Can you single-step an LSM implementation? Both are just "some executable bytes in the kernel", and debugging facilities have to be built. In the case of LSM, they have to be written into the kernel module, whereas with BPF, the user can insert their own debugging (it is not impossible, for instance, to patch bpf_trace_printk()s into a program that you only have in binary form).

> cgroups are not a replacement for AppArmor.
I wasn't saying they were.

I was merely saying that AppArmor is not a replacement for cgroup-bpf either, because if it was, cgroup-bpf wouldn't have been invented.

Re: More anti-BPF FUD from Cyberax

Cyberax — Thu, 11 Apr 2019 20:00:05 +0000

> Which in my opinion is a much better approach than every piece of kernel policy having its own virtual machine (netfilter comes to mind). If a particular policy API wants to have its own DSL, it can do that while still using eBPF for the mechanism. And because the DSL is purely a userspace thing, no-one is _forced_ to use it — e.g. when writing functional tests for XDP I found it very helpful to be able to hand-craft the assembler rather than pulling in a gigantic blob of LLVM just to do the same thing in C.
You have a production machine. You have a problem (some software doesn't work). There is no source code for BPF. How are you going to debug it? Can you single-step BPFs attached to /proc files? Is there a way to get an execution trace?

I have already had a wonderful opportunity to do exactly this. IT. WAS. NOT. FUN.

BPF violates the cardinal rule of software design - your software must be easy to troubleshoot.

> Which is why everyone is happily using AppArmor and never felt the need to invent cgroups and BPF-based security. Oh wait...
cgroups are not a replacement for AppArmor.

Re: More anti-BPF FUD from Cyberax

ecree — Thu, 11 Apr 2019 19:35:47 +0000

> BPF is NOT a language. It's a virtual machine
With the result that there are multiple different languages targeting that backend, from the high-level (bpftrace) through the low (C) right down to assembler (ebpf_asm).

Which in my opinion is a much better approach than every piece of kernel policy having its own virtual machine (netfilter comes to mind). If a particular policy API wants to have its own DSL, it can do that while still using eBPF for the mechanism. And because the DSL is purely a userspace thing, no-one is _forced_ to use it — e.g. when writing functional tests for XDP I found it very helpful to be able to hand-craft the assembler rather than pulling in a gigantic blob of LLVM just to do the same thing in C.

> with a particularly nasty instruction encoding
Firstly, there's nothing nasty about eBPF's encoding; it's cleaner and more orthogonal than most hard ISAs. Secondly, why do you care what the instruction encoding is, unless you're actually writing the toolchain?

> it's not technically Turing-complete
A deliberate design decision, ensuring that eBPF expresses procedurally only those policies that could, at the cost of far greater complexity in both interface and implementation, be expressed declaratively.

> it can be extremely difficult to understand
Only to the extent necessitated by the complexity of policies it allows the user to configure. You can write an eBPF _program_ that's difficult to understand, but that doesn't make eBPF _itself_ difficult to understand.

> Meanwhile, AppArmor policies are simple text files which are self-explanatory.
Which is why everyone is happily using AppArmor and never felt the need to invent cgroups and BPF-based security. Oh wait...

Managing sysctl knobs with BPF

Cyberax — Thu, 11 Apr 2019 17:10:46 +0000

BPF is NOT a language. It's a virtual machine with a particularly nasty instruction encoding. And while it's not technically Turing-complete, it can be extremely difficult to understand.

Meanwhile, AppArmor policies are simple text files which are self-explanatory.

Managing sysctl knobs with BPF

rahulsundaram — Thu, 11 Apr 2019 13:57:08 +0000

> That's just trading one domain-specific language (and associated knowledge) for another, what value is there in that?

Also BPF is supported by more distributions than AppArmor is ever going to be.

Managing sysctl knobs with BPF

johill — Thu, 11 Apr 2019 13:10:55 +0000

That's just trading one domain-specific language (and associated knowledge) for another, what value is there in that?

Managing sysctl knobs with BPF

Cyberax — Thu, 11 Apr 2019 08:25:40 +0000

Use an AppArmor policy for that.

Managing sysctl knobs with BPF

johill — Thu, 11 Apr 2019 08:21:36 +0000

But that only addresses the "container may not change anything at all" case.

It seems valid, and possible with the BPF approach, that a container may be allowed to change certain things, e.g. /proc/sys/net/ipv4/conf/<interface the container controls>/*.

Managing sysctl knobs with BPF

josh — Wed, 10 Apr 2019 21:14:44 +0000

I can't help but wonder if /proc/sys could just split out into a separate sysctlfs that distros can mount by default, and that containers can then prevent container-root from mounting while allowing proc.

Managing sysctl knobs with BPF

thithib — Wed, 10 Apr 2019 08:03:44 +0000

> The sysctl hook is just another example of how the kernel's API is being transformed by BPF; expect a lot more of these hooks to be added in the future.

I get the advantages of BPF over e.g. a new LSM hook in this particular case.
What I'm uncomfortable with is, I guess, the fact that it's one more step towards making BPF a mandatory skill for sysadmins trying to correctly enforce their security policy.

Managing sysctl knobs with BPF

moorray — Wed, 10 Apr 2019 05:16:07 +0000

I think you can do some of that with Landlock.

Managing sysctl knobs with BPF

roc — Wed, 10 Apr 2019 04:03:52 +0000

What really bothers me about BPF is reimplementing a general-purpose virtual machine one feature at a time.

Instead of reimplementing WebAssembly badly, just go ahead and support it in the kernel.

Managing sysctl knobs with BPF

ebiederm — Wed, 10 Apr 2019 03:25:49 +0000

New sysctl knobs are no longer exposed via the binary sysctl paths. Only the binary sysctl emulation layer even knows the old binary paths.

On Linux unlike BSD, the binary mechanism is an emulation layer of /proc/sys. While at one point the binary layer was almost equal, that is no longer the case.

The binary layer kept having conflicting paths added, was not used, was never tested, and had security holes the /proc/sys path did not. So a while ago I just reduced it to an emulation layer so we could forget about it.

Which is a long way of saying about the only thing the BSD and Linux sysctl implementations have in common is their name.

Managing sysctl knobs with BPF

wahern — Wed, 10 Apr 2019 03:00:18 +0000

The BPF train left the station years ago, LSM is just another inevitable victim.

What I find interesting is that the traditional BSD sysctl interface uses integer constants. And while most distributions have disabled the sysctl syscall, AFAIU /proc/sys was and continues to be just another interface to the same underlying mechanism. Are newer knobs no longer exposed via the old syscall interface? I ask because the new BPF interface exposes string paths, which seems not just unwise but possibly unnecessary if there already exist fixed integer identifiers.

Managing sysctl knobs with BPF

Cyberax — Wed, 10 Apr 2019 01:18:51 +0000

Please stop. Just stop.

BPF is already running amok and it's basically impossible to debug when it inevitably goes wrong. You want to limit containers? Work with existing LSM folks. If the existing LSMs are not enough then extend THEM.

Oh, I see. People think that LSM equals unusable SELinux and LSM multiplexing is still not a thing.

Managing sysctl knobs with BPF

pj — Wed, 10 Apr 2019 00:50:52 +0000

Obvious follow-on question: Can other LSM functionality be replaced with hooks for BPF programs? Are such things always better/more general/appropriate than LSMs?