LWN: Comments on "Comparing SystemTap and bpftrace"

Comparing SystemTap and bpftrace

ema — Tue, 20 Apr 2021 07:35:01 +0000

Thank you for your kind words Sanjeev. Without the suggestions and the editing work from the LWN folks it would not have been half as clear, believe me. :)

Comparing SystemTap and bpftrace

ringerc — Tue, 20 Apr 2021 01:22:24 +0000

I split the details into a child comment so the main one wouldn't be too long.

SystemTap is widely available even for older systems, though the packaged versions are usually older so it's a bit of a pain to write scripts that work with them. It's easy to compile if you're allowed to install the needed toolchain and dependencies on the target and you have the time, but that adds to the hassle. Especially when you're not hands-on and you just want the other end (customer, or whatever) to run a tapscript for you.

Additionally, for its most fully featured and default runtime (kmod) SystemTap requires kernel headers and preferably debuginfo. These are frequently unavailable for whatever older kernel point release happens to be running on the target system at the time you need to run some tracing tools. Or at best you have to go digging manually through some archive of old packages that have aged out of the main repositories for the OS. The stap-prep tool can't usually find them for you. So to reliably use systemtap's kmod runtime you need to plan ahead and install kernel headers and debuginfo whenever you update the kernel, which nobody ever does. This drastically limits its practical utility.

But lots of eBPF features and helper functions are only available in much newer kernels. On widely deployed "enterprise" system kernels it's basically useless for nontrivial userspace tracing and analysis. eBPF is quite fragile in the face of kernel version changes as soon as you step outside the canned tracepoints, and the set of helper functions is extremely limited.

Even if you can run your bpf scripts, your userspace stacks are going to look like "-" most of the time, because everything is compiled with -fomit-frame-pointer. AFAICS most bpf tools don't handle external DWARF debuginfo or use tools like libunwind to help them out. So you land up having to recompile with -fno-omit-frame-pointer and use unstripped binaries with debuginfo in the main binary. This basically means you can't do much tracing of packaged userspace binaries as are the norm on production systems.

SystemTap on the other hand will not only get you your userspace stacks using DWARF detached debuginfo, it'll now even talk to a debuginfod to download symbols for you during probe compilation. It'll walk userspace pointers chains, examine struct members, recursively print structs, handle unions and so much more using simple built-in syntax. So it's currently infinitely more powerful for userspace probing and analysis ...

... or it would be if only you could find and install the kernel headers.

SystemTap also has 'dyninst' and 'bpf' runtimes, which entirely avoid the need for kernel headers and can often be used without kernel debuginfo. But a considerable number of the built-in systemtap "tapsets" rely on embedded-C code written for kernelspace, which simply won't work for a dyninst or bpf tapscript. Or they rely on helper functions exported by the kmod runtime that are not implemented for the dyninst or bpf runtimes. So in practice most of your existing systemtap scripts won't work, and scripts are more difficult to write for the dyninst or bpf runtimes.

Additionally, the dyninst runtime requires that you wrap the target using LD_PRELOAD. So it's cool for development and QA work but for a production system it's often impractical, as you frequently want to non-intrusively trace an already-server running process.

This means you can't usually apply eBPF or use SystemTap with any of its runtimes to any system you encounter in the wild.

Comparing SystemTap and bpftrace

ringerc — Tue, 20 Apr 2021 01:21:52 +0000

Some key operational distinctions exist too, especially for userspace tracing, which is where my interests lie.

The TL;DR is that in practice the effective, reliable use of any of these tools tends to require that you be able to plan ahead in order to install necessary debuginfo, kernel headers, utilities etc well ahead of time, before they age out of repositories. For best results you'll want a much newer kernel than the enterprise-y distro defaults too. So they work best in tightly controlled farms of machines where the people who care about tracing can control how the systems are installed and updated.

* SystemTap kmod mode needs kernel headers and prefers kernel debuginfo. Both age out of repos quickly.
* SystemTap dyninst runtime requires restarting the target so it can LD_PRELOAD, and is more limited than kmod runtime
* Effective eBPF userspace tracing in practice requires quite new kernels, so bpftrace, bcc, etc are hard or impractical to apply to older kernels common in the wild
* SystemTap bpf runtime is limited by the same kernel version concerns as bcc etc *and* its own systemtap-specific limitations.
* SystemTap doesn't usually work on kernels newer than the systemtap release due to internal kernel API changes. It spews compile errors. So you usually need to get a newer systemtap to work with newer kernels.
* bpftrace and bcc don't currently handle detached DWARF debuginfo, and don't even handle binaries built with the x64 default -fomit-frame-pointer compile flag properly.
* The rich debuginfo based access to userspace state available in systemtap is mostly absent from bpf tooling targeting userspace, so access to your program state is very painful with bpf.

So on older kernels you can use systemtap, except you probably can't get the kernels headers and debuginfo installed. And you're generally not going to encounter newer bpf-friendly kernels in the wild on production systems unless you're managing your own clusters of systems. If you do have a newer kernel and want to use bpf, you get to fight with its lack of DWARF based unwinding and its primitive to nonexistent ability to understand userspace memory contents.

I find this intensely frustrating, as I get a great deal of value out of both tools in my own debugging and performance work. But systemtap sometimes breaks when I update my kernel on my laptop, and I need a bleeding-edge bcc for even some of the basic functionality I needed for simple userspace tracers.

Comparing SystemTap and bpftrace

ghane — Sat, 17 Apr 2021 04:08:24 +0000

Emanuele,

I do not use either debugging tool, or debug compiled code, but this article was a pleasure to read for its clarity.

It provides an excellent overview of the *different use cases* where one might use these tools. There are too few of these articles, I think.

Thank you
--
Sanjeev

dwarf stacktrace in bpf

mathstuf — Wed, 14 Apr 2021 14:14:55 +0000

This article[1] is the latest I can find. You can have bounded loops now, but they must be verifiable. Validating that there is an end to the stack is data-dependent and likely not allowed. One could get up to a fixed stack size though I presume.

[1]https://lwn.net/Articles/794934/

dwarf stacktrace in bpf

fuhchee — Wed, 14 Apr 2021 13:27:19 +0000

> Loops are allowed

Last I heard, this was compile-time bounded, implemented by unrolling.

Comparing SystemTap and bpftrace

rahulsundaram — Wed, 14 Apr 2021 11:03:51 +0000

This is good to see, LWN should always link to the bug reports authors file in the relevant articles. For those interested in the issues, having a place to followup is important.

Comparing SystemTap and bpftrace

hazmat — Wed, 14 Apr 2021 10:07:09 +0000

thanks to the author for posting bpftrace issues noted in the article to the upstream
https://github.com/iovisor/bpftrace/issues/created_by/ema

dwarf stacktrace in bpf

seanyoung — Wed, 14 Apr 2021 09:02:15 +0000

BPF is pretty powerful nowadays. Loops are allowed; instruction set is complete (you can work around the missing signed div) and instruction limits are high. I don't understand why dwarf stacktraces can't be generated in BPF.

Comparing SystemTap and bpftrace

fuhchee — Tue, 13 Apr 2021 23:50:08 +0000

Thank you for this fair & balanced comparison!

Comparing SystemTap and bpftrace

atnot — Tue, 13 Apr 2021 20:46:17 +0000

One note for anyone trying out bpftrace, that unfortunately makes the out-of-the-box experience a little bit more difficult than described here, is that it will fail with an permission denied error when kernel lockdown is enabled, as more distributions are doing automatically. In that case, lockdown must be disabled using a sysrq shortcut, /proc/sysrq-trigger, or a reboot with different lockdown kernel parameter.

I hope in the future some solution can be found to not have to compromise between security and debuggability in production systems in this way, but with the deep reach that kprobes have into the kernel it seems somewhat unlikely.