LWN: Comments on "Kernel analysis with bpftrace"

Kernel analysis with bpftrace

ideal — Mon, 21 Mar 2022 13:25:57 +0000

# bpftrace -e 't:timer:hrtimer_start { @[ksym(args->function)] = count(); }'

on kernel v5.16.x，need to include <linux/hrtimer.h>，or else enum hrtimer_mode is not defined.

Kernel analysis with bpftrace

flussence — Sat, 27 Jul 2019 05:56:04 +0000

I was wondering what that new CONFIG_IKHEADERS in kernel 5.2 could possibly be useful for; this scenario (kernel image living outside the known filesystem) sounds like exactly it.

Kernel analysis with bpftrace

BenHutchings — Fri, 26 Jul 2019 16:59:14 +0000

If you build the kernel with "make bindeb-pkg" you'll get those kernel headers packaged up in a linux-headers-<release> package, same as for a distribution kernel.

CTF

mjw — Fri, 19 Jul 2019 21:03:53 +0000

It is already in the upstream binutils git repo:
https://sourceware.org/git/?p=binutils-gdb.git;a=history;...

With additional work being discussed on the binutils mailinglist:
http://sourceware.org/ml/binutils/current

Support for the linker posted here:
https://sourceware.org/ml/binutils/2019-07/msg00159.html

The GCC support just saw its V4 RFC:
https://gcc.gnu.org/ml/gcc-patches/2019-07/msg01209.html

Kernel analysis with bpftrace

dw — Fri, 19 Jul 2019 15:43:49 +0000

FreeBSD is a heavy user

Kernel analysis with bpftrace

bgregg — Fri, 19 Jul 2019 15:25:01 +0000

Who are still the end users of CTF? I wanted it years ago, but now we have BTF.

Kernel analysis with bpftrace

nivedita76 — Fri, 19 Jul 2019 13:57:01 +0000

is there a doc somewhere? What is the use case for type info for binaries without debug info?

Kernel analysis with bpftrace

nilsmeyer — Fri, 19 Jul 2019 09:13:53 +0000

That still doesn't make up for Netflix cancelling shows I like.

Kernel analysis with bpftrace

dgc — Fri, 19 Jul 2019 00:44:41 +0000

Ok, so there's no /lib/modules/<vers>/build directories on any of my machines. And my test VMs don't even run a modular kernel (it's supplied externally by qemu invocation) so I'm guessing this is the issue. I'll have a bit of a look around to see if I can get this populated and whether than makes the problem go away. If I can't solve it easily, then I'll raise an issue for it.

Thanks!

-Dave.

Kernel analysis with bpftrace

bgregg — Thu, 18 Jul 2019 23:44:42 +0000

I'd raise this as a bpftrace issue please[0]; my guess is you've got an older version of LLVM, but the Debian package description does say llvm-7, which should be fine.

Using bpftrace to check how bpftrace is running that tool on my system:

# BPFTRACE_STRLEN=128 bpftrace -e 't:syscalls:sys_enter_openat /comm == "bpftrace"/ { printf("opening: %s\n", str(args->filename)); }' | grep sched
opening: /lib/modules/4.15.0-54-generic/build/include/linux/sched.h
opening: /lib/modules/4.15.0-54-generic/build/include/uapi/linux/sched.h
opening: /lib/modules/4.15.0-54-generic/build/include/asm-generic/bitops/sched.h
opening: /lib/modules/4.15.0-54-generic/build/include/linux/sched/prio.h

(We're switching from BPF stack to map storage for strings, so needing to increase BPFTRACE_STRLEN should become a thing of the past.)

[0] https://github.com/iovisor/bpftrace/issues

Kernel analysis with bpftrace

dgc — Thu, 18 Jul 2019 23:17:19 +0000

Brendan, running on debian and installing the distro bpftrace package, stuff like your off-wake.bt script doesn't work.

The problem is that your script does this:

#include <linux/sched.h>

and it assume that it picks up a kernel internal header file, not the uapi/linux/sched.h that the distro has installed in /usr/include/linux/. Hence:

$ sudo ./offwake.bt
Unknown struct/union: 'task_struct'
Unknown struct/union: 'task_struct'
$

And pointing it at the installed kernel header package via:

$ sudo ./offwake.bt -I /usr/src/linux-headers-5.0.0-trunk-common/include -I /usr/src/linux-headers-5.0.0-trunk-common/arch/x86/include/
/usr/src/linux-headers-5.0.0-trunk-common/arch/x86/include/asm/bitops.h:134:2: warning: implicit declaration of function 'barrier' is invalid in C99 [-Wimplicit-function-declaration]
.....
fatal error: too many errors emitted, stopping now [-ferror-limit=]

Throws so many warnings/errors that it aborts.

So there's some magic in your setup that is not apparent from either the article, your code samples or the documentation. Any ideas?

-Dave.

Kernel analysis with bpftrace

unixbhaskar — Thu, 18 Jul 2019 23:14:24 +0000

Wonderful! Brendan ...enjoyed very much!

Kernel analysis with bpftrace

bgregg — Thu, 18 Jul 2019 20:58:23 +0000

Don't overlook how much can be done from linux headers, without BTF. Out of the 100+ new tools I developed in the BPF book, only three of them hit issues with missing structs where I needed to manually declare them. Although, that ratio will get higher (worse) the deeper you dig into things.

Facebook have already added BTF to the build process as CONFIG_DEBUG_INFO_BTF, and I've heard that it can be included in vmlinux, so the kernel is self describing. We will need to convince distros to enable this and whatever options to deliver BTF-embedded vmlinux's by default (I can ask Canonical for Ubuntu). The BTF dedup work keeps it small, so adding it by default should be acceptable.

I don't think there is a howto about building vmlinux with BTF yet. Arnarldo did talk about the recent BTF work[0] and there's an older post from Andrii about it[1], and there's the BTF docs in the kernel [2].

[0] http://vger.kernel.org/~acme/perf/btf-perf-pahole-lsfmm-s...
[1] https://facebookmicrosites.github.io/bpf/blog/2018/11/14/...
[2] https://www.kernel.org/doc/html/latest/bpf/btf.html#btf-g...

Kernel analysis with bpftrace

SEJeff — Thu, 18 Jul 2019 20:23:41 +0000

Is there a git repo you have so we can follow along by chance?

Kernel analysis with bpftrace

nix — Thu, 18 Jul 2019 20:16:05 +0000

As a side note -- tangential but I can't resist -- CTF support has already landed in binutils (only the library at first, but the linker portion is under review now and does work in my testing). This should let us have CTF type info in the kernel as well, built from the toolchain-generated CTF info :) and it's not stripped by strip(1), either, so the thing is always present in the binaries. (I will eventually teach it to handle FreeBSD etc CTF info as well -- but BTF is off the table, it's just too different.)

(Obviously, it's non-loaded info, but it's crazy to parse this sort of stuff in kernel space anyway. This is for userspace tools to use when analysing other things... like, say, the kernel. The kernel is actually a special case: because of network booting, boots from /boot that is then unmounted etc it is quite common to boot from a kernel you can't actually access, and the kernel has *vast* amounts of type info in it, far too much to sanely load all of it at once. So we'll keep the kernel's type info in a separate, compressed, loadable archive. It's only every other binary that gets its type info in a built-in .ctf section.)

Kernel analysis with bpftrace

dw — Thu, 18 Jul 2019 19:14:22 +0000

Very excited about BTF arriving, but I'm a little confused. I expected this would mean every typical kernel would ship with type info, but read a few things in the past week to suggest this isn't quite how distros are actually going to treat it? (Sorry, lost links). The advantage of BTF for me is not having to mess with essentially getting a 'development setup' on any random machine that might need to be touched, but separate dbginfo-type package installation is exactly that

Kernel analysis with bpftrace

SPYFF — Thu, 18 Jul 2019 19:04:42 +0000

Really good introduction to bpftrace. Finally a good alternative to perf probe creation then perf trace. I would really like to see something BPF based function flow tracing/graphing, that would be a true killer!