Reliable user-space stack traces with SFrame

Posted May 23, 2023 7:25 UTC (Tue) by roc (subscriber, #30627)
In reply to: Reliable user-space stack traces with SFrame by brenns10
Parent article: Reliable user-space stack traces with SFrame

You can already emit partial DWARF debuginfo that supports stack unwinding but not other debugging features. Has anyone compared SFrame to that?

I'm worried that people who want to build binaries with full debugging information or just stack traces with parameter values are going to have to build even *bigger* binaries with both DWARF and SFrame information.

Reliable user-space stack traces with SFrame

Posted May 23, 2023 8:40 UTC (Tue) by Sesse (subscriber, #53779) [Link] (3 responses)

Uninformed guess: One would assume that the big difference then is whether you unwind when the sample comes (i.e., in the kernel; SFrame), or whether you snapshot the entire top of the stack to unwind later (i.e., in userspacer; DWARF). The DWARF standard is complex enough that I don't think anyone really wants to parse it in the privileged context of the kernel, even if one happens to use only a subset of it. Plus it can be in a separate dbgsym file, which has all sorts of other implications around needing userspace helpers?

Reliable user-space stack traces with SFrame

Posted May 23, 2023 9:52 UTC (Tue) by atnot (subscriber, #124910) [Link] (2 responses)

DWARF is indeed incredibly complex. It is basically a turing complete stack machine that can run arbitrary computations. It has to be able to do this to be able to reverse arbitrary compiler optimizations for debugging. For a trivial example, while your code may declare a pointer, the compiler might actually store it as an offset from another pointer instead. In that case some simple addition will do, but this gets a lot more complicated when you stack layers of optimizations, computations that end up being eliminated due to inlining, VLAs, varargs, etc. Answering what the value of a variable is accurately might require substantial emulation of your code.

I don't know how much of this is needed to only do unwinding, but the idea of DWARF in the kernel is a very spooky prospect to me.

Reliable user-space stack traces with SFrame

Posted May 23, 2023 9:59 UTC (Tue) by atnot (subscriber, #124910) [Link]

To answer my question, here is a paper on using malicious debug info to take over a process during DWARF stack unwinding: https://static.usenix.org/event/woot11/tech/final_files/O...

Reliable user-space stack traces with SFrame

Posted May 24, 2023 5:09 UTC (Wed) by lathiat (subscriber, #18567) [Link]

Not sure if this makes you feel better or worse, but Polar Signals/Parca are doing DWARF unwinding in BPF for continuous profiling:
https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-...
https://news.ycombinator.com/item?id=33788794

Reliable user-space stack traces with SFrame

Posted May 25, 2023 2:53 UTC (Thu) by himi (subscriber, #340) [Link]

I wonder if it'd be practical to generate the SFrame data from the DWARF data on the fly in userspace, and then present it to the kernel - probably a lot of work the first time a particular executable was processed, but it'd be fairly easy to cache it.

I think it'd be pretty similar to the dlopen() scenario, except that instead of just pointing at existing SFrame data for the object it'd generate the data from another source first.