Reliable user-space stack traces with SFrame

Posted May 23, 2023 0:32 UTC (Tue) by brenns10 (subscriber, #112114)
Parent article: Reliable user-space stack traces with SFrame

It's worth noting that SFrame is renamed from CTF Frame, which caught me off guard.

This is such an exciting project, it's the "have your cake and eat it, too" approach to stack unwinding. No extra code generated for frame pointers, no wasted register or icache. But still reliable unwinding without relying on the full DWARF debuginfo.

Hopefully this becomes standard along with CTF for lightweight introspection. Programs may want to unwind their own stack or examine the layout of data structures, so there's already good use cases. What's more, debuggers can do a lot with a symbol table, a reliable unwinder, and the basic information about types provided by CTF. While dwarf is better suited for development tasks, these smaller formats could fill a nice for basic diagnostics in production environments where debuginfo isn't available.

Reliable user-space stack traces with SFrame

Posted May 23, 2023 7:25 UTC (Tue) by roc (subscriber, #30627) [Link] (5 responses)

You can already emit partial DWARF debuginfo that supports stack unwinding but not other debugging features. Has anyone compared SFrame to that?

I'm worried that people who want to build binaries with full debugging information or just stack traces with parameter values are going to have to build even *bigger* binaries with both DWARF and SFrame information.

Reliable user-space stack traces with SFrame

Posted May 23, 2023 8:40 UTC (Tue) by Sesse (subscriber, #53779) [Link] (3 responses)

Uninformed guess: One would assume that the big difference then is whether you unwind when the sample comes (i.e., in the kernel; SFrame), or whether you snapshot the entire top of the stack to unwind later (i.e., in userspacer; DWARF). The DWARF standard is complex enough that I don't think anyone really wants to parse it in the privileged context of the kernel, even if one happens to use only a subset of it. Plus it can be in a separate dbgsym file, which has all sorts of other implications around needing userspace helpers?

Reliable user-space stack traces with SFrame

Posted May 23, 2023 9:52 UTC (Tue) by atnot (subscriber, #124910) [Link] (2 responses)

DWARF is indeed incredibly complex. It is basically a turing complete stack machine that can run arbitrary computations. It has to be able to do this to be able to reverse arbitrary compiler optimizations for debugging. For a trivial example, while your code may declare a pointer, the compiler might actually store it as an offset from another pointer instead. In that case some simple addition will do, but this gets a lot more complicated when you stack layers of optimizations, computations that end up being eliminated due to inlining, VLAs, varargs, etc. Answering what the value of a variable is accurately might require substantial emulation of your code.

I don't know how much of this is needed to only do unwinding, but the idea of DWARF in the kernel is a very spooky prospect to me.

Reliable user-space stack traces with SFrame

Posted May 23, 2023 9:59 UTC (Tue) by atnot (subscriber, #124910) [Link]

To answer my question, here is a paper on using malicious debug info to take over a process during DWARF stack unwinding: https://static.usenix.org/event/woot11/tech/final_files/O...

Reliable user-space stack traces with SFrame

Posted May 24, 2023 5:09 UTC (Wed) by lathiat (subscriber, #18567) [Link]

Not sure if this makes you feel better or worse, but Polar Signals/Parca are doing DWARF unwinding in BPF for continuous profiling:
https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-...
https://news.ycombinator.com/item?id=33788794

Reliable user-space stack traces with SFrame

Posted May 25, 2023 2:53 UTC (Thu) by himi (subscriber, #340) [Link]

I wonder if it'd be practical to generate the SFrame data from the DWARF data on the fly in userspace, and then present it to the kernel - probably a lot of work the first time a particular executable was processed, but it'd be fairly easy to cache it.

I think it'd be pretty similar to the dlopen() scenario, except that instead of just pointing at existing SFrame data for the object it'd generate the data from another source first.