LWN: Comments on "SFrame-based stack unwinding for the kernel"

Little-endian format

maskray — Mon, 08 Sep 2025 01:54:08 +0000

The format includes endianness variations that complicate toolchain support. I think we should use a little-endian format universally, regardless of the target system's native endianness. On the big-endian z/Architecture, this is efficient; the LOAD REVERSED instructions used by the bswap versions in the following program don't even require extra instructions.

#define WIDTH(x) \
typedef __UINT##x##_TYPE__ [[gnu::aligned(1)]] uint##x; \
uint##x load_inc##x(uint##x *p) { return *p+1; } \
uint##x load_bswap_inc##x(uint##x *p) { return __builtin_bswap##x(*p)+1; }; \
uint##x load_eq##x(uint##x *p) { return *p==3; } \
uint##x load_bswap_eq##x(uint##x *p) { return __builtin_bswap##x(*p)==3; }; \

WIDTH(16);
WIDTH(32);
WIDTH(64);

My blog post had a TODO chapter about CTF Frame. I spent some time today to read the spec and add my analysis here: https://maskray.me/blog/2020-11-08-stack-unwinding#sframe

SFrame logo

ibhagat — Sat, 30 Aug 2025 23:29:21 +0000

> At least, ORC hadn't been adapted to user space until the SFrame project (which truly needs an amusing Lord-of-the-Rings-inspired name) was launched.

If not the name now, may be we can make do with a cool logo for SFrame..

Some thoughts

irogers — Thu, 17 Jul 2025 05:19:14 +0000

SFrames trying to advance performance and profiling accuracy is a noble thing. The article missing LBR stack traces which have largely solved the issue in areas like AutoFDO.

Some other thoughts:
- Why a new encoding format, why not have a subset of dwarf that matches the limited stack frame encoding supported by sframes? Dwarf is compressed and there may be potential for reuse of the ELF section in areas like C++ exception delivery.
- When perf is invoked to record system wide for a period of time like `perf record -e cycles -a sleep 10` any system call that hasn't transitioned to user code will only have the kernel side of the stack trace.
- The bpf helper bpf_get_stackid with BPF_F_USER_STACK will just return a placeholder value which may break or cause additional memory usage in BPF programs.
- JITs need to be taught to generate/register/remove SFrames or perhaps some kind of JIT interface like GDB's can be devised.

"Unfortunately, frame pointers hurt performance; they occupy a CPU register, and must be saved and restored on each function call." A register can be freed by holding the frame pointer in thread-local storage. Function call cost is addressed by inlining. x86 lacks any callee-save floating point registers which is likely a greater performance issue and unfortunately not addressed by the upcoming APX changes. A greater performance issue is the -fno-omit-frame-pointers is less well optimized by C compilers, for example, missing tail call optimizations.

Last year the return of frame pointers was heralded:
https://www.brendangregg.com/blog/2024-03-17/the-return-o...

There have been calling conventions like ARM32's APCS that have very dense debug info but also would be reasonably easy to reverse engineer by walking the code forward until the function return is seen. Perhaps the code itself is the best debug format.

Naming

himi — Mon, 14 Jul 2025 23:32:34 +0000

Considering how (unintentionally?) ironic the name "DWARF" is, ENT seems perfectly reasonable for something designed to be compact and concise.

Co-routines?

Cyberax — Mon, 14 Jul 2025 21:40:47 +0000

> Isn't it enough to know "thread is in producer which was called from event loop"?

Not really. In particular, this makes debugging the coroutine-heavy code a pure nightmare. Debugger becomes worse than useless.

What I'd love is to have a weak link to the "caller frame". It's OK for it to be garbage-collected if the chain becomes too long.

Co-routines?

willy — Mon, 14 Jul 2025 14:22:29 +0000

What would you want a "stack trace" to say for coroutines? They may have called each other millions of times by the time we try to "inspect the stack". Surely you wouldn't want to have millions of repetitions of "producer called consumer called producer called ...". Isn't it enough to know "thread is in producer which was called from event loop"?

Is there any prior work in this area we can steal^W benefit from?

Naming

taladar — Mon, 14 Jul 2025 07:45:55 +0000

I guess ENT would not work that well for a format more compact than DWARF.

Co-routines?

taladar — Mon, 14 Jul 2025 07:44:54 +0000

More generally anything using continuation passing style does not really play well with stack traces and won't work with this either without some special considerations.

Naming

Sesse — Sun, 13 Jul 2025 21:27:18 +0000

SFrame obviously stands for Saruman-Frame, so we're already covered.

Naming

jemarch — Sun, 13 Jul 2025 17:16:54 +0000

It's spelled s-f-r-a-m-e but it's pronounced DURBATULUK

Naming

SLi — Sun, 13 Jul 2025 10:27:45 +0000

I approve.

CONFIG_DURBATULUK=y

Naming

excors — Sun, 13 Jul 2025 09:12:02 +0000

As this is an attempt to spread ORC into the wider world, I think the obvious answer is the word for "to rule them all" in Black Speech (the common language of Orcs and other servants of Mordor, which was itself designed to solve the problem of mutually-unintelligible dialects between tribes and help them coordinate more efficiently, just as SFrame is doing): DURBATULÛK, or Debugging Userspace Reliably By A Table of Unwinding-Location Ûseful Knowledge.

Naming

magfr — Sun, 13 Jul 2025 05:25:43 +0000

Find a nice backronym from ENT or just bite the bullet and call it MAN.

Naming

SLi — Sat, 12 Jul 2025 23:56:55 +0000

> At least, ORC hadn't been adapted to user space until the SFrame project (which truly needs an amusing Lord-of-the-Rings-inspired name) was launched.

Indeed. What can we come up with that doesn't sound completely stupid?

Smeagol? Shelob (since it unwinds tangled stacks)? Framewise Gamgee? Framebeard? BackAgain?

Co-routines?

jreiser — Sat, 12 Jul 2025 23:04:53 +0000

I do not see any support for co-routines: a RETURN is replaced by a co-CALL of some other routine in the nest, and is itself the destination of a co-CALL. Incremental producer-consumer bit packing can be expressed as a nest of two co-routines; "the stack" remembers two "leaf" activations, whose co-RETURN (co-CALL) points each vary.

SFrame us more accurate too

DemiMarie — Sat, 12 Jul 2025 18:35:50 +0000

Frame pointers can miss leaf functions. SFrame will not do that.