| |||||||||||||
Finding a profiler that works, damnitFinding a profiler that works, damnitPosted Mar 24, 2010 1:55 UTC (Wed) by foom (subscriber, #14868)In reply to: Finding a profiler that works, damnit by chantecode Parent article: KVM, QEMU, and kernel project management >Sure, how could it be another way? Without frame pointers you can't have reliable stacktraces. Or if you have a tip to go round this requirement I would be happy to implement it. The only arch I know that is able to walk the stack correctly without frame pointers is PowerPc. GCC always includes an eh_frame section (unless you explicitly pass - fnoasynchronous-unwind-tables, which of course you should never do), which allows you to unwind reliably from any point within a program. This is actually required by the X86-64 ABI. This data is never stripped, and actually is mapped into memory with the code and data of the executable, so you can read it directly from the process's memory space. It's almost the same format data as the DWARF debug info, but with a couple minor differences and with only a subset of the functionality allowed. See libunwind for a userspace library which knows how to read and interpret this data. Google-perftools uses libunwind, which is how it's able to work properly.
(Log in to post comments)
Finding a profiler that works, damnit Posted Mar 24, 2010 2:25 UTC (Wed) by chantecode (subscriber, #54535) [Link]
Ok.Thanks for this pointer, I had no idea about this.
It doesn't look usable by perf though.
We often save the stacktraces from an NMI path (covers all hardware pmu events). So this operation must be as fast and simple as possible.
This seems to me really too complicated, too slow and unlikely to work most of the time.
Finding a profiler that works, damnit Posted Mar 24, 2010 3:19 UTC (Wed) by foom (subscriber, #14868) [Link]
> This seems to me really too complicated, too slow and unlikely to work most of the time.
Shrug. Sure, it's complicated, but it's totally necessary for the callgraph profiler to be useful at all
If you don't think it's possible to implement support for EH_FRAME-based unwind, you might as
Finding a profiler that works, damnit Posted Mar 24, 2010 10:34 UTC (Wed) by nix (subscriber, #2304) [Link]
In any case, the fix is obvious. Dump an address out in the fast path, and convert it to a stacktrace later on. This is what absolutely everything else does, from GDB to valgrind to sysprof.
Finding a profiler that works, damnit Posted Mar 24, 2010 15:18 UTC (Wed) by foom (subscriber, #14868) [Link]
> Dump an address out in the fast path, and convert it to a stacktrace later on.
...huh?
You can't convert a single address to a stacktrace later on! You'd need a copy of the whole stack to
Anyways, thanks for mentioning sysprof: I hadn't heard of that one before. But looking at the
Finding a profiler that works, damnit Posted Mar 24, 2010 17:48 UTC (Wed) by nix (subscriber, #2304) [Link]
Oh crap, you're right, of course. I plead temporary insanity caused by hours of sitting through mind-numbing presentations on stuff I already knew.
So it's call the unwinder or nothing, really. Unfortunately the job of figuring out what the call stack looks like really *is* quite expensive :/ any effort should presumably go into optimizing libunwind...
Finding a profiler that works, damnit Posted Mar 26, 2010 17:55 UTC (Fri) by sandmann (subscriber, #473) [Link] I am the author of sysprof.You are right that it doesn't generate good callgraphs on x86-64 unless you compile the application with -fno-omit-frame-pointer. I very much would like to fix this somehow, but I just don't see any really good way to do it. Fundamentally, it needs to take a stack trace in the kernel in NMI context. You cannot read the .eh_frame information at that time because that would cause page faults and you are not allowed to sleep in NMI context. Even if there were a way around that problem, you would still have to *interpret* the information which is a pretty hairy algorithm to put in the kernel (though Andi Kleen did exactly that I believe, resulting in flame wars later on). You could try copying the entire user stack, but there is considerable overhead associated with that because user stacks can be very large (emacs for example allocates a 100K buffer on the stack). You could also try a heuristic stack walk (which is what an old version of sysprof - new versions use the same kernel interface as perf). That sort of works, but there is a lot of false positives from function pointers and left-over return addresses. The function pointers can be filtered out, but the return addresses can't. These false positives tend to make sysprof's UI display somewhat confusing, though not completely unusable. You could also try some hybrid scheme where the kernel does a heuristic stack walk and userspace then uses the .eh_frame information to filter out the false positives. This is what I think is the most promising approach at this point. Some day I'll try it. Finally, the distributions could just compile with -fno-omit-frame-pointer by default. The x86-64 is not really register-starved so it wouldn't be a significant performance problem. The Solaris compiler does precisely this because they need to take stack traces for dtrace. But, I fully expect to be told that for performance reasons we can't have working profilers.
Finding a profiler that works, damnit Posted Mar 26, 2010 18:06 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]
Have you tried asking about that in fedora devel list? Maybe, we can change
the compiler options for Fedora 14.
Finding a profiler that works, damnit Posted Mar 26, 2010 21:44 UTC (Fri) by foom (subscriber, #14868) [Link]
I'm probably missing something, but...
Why does it need to happen at NMI time? Why can't you just do it in the user process's context,
The setitimer(ITIMER_PROF) solution that userspace profilers use clearly works out fine for
The stack trace of the userspace half clearly can't change between when you received the NMI and
That just leaves the complication of implementing the DWARF unwinder in the kernel, but there's
Finding a profiler that works, damnit Posted Mar 26, 2010 23:28 UTC (Fri) by nix (subscriber, #2304) [Link]
There is already a DWARF unwinder in the kernel (or was, and it could be
resurrected). The tricky part is making it paranoid enough to be non-DoSable, even by hostile generators of DWARF2. IIRC the kernel unwinder was ripped out by Linus because it kept falling over when unwinding purely kernel stack frames...
Finding a profiler that works, damnit Posted Mar 27, 2010 14:10 UTC (Sat) by garloff (subscriber, #319) [Link]
Finding a profiler that works, damnit Posted Mar 27, 2010 15:12 UTC (Sat) by foom (subscriber, #14868) [Link]
Care to expand upon that link with some explanatory text?
Finding a profiler that works, damnit Posted Mar 28, 2010 0:26 UTC (Sun) by garloff (subscriber, #319) [Link]
Sorry, that was somewhat terse. The Novell Kernel Debugger has stack
unwinding features built-in; so this is something that might be leveraged in another project.
Finding a profiler that works, damnit Posted Apr 4, 2010 12:35 UTC (Sun) by chantecode (subscriber, #54535) [Link]
We need to profile from NMI if we want to profile irqs as well. Otherwise a hardware pmu event would occur at the end of an irq disabled section, not at the exact place of the event, messing completely the result.
Finding a profiler that works, damnit Posted Apr 5, 2010 1:37 UTC (Mon) by foom (subscriber, #14868) [Link]
You need to get stack traces for the *kernel* from an NMI. Surely the userspace backtracing can
wait until a more convenient time...
Finding a profiler that works, damnit Posted Mar 30, 2010 18:51 UTC (Tue) by fuhchee (subscriber, #40059) [Link]
In systemtap, we do unwinding of kernel/userspace under similar constraints. We work around the "can't page in user data" by preemptively uploading the unwind data into the kernel, so it's ready for use when needed. It costs some memory but it saves time.
|
|||||||||||||