Finding a profiler that works, damnit [LWN.net]

Finding a profiler that works, damnit

Posted Mar 24, 2010 2:15 UTC (Wed) by foom (subscriber, #14868)
In reply to: Finding a profiler that works, damnit by chantecode
Parent article: KVM, QEMU, and kernel project management

BTW, the callgraph output I'm used to reading is like this:
http://www.cs.utah.edu/dept/old/texinfo/as/gprof.html#SEC6

Or this:
http://oprofile.sourceforge.net/doc/opreport.html (section 2.3. Callgraph output)

The important points being that it shows, linearly, a section of output for every function, sorted by
time (inclusive of subroutines). Each section includes, in addition to the line for the function itself,
1) above the function's line: who called it
2) below the function's line: who it calls

Maybe your method is better for some uses, but I can't quite get my ahead around how to
use it, even after your explanation of what it means. I don't expect I'm unique in this regard, since
basically all callgraph profilers I've seen work the same way.

to post comments

Finding a profiler that works, damnit

Posted Mar 24, 2010 2:48 UTC (Wed) by chantecode (subscriber, #54535) [Link] (5 responses)

Ah I see what you mean. We have this format too. We call that "flat" callchains. It's a sequence of traditional callgraphs, linear and sorted by overhead.

Example: http://tglx.de/~fweisbec/callchain_flat.txt

We have three kind of formats: flat (as in above link), fractal (as shown in my previous example) and graph (like fractal, except that percentages of overhead are absolute wrt to the whole overhead profiling, unlike fractal where each overhead percentage is relative to its parent branch).

syntax: perf report -g flat|fractal|graph

I've never heard about someone using the flat format. Everybody seem to use the fractal (probably because it's the default, and IMHO, the most useful). And because I've never heard about it, I've not tested it for a while so it works on 2.6.32 but seem to have been broken in 2.6.33 (the graph format too), but fractal works. I'm going to send a fix for that and get it to the 2.6.33 stable branch.

>Maybe your method is better for some uses, but I can't quite get my ahead around how to use it, even after your explanation of what it means. I don't expect I'm unique in this regard, since basically all callgraph profilers I've seen work the same way.

I must confess it's the first time I hear this complaint. That said may be it's a matter of playing a bit with it before being used to its layout.

Finding a profiler that works, damnit

Posted Mar 24, 2010 3:41 UTC (Wed) by foom (subscriber, #14868) [Link] (4 responses)

> I've never heard about someone using the flat format.

I'm not surprised about that: it looks like it's basically just the same as fractal, except without
combining the similar parts of call chains together. Doesn't seem useful at all. And that also isn't
what I want. (Sidenote: I'd have expected something named "flat" to actually show me a normal
flat profile; it seems a bit odd that perf report can't actually show a normal flat profile when
callgraph info was collected.)

I just like to see a list of the *direct* callers and callees of every function. (having both easily
available is an important component). I don't actually want to be shown every unique complete
call-chain starting from the named function: it's just too much data, and yet simultaneously not
enough, since it doesn't show callees.

I think the gprof docs explain its format pretty well, and that's basically exactly what I want to
see. It's a nice, concise, easy-to-understand summarization of the data that usually has enough
information to figure out where the problem is.

Or, heck, if you get callgrind output conversion working, kcachegrind also presents the data in
this way, so you wouldn't even have to implement the gprof-like output format. :)

Finding a profiler that works, damnit

Posted Mar 25, 2010 2:52 UTC (Thu) by chantecode (subscriber, #54535) [Link] (3 responses)

Ah ok, I see what you mean know. I parsed too quickly the gprof documentation.

Yep, may be we can implement such layout mode too, or at least getting it through kcachegrind.

Finding a profiler that works, damnit

Posted Apr 1, 2010 20:42 UTC (Thu) by oak (guest, #2786) [Link] (2 responses)

Having support for perf callgraph format in this:
http://code.google.com/p/jrfonseca/wiki/Gprof2Dot

Or similar callgraph visualization support in perf itself would be nice (including filtering options, highlighting the nodes based on CPU used by given node and the nodes called by it etc).

Finding a profiler that works, damnit

Posted Apr 4, 2010 12:44 UTC (Sun) by chantecode (subscriber, #54535) [Link] (1 responses)

I really like this. We may indeed want to support it.

Finding a profiler that works, damnit

Posted Apr 5, 2010 1:39 UTC (Mon) by foom (subscriber, #14868) [Link]

Check out the google-perftools' pprof's "gv" command; it's also pretty nice. (although I still usually
prefer kcachegrind's interactive viewer)