Ktap almost gets into 3.13

Posted Nov 7, 2013 13:53 UTC (Thu) by lambda (subscriber, #40735)
In reply to: Ktap almost gets into 3.13 by karim
Parent article: Ktap almost gets into 3.13

I'm curious, what do you find wrong with perf? I've been looking for a good tracing framework recently, and figured that since perf is in mainline I should probably learn that, but I'm wondering if there are better choices out there.

It's bit frustrating that on Linux there are so many, all of which are either incomplete or out of tree; on Solaris, FreeBSD, or Mac OS X, the answer is just "dtrace", while I'm never quite sure which tracing framework to learn on Linux.

to post comments

Ktap almost gets into 3.13

Posted Nov 7, 2013 14:41 UTC (Thu) by karim (subscriber, #114) [Link] (1 responses)

Well, a whole lot of things. First and foremost it's just not the user interface I would expect. Why, exactly, do I have to give a command that it has to run, like "sleep 10", to given an upper-bound to its running? Why not just give me a "-t 10". Then the data it gives is all sample-based. Sampling is nice, but I'd find it a whole lot more useful to be able to have performance counters being read at every hit to an ftrace trace point. But I wouldn't need perf for that. The way I see it is that perf is like a better gprof. That's just not powerful enough for me. Then again, I'm not a regular user and I might have a very peculiar way of solving problems.

Ktap almost gets into 3.13

Posted Nov 7, 2013 16:31 UTC (Thu) by deater (subscriber, #11746) [Link]

What you're talking about (performance counter dump every time a tracepoint is hit) should in theory be possible with the raw perf_event interface. Just set up the tracepoint and events to measure and have them "sample" every time the tracepoint is hit. I have no idea what kind of crazy command line you'd need to try to convince the "perf" tool to actually do that though, assuming it were possible.

Ktap almost gets into 3.13

Posted Nov 12, 2013 3:58 UTC (Tue) by bgregg (guest, #46639) [Link] (1 responses)

perf is great. I've used it's profiling (sampling), static tracing, and dynamic tracing abilities, and with access to local variables. And I've written and shared documentation about perf (conference talks, books).

pref can do a lot, but there are some things where the performance hit can make me hesitate. For example, timing a busy kernel function, where I'm interested in examining its distribution, to identify outliers or multiple modes. perf has an enable/trace/dump/post-process cycle to get to the answer I want. ktap (and DTrace and SystemTap) can execute arbitrary programs in kernel context, where it can filter and summarize using per-CPU buffers (eg, via kp_percpu_data()), and then emit an answer without making a trip to user-level to post-process data. It can also (more easily) let me build real time tools.

Ktap almost gets into 3.13

Posted Nov 13, 2013 20:35 UTC (Wed) by karim (subscriber, #114) [Link]

Hi Brendan (assuming this is Brendan Gregg),

Yes, it's been a pleasure watching the videos of some of those talks and I'm glad to see your "System Performance" title is finally out, I've been waiting for it for a while. Time to order a copy.

FWIW, I'm not saying perf is useless. Just that it doesn't do what I need. Then again, as I said, I'm not the typical user: I wrote LTT in the late '90s and maintained it for a few years. That doesn't necessarily make my opinion worth more than what others have to say, but I do have a fairly good handle on what such tools can be capable of and we're certainly not where we should be and my expectations, based on historical record, are fairly low.