Ktap almost gets into 3.13

Posted Nov 7, 2013 6:49 UTC (Thu) by karim (subscriber, #114)
Parent article: Ktap almost gets into 3.13

Seriously, making this part of perf makes it much less likely that I'll use it.

to post comments

Ktap almost gets into 3.13

Posted Nov 7, 2013 13:14 UTC (Thu) by deater (subscriber, #11746) [Link] (1 responses)

> Seriously, making this part of perf makes it much less likely
> that I'll use it.

As long as Ingo maintains perf it's unlikely that any other profiling tool will get into the kernel unless it's through perf_event, even if it doesn't make sense.

It's why things like AMD Lightweight Profiling never made it into the kernel

And it's why things like ftrace tracing are part of perf, complicating the interface, which has led to recent bugs where a normal user can lock the kernel using the perf interface.

Ktap almost gets into 3.13

Posted Nov 7, 2013 14:24 UTC (Thu) by karim (subscriber, #114) [Link]

Yes, I'm fully aware of that. And while I respect Ingo's work a great deal, there are areas where it's my personal belief that his well-intentioned stewardship of key functionality is actually counter-productive. Specifically, his clout makes it so that any suggestion that runs counter to his own designs results in that proposal to be doomed to failure.

At this stage many of those that have tried are staying away. Personally I've given up on Linux ever providing meaningful online diagnostic to rival any of the other OSes out there. There's just too much kernel politics in the way. This recent idea that all new performance/monitoring stuff has to go through perf craziness is just one more on the list.

FWIW I think ftrace's user interface is kickass. It's not perfect and there are other ways of doing things, but ftrace certainly shows that it was architectured based on deep, real-life experience with tracing lots of data. Smarts just aren't enough.

Ktap almost gets into 3.13

Posted Nov 7, 2013 13:53 UTC (Thu) by lambda (subscriber, #40735) [Link] (4 responses)

I'm curious, what do you find wrong with perf? I've been looking for a good tracing framework recently, and figured that since perf is in mainline I should probably learn that, but I'm wondering if there are better choices out there.

It's bit frustrating that on Linux there are so many, all of which are either incomplete or out of tree; on Solaris, FreeBSD, or Mac OS X, the answer is just "dtrace", while I'm never quite sure which tracing framework to learn on Linux.

Ktap almost gets into 3.13

Posted Nov 7, 2013 14:41 UTC (Thu) by karim (subscriber, #114) [Link] (1 responses)

Well, a whole lot of things. First and foremost it's just not the user interface I would expect. Why, exactly, do I have to give a command that it has to run, like "sleep 10", to given an upper-bound to its running? Why not just give me a "-t 10". Then the data it gives is all sample-based. Sampling is nice, but I'd find it a whole lot more useful to be able to have performance counters being read at every hit to an ftrace trace point. But I wouldn't need perf for that. The way I see it is that perf is like a better gprof. That's just not powerful enough for me. Then again, I'm not a regular user and I might have a very peculiar way of solving problems.

Ktap almost gets into 3.13

Posted Nov 7, 2013 16:31 UTC (Thu) by deater (subscriber, #11746) [Link]

What you're talking about (performance counter dump every time a tracepoint is hit) should in theory be possible with the raw perf_event interface. Just set up the tracepoint and events to measure and have them "sample" every time the tracepoint is hit. I have no idea what kind of crazy command line you'd need to try to convince the "perf" tool to actually do that though, assuming it were possible.

Ktap almost gets into 3.13

Posted Nov 12, 2013 3:58 UTC (Tue) by bgregg (guest, #46639) [Link] (1 responses)

perf is great. I've used it's profiling (sampling), static tracing, and dynamic tracing abilities, and with access to local variables. And I've written and shared documentation about perf (conference talks, books).

pref can do a lot, but there are some things where the performance hit can make me hesitate. For example, timing a busy kernel function, where I'm interested in examining its distribution, to identify outliers or multiple modes. perf has an enable/trace/dump/post-process cycle to get to the answer I want. ktap (and DTrace and SystemTap) can execute arbitrary programs in kernel context, where it can filter and summarize using per-CPU buffers (eg, via kp_percpu_data()), and then emit an answer without making a trip to user-level to post-process data. It can also (more easily) let me build real time tools.

Ktap almost gets into 3.13

Posted Nov 13, 2013 20:35 UTC (Wed) by karim (subscriber, #114) [Link]

Hi Brendan (assuming this is Brendan Gregg),

Yes, it's been a pleasure watching the videos of some of those talks and I'm glad to see your "System Performance" title is finally out, I've been waiting for it for a while. Time to order a copy.

FWIW, I'm not saying perf is useless. Just that it doesn't do what I need. Then again, as I said, I'm not the typical user: I wrote LTT in the late '90s and maintained it for a few years. That doesn't necessarily make my opinion worth more than what others have to say, but I do have a fairly good handle on what such tools can be capable of and we're certainly not where we should be and my expectations, based on historical record, are fairly low.