User: Password:
|
|
Subscribe / Log in / New account

Perfcounters added to the mainline

Perfcounters added to the mainline

Posted Jul 2, 2009 14:59 UTC (Thu) by deater (subscriber, #11746)
Parent article: Perfcounters added to the mainline

A few comments.

* Not all Intel chips have full hardware counter support under perf. Most notably, Pentium Pro/II/III or Pentium 4. One might argue that these are old and don't matter, but they are supported by perfmon2. Pentium 4 is the troublesome one, because the performance counters for that architecture don't map well at all to the abstraction chosen by the perf developers.

* I find calling perf a "simple" command line tool to be a bit deceptive. It is quite complicated and not very well documented yet.

* There is still some lingering bitterness about how Ingo took over perfcounters, sort of the same way he took over amd64 and the CFS scheduler. Mainly because he is re-inventing everything from scratch and making many mistakes that the other implementations already learned the hard way. Also the perf implementation does a lot of things, such as abusing ioctl()s, that the perfmon2 developers were told would not be allowed in the kernel and they wasted a lot of time working around these issues, only to find out it didn't matter in the end.

* I will admit the perf developers can be helpful, especially if you bug them enough. Upon prompting, they've reduced the static aggregate count overhead in the perf tool from a few thousand instructions to near zero.

* I personally think the abstraction they chose of having "common" counters hard-wired in the kernel to be a bad one, because as they are already finding out every chip and chip revision has different counters with different issues. perfmon2 took the saner route to do this in user-space; once 2.6.31 is released the ABI is frozen and we're going to be stuck with this.


(Log in to post comments)

Perfcounters added to the mainline

Posted Jul 2, 2009 20:01 UTC (Thu) by ejr (subscriber, #51652) [Link]

Not only is he re-inventing the wheel, he's forcing users to add yet another perfctr-alike support layer. PAPI has been around a long time and has many users, but the package and its users keep being declared not to exist. Bizarre. Many performance counter tools are one-shots meant to work for a particular application or stack. That's one reason why we (users) don't have a "flagship application" to wave in front of other developers. We want a flexible way to dig at the hardware without waiting for kernel developer X to decide a particular counter is worth-while. Let us deal with things in user space.

Perfcounters added to the mainline

Posted Jul 2, 2009 21:53 UTC (Thu) by deater (subscriber, #11746) [Link]

there is hope that soon PAPI and pfmon (the perfmon2 tool) will be ported to perfcounters, so assuming the features they need are available, things might work out in the end once things stabilize for a few months/years.

It will be nice that _finally_ performance counters will be available under Linux without having to patch the kernel. It's just a shame that it happened the way it did.

Perfcounters added to the mainline

Posted Jul 3, 2009 22:54 UTC (Fri) by mingo (subscriber, #31122) [Link]

* I personally think the abstraction they chose of having "common" counters hard-wired in the kernel to be a bad one, because as they are already finding out every chip and chip revision has different counters with different issues. perfmon2 took the saner route to do this in user-space; once 2.6.31 is released the ABI is frozen and we're going to be stuck with this.

Not really. The days of weird x86 PMUs changing with every CPU model are gone, fortunately.

The AMD PMU programming low level details have stayed pretty stable since around the K7 or so - for many years.

Intel has also introduced 'architectural performance monitoring' starting with Core2 (and has extended it in Corei7) which too is a future-proof method.

Also, even assuming weird PMUs, the perfcounters ABI does not hard-code low level details like that. Proof of this is in the fact that most Power CPUs are supported by perfcounters - which all have PMUs that are wildly different from x86 PMUs.

The reason perfcounters abstracts away common events is utility: it is convenient to tools to use a 'cycles' or a 'branches executed' events regardless of which CPU model they are running on. perfmon/pfmon, despite many years of development, never achieved this kind of basic utility.

Perfcounters added to the mainline

Posted Jul 3, 2009 23:09 UTC (Fri) by mingo (subscriber, #31122) [Link]

* There is still some lingering bitterness about how Ingo took over perfcounters [...]

(Just a question - from your post i gather that you are an (ex?) perfmon developer. If yes then i'm not surprised that you feel bitter about it - still it would have been nice had you openly disclosed your direct involvement and bias in this matter.)

The thing is, perfmon, despite being available for years, never achieved any measurable usage amongst kernel developers. You can check this yourself, just type: "git log --grep=pfmon" in an upstream kernel repository. It comes up empty: no-one ever found it important to mention pfmon in an upstream kernel changelog. Not one commit out of more than 150,000 upstream kernel commits ever mentioned that pfmon was used to measure something or to solve a problem.

The reason? I cannot speak for other kernel developers but i have my guesses: i tried it, and the thing is close to unusable to kernel developers. It has way too much overhead to measure workloads with lots of tasks and lots of overhead. It relies on a fat and quirky library and takes way too much effort to install and use. Its sampling (profiling) does not read ELF symbols as far as i could see.

perfmon had many other design problems as well (not directly visible to users) - i explained the reasons in the (many) posts i wrote about perfcounters in the past ~6 months. The main failure was that it tried to abstract on a way too low level, on an almost register basis - without the kernel having any knowledge about the structure of the hardware it abstracts away.

That design choice is lethal: it has shut off many interesting capabilities that perfcounters offers here and today: inherited counters, software counters, transparent workload monitoring, nested counters, various context-switch performance optimizations, etc. etc.

Perfcounters added to the mainline

Posted Jul 3, 2009 23:24 UTC (Fri) by mingo (subscriber, #31122) [Link]

* Not all Intel chips have full hardware counter support under perf. Most notably, Pentium Pro/II/III or Pentium 4. One might argue that these are old and don't matter, but they are supported by perfmon2. Pentium 4 is the troublesome one, because the performance counters for that architecture don't map well at all to the abstraction chosen by the perf developers.

Those chips are indeed old, abandoned and dont matter. perfmon has support for them partly because perfmon was started when those chips were still relevant. Alas, IMO, this also created a wrong design and the wrong mindset for perfmon.

So in a sense, perfcounters was lucky to have come later, when saner PMUs emerged on x86.

The central concept of perfmon is to expose the PMU to user-space and to push all the complexity to user-space.

The central concept of perfcounters is to provide rich, kernel-based abstractions to measure performance characteristics of a Linux system in a coherent, unified framework - regardless of whether the information comes from a PMU, a software counter, a tracepoint or some data field somewhere.

Those are two wildly different and fundamentally conflicting sets of design goals.

But you would be wrong to suggest that P4 support is not possible under perfcounters. For example PowerPC support (which was cited as the primary counter argument against perfcounters in the perfmon vs. perfcounters discussions) is alive, well and kicking under perfcounters.

The reason why people are not rushing to implement perfcounters for P4 is probably that Core2 and later CPUs are just so much more different from a performance profile than the abandoned Netburst architecture. They are also a lot more pleasant CPUs from many perspectives.

It also makes little sense to profile on too old CPUs, as any performance optimization would have to be re-validated on more recent CPUs as well. People who care about performance tend to try to stay on the hardware edge as well, and dont tend to use obsolete systems.

As the years advance, so will perfcounter's currently 'cutting edge' PMU support create the same kind of backwards-pointing trail of CPU models. Or, if anyone cares about P4 PMU support, it can be implemented just fine as well - patches are certainly welcome.


Copyright © 2018, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds