| |||||||||||||
KS2010: Performance regressionsKS2010: Performance regressionsPosted Nov 2, 2010 13:07 UTC (Tue) by jbh (subscriber, #494)In reply to: KS2010: Performance regressions by Cyberax Parent article: KS2010: Performance regressions
Since you hesitate, I assume you're familiar with the criticisms of this test suite - that it mainly measures compiler differences, etc. Has this improved lately?
(Log in to post comments)
KS2010: Performance regressions Posted Nov 2, 2010 13:12 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]
Well, compiler differences are also important :)
Phoromatic tracker allows you to track only one variable (kernel version), while leaving everything else frozen. They even have support for btrfs snapshots to quickly revert system to a known state.
But seriously, Phoronix tracker is quite useful now.
KS2010: Performance regressions Posted Nov 2, 2010 13:54 UTC (Tue) by jbh (subscriber, #494) [Link]
I had a brief look at the kernel tracker, and it does look more useful than I remembered. The standard tests there (dbench and so on), and although it looks like a lot of the tests are a bit pointless (basic single-thread cpu-heavy workload), they can probably be disabled.
But the arbitrary mix of operations-per-seconds and seconds-to-complete is very annoying, it means I have to read the fine print on every graph to parse it. Gah!
KS2010: Performance regressions Posted Nov 2, 2010 15:15 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]
"I had a brief look at the kernel tracker, and it does look more useful than I remembered. The standard tests there (dbench and so on), and although it looks like a lot of the tests are a bit pointless (basic single-thread cpu-heavy workload), they can probably be disabled."
Single-threaded benchmarks are not pointless. I had regressions in single-thread workloads caused by 'too clever' locking which had higher overhead than good old lock_kernel.
Anyway, it's certainly possible to disable uninteresting benchmarks in Phoromatic.
KS2010: Performance regressions Posted Nov 2, 2010 21:29 UTC (Tue) by jbh (subscriber, #494) [Link]
If you got burned by locking changes, I suspect it wasn't a cpu-bound workload to begin with. But if you find them useful, good for you!
KS2010: Performance regressions Posted Nov 3, 2010 5:32 UTC (Wed) by mtippett (guest, #70976) [Link]
Don't confuse the use of the Phoronix Test Suite and the Phoronix Test Suite itself. I assume you are talking about the reporting on results from particular runs of the Phoronix Test Suite.
Phoronix Test Suite is just a system to run tests in a repeated manner. If you keep the compiler consistent between kernels you are only testing the kernel. People usually raise issues when there are multiple variables changing between systems under tests (some say the kernel, some say the compiler, some say the filesystem).
KS2010: Performance regressions Posted Nov 3, 2010 8:01 UTC (Wed) by jbh (subscriber, #494) [Link]
I'm not sure I'm confusing anything. I wasn't thinking about multiple variables. A number of the tests in the test suite seem to test mainly the compiler optimisations, since they are cpu bound. Hence, pointless (not *wrong*, just not adding any information) for the job to which they are often put. Of course you can argue that this is the users' fault: Let's call it "user education", then, instead of "criticism".
|
|||||||||||||