KS2010: Performance regressions

Posted Nov 2, 2010 12:43 UTC (Tue) by Cyberax (✭ supporter ✭, #52523)
Parent article: KS2010: Performance regressions

I hesitate to say this, but why not use Phoronix Test Suite?

KS2010: Performance regressions

Posted Nov 2, 2010 13:07 UTC (Tue) by jbh (guest, #494) [Link] (6 responses)

Since you hesitate, I assume you're familiar with the criticisms of this test suite - that it mainly measures compiler differences, etc. Has this improved lately?

KS2010: Performance regressions

Posted Nov 2, 2010 13:12 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

Well, compiler differences are also important :)

Phoromatic tracker allows you to track only one variable (kernel version), while leaving everything else frozen. They even have support for btrfs snapshots to quickly revert system to a known state.

But seriously, Phoronix tracker is quite useful now.

KS2010: Performance regressions

Posted Nov 2, 2010 13:54 UTC (Tue) by jbh (guest, #494) [Link] (2 responses)

I had a brief look at the kernel tracker, and it does look more useful than I remembered. The standard tests there (dbench and so on), and although it looks like a lot of the tests are a bit pointless (basic single-thread cpu-heavy workload), they can probably be disabled.

But the arbitrary mix of operations-per-seconds and seconds-to-complete is very annoying, it means I have to read the fine print on every graph to parse it. Gah!

KS2010: Performance regressions

Posted Nov 2, 2010 15:15 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

"I had a brief look at the kernel tracker, and it does look more useful than I remembered. The standard tests there (dbench and so on), and although it looks like a lot of the tests are a bit pointless (basic single-thread cpu-heavy workload), they can probably be disabled."

Single-threaded benchmarks are not pointless. I had regressions in single-thread workloads caused by 'too clever' locking which had higher overhead than good old lock_kernel.

Anyway, it's certainly possible to disable uninteresting benchmarks in Phoromatic.

KS2010: Performance regressions

Posted Nov 2, 2010 21:29 UTC (Tue) by jbh (guest, #494) [Link]

If you got burned by locking changes, I suspect it wasn't a cpu-bound workload to begin with. But if you find them useful, good for you!

KS2010: Performance regressions

Posted Nov 3, 2010 5:32 UTC (Wed) by mtippett (guest, #70976) [Link] (1 responses)

Don't confuse the use of the Phoronix Test Suite and the Phoronix Test Suite itself. I assume you are talking about the reporting on results from particular runs of the Phoronix Test Suite.

Phoronix Test Suite is just a system to run tests in a repeated manner. If you keep the compiler consistent between kernels you are only testing the kernel. People usually raise issues when there are multiple variables changing between systems under tests (some say the kernel, some say the compiler, some say the filesystem).

KS2010: Performance regressions

Posted Nov 3, 2010 8:01 UTC (Wed) by jbh (guest, #494) [Link]

I'm not sure I'm confusing anything. I wasn't thinking about multiple variables. A number of the tests in the test suite seem to test mainly the compiler optimisations, since they are cpu bound. Hence, pointless (not *wrong*, just not adding any information) for the job to which they are often put. Of course you can argue that this is the users' fault: Let's call it "user education", then, instead of "criticism".