LWN: Comments on "KS2010: Performance regressions"

KS2010: Performance regressions

sethml — Tue, 09 Nov 2010 16:26:55 +0000

Google's systems are a lot more "special" than they used to be. I suspect they're not real keen on mailing out kernel logs which might reveal exactly how.

KS2010: Performance regressions

jengelh — Wed, 03 Nov 2010 11:38:41 +0000

>Google's Mike Rubin agreed with all of this, saying that he would like to set up a group of machines running normal hardware (instead of Google's special systems)

Given Google's tendency to use low-cost systems (but a huge array of them), the "special" systems don't seem all that special.

KS2010: Performance regressions

jbh — Wed, 03 Nov 2010 08:01:37 +0000

I'm not sure I'm confusing anything. I wasn't thinking about multiple variables. A number of the tests in the test suite seem to test mainly the compiler optimisations, since they are cpu bound. Hence, pointless (not *wrong*, just not adding any information) for the job to which they are often put. Of course you can argue that this is the users' fault: Let's call it "user education", then, instead of "criticism".

KS2010: Performance regressions

mtippett — Wed, 03 Nov 2010 05:32:21 +0000

Don't confuse the use of the Phoronix Test Suite and the Phoronix Test Suite itself. I assume you are talking about the reporting on results from particular runs of the Phoronix Test Suite.

Phoronix Test Suite is just a system to run tests in a repeated manner. If you keep the compiler consistent between kernels you are only testing the kernel. People usually raise issues when there are multiple variables changing between systems under tests (some say the kernel, some say the compiler, some say the filesystem).

KS2010: Performance regressions

jbh — Tue, 02 Nov 2010 21:29:33 +0000

If you got burned by locking changes, I suspect it wasn't a cpu-bound workload to begin with. But if you find them useful, good for you!

KS2010: Performance regressions

Cyberax — Tue, 02 Nov 2010 15:15:47 +0000

"I had a brief look at the kernel tracker, and it does look more useful than I remembered. The standard tests there (dbench and so on), and although it looks like a lot of the tests are a bit pointless (basic single-thread cpu-heavy workload), they can probably be disabled."

Single-threaded benchmarks are not pointless. I had regressions in single-thread workloads caused by 'too clever' locking which had higher overhead than good old lock_kernel.

Anyway, it's certainly possible to disable uninteresting benchmarks in Phoromatic.

KS2010: Performance regressions

jbh — Tue, 02 Nov 2010 13:54:41 +0000

I had a brief look at the kernel tracker, and it does look more useful than I remembered. The standard tests there (dbench and so on), and although it looks like a lot of the tests are a bit pointless (basic single-thread cpu-heavy workload), they can probably be disabled.

But the arbitrary mix of operations-per-seconds and seconds-to-complete is very annoying, it means I have to read the fine print on every graph to parse it. Gah!

KS2010: Performance regressions

ajb — Tue, 02 Nov 2010 13:47:07 +0000

"The best thing, of course, is a bisection which fingers the guilty commit. Doing that requires highly repeatable tests, though."

There is a bisection algorithm for intermittent bugs: http://github.com/ealdwulf/bbchop/. It probable wouldn't be hard to adapt it for perfornance regressions.

KS2010: Performance regressions

Cyberax — Tue, 02 Nov 2010 13:12:02 +0000

Well, compiler differences are also important :)

Phoromatic tracker allows you to track only one variable (kernel version), while leaving everything else frozen. They even have support for btrfs snapshots to quickly revert system to a known state.

But seriously, Phoronix tracker is quite useful now.

KS2010: Performance regressions

jbh — Tue, 02 Nov 2010 13:07:12 +0000

Since you hesitate, I assume you're familiar with the criticisms of this test suite - that it mainly measures compiler differences, etc. Has this improved lately?

KS2010: Performance regressions

Cyberax — Tue, 02 Nov 2010 12:43:15 +0000

I hesitate to say this, but why not use Phoronix Test Suite?