Anatomy of a flawed microbenchmark (IBM developerWorks) [LWN.net]

Brian Goetz discusses benchmark shortcomings on IBM developerWorks. "Software engineers are notoriously obsessed, sometimes excessively, with performance. While sometimes performance is the most important requirement in a software project, as it might be when developing protocol routing software for a high-speed switch, most of the time performance needs to be balanced against other requirements, such as functionality, reliability, maintainability, extensibility, time to market, and other business and engineering considerations. In this month's Java theory and practice, columnist Brian Goetz explores why it is so much harder to measure the performance of Java language constructs than it looks."

(Log in to post comments)

Anatomy of a flawed microbenchmark (IBM developerWorks)

Posted Feb 24, 2005 23:17 UTC (Thu) by ncm (subscriber, #165) [Link]

A curious article. In the first part the author demonstrates his contempt for software engineers and for concerns about performance. Then he presents a bad benchmark followed by a laundry list of what's wrong with it. Finally he explains that benchmarking Java is futile, unless you have written an optimizing JIT JVM.

The article reveals some good points, but the most interesting points are the ones the author studiously avoids mentioning. Chief among them is that if you're interested in performance, you won't be using Java, and you won't be using thread synchronization. He does note that if you're interested in measuring performance (to compare, I assume, to the same program in a real language) you'll be hard-pressed to get repeatable numbers from a threaded program. Therefore, his contempt for his audience is justified, but not for any reason he can afford to say in the article.

Among other points he fails to mention is that benchmarks routinely fail to account for cache and memory footprint effects. Such effects are difficult to measure because they are not itemized in profiles. A function that seems slower than it should be, considering how much it does, might actually spend most of its time just waiting for main memory to load into cache, or off disk. More commonly the footprint cost is distributed randomly among all the functions, so there's no obvious bottleneck to concentrate on.

Anatomy of a flawed microbenchmark (IBM developerWorks)

Posted Feb 25, 2005 16:46 UTC (Fri) by shane (subscriber, #3335) [Link]

I disagree. Consider a large class of applications: on-line databases. This could be anything: a web store, airline reservation system, and so on. These systems are often limited by poor database architecture and design more than the language used to implement the actual logic.

Do you care about performance on these systems? Absolutely. Could you write it in Python and get adequate speed? Absolutely.

I agree with the author in that most benchmarking is flawed, and that developers usually worry about optimisation in inappropriate ways. A little more understanding of the systems involved and judicious application of Amdahl's Law would help far more.

[OT-ish] When to optimize (hey, I'm USian :-)

Posted Feb 25, 2005 19:57 UTC (Fri) by Max.Hyre (subscriber, #1054) [Link]

[D]evelopers usually worry about optimisation in inappropriate ways.

I like the Laws of Optimization which I've seen attributed to Michael Jackson*:

First Law: Don't.

Second Law (for experts only): Don't, yet.

In many projects I've seen flouting them lead to, at least, wasted time, and, all too often, introduction of bugs.

* I once heard a rep for his SW development methods describe him as ``two-glove Michael Jackson''. (BTW, I've not read deeply into his work, but the chunks I've read have been very useful.)