Open64 5.0 released [LWN.net]

Open64 5.0 released

Posted Nov 11, 2011 20:13 UTC (Fri) by daglwn (guest, #65432)
In reply to: Open64 5.0 released by nmav
Parent article: Open64 5.0 released

These tests are pretty meaningless for at least a few reasons.

Clang doesn't support OpenMP at all so gcc "speedups" on those tests reflect the use of threading. Now, one could argue that's a fair comparison as it demonstrates features lacking in Clang, but the article doesn't make that point.

The more serious problem is that there's no attempt whatsoever to account for differing optimization levels. -O2 in gcc means something very different from -O2 of Clang/LLVM and likely means something entirely different again in Open64. A "real" benchmark or performance-critical production code will have tuned flags for each compiler expected to process it and those flags will be quite different for each compiler.

I've long been an advocate of having compilers default to their idea of the best optimization strategy (rathern than defaulting to -O0) and then compare compilers based on that decision. Everything else is throwing switches which can produce wildly different results.

to post comments

Open64 5.0 released

Posted Nov 11, 2011 21:47 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (13 responses)

These tests are meaningful. They show that if my distro were compiled using Open64 then I'd get a noticeable speedup.

I've tested it with my software, it's at least as fast as Intel's compiler.

Open64 5.0 released

Posted Nov 12, 2011 3:38 UTC (Sat) by daglwn (guest, #65432) [Link] (10 responses)

They show no such thing. They show some data for a few programs using a set of compiler flags probably chosen for gcc.

It is almost invariably true that Open64 will be better on some codes, gcc on others, Intel on still others, etc. If Open64 is better on the bulk of codes important to you, that's great. But the presented data in no way demonstrate that.

Open64 5.0 released

Posted Nov 12, 2011 4:51 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (9 responses)

I have just checked - most of my Linux software is compiled with -O2 and at most a couple of other optimization flags. So this comparison is very relevant.

>-O2 in gcc means something very different from -O2 of Clang/LLVM and likely means something entirely different again in Open64.

That is also incorrect. -O2 is a generic flag which means "turn on all sane optimizations" in all three compilers. Of course, optimizations used are going to be different - but that's the point.

And Open64 seems to fare significantly better than GCC in most of the tests.

Open64 5.0 released

Posted Nov 12, 2011 15:16 UTC (Sat) by endecotp (guest, #36428) [Link] (8 responses)

> -O2 is a generic flag which means "turn on all sane
> optimizations" in all three compilers.

"Citation required", as they say. That certainly isn't what "man gcc" says, for example.

Surely it is pointless to compare the performance of compilers based on anything other than their maximum optimisation level.

BTW, I have always found it strange that binary distributions compile by default at -O2. Do they not have the capacity in their build farms to use a higher level?

There are no such thing...

Posted Nov 12, 2011 15:57 UTC (Sat) by khim (subscriber, #9252) [Link] (4 responses)

Surely it is pointless to compare the performance of compilers based on anything other than their maximum optimisation level.

This is most pointless comparison of them all. Nobody uses -O3. Either people use -O2 (as "generally safe" optimization level) or they carefully tune flags and test the result.

BTW, I have always found it strange that binary distributions compile by default at -O2.

If you don't know why and when people use different options then perhaps you should not proclaim that you unjustified beliefs have are relation to real world?

Do they not have the capacity in their build farms to use a higher level?

This is not about build farms. This is about Q&A. Because "normal" optimization level is -O2 upstream developers often close bugs related to -O3 level as "WONTFIX" or "INVALID". This makes it absolutely unpractical to use -O3 for most packages. Only handful of packages where upstream actually test everything as -O3 (plus some hand-picked additional options) are compiled as -O3. This is the only realistic possibility. Gentoo allows you to recompile everything with -O3 but even there bugreports from such people are viewed as suspicious.

There are no such thing...

Posted Nov 12, 2011 17:07 UTC (Sat) by endecotp (guest, #36428) [Link] (2 responses)

> Nobody uses -O3.

Well, I use it as my normal setting for "release" builds.

Again, "citation required". Maybe you're right and (almost) everyone uses -O2 - but I would like to know where that info comes from.

> Because "normal" optimization level is -O2 upstream developers
> often close bugs related to -O3 level as "WONTFIX" or "INVALID"

Could you point me to a couple of examples of that? Thanks.

There are no such thing...

Posted Nov 14, 2011 2:10 UTC (Mon) by rsidd (guest, #2582) [Link]

It's not that -O3 is buggy (it can be but usually isn't) - it's that -O3's additional optimisations are not always of benefit. Source - gcc docs. On my code I find no significant speed difference in -O2 and -O3.

There are no such thing...

Posted Nov 16, 2011 19:32 UTC (Wed) by tetromino (subscriber, #33846) [Link]

> Maybe you're right and (almost) everyone uses -O2 - but I would like to know where that info comes from.

Take a look at *any* distribution's default optimization flags. For rpm-based distros, the default /usr/lib/rpm/rpmrc says 'optflags: amd64 -O2 -g'. In Debian, dpkg-buildflags returns 'CFLAGS=-g -O2' by default. Even in Gentoo, /usr/portage/profiles/arch/amd64/make.defaults says 'CFLAGS="-O2 -pipe"'.

There are no such thing...

Posted Nov 12, 2011 17:57 UTC (Sat) by cowboy (guest, #2231) [Link]

> Nobody uses -O3.

Wrong.

I use -O3 on Linux, and -O4 on AIX (xlc) for my product builds.

To be fair, there a few modules that don't fair well at -O4, but only because I've not had time to find out what is wrong with them (usually uninitialized variables).

My next step on AIX is IPA at link time, it will do inlining and other code movement that couldn't be done at compile time, due to the plethora of little pieces, all compiled separately.

Open64 5.0 released

Posted Nov 12, 2011 20:10 UTC (Sat) by nix (subscriber, #2304) [Link]

Given that -O3 under GCC means 'carry out optimizations that should reduce cycle count regardless of effects on code size', it fairly often leads to icache bloat or register pressure not compensated for at later stages and corresponding excessive stack spilling which end up slowing the code down.

So you end up with larger and/or slower code than under -O2.

Open64 5.0 released

Posted Nov 14, 2011 20:46 UTC (Mon) by daglwn (guest, #65432) [Link] (1 responses)

> Surely it is pointless to compare the performance of compilers based on
> anything other than their maximum optimisation level.

That would be no better and quite possibly much worse.

The "maximum" optimization level is very code-dependent. It takes a long time to tune the optimization flags to make your code run as fast as possible. Most Free Software projects don't bother. But in places like HPC where performance is everything, people spend a lot of time trying various flags and directives to squeeze out performance.

The only reasonable way to compare compilers is to tune the flags to get the best result for each and then see where you're at.

Open64 5.0 released

Posted Nov 22, 2011 17:54 UTC (Tue) by JanC_ (subscriber, #34940) [Link]

The only reasonable way to compare compilers is to tune the flags to get the best result for each and then see where you're at.

I think the only reasonable way to compare compilers is to test them the way you intend to use them:

In case of a linux distro, that means testing them with some default options that were chosen because they work well with all (or at least most) packages. You don't want to tune compiler options for 30000 packages one by one...
In case of a HPC environment, that means you tune each compiler to give its best result for the type(s) of code you need to run as fast as possible.

Open64 5.0 released

Posted Nov 12, 2011 17:40 UTC (Sat) by ballombe (subscriber, #9523) [Link] (1 responses)

A meaningful test would be to try to rebuild your distribution and count the number of compiler-related failure (including wrong code generation) with each compilers.
For a distribution, reliability is as much important than performance.

Open64 5.0 released

Posted Nov 12, 2011 21:18 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

Well, with gcc this number is near zero. So it's not interesting.

However, Open64 can lead to better performance in real-world apps just by changing gcc to Open64.

Open64 5.0 released

Posted Nov 14, 2011 5:34 UTC (Mon) by rsidd (guest, #2582) [Link] (4 responses)

I've long been an advocate of having compilers default to their idea of the best optimization strategy (rathern than defaulting to -O0)

Given that compilers are used numerous times during development, and only once (per file) during product delivery, and given that debugging is unreliable or impossible with anything other than -O0, I'm happy with the current default.

Open64 5.0 released

Posted Nov 14, 2011 15:29 UTC (Mon) by epa (subscriber, #39769) [Link]

I guess the answer is an --optimize-for-benchmarking flag which instructs the compiler to do the best it can, with no further information given. You could use that flag for building your released executables and it would probably do pretty well, or you could get into -fstrict-beard-alignment to squeeze out a little bit more performance. The point is to have a *single* setting both to give a fair comparison in benchmarks and to give a good enough default for those who don't want to become compiler experts.

For gcc, --optimize-for-benchmarking could be an alias for -O2, or perhaps -O3 or -O1... I'm not sure which, and that is the point.

Open64 5.0 released

Posted Nov 14, 2011 15:47 UTC (Mon) by endecotp (guest, #36428) [Link] (1 responses)

> debugging is unreliable or impossible with anything other than -O0

My experience is that I can normally get enough information to find a bug from a -O3 stack trace. If I can't, then it's possible to fall back to -O0.

I worry more about the possibilities that (a) some warnings aren't shown if optimisation is off, since apparently the data needed to generate the warning is only computed during an optimisation pass, and (b) some errors might only manifest themselves when optimisation is on, e.g. relying on the value of an uninitialised variable, or strict aliasing issues.

Open64 5.0 released

Posted Nov 14, 2011 16:17 UTC (Mon) by epa (subscriber, #39769) [Link]

You might be interested in kcc which is a full specification of the semantics of the C language, made executable. You can run your code under it and it will warn about undefined or implementation-specific behaviours your code is relying on.

Open64 5.0 released

Posted Nov 20, 2011 21:55 UTC (Sun) by BenHutchings (subscriber, #37955) [Link]

...debugging is unreliable or impossible with anything other than -O0...

I'm aware that the GNU toolchain sucks for source-level interactive debugging of optimised code, though this is slowly improving. But so long as you use C then many bugs are due to invoking 'undefined behaviour'. If your program has undefined behaviour then its actual behaviour cannot be understood in terms of the source language and will likely be inconsistent between the 'debug' and optimised build configurations. You have to learn to read assembly and deal with the crappy tools.