If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

CompilerDeathMatch 64bit Final results

02-13-2011, 04:11 PM

I hereby present the final results from my compiler death match comparisons. Based on the experiences obtained during the generation of this dataset, I have decided to present it sorted and merged per optimization setting. If people are interested, I can also add merges of the data sorted per compiler (comparing different settings, I already have such data for PCC and Clang).

All the comparisons are now finished, with the exception of "Custom" that upon request still can be changed further.

Comment

There are missing compilers in some graphs. Worst example in the timed compilation of "custom".

I am not entirely sure about what you mean. That some compilers are missing from some tests are often due to that they failed that test (for example the compilation tests, where it was common that I got a compilation time but with a message like "exited with a non-zero status" or something like that. Those results were automatically omitted by PTS).

For Custom, I have only had requests for specific settings of GCC, ICC, Open64 and the flags I got as a tip from the PCC developer. PCC and TCC can not be included in the -O tests since they do not have those options (yet).

Comment

What I mean is, for example the "Timed Apache Compilation" only lists GCC and ICC. But later on, in the "Apache Benchmark", it also lists Open64 and PCC.

So from that one wonders how it's possible to do the "Apache Benchmark" with Open64 and PCC if those compilers weren't able to actually build Apache? (Since they're missing from the "Timed Apache Compilation" bench.)

Comment

What I mean is, for example the "Timed Apache Compilation" only lists GCC and ICC. But later on, in the "Apache Benchmark", it also lists Open64 and PCC.

So from that one wonders how it's possible to do the "Apache Benchmark" with Open64 and PCC if those compilers weren't able to actually build Apache? (Since they're missing from the "Timed Apache Compilation" bench.)

That is indeed a strange thing with the PTS tests. As I said in the previous reply, it often seemed like the compilations actually happened during compilation tests, but that they failed some sort of quality test and were omitted from the results. Perhaps the resulting binaries were still possible to use in the functional tests... The author of the "compiler" suite could probably explain this way better than I can.

Comment

Ohhh, i like custom!
Then gcc suddenly wins it all again. Look at the apache static page benchmark. With custom it gets a whopping 4117 requests per second! In your benchmarks i've never seen a higher number then that. I do wonder why open64 dropped... it seems to decrease performance with those flags.

in fact, gcc seems to win (nearly) all with those flags which leads me to thing that the CO: CFLAGS has something odd in it since open64 can perform better at just -o2 ..

In general, It seems that GCC performs very well, I have made a few tests by my own of GCC vs ICC and I have found similar results, the only cases where ICC seems to really outperform GCC is on multi-threaded applications like mplayer-mt or c-ray-mt.