> > The non-profit Standard Performance Evaluation Corporation and
> volunteers
> > who work for it (including me) would be pretty dismayed that anyone
> > believes that it has no relevance to performance evaluation.
>> SPEC self-limits its relevance by refusing to recognize that it should
> be open-source. being open-hostile means that it has very limited
> numbers
> of data points,
Yup, only 9,335 submissions indexed on this page:
http://www.spec.org/cpu2006/results/cpu2006.html
I'm not getting into a debate. I'm glad there are open source benchmarks available too. I'll just provide some facts and let readers decide for themselves.
> very minimalistic UI (let alone data mining tools),
It is a limited text-based UI -- that runs on Linux, Windows, and proprietary Unixes. Portability was/is a major goal of SPEC.
The search form:
http://www.spec.org/cgi-bin/osgresults?conf=cpu2006&op=form
is useful for data mining.
> and perhaps most importantly, slow adaptation to changes in how
> machines
> are used (memory footprint, etc).
True. But SPEC MPI2007 v2.0 is a 2.4 GB package of software [larger datasets, and a 128 GB minimum RAM / 64 cores (min) requirement to run the Large suite; the Medium suite (16 GB, ~8 core minimum) is still part of 2.0].
I just downloaded Phoronix, and the tarball was ~450KB.
It looks like a good collection of tests.
Like HPC Challenge, and NAS Parallel, it does not provide a single number as a metric of performance. There are always compromises and knashing of teeth in coming up with a formula for that single number, but
SPEC and the Linpack/Top500 maintainers have found that people like it.
-Tom