Frequently Asked Questions (FAQs) About SPEC OMP2001 Software

Q1: What is SPEC OMP2001?

A1: SPEC OMP2001 is a software benchmark product produced
by the Standard Performance Evaluation Corp.'s High-Performance Group (SPEC/HPG).
SPEC is a non-profit organization that includes computer vendors, systems
integrators, universities, research organizations, publishers and consultants
from around the world. The benchmark is designed to provide performance measurements
that can be used to compare compute-intensive parallel workloads on different
parallel computing systems.

SPEC OMP2001 contains two benchmark suites. The first benchmark, SPEC OMPM2001,
measures performance of shared-memory systems with between four and 32 processors.
The second, SPEC OMPL2001, is designed to measure performance of systems with as
many as 512 processors.

Q2: What is a benchmark?

A2: A benchmark is a standard of measurement or evaluation.
SPEC was formed to establish and maintain computer benchmarks for measuring
and comparing component- and system-level computer performance.

Q3: What components does SPEC OMP2001 measure?

A3: Since the benchmarks are designed to reflect applications
requiring compute-intensive parallel processing, they measure performance
of the computer's processors, memory architecture, operating system, and
compiler. It is important to remember the contribution of the latter three
components.

Q4: What component performance is not measured by SPEC OMP2001?

A4: The OMP2001 benchmarks do not stress I/O (disk drives),
networking or graphics. It might be possible to configure a system in such
a way that one or more of these components impact the performance of OMP2001,
but that is not the intent of the suites.

Q5: What is included in the SPEC OMP2001 packages?

A5: SPEC provides the following:

SPEC OMP2001 tools for compiling, running and validating the benchmarks
for a variety of operating systems

source code for the tools, so that they can be built for systems not
covered by the pre-compiled tools

source code for the benchmarks

tools for generating performance reports

run and reporting rules defining how the benchmarks should be used
to produce standard results

SPEC OMP2001 documentation

SPEC OMP2001 includes tools for most UNIX operating systems. Additional products
for Windows NT and other operating systems will be released later if SPEC
detects enough demand. SPEC OMPM2001 and SPEC OMPL2001 will be shipped on
separate CD-ROM disks.

Q6: What does the SPEC OMP2001 user have to provide?

A6: The user must have a computer system with a CD-ROM
drive that has a UNIX based O/S with both C and FORTRAN90 compilers supporting
OpenMP. Depending on the system under test, approximately 3GB will be needed
on a hard drive to install, build and run SPEC OMPM2001; twice as much is
needed for SPEC OMPL2001.

The system should have at least 2GB of RAM for SPEC OMPM2001 and 8GB for
SPEC OMPL2001 to ensure that benchmarks remain in memory and paging does
not occur. Systems with large processor accounts will require greater amounts
of memory.

Q7: What are the basic steps in running the benchmarks?

A7: Installation and use are covered in detail in the
SPEC OMP2001 User Documentation. The basic steps are as follows:

Install SPEC OMPM2001 or SPEC OMPL2001 from media.

Run the installation scripts specifying your operating system.

Compile the tools if executables are not provided in SPEC OMP2001.

Determine which metric you want to run.

Create a configuration file for that metric. In this file, you specify
compiler flags and other system-dependent information.

Run the SPEC tools to build (compile), run and validate the benchmarks.

If the above steps are successful, generate a report based on the run
times and metric equations.

Q8: What source code is provided? What exactly makes up these suites?

A8: SPEC OMPM2001 and SPEC OMPL2001 are based on compute-intensive,
parallel-processing applications that are provided as source code containing
OpenMP directives. A medium and large version of the same 9 applications
(seven FORTRAN90 and two C) are used within the two benchmark suites along
with an additional FORTRAN90 and an additional C code for the medium version:

The numbers in the benchmarks' names serve as identifiers to distinguish
programs from one another and from similar codes in SPEC CPU2000. More detailed
descriptions of the benchmarks can be found in the individual benchmark directories
in the SPEC benchmark tree.

Q9: What metrics can be measured?

A9: The benchmark suites can be used to measure and calculate
the following metrics:

SPEC OMPM2001SPECompMpeak2001: The geometric mean of 11 normalized ratios (one
for each benchmark) when compiled with "aggressive" optimization and possible
code modification.SPECompMbase2001: The geometric mean of 11 normalized ratios (one
for each benchmark) when compiled with "conservative" optimization (all
benchmarks compiled with the same flags and no modifications to source
code).SPECompM2001: greater of base & peak metric.

SPEC OMPL2001SPECompLpeak2001: The geometric mean of 11 normalized ratios (one
for each benchmark) when compiled with "aggressive" optimization, and possible
code modification. SPECompLbase2001: The geometric mean of 12 normalized
ratios (one for each benchmark) when compiled with "conservative" optimization
(all benchmarks compiled with the same lags and no modifications to source
code).SPECompL2001: greater of base & peak metric.

The ratio for each of the benchmarks is calculated using a SPEC-determined
reference time and the actual run time of the benchmark. A higher score means "better
performance" on the given workload.

Q10: What is the difference between a "conservative" (base) metric
and an "aggressive" (non-base) metric?

A10: In order to provide comparisons across different
computer hardware, SPEC provides benchmarks as source code. This means they
must be compiled before they can be run. There was agreement within SPEC
that the benchmarks should be compiled the way users compile programs. But
how do users compile programs? On one side, people might just compile with
the general high-performance options suggested by the compiler vendor. On
the other side, people might experiment with many different compilers and
compiler flags to achieve the best performance. So, while SPEC cannot match
exactly how everyone uses compilers, it can provide metrics that represent
the general characteristics of these two groups.

The base metrics (e.g., SPECompMbase2001) are required for all reported results
and have set guidelines for compilation (e.g., the same flags must be used
in the same order for all benchmarks of the same language, no assertion flags).
The assumed model uses performance compiler flags that a compiler vendor
would suggest for a given program knowing only its own language.

The non-base metrics (e.g., SPECompMpeak2001) are optional and have less-strict
requirements (e.g., different compiler options and code modifications related
to parallel performance can be used on each benchmark).

A full description of the distinctions can be found in the SPEC OMP2001 run
and reporting rules.

Q11: How should I use SPEC OMP2001?

A11: Typically, the best measurement of a system is the
performance of your own application with your own workload. Unfortunately,
time, money and other constraints make it very difficult to get a wide base
of reliable, repeatable and comparable measurements on different systems.

Benchmarks act as a reference point for comparison. It's the same reason
that gas mileage ratings exist, although probably no driver gets exactly
the same mileage as listed in the ratings. If you understand what benchmarks
measure, they're useful.

It's important to know that SPEC OMPM2001 and -large focus on parallel-processing
not system performance. They concentrate only on some of the factors that
contribute to applications performance. A graphics or network performance
bottleneck within an application, for example, will not be reflected in these
benchmarks.

Understanding your own needs helps determine the relevance of the benchmarks.

Q12: Why was SPEC OMP2001 developed?

A12: As more multi-processor systems became available,
the SPEC High Performance Group saw a need for a benchmark suite to measure
parallel performance, and more specifically, the performance of shared-memory
parallel systems. The OpenMP directives were used because they have become
the de facto standard for implementing this type of parallelism.

Q13: What criteria were used to select the benchmarks?

A13: In the process of selecting applications to use as
benchmarks, SPEC considered the following criteria:

benchmarks should produce scalable parallel performance over several
architectures.

benchmarks should not include measurable I/O

benchmarks should not include networking or graphics

benchmarks should run in 2GB of RAM for SPEC OMPM2001 and 8GB of RAM
for SPEC OMPL2001 without swapping for a single CPU.

no more than five percent of benchmarking time should be spent processing
code not provided by SPEC.

Q14: Weren't most of the SPEC OMP2001 benchmarks in SPEC CPU2000?
How are they different?

A14: Although some of the benchmarks from SPEC CPU2000
are included in OMP2001, they all have been given larger workloads. Also,
all of the codes have been modified for parallelism, including the insertion
of OpenMP directives. The revised benchmarks have been assigned different
identifying numbers to distinguish them from versions in previous suites
and to indicate that they are not comparable with their predecessors.

Q15: Why does SPEC use a reference machine for determining performance
metrics? What machine is used for SPEC OMP2001 benchmark suites?

A15: SPEC uses a reference machine to normalize the performance
metrics used in the OMP2001 suites. Each benchmark is run and measured on
this machine to establish a reference time for that benchmark. These times
are then used in the SPEC calculations.

SPEC OMP2001 uses an SGI 2100 with four 350MHz processors as the reference
machine. It takes approximately one-and-a-half days to do a SPEC-conforming
run of SPEC OMPM2001 using all four processors on this machine. The performance
relation between two systems measured with the OMP2001 benchmarks would remain
the same even if a different reference machine was used. This is a consequence
of the mathematics involved in calculating the individual and overall (geometric
mean) metrics. The SGI reference machine performs at about 1000 SPECompMbase2001.

Q16: How long does it take to run the SPEC OMP2001 benchmark suites?

A16: It depends on the suite and the machine that is running
the benchmarks. In particular it will depend on the number of processors
in the target machine.

Q17: What if the tools cannot be run or built on a system? Can
they be run manually?

A17: To generate SPEC-compliant results, the tools used
must be approved by SPEC. If several attempts at using the SPEC tools are
not successful for the operating system for which you purchased OMP2001,
you should contact SPEC for technical support. SPEC will work with you to
correct the problem and/or investigate SPEC-compliant alternatives.

Q19: Can SPEC OMP2001 results be published outside of the SPEC
web site?

A19: Yes, SPEC OMP2001 results can be freely published
if all the run and reporting rules have been followed and the results are
reviewed by SPEC/HPG for a nominal fee. Yes, SPEC OMP2001 results can be
freely published if all the Run and Reporting Rules have been followed and
reviewed by SPEC/HPG for a nominal fee. The SPEC OMP2001 license agreement
binds every purchaser of the suite to the run and reporting rules if results
are quoted in public. A full disclosure of the details of a performance measurement
must be provided to anyone who asks. See the SPEC OMP2001 Run and Reporting
Rules for details.

SPEC strongly encourages that results be submitted for the web site, since
it ensures a peer review process and uniform presentation of all results.
The Run and Reporting Rules contain an exemption clause for research and
academic use of SPEC OMP2001. Results obtained in this context need not comply
with all the requirements for other measurements. It is required, however,
that research and academic results be clearly distinguished from results
submitted officially to SPEC.