SPEC SFS97 Q&A

Q.) What is SPEC SFS 2.0 and how does this benchmark compare to other
network file system (NFS) benchmarks?

A.) SPEC SFS 2.0 is the latest version of the Standard Performance
Evaluation Corp.'s benchmark that measures NFS file server throughput
and response time. It differs from other NFS benchmarks in that it provides
a standardized method for comparing performance across different vendor
platforms. The benchmark was written to be client-independent and
vendor-neutral. Results undergo peer review before publication on SPEC's public Web site and in
its hardcopy newsletter.

Q.) Does this benchmark replace the SPEC SFS 1.1 suite?

A.) Yes. Now that SPEC SFS 2.0 is available, SFS 1.1 licenses are no longer
being sold. SPEC is providing a six-month transition period from the date
of the SFS 2.0 announcement (December 19, 1997). During this period, SPEC
will accept, review and publish results from both benchmark versions. After
this period, results from SFS 1.1 will no longer be accepted by SPEC for
publication.

Q.) Can SPEC SFS 2.0 results be compared to SFS 1.1 results?

A.) No. Although the benchmarks are similar, they cannot be compared, since
SFS 2.0 uses a different workload.

Q.) What improvements have been made to SPEC SFS 2.0?

A.) In addition to general code improvements, SPEC SFS 2.0 includes five
major enhancements: 1. It measures NFS protocol version 3 results in
addition to those from NFS protocol version 2. 2. It adds support for TCP;
either TCP or UDP can be used as the network transport. 3. SPEC SFS
2.0's NFS operation mix more closely matches today's real-world NFS
loads. 4. The benchmark distribution CD contains pre-compiled and tested
binaries. 5. It has an improved interface to accommodate both accomplished
and novice users.

Q.) How was the SPEC SFS 2.0 workload determined?

A.) The SPEC SFS 2.0 workload is based primarily on a survey of more than
1,000 servers in different application environments. The survey found that
60 percent of these users have similar mixes of NFS operations.

Q.) What is the metric for SPEC SFS 2.0?

A.) SPEC SFS 2.0 has two performance measurement metrics: SPECsfs97.v2 for
NFS protocol version 2 and SPECsfs97.v3 for NFS protocol version 3. Both
metrics include a throughput measure (in operations per second) and an
overall response time measure (the average response time per operation).

Q.) Are the metrics for SPEC SFS 2.0 different than the metric for SFS 1.1?

A). Yes. SFS 2.0 removes the SFS 1.1 metric for response time at peak
measured throughput and replaces it with the overall response time and peak
throughput. The larger the peak throughput the better. The lower the
overall response time the better. The overall response time is an indicator
of how quickly the system under test responds to NFS operations over the
entire range of the tested load. In real-world situations, servers are not
run continuously at peak throughput, so peak response time provides only
minimal information. The overall response time is a measure of how the
system will respond under an average load. Mathematically, the value is
derived by calculating the area under the curve divided by the peak
throughput. Below the first data point is assumed to have a constant
response time equal to that of the first data point.

Q.) How widespread is NFS version 3?

A.) NFS version 3 has been shipping on systems for more than three years
and is available for most systems that support NFS version 2.

Q.) What is the correlation between the TPC (Transaction Processing
Council) benchmarks and SPEC SFS 2.0?

A.) There is no correlation; the benchmarks measure totally different
aspects of system performance.

Q.) Is SPEC SFS 2.0 a CPU- or I/O-intensive benchmark?

A.) SPEC SFS 2.0 is a system-level benchmark that heavily exercises CPU,
mass storage and network components. The greatest emphasis is on I/O,
especially as it relates to operating and file system software. To obtain
the best performance for a system running SFS 2.0, the vendor will
typically add additional hardware -- such as memory, disk controllers,
disks, network controllers and buffer cache -- to help alleviate I/O
bottlenecks and to ensure that server CPUs are used fully.

Q.) For what computing environment is SPEC SFS 2.0 designed?

A.) The benchmark was developed for load-generating clients running in the
UNIX environment. But since the load-generating clients execute the
benchmark code, SPEC SFS 2.0 can be used to test the performance of any NFS
server, regardless of the underlying environment. Porting is required,
however, for non-UNIX environments.

Q.) Can users measure NFS performance for workloads other than the one
provided within SPEC SFS 2.0?

A.) Yes, users can measure their own workloads by making changes to the
SPECsfs97 benchmark mix parameters to reflect the new measurements. The
SPEC SFS 2.0 User's Guide details how this can be done. Workloads
created by users cannot, however, be compared with SFS 2.0 results, nor can
they be published in any form, as specified within the SFS 2.0 license.

Q.) To what extent is the server's measured performance within SPEC SFS
2.0 affected by the client's performance?

A.) SPEC has written SFS 2.0 to minimize the effect of client performance
on SPECsfs97 results.

Q.) Why have only three companies reported SPECsfs97 results in conjunction
with this announcement?

A.) SPEC SFS 2.0 is a system-level benchmark that requires scheduling
substantial resources for testing. SPEC expects other member companies to
report results in the near future.

Q.) How does SPEC validate numbers that it publishes?

A.) Results published on the SPEC Web site and in the SPEC newsletter have
been reviewed by SPEC members for compliance with the SFS 2.0 run and
disclosure rules, but there is no monitoring beyond that compliance check.
The vendors that performed the tests and submitted the performance numbers
have sole responsibility for the results. SPEC is not responsible for any
measurement or publication errors.

Q.) Are the reported SFS 2.0 configurations typical of systems sold by
vendors?

A.) Yes and no. They are similar to large server configurations, but the
workload is heavier than that found on smaller server configurations. SPEC
has learned from experience that today's heavy workload is
tomorrow's light workload. For some vendors, the configurations are
typical of what they see in real customer environments, particularly those
incorporating high-end servers. For other vendors, SFS 2.0 configurations
might not be typical.

Q.) Do the SFS 2.0 run and disclosure rules allow results for a clustered
server?

A.) Yes, cluster configurations are allowed as long as they conform
strictly to the even distribution of all resources as defined by the SFS
2.0 run and disclosure rules.

Q.) Why do so few published results approach SPEC's response-time
threshold cutoff of 40 milliseconds?

A.) It is important to understand first that SPECsfs97 run rules do not
require that the throughput curve be carried out to 40 ms; they only state
that the results cannot be reported for a response time higher than 40 ms.
There are several reasons why results do not approach the threshold cutoff.
Optimally configured servers often will achieve their maximum throughput at
response times lower than the cutoff. Additionally, some vendors emphasize
maximum throughput while others concentrate on fast response time. It does
not indicate a problem with the results if the curve is not carried out to
40 ms, and those reviewing results should not try to predict what the
throughput curve might be past the reported point.

Q.) Why was the response-time threshold reduced from 50 ms for SFS 1.1 to
40 ms for SFS 2.0?

A.) The lower response-time threshold reflects advances in server
technologies since the release of SFS 1.1 in January 1995.

Q.) What resources are needed to run the SPEC SFS 2.0 benchmark?

A.) In addition to a server, a test bed includes several clients and an
appropriate number of networks. The server must have enough memory, disks
and network hardware to saturate the CPU. The test bed requires at least
one network and each network must have sufficient client capacity to
saturate the network(s). A minimum of 64 MB of memory is required for each
client, although in most cases 128 MB is needed. Requirements are detailed
in the SFS 2.0 User's Guide. To facilitate accuracy of reported vendor
results, SFS 2.0 includes an entire NFS implementation. Examples of typical
load-generating configurations can be found on the SPEC Web site.

Q.) What is the estimated time needed to set up and run SPEC SFS 2.0?

A.) Hardware setup and software installation time depend on the size of the
server and the complexity of the test beds. Many servers require large and
complex test beds. The SFS 2.0 software installs relatively quickly. A
SPECsfs97 submission from a vendor includes at least 10 data points, with
each data point taking about 20 to 30 minutes to complete.

Q.) What shared resources does SPEC SFS 2.0 use that might limit
performance?

Q.) SPEC's CPU95 benchmark defines compiler optimization flags that can
be used in testing. Does SPEC SFS 2.0 set tuning parameters?

A.) When submitting results for SPEC review, vendors are required to supply
a description of all server tuning parameters within the disclosure section
of the reporting page.

Q.) Can a RAM disk be used within a SPEC SFS 2.0 configuration?

A.) SPEC enforces strict storage rules for stability. Generally, RAM disks
do not meet these rules, since they often cannot survive cascading
failure-recovery requirements unless an uninterruptible power supply (UPS)
with long survival lines is used.

Q.) How will the choice of networks affect SFS 2.0 results?

A.) Different link types and even different implementations of the same
link type might affect the measured performance -- for better or worse --
of a particular server. Consequently, the results measured by clients in
these situations might vary as well.