Open Systems Group

The Open Systems Group is the oldest group within SPEC. The OSG is
the current embodiment of the original founders of SPEC. The OSG is
governed by the policies and principles described in the SPEC
Open Systems Group Policies and Procedures Document. The Open Systems
Steering Committee (OSSC) provides oversight and management for a number
of technical subcommittees that investigate and develop the benchmarks,
metrics, run and reporting rules, and so on.

OSG focus is on component- and systems-level benchmarks for desktop
systems, workstations and servers running open operating system environments.
This is the group responsible for the processor metrics SPECint2006
and SPECfp2006 (and their predecessors in 2000, 1995, 1992 and the
original SPECmarks from 1989).

In addition to the SPEC CPU2006 suite, the
group has also developed Java benchmarks, a SIP benchmark, web server
benchmarks, mail server benchmark, file server benchmarks, a power
and performance benchmark, and the virtualization benchmarks. Currently
in development are an updated CPU benchmark suite, a handheld device
performance benchmark, a SOA benchmark, and a cloud services benchmark.
More in-depth information may be found in the OSG
FAQ and under the benchmark specific links below.

SPECpower_ssj2008 is the first industry-standard SPEC benchmark
that evaluates the power and performance characteristics of volume
server class computers. The first subset of server workloads addresses
the performance of server-side Java; additional workloads are planned.

A standardized performance test designed to evaluate a system's
ability to act as a SIP server supporting a particular SIP application.
The application modeled is a VoIP deployment for an enterprise,
telco, or service provider, where the SIP server performs proxying
and registration.

SPEC's updated benchmark addressing performance evaluation of
datacenter servers used in virtualized server consolidation, measuring
end-to-end performance of all system components including the hardware,
virtualization platform, and the virtualized guest operating system
and application software. In addition to major workload upgrades,
the SPECvirt web server workload has been modified to require SSL
(HTTPS) between the client system and the web server.

Intended to measure server energy efficiency, initially as part
of the second generation of the US Environmental Protection Agency
(EPA) ENERGY STAR for Computer Servers program. Designed to be simple
to configure and use via a comprehensive graphical user interface,
the SERT uses a set of synthetic worklets to test discrete system
components such as memory and storage, providing detailed power
consumption data at different load levels.

Chauffeur was designed to simplify the development of workloads
for measuring both performance and energy efficiency. Because Chauffeur
contains functions that are common to most workloads, developers
of new workloads can focus on the actual business logic of the application,
and take advantage of Chauffeur's capabilities for configuration,
run-time, data collection, validation, and reporting.

Chauffeur was initially designed to meet the requirements of the
SERT. However, SPEC recognized that the framework would also be
useful for research and development purposes. The Chauffeur framework
is now being made available as the Chauffeur Worklet Development
Kit (WDK). This kit can be used to develop new workloads (or "worklets" in
Chauffeur terminology). Researchers can also use the WDK to configure
worklets to run in different ways, in order to mimic the behavior
of different types of applications. These features can be used in
the development and assessment of new technologies such as power
management capabilities.

Future Benchmarks

OSG has formed a subcommittee to develop a cloud services benchmark.
The intended audience for this benchmark includes hardware and software
vendors providing products that enable cloud services; cloud providers
that offer cloud services such as IaaS, PaaS or SaaS; and business
customers who would use benchmark results to help them select cloud
providers.

OSG has formed a subcommittee to develop a compute-intensive benchmark
suite for general consumer handheld devices which support an open
community of application developers with access to tools and libraries
for native applications.

SPEC has formed a new subcommittee to develop standard methods
of measuring performance for typical middleware, database and hardware
deployments of applications based on the Service Oriented Architecture
(SOA).

SPEC's industry standard benchmark designed to measure the performance
of corporate e-mail servers processing requests based on the Internet
standard protocols SMTP and IMAP4, with support for SSL v3.0 and
TLS 1.0.

SPEC's first industry-standardized mail server benchmark designed
to measure a system's ability to act as a mail server processing
email requests, based on the Internet standard protocols SMTP and
POP3.

System-MultiTasking. The next generation for the SDM benchmarks. A flexible workload harness with
a workload based upon public domain system utilities (e.g. gcc,
groff, gzip, etc.) performing basic code development activities.
Should provide a good measure of a system's ability to manage
large numbers of active users.

SPEC's benchmark addressing performance evaluation of datacenter
servers used in virtualized server consolidation, measuring end-to-end
performance of all system components including the hardware, virtualization
platform, and the virtualized guest operating system and application
software.

A standardized performance test for web servers, the benchmark
consists of different workloads (both SSL and non-SSL), such as
banking and e-commerce. The web server also communicates with a
lightweight backend to simulate an application/database server.
New in this benchmark is the addition of a power workload with a
measurement methodology taken from the SPECpower group.

A standardized performance test for WWW servers, successor of
SPECweb99 and SPECweb99_SSL, the benchmark consists of different
workloads (both SSL and non-SSL), such as banking and e-commerce,
and writes dynamic content in scripting languages to more closely
model real-world deployments. The web server also communicates with
a lightweight backend to simulate an application/database server.

Joining SPEC/OSG

We welcome your interest in joining the Open Systems Group. For more
detailed information on benefits, membership fees, and the application
process, please go to our membership page.

The SPEC Newsletter

The SPEC Newsletter was the official voice of
the OSG. This included news and analysis along with each quarter's
performance results. Results are now published electronically on the SPEC
web site.

Questions

General questions about the OSG benchmarks can be answered by SPEC's
main office staff, most easily reached as info@spec.org.

Technical questions about one of OSG's benchmark products should
be directed to SPEC's tech-support staff at support@spec.org.