TPC Benchmark Status
February 2004

Overview

The TPC held a General Council meeting
on December 11 in Dana Point, California. The main focus
of the work was on refining existing benchmarks and laying
the
foundation
for new benchmarks. On the OLTP front,
work continues on the
new TPC-E specification with a focus in two general areas:
TPC-provided source code and new/modified features of the
benchmark. The TPC-C benchmark
has been updated to Version 5.2.
With respect to decision support, progress continues on
the
new DS
benchmark on dbgen output modifications,
benchmark metrics, data maintenance, and pricing. For the TPC-H
and TPC-R benchmarks, solutions
are being explored to address dbgen issues with 64 bit
architectures.
As for the
transactional
web e-Commerce benchmark,
a draft specification of the next version of TPC-W is
now available for public review.

Current Benchmarks

TPC-C

The TPC-C Maintenance Subcommittee
completed work on Version 5.2 of the TPC-C Specification.
This version will become effective on February 9, 2004. The
changes in Version 5.2 include:

Wording to clarify certain clauses in the specification

Definition of a third-party

Revising component availability
dates

Line item pricing requirements

Report performance (tpmC) as a whole number (previously
shown with 2 digits after the decimal point)

Results obtained with TPC-C Version
5.2 are comparable with those obtained with TPC-C Version 5.1

TPC-H and R

The TPC-H/R subcommittees
identified a problem in dbgen when compiled for 64 bit architectures:
Due to a data type overflow in dbgen, customer and supplier
address fields are generated differently on 32 and 64 bit
architectures. The code causing this problem is in rnd.c.
A simple workaround allows users to generate the correct
address fields. The subcommittee has proposed a code change
which is currently being implemented and scheduled for vote
during the February meeting. It is believed that there is
no performance impact due to this behavior. A detailed investigation
is under way.

New Benchmarks

TPC-E (OLTP)

The TPC-E Development Subcommittee continued development of a new OLTP
workload. The workload is designed to challenge computing
environments of today and the future. The subcommittee reviewed
several presentations
from member companies regarding the current design of the
benchmark environment and ideas to expand functionality. Early
prototype data were presented and
work items have been identified to address concerns raised
by the data. An updated version of the source code
was delivered
and work items for the next version have been identified.
Also, an updated version of the draft specification was approved
along with ongoing changes
and development of the specification.

TPC-DS

The subcommittee continued its work
on developing TPC-DS in the areas of dbgen output modifications,
metric, data maintenance and pricing.

Dbgen output can be
modified. The resources (time, systems) to generate and
modify dbgen output are not included in any metric. The subcommittee
agreed on a basic throughput based metric. It further developed
a catalog of criteria as a framework to finalize the metric.
A group of members have been tasked to finalize the metric
and present a proposal at the next meeting. The data
maintenance portion of the specification has been overhauled
according
to motions made at the last meeting. Pricing in TPC-DS
will
follow closely the general pricing document that the TPC
is currently developing with minimal TPC-DS specific clauses.
In this process the subcommittee also identified some minor
issues with the pricing document, which will be forwarded
to the pricing subcommittee.

TPC-W

The TPC-W subcommittee continued its development of the
next version of TPC-W. Version 2 is
a transactional web service benchmark consisting of
database interactions displaying ACID properties running
against a commercial
application server. The primary metrics
are SIPS (Service Interactions Per Second), Price Performance
which is $/SIPS, and the system availability date.

Components that must be commercially available have been
enumerated and defined. The subcommittee has also adopted
additional requirements such as adhering to the WS-I basic
profile (BP1.0) for web service interoperability. Strict
restrictions have been placed on user written/modified
code regarding operations resulting in caching of data.
All caching of durable data must maintain full atomicity,
consistency, and isolation (ACI).

All benchmark clauses
have been defined and accepted. This includes processing
requirements, database schema and population,
test run requirements, and steady state requirements for
the test run. It also includes ACID tests which demonstrate
compliance of the database system as well as that of distributed
transactions and message queues. Reporting requirements,
and audit requirements for the web service interactions
have also been defined and accepted.

Two prototype efforts
have thus far yielded three initial sets of prototype data
(2 .NET, 1 Java). Additional prototype
data is expected as time proceeds. Prototype performance
is trending upward as the implementations evolve. The prototypes
demonstrate the specified workload can be implemented using
both Java and .NET.

The subcommittee presented the
draft specification to the TPC General Council and received
unanimous approval
to proceed to Company and Public review with comments accepted until February 20, 2004.

Other TPC Activities

Public Relations Committee

The PR Committee continues its work on the campaign to promote
the TPC and the TPC benchmarks to the industry. In response
to the increasing publication of TPC results in currencies
other than US dollars, the TPC web site has been enhanced
to display results grouped by currency.

Pricing

The TPC formed a pricing workgroup on October 9, 2002 to
recommend revisions to the existing pricing methodology so
that prices
used in published TPC results are clear, consistent and verifiable
for all the TPC benchmarks. The revisions do not change what
is priced. Its adoption is expected to produce results comparable
with current results. The requirements for reporting and
ordering are enhanced and strengthened, allowing improved
verifiability of pricing. The pricing specification would
be referenced by all existing and future TPC specifications
for a consistent methodology.

The draft TPC pricing specification
is now available for both public and member company review,
with comments accepted
until February 29, 2004.