SPECjAppServer2004 is a J2EETM benchmark
to measure the scalability and performance of J2EE servers and containers
and is based on the J2EE1.3 Specification. SPECjAppServer2004 strives
to stress the middle-tier rather than the client tier or the database
server tier.

This document is a practical guide for setting
up and running a SPECjAppServer2004 benchmark. This user guide discusses
some, but not all, of the rules and restrictions pertaining to SPECjAppServer2004.
Before continuing with the benchmark, we strongly recommend you read
the complete SPECjAppServer2004 Run and Reporting
Rules contained in the kit. For an overview of the benchmark architecture,
see the SPECjAppServer2004 Design Document also
contained in the kit.

"SPECjAppServer2004 JOPS" is the SPECjAppServer2004
performance metric and denotes the jAppServer Operations Per Second completed
during the Measurement Interval. "SPECjAppServer2004 JOPS" is
composed of the total number of business transactions completed in the
Dealer Domain, added to the total number of workorders completed in the
Manufacturing Domain, normalized per second.

EAStress2004 is a research workload
created on the basis of SPECjAppServer2004 benchmark. It is only provided
as a tool for research use such as testing and experimentation in situation
where the benchmark run rules cannot be met due to technical reasons
(e. g., usage of projects under development, usage of experimental platforms).
EAStress2004 results are not reviewed by SPEC. EAStress2004 results can
in no way be compared with SPECjAppServer2004 results.

"EAStress2004 HASOPM" is the EAStress
performance metric and denotes the Hundreds of Application Server
Operations Per Minute completed during the Measurement Interval. "EAStress2004
HASOPM" is composed of the total number of business transactions
completed in the Dealer Domain, added to the total number of workorders
completed in the Manufacturing Domain, normalized to hundreds per minute.

The Domain is a logical entity that
describes a distinct business sphere of operations. The four SPECjAppServer2004
domains are: Dealer, Manufacturing, Supplier and Corporate.

The Driver refers to the client code
that drives the benchmark, monitors requirements, and reports results.

SPECjAppServer2004 Kit refers to the
complete kit provided for SPECjAppServer2004. This includes the SPECjAppServer2004
Reference Beans, the Driver, load programs, and documentation.

SPECjAppServer2004 Reference Beans refers
to the implementation of the Enterprise Beans provided for the SPECjAppServer2004
workload.

The Supplier Emulator is a Java Servlet
that can run inside any Java enabled web server, and emulates the process
of sending and receiving orders to/from suppliers.

The SUT is the System UnderTest
and is comprised of all components that are being tested. This includes
Application Servers/Containers, Database Servers, network connections,
etc. The Driver and Supplier Emulator are not part of the SUT.

The SPECjAppServer2004 workload emulates an
automobile manufacturing company and its associated dealerships. Dealers
interact with the system using web browsers (simulated by the driver)
while the actual manufacturing process is accomplished via RMI (also
driven by the driver). This workload stresses the ability of Web and
EJB containers to handle the complexities of memory management, connection
pooling, passivation / activation, caching, etc. The SPECjAppServer2004 Design Document includes
a complete description of the workload and the application environment
in which it is run. This section of the user's guide describes software
and hardware environment required to run the workload.

Although SPECjAppServer2004 can be run on a
single machine for testing purposes, compliance with the SPECjAppServer2004 Run and Reporting Rules requires
that the driver and supplier emulator be run on a machine outside the
SUT. Therefore, a compliant hardware configuration must include a network
and a minimum of two systems – one or more systems to run the components
within the SUT and at least one system to run the driver and supplier
emulator outside the SUT. A typical configuration is illustrated below.

SPECjAppServer2004 is a J2EE application that
requires a J2EE 1.3 compatible application server as well as a Relational
Database Management System (RDBMS) to run as part of the SUT. Outside
the SUT, a Java Runtime Environment (JRE) version 1.3 or later is required
for the driver and supplier emulator. A web server supporting HTTP 1.1
is also required for the supplier emulator

The SPECjAppServer2004 Kit is supplied as either
a tar, gzip, or zip file that should be extracted into a directory. We
will call this top level directory ${JAS_HOME}.

SPECjAppServer2004 can be run on any Application
Server that is compliant with the J2EE 1.3 Specification and
has passed the Certification Test Suite (CTS) for that specification.
Most of the code is implemented in Java, however there are a few scripts
used to start certain programs. These scripts are provided in .sh format
for Unix platforms and in .bat format for Windows.

bin : Driver and database load scripts.
All additional scripts except specific database related scripts will
be in this directory.

classes : This is a directory where
compiled Java classes are placed. It comes with all the necessary class
files necessary to run SPECjAppServer2004. Re-compilation of the source
files is unnecessary to run the application.

classes/META-INF : Temporary place
for holding deployment descriptors just before packaging into JAR format
files.

classes/WEB-INF : Temporary place
for holding web deployment descriptors just before packaging the WAR
files.

config : Environment configuration
and properties files used by the Driver, make and load programs. Once
SPECjAppServer2004 has been successfully deployed, this is the only place
where the configuration should be modified.

config/security : Security policy
files used for the benchmark programs. These should not be modified.

docs : The documentation for all SPECjAppServer2004
beans generated by running 'make doc'.

jars : The JAR files created by the
make files. The files distributed in this directory are driver.jar, launcher.jar,
charts.jar and xerces.jar. The application server specific jars for deployment
need to be generated by the user.

License : Contains license document
files for the product

output : Default directory in which
output of runs go.

reporter : Contains the source for
the reporter which consumes results from a run to create the submission
package for SPEC.

schema : Schema information and static
data to be loaded into the database.

schema/sql : Generic SQL scripts for
creating the tables, etc. in the database. These should be copied and
modified as necessary for a particular database product.

The beans adhere to the EJB 2.0 specification.
This package only provides beans that are deployable as Container Managed
Persistence (CMP) entities.

Of the packages listed above, corp,orders,mfg,
and supplier contain the beans used for the four SPECjAppServer2004
domains respectively; the util package is included once with each
of the other domains, more on this later. Each domain package contains
distinct sub-packages for each EJB and its helper classes. The EJB package
contains a sub-package named "ejb" which holds the bean implementation
including the home and remote interfaces. For instance, the following
shows the fully qualified class names for the OrderLineEnt entity bean:

org.spec.jappserver.orders.orderlineent.ejb.OrderLineEntHome

org.spec.jappserver.orders.orderlineent.ejb.OrderLineEnt

org.spec.jappserver.orders.orderlineent.ejb.OrderLineEntPK

org.spec.jappserver.orders.orderlineent.ejb.OrderLineCmpEJB

The helper classes for the orders domain are
in the helper sub-package under orders as follows:

The common package contains the source
code for helper classes used by all the beans (i.e. it is common across
different domains as well as beans).

The util package contains the beans
used to generate primary keys for the various domains.

The supplier.emulator package contains
the Supplier Emulator Servlet.

The driver package contains the SPECjAppServer2004
drivers.

The load package contains the database
data generation and load programs required to load the SPECjAppServer2004
database(s) according to the benchmark scaling rules.

xerces.jar is used by Delivery Servlet
in Supplier domain and the Supplier Emulator Servlet. Add this jar file
in the Application server classpath if XML DOM Parser packages are not
part of your application server. To include this jar in the Tomcat server,
copy xerces.jar file to ${TOMCAT_HOME}/lib directory.

There are several components required to build,
deploy, test and run SPECjAppServer2004. These are :

An EJB Container to deploy all the SPECjAppServer2004 Beans.

A Web Container to deploy the JSPs for the HTTP Clients in the Dealer
Domain as well as the Delivery Servlet in the Supplier Domain.

An external Web Server to run the Supplier Emulator Servlet. This
web server should turn off the keep-alive option. The reason
for this is the Emulator uses short, bursty, transactions and does
not benefit from keep-alives as most Web Applications do.
The emulator is not part of the SUT and may not be deployed on the
same system as the EJB or Web containers running within the SUT.

The SPECjAppServer2004 Driver that runs on client systems external
to the SUT

Building and deploying the benchmark requires Ant
1.6.2, which is provided as part of the SPECjAppServer2004
kit.

Several steps must be accomplished prior to
running the SPECjAppServer2004 benchmark:

Create the SPECjAppServer2004 database(s)

Configure the environment

Build the .jar, .war, and .ear files

Load the database

Deploy the .ear files

Deploy the emulator

Test the SPECjAppServer2004 deployment using the Web interface

Configure the driver to run SPECjAppServer2004

3.1.1 Create the Database(s)

For the Standard Workload, you can create a
single database that houses all the 4 Domains. For the Distributed Workload,
you must create 4 separate databases, one for each domain.

Standard SQL scripts for creating the database
schema are provided in schema/sql. These are intended to give
a starting point for creating schemas for other database products. It
is beyond the scope of this user guide to provide guidance for the installation
and configuration of the various RDBMs available on the market. For convenience
database creation scripts are provided for Oracle 9i, DB2 and Sybase
in schema/<DBMS>. These are example scripts provided by
the RDBMs vendors and can be modified, but serve as a starting place
for deployment. However, see section 2.4 of the SPECjAppServer2004
Run and Reporting Rules for restrictions on modifications to the
schema scripts.

Loading of the database's is deferred until
the environment file and associated batch / script files have been updated
in subsequent steps.

This section describes the steps required to
build the SPECjAppServer2004 ear in preparation for deployment. These
instructions assume that the target application server has been installed
and the environment variable ${J2EE_HOME} points to the directory
above the lib directory that contains the j2ee.jar. This
is generally the home of the AppServer being tested. Please refer to
your specific application server's installation instructions for information
about a specific product.

In the ${JAS_HOME}/config directory there
are files named appserver and default.env which contain
the environment variables used to deploy and run the benchmark.

The file $JAS_HOME/config/appserver is
used to locate the correct env file for deployment and execution.
The contents of this file are appended with .env and the resulting
file becomes the properties file used. You may either update the supplied default.env file
or make a copy of it. For example, if you copy default.env to myAppServer.env update
the file $JAS_HOME/config/appserver to contain myAppServer.

Specifies the root directory
for the SPECjAppServer2004 kit. For example, if deploying
the benchmark on Windows in C:\working\SPECjAppServer2004
update the environment variable like:

JAS_HOME=C:/working/SPECjAppServer2004

FS

This variable defines the File
Separator for the deployment / driving system. For users
deploying on Windows you must update the ${FS} environment
variable from ':' to ';' otherwise you will get "Class not
Found" Exceptions when running the scripts provided with
this benchmark.

JAVA_HOME

Normally the Application Server
being tested provides a JRE to run their application server.
If you choose to use the supplied one update this environment
variable to root directory for the JRE. Otherwise, you can
point to the JRE of your choice.

Note:The JDK pointed to here is the one used to drive
workload. This is independent of the JRE used to start the
Launcher which is started in the driver.sh or driver.bat scripts
depending on your driving system.

J2EE_HOME

The Application Server being
tested should provide a library (e.g. j2ee.jar) containing
its implementation of the J2EE specification. This directory
specifies the directory above the lib directory.

JAVAX_JAR

The path to the Application
Server's library (e.g. j2ee.jar) containing its implementation
of the J2EE specification.

Note: In versions of the kit up to 1.03, the JAVAX_JAR
property was not included in the sample properties file ${JAS_HOME}/config/default.env.
Make sure you don't forget to set this property in your env file.

BIND_WAIT

Specifies the number of seconds
that the Launcher should wait after starting all processes
to allow them to initialize and connect to the controller.
Default is 5 seconds if it is not specified.

CLASSPATH
related Entries

Note: The Launcher (invoked in driver.sh / driver.bat
) prepends the file ${JAS_HOME}/jars/driver.jar to
the beginning of the classpath so that the driver cannot
be overridden.

CLASSPATH

Contains the various jar files
required by the driver. Note: The corp.jar, mfg.jar, orders.jar,
and supplier.jar are assumed to contain the stubs for the
AppServer being tested. In most cases these are the jars
that are created during the deployment process for the AppServer
being tested. Review the default.env file for an example
of this variable.

JDBC_CLASSPATH

Provides the path and jar files
needed to access the database; this will be unique for each
RDBMS. This path is referred to in the load.sh or load.bat script
file as appropriate for your operating environment

SUT
AppServer Information

Specify the name of the host where the Emulator will find
the DTD for the POs and for the Audit Beans to locate
and reset logs.

JAS_HOST

Identifies the system(s) that
house the DeliveryServlet as well as specifying the DTD that
the Emulator will need to read to validate the PO XML sent
as part of Supplier interaction with the Emulator.

JAS_PORT

The port number of the WebContainer
/ HTTP entry point to the SUT. Default is port 80.

JAS_PREFIX

Set this to prepend a prefix
for the Web URL. This is not necessary in most deployments.Provided
to improve flexibility for testing.

Emulator
Information

Like the similar variables used for the SUT these variables
provide information to the SUT on how to access the Emulator
as well as for the auditing Beans to determine the number
of emulator interactions. These parameters are used during
deployment as well as at runtime.

EMULATOR_HOST

Specifies the name of the system
that is acting as the emulator.

EMULATOR_PORT

Default port for HTTP requests
made to the Emulator.

EMULATOR_PREFIX

Used to prepend a prefix for
the Emulator URL. This is not necessary in most deployments
but is provided for flexibility in testing and research.

RMI
Settings

These variables are used during deployment and runtime
by the driver. They are Application Server specific.
Refer to the documentation provided by your Application
Server vendor for specific information. Appendix A provides
the settings provided by some Application Server vendors
for these values.

NAMING_PROVIDER

Identifies the URL of the Remote
RMI endpoint to provide name resolution and to send RMI requests.
Consult the documentation for the tested AppServer for the
correct setting.

EMULATOR_PORT

Default port for HTTP requests
made to the Emulator.

JNDI_CLASS

This setting (like the NAMING_PROVIDER)
is unique to the AppServer. Each AppServer should provide
a class name to effect JNDI lookups.

Driver
JVM Settings

These settings are used for the launched components of
the driver.

JVM_OPTIONS

Identifies the URL of the Remote
RMI endpoint to provide name resolution and to send RMI requests.
Consult the documentation for the tested AppServer for the
correct setting.

JAVA

Used by the Launcher to start
the various subprocesses for the Orders Domain (HTTP) as
well as the Manufacturing Domain (RMI)

3.1.2.2 Update the setenv scripts

The ${JAS_HOME}/bin directory contains
scripts to setup the shell environment for subsequent steps. There are
two files provided; a .sh for Unix environments and a .bat file
for Windows. Below is a sample .sh file. Update these variables
to the values as specified in the env file above and then execute
the shell script. For example: ../setenv.sh

3.1.2.3 Create the descriptor's directory

During subsequent build steps the classes of
the SPECjAppServer kit will be combined with the set of J2EE deployment
descriptors used to create a deployable ejbjar file as well as combine
the various EJB jars into a deployable unit called an Enterprise ARchive
(EAR). The various Deployment Descriptors (DDs) used in the kit need
to have some information in them updated with some of the information
provided in the environment file. As a result, during the build process
the ANT utility will look in the directory ${JAS_HOME}/src/deploy/<appserver> to
find the appropriate deployment descriptors.

In the ${JAS_HOME}/src/deploy/ directory
create a directory which follows the value you specified in step 3.1.2.1
and copy the reference DDs into this new directory. If the file ${JAS_HOME}/config/appserver file
had been updated to myAppServer the following commands on UNIX
would need to be executed:

Application Server vendors may provide additional
DDs required for their product to complete the deployment process. They
should also provide a set of instructions that work within the context
of this build environment to complete the steps for their products.

3.1.2.4 Build the .JAR, .WAR and .EAR files

The following commands for both Unix and Windows
can be used to build the EJB Jar, Web Application
aRchive (WAR)and Enterprise ARchive (EAR) files.
We assume that the file ${JAS_HOME}/config/appserver has been
updated to myAppServer in step 3.1.2.1.

Unix

Windows

$ cd ${JAS_HOME}

C> cd %JAS_HOME%

$ ant/bin/ant clean

C> ant\bin\ant clean

$ ant/bin/ant -Dappserver=myAppServer

C> ant\bin\ant -Dappserver=myAppServer

If any errors occur at this step it is generally
a setup / directory mis-match problem. Please review all settings carefully
prior to contacting SPEC for support.

3.1.2.5 Build Load Programs and Initialize
the Database(s)

There are several files in the $JAS_HOME/config/ directory
named *db.properties. These files are used by the load utility
programs to find and connect to the database server you will be using
for your tests. You will need to know the correct URL for your JDBC driver;
please refer to your JDBC driver's documentation for format information.
You will also need to know the user and password to connect to the database.

Note: In case you are using different
database username/password than the default spec/spec you will also have
to edit the dbUser and dbPassword properties in the above
files.

The properties files provided in the $JAS_HOME/config/ directory
contain some URLs provided by some DB vendors.

Now load the database:

$ cd $JAS_HOME$ ant/bin/ant loaddb

Note: To load the database for higher
Injection Rate runs use the bin/loaddb.sh script.

Usage:

loaddb <scale> [<numThreads> <-doFlatFiles>]

Where:

<scale> required,
is the injection rate at which the benchmark will run<numThreads> is optional, if not specified,
the default is 1<-doFlatFiles> is an optional flag, and if
not specified the loader doesn't generate flat files.

When -doFlatFiles option is specified, the loader
will generate the files as follows

${pipeDir}/datafiles/Ir_YYY/tableName/tableName_XXXX

YYY in Ir_YYY represents
the injection rate specifedXXXX in tableName_XXXX represents
a four digit number starting from 0000 and could have as many as the
number of threads specified.pipeDir is a configurable property and by default
is set to /tmp

The remaining steps to deploy the SPECjAppServer.ear
file created in the ${JAS_HOME}/jars/ directory depend on the
Application Server you are using and whether or not you are using extended
DDs provided by them. Each Application Server is unique and it is impossible
here to capture the exact instructions to complete the deployment. Here
are a list of steps that are required to complete the deployment if not
using vendor supplied DDs. Please refer to Application Server documentation
for help in accomplishing these steps:

Database to CMP / CMR field mapping.

This step tells the application server how to map the Java names
used for CMP / CMR fields to the actual column names used in the
Schema's provided.

Provide JNDI name mapping.

Tells the Application Server what JNDI name to use for the reference
beans as well as provides a mapping of reference names to their correct
target bean.

Map JMS Queues and Topics

Like JNDI binding this step maps the JMS queue names to the underlying
nomenclature used by the Application Server for the JMS resources
defined in it.

Running SPECjAppServer2004 requires that the
driver be configured for your environment and particular application
server. Configuring the driver properly requires understanding how the
driver works, which is described below.

The SPECjAppServer2004 Driver consists of several
Java programs and is designed to run on multiple client systems, using
an arbitrary number of JVMs to ensure that the Driver has no inherent
scalability limitations. Note that none of the client systems are part
of the SUT.

The actual applications that implement the workload are defined in
the specifications. These are OrderEntry, PlannedLine and LargeOrderLine.

There is one Agent type for each of the applications. The
Agents are OrdersAgent, MfgAgent and LargeOLAgent.
You can configure as many OrdersAgent and MfgAgent agents as you
wish on any number of client systems. There must be at least one
of each Agent type on the Master machine if OrdersAgent and/or MfgAgent
are to be run.

There is one Controller which runs on the Master machine and
with which all the Agents register.

There is one Driver which runs on the Master machine and which
is responsible for doing a benchmark run.

The Driver communicates with all the Agents
using RMI. The Driver reads the run properties and configures the Agents
appropriately. Each Agent will then run as many threads of their respective
workload. For example, the DealerAgent will run DealerEntry. The number
of threads is determined by the scaling rules of the specification and
are equally distributed amongst all Agents. Each thread runs independently,
executing the workload according to the rules defined in the spec. When
the run completes, the Driver co-ordinates with the Agents to retrieve
all the stats and prints out the reports.

In the past, some people have had problems with
the way the Driver gets its InitialContext and lookups. This section
explains exactly how this is done.

The Driver obtains the JNDI InititalContext
by executing the following code in each of the client threads in Agent
processes:

try {

context = new InitialContext();

} catch (NamingException ne) {

throw new RemoteException(InitialContext failed. " +
ne);

}

Each of the bean home interfaces that the applications
require is looked up using the absolute home name of the bean. An example
is shown below:

OrderSesHome orderSesHome =

(OrderSesHome) PortableRemoteObject.narrow(

context.lookup("OrderSesHome"), OrderSesHome.class);

If your app server needs extra properties to
be set for the InitialContext, this can be done by setting them in jndi.properties in
the CLASSPATH or changing the variable JAVA in config/<appserver>.env.
The example below shows a generic implementation. Specific values for
several application servers are provided in the subsequent sections.

This step assumes you have already completed
the steps in 3.1.2.1. Make sure that the config directory on the driving
system is synchronized with the updates made earlier. In addition
to those changes you must also:

Edit $JAS_HOME/config/run.properties and set the SPECjAppServer2004
run parameters appropriately. The parameters are described in comments
in this file. Ensure that dumpStats is 0 if you are not running
the charting application, as otherwise the Driver will hang waiting
for the reader on the pipe. Ensure that researchMode is set
correctly to run either official or EAStress run.

Check that the method used by the Driver to get its InitialContext
and do lookups will work for your application server.

Now, run the driver by running the script bin/driver.sh
[<driver host>] from the $JAS_HOME directory.

You can abort a run at any time by pressing Ctrl-C.
This will cause all the processes to exit gracefully.

If there were errors during pervious run, M_largeorder
table may have entries which will affect the "SPECjAppServer2004 JOPS" of
the run. So make sure before each run the M_largeorder table is empty
by reloading the database or by deleting entries from M_largeorder table
using a SQL command.

We have sometimes experienced binding exceptions
at the start of the first time run. For now, just cancel the script (by
pressing Ctrl-C) and rerun it again. This problem usually does
not persist to the second run.

The driver.sh script starts all the Agents
on the Master machine. If you want to start the Agents on different client
systems, the script needs to be edited appropriately. "bin/driver.sh <driver
host>" must be started on the other client systems after starting
the script on the Master machine. <driver host> is the name
of the Master machine. The properties file must be configured correctly
on the client systems, with the appropriate configurations in the $JAS_HOME/config/run.properties for
the OrderAgent and the MfgAgent to be run on the client systems. In the run.properties file
the runOrders= and runMfg= properties actually specify the number of
Orders / Manufacturing agents to run on the target system.

A driver.bat script is provided for running
SPECjAppServer2004 on MS Windows systems.

After completing a run, the SPECjAppServer2004
driver will generate a result.props file in the output directory
containing the raw results from the benchmark. The contents of result.props are
used in the Submission File from which the SPECjAppServer2004 report
is generated.

SPECjAppServer2004 includes a utility for generating
a results page called the reporter. The report contains a summary of
the key results and a complete system description in a format designed
to be consistent with other SPEC reporting pages.

To run the reporter, use the following command:

$ java -classpath reporter.jar
reporter [-a] <submission file>.txt

Where <submission file>.txt is
the Submission File created in the previous step.

An HTML file named <filename>.report.html is
created by default.

The "-a" option will create a text report page
named <filename>.report.txt.

Every submission goes through a minimum two-week
review process, starting on a scheduled SPEC OSG Java sub-committee conference
call. During the review, members of the committee may ask for additional
information or clarification of the submission. Once the result has been
reviewed and accepted by the committee, it is displayed on the SPEC web
site at http://www.spec.org/.