Disk Performance Testing with Jetstress 2007

Introduction

Although I/O throughput requirements for Exchange Server 2007 have significantly dropped (more than 70%) when compared with previous versions, the truth is that the disk subsystem is still a critical component of any Exchange solution.

The newest version of Jetstress supports Exchange 2000, 2003 and 2007. No matter which Exchange version you’re using, Jetstress will help verify the performance and stability of the disk subsystem before putting an Exchange server in a production environment. Jetstress works with the Exchange Server Database Storage Engine to simulate the Exchange database and log disk I/O load.

There are two categories of test scenarios: disk subsystem throughput and the Exchange mailbox profile.

In the disk subsystem throughput test scenario, you can perform the following types of tests:

Performance of database transactions (the performance test becomes a stress test when its duration is longer than six hours)

Streaming backup

Soft recovery

In the Exchange mailbox profile test scenario, you can specify the number of mailbox users and I/O per seconds per mailbox to simulate the profiled Exchange mailbox load.

Although Jetstress produces a summary report, it is recommended that you use System Monitor and Event Viewer in conjunction with Jetstress to validate that the performance criteria is met or exceeded and that no errors are logged.

First Impressions

The first things I noticed from this new version is that the tool now has a help file (Microsoft Exchange Server Jetstress.chm) that replaced the Word document with instructions that came with previous versions, and that the look and feel is now similar to the rest of the Exchange Analyzers. But of course there are some other improvements:

Simplified configuration - The GUI version of Jetstress, JetstressWin.exe, has simplified the process of configuring a test. More advanced parameters can be used, by editing the XML configuration file.

Test Pass/Fail criteria - the pass/fail criteria now depends of what type of architecture you’re using: 32 or 64 bit.

Memory requirements - Jetstress limits the number of storage groups based on the physical memory of the system. In order to run Exchange 2007 tests you’ll need at least 1 GB plus 256 MB for each storage group. Jetstress does not require that much physical memory for the Exchange 2000 Server and Exchange Server 2003 Database Engine libraries.

Disk I/O ratios and patterns - Jetstress is designed to match the Exchange 2007 server disk I/O ratios and burst patterns. This is approximately a 1:1 Database Read: Write ratio and a .5-1:1 Log Write: DB Write ratio.

Maximum/Minimum Cache Size - Jetstress sets the maximum database cache size to be 256 MB multiplied by the number of storage groups. The minimum database cache size is set to be 32 MB multiplied by the number of storage groups.

Throttled Load Generation - Jetstress artificially throttles the amount of disk I/O that a thread generates in order to support the fine grained control on the overall I/O throughput and to accommodate a case of 50 storage groups where a few threads could lead to overly large changes in I/O per seconds.

Suppress Tuning and Use Thread Count - Jetstress may frequently fail the automatic tuning if you decide to host many storage groups on the same disk drive, so now there’s an option to suppress tuning

Selecting Time Range of Performance Log - Jetstress saves a performance log for the whole time range of a test run. But the test report accounts for a valid time range of performance samples from a point where it has reached 90% of the maximum cache size.

Installing Jetstress

Preferably, Jetstress testing should be performed before you install Exchange on the server. There are some well-known risks associated with running Jetstress on a machine with Exchange installed. If you use a version of Jet (ESE.DLL and ESEPERF.*) different than the version installed with Exchange, the registration of the Jet database counters in the Jetstress install directory will break the database counters for Exchange after Jetstress is removed. You may have to uninstall the existing Exchange Database Engine counters before you run Jetstress by running unlodctr ESE at a command prompt. However, you will still have to uninstall and reinstall the Exchange version of the Exchange Database Engine counters when testing is completed

To install Jetstress follow these steps:

Ensure that installation requirements are met:

Jetstress 2007 requires that the server that is running it has .NET Framework Version 2.0 installed.

Jetstress is supported only on Windows 2000 Server and Windows Server 2003 (32 or 64 bit).

Configure the storage subsystem:

Although not really a prerequisite, consider following these recommendations: use Windows Hardware Quality Labs (WHQL) certified hardware. If the server is a cluster, the whole server/storage configuration must be Cluster Certified.

Verify that all the storage components have the latest firmware.

Format the LUNs within Windows with NTFS file system (default allocation size) and consider aligning the disks with diskpart (or information about how to run diskpart, read Disk Geometry).

Verify that the HBA/SAN specific configuration is set correctly. Many HBAs use registry keys to customize the configuration to a specific SAN platform (for example, Queue Depth).

Download Jetstress and start the installer by double-clicking the Jetstress.msi file. Follow the prompts to install the tool on your computer.

Copy the following ESE database modules to the directory where you installed Jetstress.

ESE.DLL

ESEPERF.DLL

ESEPERF.INI

ESEPERF.HXX

If you forget to copy these files, you’ll get the following warning when you try to run the tool:

Figure 1: Validating required files

Running the tool will trigger some validation tests, such as the one mentioned in the previous step. Checking if the LogicalDisk counters are enabled is another test that is run and that will be automatically fixed.

Figure 2: Validating performance counters

Running the Test

Before running the test, there are some settings you should consider and that will have impact on your system’s performance:

Number of storage groups and databases

Database size

Mailbox size limit

User profile (estimated IOPS per mailbox)

Data replication

The way that each of these factors influence performance is not in the scope of this article, but if you want to know more about this subject, there is some detailed information in the Jetstress 2007 help file.

To start Jetstress simply click the shortcut available at the Start Menu or start JetstressWin.exe. On the Open Configuration page, you can either create a new test configuration or open an existing configuration file.

Figure 3: Test configuration

There are two test scenario categories listed here that you can select from the Define Test Scenario page:

Test a disk subsystem throughput

Test an Exchange mailbox profile

In this example we’ll use the first option, test a disk subsystem throughput.

Figure 4: Define test scenario

The next screen is where you select the kind of test you want to perform: Performance, Stress or Streaming Backup.

Performance - Generates the Exchange type of I/O by accessing a database that has Jet transactions.

Streaming backup - Measures the performance of a backup solution. Be aware that Jetstress can perform a streaming backup only to a device that can be mounted with a drive letter.

Soft recovery - Measures the log replay rate.

Figure 5: Test type

Since we’re performing a disk subsystem throughput test, the Select Capacity and Throughput page is available. There are options to size the test databases using the percentage of the maximum storage capacity, and target I/O throughput (IOPS) by the percentage of the maximum throughput capacity of the disk subsystem. Jetstress reserves 25% of the initial database file size for its future growth during test runs. You can also suppress tuning and use thread count, if the automatic tuning frequently fails.

Figure 6: Capacity and throughput

The Define Test Run page has the following options:

Output path for test results enables you to specify the directory where Jetstress will save the performance logs and test reports.

Test duration (hours) enables you to specify the period for performance sample gathering. The performance test becomes a stress test when the duration is longer than six hours so that it uses a lenient set of the disk I/O latency thresholds.

Figure 7: Test duration

As I wrote before, Jetstress limits the number of storage groups based on the physical memory of the system. You can enter four storage groups if you are using Exchange 200x Database Engine. For Exchange 2007, there’s a requirement of 256MB per storage group plus an additional 1 GB in order to run Jetstress. There is also a check box selection to use network-attached storage (remember that NAS isn’t supported with Exchange 2007).

Figure 8: Define Storage Group

The Select Database Source page has three options:

Create new databases

Restore backup databases

Attach existing databases

Since we are building this test from scratch, we’ll select Create new databases. If we had previously prepared this test, we could just attach existing databases, saving some time.

Figure 9: Database source

The Review & Execute Test page gives you a summary of the test scenario that Jetstress will run. You have the following options available:

Save test - Saves the settings that you have configured to a new or an existing configuration file.

Figure 10: Review & Execute Test

When you hit Execute test, you get the Test in Progress page, where you have the option to cancel the test and exit from the application.

Figure 11: Test in progress

Analyzing Results

After the test is completed, the performance data is analyzed and reported in a summary report. Results will be saved to Performance_(DateTime).html file. All the performance counters collected will be gathered in a counter log file named Performance_(DateTime).blg that you can use for some more advanced analysis.

Consider the following guidelines when examining the data collected.

Performance Counter Instance

Guidelines for Performance Test

Guidelines for Stress Test

Database Avg. Disk sec/Read

The average value should be less than 20 ms (.20) and the maximum values should be less than 50 ms.

The maximum value should be less than 100 ms.

Log Avg. Disk sec/Write

Log disk writes are sequential, so average write latencies should be less than 10 ms, with a maximum of no more than 50 ms.

The maximum value should be no more than 100 ms.

%Processor Time

Average should be less than 80% and the maximum should be less than 90%.

Same as for Performance test.

Available Mbytes (32-bit Windows Server)

Minimum should be no less than 50 MB.

Same as for Performance test.

Free System Page Table Entries (32-bit Windows Server)

Minimum should be no less than 5000.

Same as for Performance test.

Transition Pages Repurposed/sec (Windows Server 2003)

Average should be less than 100.

Same as for Performance test.

Pages/sec (Windows 2000 Server)

Average should be less than 100.

Same as for Performance test.

Pool Nonpaged Bytes (32-bit Windows Server)

Maximum should be less than 75 MB.

Same as for Performance test.

Pool Pages Byes (32-bit Windows Server)

Maximum should be less than 180 MB.

Same as for Performance test.

Database Page Fault Stalls/sec.

Maximum should be less than 1.0.

Same as for Performance test.

Table 1: Guidelines for examining Jetstress 2007 analysis reports

If you open Performance_(DateTime).html you’ll see a nice HTML report with some tables with performance analysis, like the following example:

Table 2: Disk subsystem performance summary

How to interpret the summary report:

The Test Summary legend summarizes the test run.

The Test Issues legend indicates whether the test passed or failed. It also indicates any performance sample averages and spikes that have gone out of thresholds.

The Jetstress System Parameters legend explains the parameters that relate to the Exchange Database Engine.

The Disk Subsystem Performance and Host System Performance summarizes the performance samples and displays errors.

Since I made this test using a virtual machine on my laptop, the test just failed. That was expected, since I don’t have enough performance to run Exchange in a production environment.

Conclusion

Jetstress is a valuable tool, but you must fully understand the results produced in order for it to become an effective help. A good knowledge of Exchange storage requirements is necessary, so that you won’t have bad surprises later (see Related Links further ahead).

Also don’t forget to carefully read the help file that comes with the tool and where you’ll find much more deep technical stuff.

A final word of appreciation for Hyungmin Lee from Microsoft, who helped me out with some issues I had during the tests. Read his blog, you’ll find valuable information about Jetstress and some other great tools.