People working in the
Assay Office during the California Gold Rush were obliged by law to
wash and dry the scale, avoid breezes by making sure the windows were
closed and verify the scale's accuracy with an object of known weight
before each use. Today, McDonald's Corp.'s restaurants have similarly
stringent rules for weighing precooked Quarter Pounders when government
inspectors descend on one of its fast food outlets.

In our labs, we're just as careful about weighing things. Instead of
gold dust and hamburgers, we balance network hardware and software on
the scales. We have standards for creating a level playing field when
testing products head to head.

When we test with Windows, we make sure background processes, disk
utilization and file position, network configurations and even monitor
resolutions are the same across all our tests.

Services running in the background are a big factor in configuring a
fair contest environment. Take Microsoft's Index Server process
(cisvc.exe), which prepares Web-accessible indexes for use by a search
engine. Because its periodic background examination of files can be
momentarily disk- and CPU-intensive, Index Server can profoundly
influence the performance of a product we're trying to test.

A protocol analyzer trace confirms each test occurs in as quiet an
environment as possible.

When we're testing
multiple products on the same computer, we reinstall and reconfigure
Windows for each product. Just uninstalling one product before
installing the next isn't enough. Most uninstall procedures leave at
least some residue in the computer in the form of obsolete hidden
files, registry entries or system configuration changes. The residue
might cause the next product's files to install in different locations
on the disk, which can sometimes affect performance. Moreover, if a
product sets up a new Windows service or replaces some of Windows'
system files, the uninstall procedure often leaves these components on
the computer.

If we run tests on multiple computers, we use duplicate machines for
data storage, business logic and presentation logic or software agents.

If one product requires a monitor resolution of at least 800 by 600 and
another requires at least 1,024 by 768, we configure Windows to use the
higher resolution for all tests. Video drivers can consume considerable
CPU time, especially at higher resolutions.

Creating a fair testing environment on Unix or Linux presents basically
the same challenges as on Windows, but the terminology is different.
For instance, we remove entries from .RC files that start unnecessary
background processes (such as daemons) and then reboot.

Following these guidelines can help ensure your testing is always fair
and accurate.