DirectX 10 Gaming Performance Review - High End GPUs

Testing Methodology and System Setup

Testing Methodology

Graphics card testing has
become the most hotly debated issue in the hardware enthusiast
community recently. Because of that, testing graphics cards has become
a much more complicated process than it once was. Before you might
have been able to rely on the output of a few synthetic, automatic
benchmarks to make your video card purchase, that is just no longer the
case. Video cards now cost up to $500 and we want to make sure that we
are giving the reader as much information as we can to aid you in your
purchasing decision. We know we can't run every game or find every bug
and error, but we try to do what we can to aid you, our reader, and the
community as a whole.

With that in mind, all the
benchmarks that you will see in this review are from games that we
bought off the shelves just like you. Of these games, there are two
different styles of benchmarks that need to be described.

The first is the
"timedemo-style" of benchmark. Many of you may be familiar with this
style from games like Quake III; a "demo" is recorded in the game and a
set number of frames are saved in a file for playback. When playing
back the demo, the game engine then renders the frames as quickly as
possible, which is why you will often see the "timedemo-style" of
benchmarks playing back the game much more quickly than you would ever
play the game. In our benchmarks, the FarCry tests were done in this
matter: we recorded four custom demos and then played them back on each
card at each different resolution and quality setting. Why does this
matter? Because in these tests where timedemos are used, the line
graphs that show the frame rate at each second, each card may not end at the same time precisely
because one card is able to play it back faster than the other -- less
time passes and thus the FRAPs application gets slightly fewer frame
rates to plot. However, the peaks and valleys and overall performance
of each card is still maintained and we can make a judged comparison of
the frame rates and performance.

The second type of benchmark
you'll see in this article are manual run throughs of a portion of a
game. This is where we sit at the game with a mouse in one hand, a
keyboard under the other, and play the game to get a benchmark score.
This benchmark method makes the graphs and data easy to read, but adds
another level of difficulty to the reviewer -- making the manual run
throughs repeatable and accurate. I think we've accomplished this by
choosing a section of each game that provides us with a clear cut path.
We take three readings of each card and setting, average the scores,
and present those to you. While this means the benchmarks are not
exact to the most minute detail, they are damn close and practicing
with this method for many days has made it clear to me that while this
method is time consuming, it is definitely a viable option for games
without timedemo support.

The second graph is a bar
graph that tells you the average framerate, the maximum framerate, and
the minimum framerate. The minimum and average are important numbers
here as we want the minimum to be high enough to not affect our gaming
experience. While it will be the decision of each individual gamer
what is the lowest they will allow, comparing the Min FPS to the line
graph and seeing how often this minimum occurs, should give you a good
idea of what your gaming experience will be like with this game, and
that video card on that resolution.

Our tests are completely based around the second type of benchmark method mentioned above -- the manual run through.

System Setup and Benchmarks

Since we are testing DX10 titles, obviously running Windows Vista is a requirement. Luckily (or rather, by design) we've been using Vista x64 for our graphics card testing for quite some time now, so I am very familiar with the OS and any quirks / issues it might have. Our test system is pretty standard as it has been for a while: a 680i motherboard for the NVIDIA graphics cards and an Intel 975X motherboard for the AMD graphics cards.

For our first wave of DX10 testing, I'll be focusing on the high-end graphics card from each vendor. I have tested the NVIDIA 8800 GTX, 8800 GTS 640MB and the AMD HD 2900 XT cards in both single and multi-GPU modes. I also decided to stick to the most popular resolutions for gaming, 1600x1200 for standard ratio monitors and 1920x1200 for widescreen monitors.

I am using the latest AMD and NVIDIA drivers - 163.69 Detonators and the 7.9 Catalyst drivers with the latest Hotfix installed.

Our DX10 titles have already been discussed, but there was another one we almost included: PTBoats. You may have not heard of it: it is a synthetic benchmark that was put out by a company known as Akella. A graphical simulation of a WWII battleship fight, it is fairly impressive to look at, but unfortunately wouldn't run on the AMD cards over a few frames per second. We'll wait for revisions from all sides before including it in our tests.