Monday, October 28, 2013

Windows 8.1 Exhibits Gaming Performance Deficit

Microsoft just released Windows 8.1, and with that I subjected the latest version of the operating system to some gaming performance testing. The results illustrate a small decline in the performance of Windows when moving from 8.0 to 8.1.

System as Tested

Asus z87-Pro Motherboard

Intel i7-4770k @ 4.7Ghz/4.4Ghz Uncore

Corsair Dominator Pro 16GB (8x2) @ 2133, 10-11-11-27

Nvidia 670 2GB @ 995/3204

ForceWare 320.49 on both 8.0 and 8.1

Sound Blaster ZxR Rendering in 5.1

"Upgrade Install" of 8.1 from 8.0. 8.0 was installed fresh.

All drivers the same

App/Service footprint nearly identical

Testing Methodology

For the last 13 years or so, I've used a benchmarking script to test
gaming performance. This script performs automated testing and logs
results from each game to a central file. The script runs each game n
(i.e. as many as I would like) times and then generates a
median (rather than average). I have added games to this
periodically since its inception.

I say all this
because you'll notice there are a few old games listed. I have found
including these results useful to judge CPU/Memory performance and
identify bottlenecks specific to different APIs.

For these tests, each game was run at least three times (most more) for each set of settings. There are 4 tested settings that apply to most games: 4xAA/16xAnisotropic filtering, 0x/0x, and all combinations thereof. Normally these results would be split, but for the purposes of this platform comparison they have been combined. Bioshock Infinite does not support any AA, so that is not affected by the AA setting. All tests are performed at 1600x1200 which maintains compatibility with the older titles and while being a less than ten percent deviation from the 1080p effective pixel count.

For the first set of results there were over 250 test runs. These runs have been compared in total between Windows 8.0 and Windows 8.1 on a per game basis to generate a % performance change.

Games tested as follows: (in order of age, oldest->newest)

Quake 3 (1999, OpenGL)

Comanche 4 (2001, D3D8)

Doom 3 (2004 OpenGL)

Serious Sam 2 (2005 OpenGL)

Company of Heroes (2006 DX10)

Crysis (2007 DX10 Very High Settings)

Dirt 2 Ultra (2009 DX11Ultra Settings)

Alien vs. Predator (2010 DX11)

Bioshock Infinite Ultra (2013 DX11 Ultra Settings)

Unigine Valley (2013 DX11)

Results

As illustrated in the bar chart below, the results indicate that Windows 8.1 lags in performance when compared to Windows 8.0 original release.

Game; All Scores Median of Multiple Runs

8.0 4xAA
16xAni

8.1 4xAA
16xAni

% Diff

8.0 0xAA
0xAni

8.1 0xAA
0xAni

% Diff

8.0 4xAA
0xAni

8.1 4xAA
0xAni

% Diff

8.0 0xAA
16xAni

8.1 0xAA
16xAni

% Diff

Quake3

904.4

904.73

0.04

912.37

910.97

-0.15

903.43

903.8

0.04

911.1

908.6

-0.27

Comanche4

198.47

198.5

0.02

198.53

198.97

0.22

198.33

199.87

0.78

199.5

198.93

-0.29

Serious Sam 2

486.7

469.6

-3.51

549.3

536.2

-2.38

521.5

507.5

-2.68

547.9

538.7

-1.68

Doom3

338

334.7

-0.98

473.2

472.1

-0.23

339.8

337.7

-0.62

468.5

472.7

0.90

Company of Heroes_DX10

261.55

246.5

-5.75

290.25

268.05

-7.65

267.3

248.7

-6.96

285.15

264.05

-7.40

Crysis_Island_DX10

63.06

62.12

-1.49

81.16

79.59

-1.93

66.03

64.73

-1.97

77.82

76.32

-1.93

Dirt 2 DX11_Ultra

129.46

127.62

-1.42

167.47

165.05

-1.45

142.65

139.97

-1.88

155.99

154.13

-1.19

Alien vs. Predator DX11

63

62.2

-1.27

110.3

108.6

-1.54

65.4

64.6

-1.22

104.6

103.3

-1.24

Bioshock Infinite Ultra_DX11

100.4

99.29

-1.11

99.73

99.11

-0.62

99.53

97.93

-1.61

100.2

98.48

-1.72

Unigine_Valley

58.63

58.28

-0.60

79.75

79.67

-0.10

60.13

59.08

-1.75

78.25

77.6

-0.83

While most percentages are relatively low, the trend appears consistent and the high sample size lends to the reliability of the data. I find this a bit troubling since there are no obvious reasons performance degradation should be introduced when looking at the 8.1 changes. Note there is still work to do; I'd like to get additional opinions including ATI results.

Now let's take a look at Futuremark 3dMark Advanced Edition. These tests were run through a couple times each and averaged. They were performed on default settings. (1080p/720p/720p)

3DMark Test

win 8.0

win 8.1

% Diff

Fire Strike

6560

6502

-0.88

Cloud Gate

25138

24331

-3.21

Ice Storm

179351

176476

-1.60

Again, consistent downturns in performance.

This is where I was going to publish this article, but as my finger hovered over the "Publish" button I had one lingering concern: What if I'm wrong? What if I missed a setting or something in the upgrade that might have this impact?

So Then We Did it Over Again

I wouldn't feel right publishing this article without doing a clean install and absolute parity of all operating systems involved. For the sake of being complete, I included a Windows 7 install as well. What I found stunned me... 8.1 isn't the start of a downturn, it's a continuation. On a smaller subset of tests I performed the same amount of runs. (average of 14 runs per title) The operating system/drivers were as follows:

The "Fire Strike" test gives us the only real hope of progress in Windows 8, but Windows 8.1 gives that gain right back. Overall, more troubling results.

This, coupled with the lingering mouse and input issues make for a bit of a shaky upgrade to 8.1 from a gaming perspective. As I outlined in my previous article, I believe it's time for Microsoft to step up their game (hah!) in light of new market pressures. This trend is unfortunate since there are a lot of minor changes in 8.1 that are a step in the right direction. (UI, etc.) Let's hope they iron out this performance issue quickly.

Update (11/4/2013): It looks as if Microsoft has acknowledged the mouse issue which I expect means a fix would be coming. Let's hope performance and other issues are addressed as well.

Thanks Nathan! I've removed your first comment because I want to make sure your e-mail is protected should any spammers comb the site. :) I appreciate your kind words. Here is a link on how to follow Blogger blogs. As for blocking a gaming site on the router the easiest way would probably be to redirect the desired URL to a bad address with dnsmasq, but that's a discussion for a different post. :)

I reckon it comes down to hardware and settings and what you add or use with the game in terms of thirdparty antialiasing and mods

For me there is no noticeable difference on my system and I run dxtory 24/7 so im aware of how games run on my comp...

In the end Windows 8/8.1 is so similar to Windows 7, the general public would disagree, but if I got a Windows 7 user to use my comp they would not be able to tell the difference unless they checked System Properties.. basically killed metro and charms and all that "crap" they wanted to add to try and make Windows Accessible.