Metacritic Scores vs. Sales: 2009 Holiday Season Analysis

This post has not been edited by the GamesBeat staff. Opinions by GamesBeat community writers do not necessarily reflect those of the staff.

Recently in GamePro, Julian Murdoch examined the reviews aggregator site Metacritic. His story opens with a rather startling observation from Activision Vice President of Marketing Robin Kaminsky at the 2008 Design, Innovate, Communicate, and Entertain (DICE) Summit. The opening slide to her presentation read: "For every additional five points over an 80 percent average review score, sales may as much double."

Incredible! If true, game sales must increase exponentially as they score above an aggregate of 80 on the website. Surely, an employee of Activision — let alone a VP of marketing — must have solid data to back up such a bold declaration.

But I'm the skeptical type, so I did a little of my own research. I’m here to tell you that Kaminsky's statement is a load of horse shit, and that publishers' insistence on a correlation between scores and sales is unfounded.

Murdoch’s article, which is well worth reading, focuses not on Metacritic’s influence (or lack thereof) on sales, but on the connection that publishers make between scores and sales. In other words, he investigates the shady practice of tying developer compensation to review scores.

My goal, however, is to see if any meaningful correlation between Metacritic scores and unit sales exists at all. When I previously studied this issue, I concluded that the former have virtually no influence on the latter. Critically acclaimed games, like the Wii’s Boom Blox: Bash Party (which received a Metacritic score of 86), sold poorly, while much lesser-scoring games, such as Night at the Museum or Terminator: Salvation, sold significantly more.

My sample size for that examination was small — I used May 2009 releases and 10 weeks of their sales data. To investigate this matter further, I decided to collect a much larger, more representative sample. My analysis for this article includes everything that came out between October 1 and December 31, 2009.

For each title, I collected 10 weeks of sales data as it was posted on VGChartz. Any releases that did not have the full 10 weeks and a Metacritic score were not included. Out of 364 titles (including all system ports between the Nintendo DS, Nintendo Wii, PlayStation 2, PlayStation 3, PlayStation Portable, Xbox 360), 218 made the cut. PC, iPhone, and digital releases were not included because of the lack of sales data.

Read on to page two for the breakdown.

First, I’ll analyze the data in the exact same fashion I did back in September. I combined cross-platform game sales and averaged Metacritic scores between platforms to arrive at one score for all versions of a title.

Click for a larger view

We have a few clear outliers — Modern Warfare 2, New Super Mario Bros. Wii, and Assassin’s Creed 2. These three were the only titles to sell more than three million copies (approximately 9.5 million, 5 million, and 3.3 million, respectively). They were also some of the highest scoring games (94, 87, 84, respectively). We can’t really tell much from the data by looking at this chart, so let’s remove the outliers from the analysis.

Click for a larger view

Here we get a slightly clearer picture, but we still have this giant mass of games in the top-left corner of the chart. Like my previous study, this data gives a similar impression — a few games sell well and generally score above 60, while most others sell poorly regardless of their score. Again, no games score extremely low and sell very high.

A few interesting examples, though, are in this chart. Tony Hawk: Ride (PS3, Xbox 360, and Wii) sold about 943,000 copies with an averaged review score of 46. On the other hand, the Xbox 360 exclusive Forza Motorsport 3 sold similarly (about 960,000 units) but scored twice as high — a 92. What’s going on here?

Another comparison that gives me pause — the PSP version of Grand Theft Auto: Chinatown Wars sold more than 120,000 copies with Metacritic score of 90. But Rogue Warrior, which appeared on the PS3 and Xbox 360, sold nearly the same with an aggregate score of 28. Yes, you read that right.

The cluster of games in the top-left corner deserves closer examination. It looks like most of these titles sell less than approximately 500,000 copies, so let’s zoom the chart in there.

Click for a larger view

The data shows no clear trendline. We have a large scatter of points between scores of 40 and 80, with few outliers. Again, I’ll make another comparison to GTA: Chinatown Wars — look at Star Wars: The Clone Wars: Republic Heroes, which sold almost four times as many copies with an averaged score of 45 — fully half that of Chinatown Wars.

But it may not be fair to combine cross-platform data like that and make valid comparisons — handheld and Wii versions can vary greatly from their PS3 and Xbox 360 counterparts. Certainly, that data is interesting, but let's review scores and sales for each system.

Click for a larger view

Again, the full chart doesn’t tell us much, other than that most games are going to sell less than a million copies, regardless of their Metacritic score. We see the same outliers, too. Let’s zoom right in to those that sold 500,000 copies or less.

Click for a larger view

Once again, no clear trends come out of the data. Each system’s data points scatter widely, with no clear correlation apparent between sales and scores. Interesting to note is that PSP games sell especially poorly (LittleBigPlanet is the handheld’s highest selling game with approximately 260,000 copies).

Below are tables with each system's top five games ranked by Metacritic scores (left) and unit sales (right).

PS3

Rank

Total Sales

Score

PS3

Rank

Total Sales

Score

Uncharted 2: Among Thieves

1

1,118,574

96

Modern Warfare 2

1

3,310,903

94

Modern Warfare 2

2

3,310,903

94

Assassin's Creed II

2

1,285,815

91

Assassin's Creed II

3

1,285,815

91

Uncharted 2: Among Thieves

3

1,118,574

96

God of War Collection

4

554,536

91

God of War Collection

4

554,536

91

FIFA Soccer 10

5

280,984

91

Dragon Age: Origins

5

455,755

87

PSP

Rank

Total Sales

Score

PSP

Rank

Total Sales

Score

GTA: Chinatown Wars

1

123,695

90

LittleBigPlanet

1

260,086

87

LittleBigPlanet

2

260,086

87

Assassin's Creed: Bloodlines

2

219,983

63

Half-Minute Hero

3

34,287

84

Star Wars Battlefront: Elite Squadron

3

149,392

63

FIFA Soccer 10

4

97,475

83

GTA: Chinatown Wars

4

123,695

90

Tekken 6

5

70,748

82

Yu-Gi-Oh! 5D's Tag Force 4

5

121,150

70

Wii

Rank

Total Sales

Score

Wii

Rank

Total Sales

Score

New Super Mario Bros. Wii

1

5,042,122

87

New Super Mario Bros. Wii

1

5,042,122

87

DJ Hero

2

240,915

87

Wii Fit Plus

2

1,967,654

80

Pro Evolution Soccer 2010

3

57,190

82

Mario & Sonic at the Olympic Winter Games

3

893,648

72

Wii Fit Plus

4

1,967,654

80

Modern Warfare: Reflex Edition

4

549,603

76

EA Sports Active: More Workouts

5

194,325

80

Just Dance

5

470,787

48

DS

Rank

Total Sales

Score

DS

Rank

Total Sales

Score

The Legend of Zelda: Spirit Tracks

1

788,519

87

The Legend of Zelda: Spirit Tracks

1

788,519

87

Might & Magic: Clash of Heroes

2

40,516

86

Mario & Sonic at the Olympic Winter Games

2

520,568

76

Space Invaders Extreme 2

3

43,989

82

Bakugan: Battle Brawlers

3

501,933

60

LEGO Rock Band

4

107,884

80

Pokémon Mystery Dungeon: Explorers of Sky

4

348,233

53

Mario & Sonic at the Olympic Winter Games

5

520,568

76

Drawn to Life: The Next Chapter

5

309,289

72

X360

Rank

Total Sales

Score

X360

Rank

Total Sales

Score

Modern Warfare 2

1

6,232,345

94

Modern Warfare 2

1

6,232,345

94

Forza Motorsport 3

2

963,155

92

Left 4 Dead 2

2

1,938,107

89

Assassin's Creed II

3

1,936,410

91

Assassin's Creed II

3

1,936,410

91

FIFA Soccer 10

4

384,540

90

Forza Motorsport 3

4

963,155

92

Left 4 Dead 2

5

1,938,107

89

Dragon Age: Origins

5

938,898

86

From this data, what can developers and publishers expect after a game's release?

Based on Metacritic’s own scoring system, I split the data into four categories: universal acclaim (a score of 90 or more), generally favorable reviews (between 89 and 75), mixed or average reviews (between 74 and 50), and generally unfavorable reviews (49 or less). I decided not to use "overwhelming dislike" because no games scored that low in my sample.

Overall, most titles — 58 percent — received mixed reviews. Additionally, most sold less than 100,000 units (62 percent).

Even more depressing, a plurality of games — 36 percent — sold less than 100,000 copies and received mixed reviews. Twenty percent of all games studied sold between 100,000 and 500,000 fell into the same scoring category. Only three percent of all games analyzed sold more than 1 million copies and received Metacritic’s most coveted rating — universal acclaim.

Click for a larger view

Includes sales data for games studied without a Metacritic score. Click for a larger view.

I suppose that Kaminsky gave herself an out by qualifying her statement with “may.” If what she proposes were actually true, we’d expect to see the above scatter charts’ sales data curve upwards exponentially above a Metacritic score of 80. In fact, what we actually see is most games lingering in the top-left corner of the chart (i.e., low sales, moderate scores) with a few runaway titles garnering millions in sales.

My analysis demonstrates that no link between Metacritic scores and sales exists. So why do publishers insist on such a connection? I can’t really say. That reviews scores would influence sales does feel intuitive. Perhaps publishers also feel reassured that they've based their decisions on a quantifiable number.

But I won't pretend that I have all the answers. I don't really know why or why not a particular game sells well or poorly. My data isn't equipped to answer that question, either. What I do know is that no correlation between Metacritic scores and unit sales exists, and I think it's foolish for publishers to tie monetary compensation to such scores.