Gaming & Culture —

Game reviews on Metacritic: why we avoid inclusion

Metacritic can bring readers and more links to your reviews, so why wouldn't …

The world of game reviews is often difficult to navigate. Everyone uses different scores, and a large emphasis is placed on the single score given to games by Metacritic, a review-aggregation site. Metacritic uses a scale of 1 to 100 for reviews, a figure calculated by averaging multiple scores. What comes out after that averaging is seen as something akin to a gold standard for judging the quality of a game. We've been asked numerous times why we're not included in the game rankings given by Metacritic: our reviews aren't linked from the site, and we're not included in the final uber-score. That's by design.

I had the good fortune of being a guest on the GameShark podcast called "Jumping the Shark," and the conversation centered on the games we were playing and the issue of game reviews. One of the details shared on the podcast was rather surprising, especially since it seems to be a common practice.

"THQ pulled this... this was funny," one of the hosts said. Their reviewer received an early copy of the new THQ UFC title early for review. THQ, oddly, gave some rules about when the review could go live. The review could be published early if the Metacritic score was above 85 percent, but if it was under that minimum the review couldn't be released until the game's release. "This is not just THQ; this is very common," I was told. The score failed to hit the magic number, so the review was not released as of the recording of the podcast.

Here's where things get slightly crazy. The score that GameShark gave the game in the review, according to the podcast, was a B+. That, according to Metacritic, does not equal an 85 percent. Take a look at the FAQ from Metacritic: it breaks down how scores given by review outlets are interpreted. Giving a game a B+ may seem like a good score, but Metacritic actually takes that to mean you gave the game an 83. An A- would have been necessary to equal the 85 percent. If you're willing to up your score to what Metacritic considers an 85, then the company is willing to let you release your review earlier.

Metacritic's take on non-numerical scores

Being on Metacritic can increase your readership, but that comes with a number of caveats that frankly don't seem to be worth the trade-off. Going along with this system means a reviewer loses control on a number of levels. You lose control over what your score means, because Metacritic has locked down a numerical score that tries to take non-numerical scoring systems into account. You lose control over when you can release your review, because companies give you ultimatums based on how Metacritic interprets that score. If you take your reviews seriously, both of those restrictions seem to be intolerable.

"We use a consistent A-F conversion scale which has been in place since we launched in 2001," a Metacritic representative told us when we contacted them for clarification. "Unlike the American school letter grade conversion which covers a 42 point range starting at 58 ('F'), we convert letter grades to the full 100 point scale so that the worthless 'F' score is equated to zero (not 58), C is equated to 50, etc."

Metacritic declined to offer any thoughts on the practice of its scores being used to determine the release dates of reviews, asking us instead to contact the publisher in question for clarification.

Our practice of giving games either a "Buy, Rent or Skip" verdict is too vague for Metacritic to arbitrarily take our score and tell its readers what it thinks we meant. Since we're not weighted in the Metacritic score, no one bugs us about what rating we're going to give a game, and we've never been presented a variable embargo based on the score. In other words, not being a part of that scoring system removes a lever that publishers can use to try to change how their games are covered. Based on the number of publishers willing to send us early games to review, they don't appear to mind.

We're very happy to keep the meat of our reviews in the text itself, where it belongs. We don't want someone else to define our score, weigh the value of our feedback, and use our experiences to form a meta-score. We'd much rather speak for ourselves, and only ourselves and believe our current system is the best way to do that.

Be sure to listen to the whole podcast. We dig into the issue of game reviews a little deeper. It was a good, not to mention personally enlightening, discussion.

Latest Ars Video >

War Stories | Ultima Online: The virtual ecology

When creating Ultima Online, Richard Garriott had grand dreams. He and Starr Long planned on implementing a virtual ecology into their massively multiplayer online role-playing game. It was an ambitious system, one that would have cows that graze and predators that eat herbivores. However, once the game went live a small problem had arisen...

War Stories | Ultima Online: The virtual ecology

War Stories | Ultima Online: The virtual ecology

When creating Ultima Online, Richard Garriott had grand dreams. He and Starr Long planned on implementing a virtual ecology into their massively multiplayer online role-playing game. It was an ambitious system, one that would have cows that graze and predators that eat herbivores. However, once the game went live a small problem had arisen...