The recent revelation that a slightly-too-low Metacritic score prevented …

When we game critics get together and talk at our clandestine meetings underneath our secret mountain base, we often debate whether our reviews actually have a tangible impact on how many people buy a game. Some will point to low-selling critical darlings like Psychonauts and healthy sales for crappy licensed games as proof that the public doesn't, as a whole, listen to reviewers all that much. Those on the other side will point to strong reviews that helped propel games like Super Meat Boy to sales success, or critical drubbings that seem to have hurt sales for games like Too Human.

Regardless of how much the gamers themselves seem to care, many publishers seem to give outsized attention to review scores, going so far as to use them in determining bonuses for developers. It's a practice that puts too much focus on those scores as a final arbiter of a game's quality, and one that really needs to stop.

Obsidian's near miss

Linking a developer's bonus to a game's Metacritic score is nothing new—the practice has been widely reported in and around the industry for years. But the issue constantly pops back into the public attention as new instances of the practice come to light.

Today, the focus of that attention is developer Obsidian Entertainment, whose creative director and chief creative officer Chris Avellone recently tweeted that the developer didn't receive a bonus for 2010's Fallout: New Vegas from publisher Bethesda because the game didn't reach a target Metacritic score of 85. The actual Metacritic average for the game currently stands at 84 (that's for the PC and Xbox 360 versions; the PS3 version sits at a slightly worse 82).

Improper tool use

By tying a bonus to a specific Metacritic score, publishers seem to be saying that they think positive reviews will have a direct, positive impact on their bottom line through increased sales. To be fair, there is some evidence that review scores do broadly have an effect on game sales, both in a laboratory setting and in real-world examples.

But it's hardly a direct, repeatable relationship, as proven by the numerous situations where review scores and sales levels fail to line up. A game with an 85 average on Metacritic is definitely not guaranteed to sell better than a game that scores an 84, even if things like genre, franchise popularity, and marketing budget were all somehow set exactly equally.

You could argue that review scores are the best way to judge a developer's output independently of other factors that might influence sales, but that's not always the case. Any critic will tell you that being forced to distill the complex and involved experience of playing a game into a single number is often a fool's errand, and that the difference between a game that gets a 7 out of 10 and one that get an 8 is often incredibly slight. Aggregating these scores into a single weighted average, as Metacritic does, just adds another layer of abstraction from the critic's actual determination.

More broadly speaking, though, focusing on a single number can obscure a success that's sitting right in front of one's nose. Out of 81 reviews for the Xbox 360 version of Fallout: New Vegas listed on Metacritic, 72 are characterized as "positive," with nine said to be "mixed" and zero "negative." The game went on to ship 5 million copies and attract $300 million in revenue in its first month. If that isn't a critical and commercial success worthy of a bonus for the developer, I don't know what is.

To be clear, I don't have a problem with Metacritic itself in this matter. The site is just a tool, and one that can be useful as a quick, rough, heuristic guide to the general critical consensus about a game. It's the publishers that are using this tool improperly as some sort of final, objective arbiter for the quality of games its developers are putting out. While it's tempting to look to a third party source that can be used for this kind of important determination, Metacritic isn't it.

83 Reader Comments

What bothers me most about Metacritic is the user score. I am so sick and tired of people zero-bombing a game for no reason, or for reasons that hardly constitute a fair review.

Presently the biggest offender are user reviews for Mass Effect 3. I understand the controversy behind the ending and the day-one DLC, but are these issues that justify blatant ZERO scores? A bad ending is enough to inspire a negative review, but is it too much to ask to consider the merits of the rest of the game before nonchalantly giving the game a big fat ZERO? And the day-one DLC? That's not even related to the core gameplay experience. That's like giving a Call of Duty game a zero for a crappy level on a post-release map pack.

If you must write a user review, then kindly make it a REVIEW, or at least an attempt at one. Look at the game as a whole experience. Don't just zero-bomb the game because you're all butthurt over a single issue, and certainly don't do so over a chunk of controversial DLC that you don't even have to buy if you don't want to.

I've been saying this for a long time.... Metacritic is bad because Metacritic is lazy.

It should eliminate those scores with very high standard deviations. For example, if most people gave a game a 7, but a bunch gave it 0's, then it's much safer to say the game really was a 7. Not a 5.

2)Metrics are not always linear. 2 stars doesn't mean it's twice as good as a 1 star. And a 5 star isn't necessarily 25% better than a 4 star. It'd be nice to get everyone on the same page, but good luck with that. These reviewers should be able to log in to Metacritic and WEIGHT their metric.

3)Positive/Neutral/Negative should be factored in as well. For example, ZERO Negatives for FONV? I'm surprised. I loved the game, but you'd think there would be one person who thought it was bad.

This is why Rotten Tomatoes uses the Tomatometer. A fanboy might have loved it, but a neutral person may have thought it was "good." But those should both count equally as positives. Though this is also why RT gives you the "Average Score" in addition to the Tomatometer.

A hybrid of both systems is ideal, and it's not too hard to come up with a formula that will fix these problems... but I'm pretty sure the people at Metacritic put far less thought into their site than even I have.

If you go by sales numbers, Beyond Good & Evil and Psychonauts get spanked, while "Madden: Same Game New Box Edition" makes its devs rich. People playing makes Zynga's IP theft look like the way to go.

Well, since investors don't give a rats ass about review scores if the revenue sucks, bonuses *should* be paid based on net profit.

The article was "Why linking developer bonuses to Metacritic scores should come to an end". My arguement was how the current metric gives gamers better games. As this is a tech site and not an investor site I think it still stands. If we were on Forbes or Rich Investor Guy, I would argue for a straight profit metric.

Quote:

The movie industry's way of dealing with this issue isn't perfect, but it seems to work. You get paid more if you make lots of money by making popular movies like the Transformer series. If you want artistic respect from your peers, you make films that win awards at Sundance/Cannes.

Er.. Hollywood is famous for forcing people involved in a movie to accept part of the net, as part of their payment for their work. Hollywood then goes on to say that no movies actually make a net. So people get nothing. Very few people involved with a movie get part of the gross.Sometimes a big dog like Stan Lee sues and wins but most people just get hosed. "Hosed" is not a good/working system.

When your developers love the sound of deadlines whooshing by as much as they do at Obsidian

That's actually pretty funny.

This leads back to why an 84 gets no bonus but an 85 doesn't. The reason? 85 is the number specified in the contract.[....]Relating to this, a 100-point scale is wasted on gamers and game journalists — I don't think most people trust buying anything beneath a 70, but you still get a big dropoff at the 80 mark (less so at the 90 mark). When dealing with poor quality, does a 1-69 point variance really matter?

I'm with you, and that's exactly the point. Both parties signed a contract that said if they hit 85, they would get a bonus. Obsidian knew exactly what metacritic was, warts and all, when it signed the agreement. Whining about it to the public when they didn't hit the agreed upon contract number tells me that either A.) Someone didn't understand their own contract or B.) They're whining about the terms of a legally fulfilled contract after the fact, which is just unprofessional as hell. Either way it makes them look amateurish, and like they don't understand their own business (and which is probably why the business guys at Obsidian are currently explaining to Chris Avellone why you don't complain about your business on your personal social networking accounts).

Plus, the whole 'you can't apply a number to this' concept has been beaten to death and back repeatedly for the last 200 years. You can't apply a 1-100 score to a computer game review, but its pretty much universally acceptable to encapsulate an entire semesters worth of college education in a 1-100 score? Chris and team should have worn a shorter skirst and sat at the front of the class if they're expecting to skirt by with the 'just need one more point for an A' argument.

Here's a novel idea... instead of useless executives pocketing most of the profits and offering the real workers measely little "bonuses" tied to rigged performance metrics, companies should use a simple egalitarian formula:

Kyle Orland / Kyle is the Senior Gaming Editor at Ars Technica, specializing in video game hardware and software. He has journalism and computer science degrees from University of Maryland. He is based in Pittsburgh, PA.