Coffee Hour Brain Explosion: Content Curation and Discovery

As I put on my morning pot of coffee today, I couldn’t help but get lost in thought about the myriad problems with content curation and discovery in digital game download services. This type of thinking happens on a pretty regular basis for me. Well, not content curation issues specifically, but daydreaming about video games in general. In this week’s column, I’ll share the results of my most recent coffee hour brain explosion with you.

What specifically prompted this brainstorming session was an article that NintendoLife posted yesterday asking whether Nintendo should enact quality control measures on the Wii U and 3DS eShop. The barrier to entry for developing a game for a Nintendo system – indeed, any gaming device – has never been lower, and the natural result is an increase in the number of terrible games being foisted on the masses. The question is – what, if anything, should be done about these games?

Nintendo, like Microsoft and Sony, have technical control on releases. If a game doesn’t compile and pass basic crash testing, it can’t be released. But should they also enact quality controls? Should they limit who can develop for their systems or require that games pass a review board? Is it acceptable to allow a game like The Letter on the shop?

This is actually a really contentious issue, and you’ll find viewpoints all over the spectrum. My personal opinion is a resounding “No.” Nintendo should never reject a game for not meeting some content threshold. The thing is, quality is almost entirely an inherently subjective matter. Any time that a big game is released you’ll find countless gamers decrying this or that review for being “biased” or “not objective enough.” The ideal of an objective game review is a myth, utter madness. A review – a true critique – of a game is an opinion. You may agree or disagree with a reviewer’s opinion but to automatically label a review as “clickbait” or “biased” is ignorant. They aren’t being paid off by some megalomania corporation. They have a different opinion than you, they aren’t biased in the way you’d prefer.

However, that is the core problem with attempting to gate games for your shop based on some quality scale. Quality is subjective. Who gets to judge whether a game gets released? One person? A panel? Either way you sure as hell aren’t going to end up with enough ratings to achieve any sort of statistical significance.

To add another cautionary tale to the pile, Sega of America actually implemented quality controls on the Sega Saturn in the 1990′s. According to Sam Pettus’ Service Games: The Rise and Fall of SEGA, all Saturn games from both internal teams and 3rd party publishers had to pass the “Five Star Games Scale.” If a game did not receive a composite score across all categories of 90 or more out of 100, the game could not be released in America. This policy was a complete and utter disaster, and has been repeatedly blamed for the dearth of quality Saturn games in America. In particular, this scale was accused of being unfair to Japanese games, possible resulting in the lack of a US release for Sakura Taisen, Radiant Silvergun, and other well-regarded titles.

The nasty subjectivity issue rears its head again whenever a quality scale is used as an excuse for passing on games that weren’t safe sales bets. If the answer is not to limit releases, what is it? How can naive consumers be steered away from terrible games?

I actually think that the key to solving this “problem” is to look at another major issue facing these digital game stores, how to surface the good content. Some would argue that there are few enough games being released on their platforms that Nintendo can’t afford to turn away developers. Whether or not that is true the number of games is starting to add up, particularly on the 3DS. It can be hard to find the best games, and older games might be permanently buried beneath newer content – possibly limiting the lifetime sales of a game. The problem is only compounded on platforms like Steam, where dozens to hundreds of games are released every week. Perhaps more pressing than how to steer shoppers away from bad games is how to steer shoppers towards the good ones.

Rather than stricter content curation tactics, what we need are better content discovery mechanisms.

Thomas Whitehead, the author of the NintendoLife piece that inspired this discussion, came to the same conclusion. His suggestions included implementing a tiered system separating, for example, established developers from newcomers. This isn’t a bad idea at all. By default, the Wii U and 3DS eShops need better organization and categorization methods.

However, I would take the process of content filtering a step further and embrace that idea of subjectivity. Rather than seeking a method of steering a gamer away from a panned game, I think the more important question is, should they be steered away from that game? What if they’d actually enjoy that game? We frequently tease Antonio for liking games that the rest of us wouldn’t touch with a ten-foot pole. Should he be warned away from those games? If he had a great time with an “awful game” why not recommend similar games to him regardless of if they are universally panned or beloved?

All of these digital storefronts face the same challenges of surfacing the good games and repressing the bad ones, and I think that the way forward is to take advantage of personalization to show users the games that they might enjoy, regardless of what the “universal” opinion is of their quality.

At this point, many of you are likely thinking, “Yes, of course, but isn’t that going to be nearly impossible?” You’re not wrong – delivering accurate, customized recommendations for every user is a challenging task to pull off, but fortunately we’re not going in blind. There are a number of techniques, collectively known as “recommender systems,” to perform this very task. By this point, you’ve almost certainly interacted with a number of these things. Ever noticed how Amazon displays a bunch of products you might want to try? Booted up Netflix to see a bunch of movies you might enjoy? Those are recommender systems in action.

Such systems typically come in one of two forms – content-based filters or collaborative filters – or as a hybrid of the two. Content-based filtering take an item of interest and offers similar items based on the properties of that item. Take a game like Super Mario Bros.. If you think about Super Mario Bros. for a few minutes, you could probably come up with a bunch of keywords to describe the game. That set of keywords could be compared to the keywords for other games on the eShop, and the most similar games could be returned as recommendations. For instance, if you previously bought a Mario game, the eShop might suggest other 2D platformers, or other NES games, or other games released in the 1980s, or other games featuring famous mascots.

Now, just returning games similar to what you’ve bought doesn’t entirely solve the issue at hand. You might hate the new game. The keywords used to describe something like Super Mario Bros might not sufficiently capture what you enjoy in a good 2D platformer. A past purchase also doesn’t automatically suggest that you liked the game – we’ve all gone through a good bit of buyer’s remorse. This is where feedback comes in.

Feedback mechanisms are central to the success of most recommendation systems. Ever used Pandora Radio? Pandora operates on this same content-based model. You enter an artist or song that you like, and it tries to play similar music. As songs appear, you can tell the system whether or not you enjoyed them, and as you offer more and more ratings, the new songs that appear should match your preferences more and more closely. Similarly, as you rate more 2D platformers, the eShops’ recommendations on new platformers should more closely match your actual interests.

Of course this also highlights a key weakness of recommender systems – their accuracy depends on how much time the user is willing to spend providing feedback. Without any refinement from the user, this new eShop woud be just as bad as the old one at surfacing interesting content.

Collaborative filtering systems can also help alleviate this weakness. Rather than basing recommendations on the properties of an item, these systems offer recommendations based on how users interacted with an item. If you go to Amazon and look at something, they’ll recommend a bunch of other products that you might also like to buy. Their recommendations are based on a collaborative model, where they look at the item you’ve pulled up and take a look at what purchasers of the same item have picked up. Let’s say that you’re eyeing a nice new french press. Well, Amazon might also suggest you order a pallet of Starbucks dark roast to go with it, because people that bought that french press ordered coffee to go with it. Going back to our eShop example, if you pull up the page for Super Mario Bros., it might suggest that you also buy Mega Man 2 because people that bought Mario tend to also purchase Mega Man games.

These two models of content filtering can and should be combined. Take Netflix, for example. Netflix employs what is ostensibly a content-based model, recommending movies that are similar to what you’ve both watched and rated highly before. However, Netflix also incorporates what other users have watched and rated highly into that same calculation, resulting in what is arguably one of the most well-refined recommender systems out in the wild.

Our new hypothetical eShop would employ a similar model, using collaborative filtering to surface games beloved by a widespread audience, mixed with content-based filtering and your own feedback history to present content relevant to your own personal interests. Two birds – surfacing the best and suppressing the worst content – killed with one stone (albeit an insanely complicated stone).

Recommender systems aren’t necessarily a cure-all for the content discovery problem, and a poorly-implemented system might end up being even worse than the current eShop. Such systems need to be carefully implemented and constantly tweaked, and the success of a personalized content discovery system depends entirely on the volumes of data that can only come from an active community that actually purchases and rates games (trust me, I know this one from experience). Still, when I think of active communities, ones where the users are engaged and willing to make themselves heard, Nintendo is one of the few companies that comes to mind with such a fanbase.

What I’m saying is that it’s a longshot, but better solutions are desperately needed for content filtering on digital game stores. And outside of Valve, I think that Nintendo is the one company that cares enough and has the dedicated community to be able to pull it off.