Mirror Mirror On The Wall Who Makes The Best Recommendations Of Them All?

When I was a kid, I grew up in middle of the sticks where my parents didn’t have access to cable TV. When satellite dishes first started to show up, my folks were early adopters and bought a ginormous pirate dish (that you’d actually have to crank by hand when you wanted to jump to different channels), but for the most part, my childhood television experience consisted of two fuzzy OTA choices, NBC and CBS.

Back then deciding what to watch was rarely a problem. If I didn’t like what was on one channel, I’d simply live with whatever was on the other. Of course since then, technology (and my life) have changed a lot.

In thinking about my home entertainment setup today, I’m amazed at how many choices I actually have. It’s as if I’ve overcompensated for my lack of choice growing up, by deluging myself with every media gadget or service that comes out today. Between Netflix, Comcast, Amazon, YouTube, a dual tuner HD TiVo (not to mention my dual tuner media PC) and a little friend I like to call Bit Torrent, there is no shortage of content to watch.

While having plenty of choices is fun, having too many can be just as paralyzing as having too few and as more and more technology companies continue to take their shot at crossing into the living room, managing all of this content is going to become an even more important task.

Over the years, I’ve signed up for a plethora of movie recommendation services, but last summer I realized that it was too difficult to sync my ratings between them all. Since I strongly believe that TV and movie suggestions is a crucial piece of the new media experience, I wanted to make sure that my metadata activity was giving me the most bang for my click. In order to test out the various movie sites, I decided to create a Netflix challenge and conducted a “blind” taste test to figure out what site actually makes the most relevant recommendations.

To set up my experiment, I randomly choose 5% of the 2,000 movies and TV shows that I’ve rated through the Netflix service. I then set up a new profile on Netflix, Blockbuster, Spout and Criticker and manually entered my ratings for these 100 films, into each service. According to Spout, these ratings represented 11.6 days and 181 minutes of time spent watching TV

After I had setup an identical profile on all four services, I then took a look at the first 50 suggestions from each site and compared it to what I had actually rated the film on my main Netflix account.

If the service suggested a movie that I hadn’t already seen, I disregarded it toward calculating the final scores. Essentially, what I wanted to figure out was which movie recommendation service provided the best recommendations based on my actual real life viewing data. After taking a look at all four sites, it was clear that Netflix easily won this challenge.

Of the 50 recommendations that Netflix made, 28 of them (56%) were for films that I had rated 5 stars on my main account. While they did include two 1 star recommendations (Fight Club & American Psycho), the average rating for their recommendations weighed in at 4.18. Critcker came in second place with an avg of 4 stars per recommendation, followed by Blockbuster at 3.96 and Spout at 3.87.

Perhaps even more interesting then the final scores though, were the services that helped me find movies that I had never seen before. Netflix’s results may have been higher then anyone else, but of the 50 recommendations that they made, there were only 11 films that I hadn’t already seen. This compares to 24 unknown content suggestions (48%) from Blockbuster and an astonishing 44 unknown recommendations (88%) from Criticker. In taking a look at the types of movies Blockbuster was recommending, I wasn’t all that impressed with the list of unknown films. Most of them were kids movies that I simply missed because I had no interest in them at all. Trying to decipher Criticker’s picks was a little more difficult because most of their picks were for indie or foreign based films. At first glance it would seem that while Netflix may make more accurate suggestions, Criticker may actually be the best place for finding films that go beyond mainstream audiences.

I’ve listed the breakdown of my results below, but I would encourage you to take these results with a grain of salt. With a sample size of 1, my survey isn’t very scientific and because the ratings of my Netflix sample profile were 100% identical to the ratings on my main profile, it could have influenced my results from Netflix. Nonetheless, at the end of the day, I was pleased to discover that Netflix appears to be the best place for me to be rating and interacting with movies. While I’m hopeful that their open API will eventually allow consumers to port their data between services, it feels good to know that my primary movie recommendation service is the most optimal one for me.

1 Comment

Interesting experiment! I especially enjoyed your reflections on the different aspects of good recommendations – accuracy, diversity, and so on. If you’re still feeling experimental, you might like to have a look at Jinni (http://www.jinni.com); we just opened in private beta and offer both search and recommendations based on the Movie Genome.