The Best Super Bowl Ads: Opinions vs. Quantitative Super Test

Well, it is the time of year when America’s greatest sporting event takes place. I speak of course about the race to determine which Super Bowl ad is the best. Over the years there have been many ways to accomplish this, but like so often happens in research today, the methods are flawed.

First there is the “party consensus method”. Here people gathered to watch the big game call out their approval or disapproval of various ads. Beyond the fact that the “sample” is clearly not representative, this method has other flaws. At the party I was at we had a Nationwide agent, so criticism of the “dead kid” ad was muted. This is just one example of how people in the group can influence each other (anyone who has watched a focus group has seen this in action). The most popular ad was the Fiat ad with the Viagra pill…not because it was perhaps the favorite, but because parties are noisy and this ad was largely a silent picture.

Second, there is the “opinion leaders” method. The folks who have a platform to spout their opinion (be it TV, YouTube, Twitter or Facebook) tell us what to think. While certainly this will influence opinions, I don’t think tallying up their opinions really gets at the truth. They might be right some of the time, but listening to them is like going with your gut…likely you are missing something.

Third, there is the “focus group” approach. In this method a group of typical people is shuffled off to a room to watch the game and turn dials to rate the commercials they see. So, like any focus group, these “typical” people are of course atypical. In exchange for some money they were willing to spend four hours watching the game with perfect strangers. Further, are focus groups really the way to measure something like which is best? Focus groups can be outstanding at drawing out ideas, providing rich understandings of products and so on, but they are not (nor are they intended to be) quantitative measures.

The use of imperfect means to measure quantitative problems is not unique to Super Bowl ads. I’ve been told by many clients that budget and timing concerns require that they answer some quantitative questions with the opinions of their internal team, or their own gut or qualitative research. That is why we developed our agile and rigorous tools, including Message Test Express™ (MTE™).

MTE™ uses our proprietary Bracket™ method and a standardized approach to deliver quantitative results in a week’s time and starting from $9,900. So, I decided to see if it would work for Super Bowl ads. Monday morning we launched a survey which measured the relative merit of 36 Super Bowl ads. Respondents were asked to choose their most and least entertaining from groups of 3 ads. The “winners” then played off against each other until the respondent had picked their favorite. Through the use of Hierarchical Bayesian Estimation (the same math that makes Max-Diff or Discrete Choice conjoint work) we were able to produce utilities (which divide 100 points among all the ads tested) and with that quantitatively determine which ad was the most entertaining and how much better it was than the next ad.

The winner was the Budweiser Puppy Ad, which is no great surprise (if you’ve watched the coverage). The margin of victory, however, might be. As you can see in the chart above, Budweiser’s ad got over four times as high a utility score as second place Fiat/Viagra did. In fact, Budweiser’s ad was the favorite of roughly one in three respondents and ranked high for the vast majority. The ad did much better among women than it did among men (though it won with both groups) and scored well with both beer drinkers and non beer drinkers.

The Toyota “Being a Dad is more than being a father” ad came in fourth, but importantly it scored far better among those looking to buy a car in the coming months than among those who do not. Easy enough to forget that the point of these ads is to sell product!

I found it interesting that the Snickers Brady Bunch ad scored well with all age groups under 65. I had theorized that it would score better with those over 50 (who saw it as kids), but the show somehow still resonates 40 years after it went out of production. Only those too old to have “appreciated” the show when it first aired ranked the ad low.

One of my favorites, which hasn’t shown up on many lists, was the Doritos Airplane ad, which our study ranked 5th. Perhaps I’ve just been traveling a lot lately but it spoke to me.

The Victoria’s Secret ad ranked 6th and suprisingly it scored equally among both men and women.

Finally, an ad that I’ve heard a lot of commentators and Facebook posts talk about is the Squarespace “Dreaming with Jeff Bridges Ad” With apologies to the “Dude”, our representative sample ranked it as 35th (out of 36 tested).

We’ll be digging further into the results of this research over the coming days so stay tuned for more and in the meantime if you would like to know more about the methodology or the results, feel free to reach out.

2015 Super Bowl Ads Ranked from the Most to Least Entertaining

#1 Budweiser – Lost Dog

#2 – Fiat 500X Crossover – Viagra in the gas tank

#3 – Snickers – Brady Bunch – You’re not you when you’re hungry

#4 – Toyota – Being a dad is more than being a father

#5 – Doritos – Airplane seat saver with mom and baby

#6 – Victoria’s Secret – I’m in the mood for love, models posing

#7 – esurance – 2 ads: Lindsay Lohan and Walter White

#8 – Bud Light – Life-size Pac-Man game

#9 – BMW i3 – Katie Couric and Bryant Gumbel

#10 – Always – What it means to be a girl

#11 – Clash of Clans – Liam Neeson doesn’t like to lose

#12 – Coca-Cola – Spilled Coke changes the internet to make the world happy

I also found that they did a good job at making certain there was inventory at retail. There was a big in-aisle display in the fresh food section. They didn’t do that well though on signage to connect the ad with the in-store display.

Actually, I did notice from your list, that the only brand that advertised that wasn’t what I would call a big, mature brand was Clash of Clans. And you could question whether this is a ‘big’ brand or not. So actually, for a ‘small’ brand, the Avocados of Mexico did really well.

While I see the value of being able to assess an ad’s entertainment value, and using advanced Bayesian analytics to provide greater differentiation in the data, I’m not sure if that really provides marketers and advertisers with the answers they SHOULD be most concerned with.

If the goal and the role of advertising is to brand build and/or to create a (sometimes immediate) call to action, I’m not sure if assessing Likeability/Entertainment answers whether or not that has been achieved.

In fact, I would argue that in the case of Budweiser, the high entertainment/breakthrough element of the copy is actually working against them. In my recent blog write up on my company’s website (Lieberman Research Worldwide – LRW), I talked about how “Budweiser’s cute and cuddly communication, for the most part, contradicts the majority of the beer’s other messages and sponsorships.” In other words, it goes against what we refer to as their inherent Brand Stereotype®. That blog can be found here – http://lrwblog.net/lost-dog_im-lovin-it_super-bowl-ads

Our research has shown that if the goal for a brand is to shift their brand’s stereotype, it takes a lot of time and continued reinforcement. In Budweiser’s case, this this kind of intermittent — if once a year qualifies as intermittent — messaging likely does the opposite of what Budweiser intends, by causing implicit confusion from a branding standpoint.

One of the ads that I believe did the best job of leveraging its less-conscious perceptions to create a call to action was McDonald’s. By leveraging their inherent positive associations, or in other words, the things consumers give them permission to talk about, into their advertising discussion, it bolsters the power of the communication to have a stronger effect on desired behaviors – to go buy more McDonald’s. Interestingly, this is one of the ads that fell pretty far down on the entertainment list generated by your analytics (#18)

So again, while it may be very interesting to look at Entertainment value of an ad, especially for those that ran during the Super Bowl, I think it is important to remember that this is but a very small piece of the puzzle that marketers are trying to solve for, and that much more needs to be taken into account when assessing the potential success of a company’s advertisements.

Andrew: Interesting analysis. Agree that the purpose of advertising is to advance the brand and ultimately sell. The purpose of our research was to show that an advanced method could take a very long list (36 items/descriptions) and rank them effectively. With that said, we did wonder if entertainment value differed from the actual value of the ad toward selling more product. To that end, we are collecting data to see if a different framing statement (which ad made you more likely to buy the product?) will generate different results. Still imperfect of course, but should be interesting.