20121108

bing it on challenge

I'm glad that Microsoft for trying to break into the search sphere with Bing. It's good to keep search competitive so things continue to improve instead of stagnating.

That said, I find the "Bing it on" ads and marketing campaign to be misleading. I think that their Bing vs. Google test favors Bing (surprise, surprise) for three main reasons.

First is Google's info sidebar that now accompanies search. Google Obama, for instance, and you get the typical search engine list of blue links and their summaries, but you also get a really helpful side panel with a picture and biographical info. This side panel is not present at all in the Bing challenge, so you're not using the full experience for the comparison. Sure, Bing has it's own side panel that it's excluding, but it's social networky, hard to parse, and really uninformative.

Second, my hunch is that the kind of searches people do for these kind of challenges are really generic, like "butterflies," "red dresses," or "sports cars." These will all result in picture-heavy results, and Bing puts pictures closer to the top than Google does, and people like pictures. Further, these generic searches don't really represent what people actually search for. If you do this challenge, try to use something specific that you'd actually look for, like a professional colleague's name, an academic paper or book title, or a food dish that you'd like to cook.

Finally, if you do the challenge on someone else's computer, like for all the commercials and stats they're showing you, you're not signed into Google, and so the challenge isn't showing personalized results. Doing the challenge at home, this isn't an issue, but the stats and commercials are crucial to the marketing campaign. Google is really smart about knowing what you like and what's relevant to you, and as far as I can tell, Bing isn't, but I might not have used it enough to pass verdict. Bing hooks into Facebook, but that's not relevant information for personalizing search. (It is, however, how they track you from session to session, the way Google does with your Gmail or your Google account.) What is relevant information is what you've searched for and clicked on in the past, which Google has by the bucketful.

It's possible that Bing is really great. Lots of people like it for the pretty search page, but I always search by hotkey, so that doesn't matter to me. It's also entirely possible that Bing can give great personalized results once they build up enough data about you. But I don't want to waste my time training them, when I already have a trained engine. I also use a bazillion other Google products: Gmail, Calendar, Drive, Blogger...the list goes on, so I'm going to be signed into Google anyway. I don't really use Facebook that much, and don't like the idea of my search experience being tied to my Facebook account. I want the freedom to delete that account without losing quality of search.

I'm obviously a Google loyalist, but I think Bing is a great search engine and I'm glad people can pick. I'm not objecting to Bing, but to the marketing for it.

3 comments:

In response to this post, a friend asked me, along with many much harder questions, "What *should* their ads say?"

I think that the Bing ads can say what they do right now, but be more explicit. They could give examples of when Bing does better and when Google does better, which they probably now have because of their online test. I think it would also be nice if their numbers updated as people took the challenge, rather than just being a static 2:1 from the initial test. While they're relatively open about methodology, I'm asking for more transparency in their results.

"if you do the challenge on someone else's computer, like for all the commercials and stats they're showing you, you're not signed into Google, and so the challenge isn't showing personalized results. Doing the challenge at home, this isn't an issue, but the stats and commercials are crucial to the marketing campaign"

I don't think the results are personalized for google if you're on your own computer. For example, a search for "weather" brings up weather for Redmond WA and a search for a friends name doesn't bring up stuff by them the way it does logged in as me.

" I think that their Bing vs. Google test favors Bing (surprise, surprise) for three main reasons."

What about selection bias? If Google had won in Bing's independent evaluation they wouldn't have spent lots of money promoting it. They could easily have been running these tests every so often as they made changes to Bing to improve it, and then when they got a good result decided to market it. Without knowing how many tests they ran and didn't publish their saying "The overall sampling error rate for the study is +/- 3 percent at the 95 percent confidence level" is highly misleading.

(But, as you know, I work for Google, so am going to have a hard time not seeing Google's side.)

When I tried the challenge at home, I was logged in in the same browser as I took the challenge, and I thought it was picking up personalized results. If that's not actually the case, then the challenge is even worse than I thought.

And of course there can be and probably was selection bias in their testing, as with so many things. User testing is great for product development, but for marketing? I'm thoroughly skeptical, even more so for products that I feel I understand well.