Posted
by
Zonkon Thursday June 28, 2007 @05:26PM
from the weighty-logos dept.

amigoro writes "Here's an interesting experiment: Copy Google results pages from four different e-commerce queries. Tell 32 test subjects who are going to evaluate the results that the results were from four different search engines: Google, MSN Live Search, Yahoo! and an in-house engine created for the study. Then see which ones they rate as the best. As it turns out Google and Yahoo! win hands down, proving that even on the Internet it's all about branding."

I'd say it is due to the fact that people perceive Google and Yahoo as "Good" companies. While most perceive Microsoft to be evil. Or maybe Google has finally learned how to hypnotize people with.gif images...
-
Only time will tell.

Maybe they perceive them as good companies because they have tried all 3 of them and found out that MS' search engine doesn't deliver? MSN/live.com got better over the years, but as long as they are not as good as google they won't gain any marketshare except from those using IE and being unable to switch their default home page.

But that's just circular reasoning now. The study had them try all three when they were all artificially exactly as good as Google, and they still thought MS sucked. MS doesn't just have to be as good as Google, they have to be enough better than Google to overcome the confirmation bias of Google/Yahoo fans....of course, they really aren't as good as Google;).

This is not circular reasoning at all: Everyone rates information based on prior experience. Do you believe the guy making up crazy stuff more than someone who is right most of the time? Traditionally we judge a lot based on the reputation of the source. It's actually a very effective way to decide...

One could easily give credit to the results of past performance. Yahoo was the #1 search engine for years because it delivered good results. Google became first because it delivered beter results, and generally still does. MS search got a bad reputation from stories about manipulating search results. And the "in house" engine has no past performance - who would trust that at all?

When Yahoo was number one, they were using Google for search results. They lost that position (in part) because they started selling result positioning, making them far less accurate. Throw in a clunky interface and Google starts looking even better...

Google became first because it delivered beter results, and generally still does.No more!Overture powered Yahoo! has surpassed it.I can say it because I have seen my blog featuring on Google's first page for search queries that were mostly irrelevant to my content.

The only point in history that Yahoo was the #1 search engine was the period during which they used Google as their search engine.
Before that, AltaVista and the like dominated. Not that it mattered, mind you, since most search engines were utter crap and most people just used whatever was on their home page.
These days, the search results are pretty much the same, but Google still does better because people associate them with search, because they're sexy, and because their results come back a lot faster

The experiment is confused. Are the evaluators supposed to do their own research on the products, or respond with their trusted search engine? If the former, they shouldn't care what the raw data is from. If the latter, then it's not just branding, since each company is different and the point of trusting a company is that you accept their results without scrutiny (at least, with less); after all, what's the purpose of using their services if you feel you have to verify everything yourself anyway?

Who is stupid enough to go through a page of identical results and claim one is better than the other? That is why the design of the experiment is so important - it would answer WHY people did that, and how valid the results are.

It sounds like they gave a number of people the same page with different logos. It doesn't sound to me like they gave the same people the same page with different logos. So they asked people to rate the reliability of the results, and people who were given the Google logo rated the same results better than people who were given a no-name logo.

It was the same results with the same layout, only the logo at the top changed.

This is also how I interpreted the article. It doesn't seem like a good way to set up the experiment. Wouldn't anyone who actually used the search engines (the people that branding would mean something to) realize that the format of the search results didn't match what they normally see from that search engine? Or did the experimenters choose a neutral format that none of the search engines use, including the logos only as an identification of which search engine the results supposedly came from?

The overall results of this study are that performance and reputation trump marketing money. M$ spends close to a billion dollars a month on advertising, dwarfing the combined spending of the rest, but people clearly think their stuff is second rate. This is how a the market should work and it's encouraging. People are not nearly as dumb as M$ thinks they are.

Reputation is a legitimate decision factor in information services. It's right for people to think M$'s search engine is goofey when M$ is such a dishonest company, their results have been poor in the past and they admit to selling placement. It's also right for people to have a neutral or favorable view of Google and Yahoo given the performance record of both companies in search. The neat thing about search is that things that don't look like useful results often are, at least when you use a good engine.

There were a few problems of sample size. The group size is to small and the responses were too poor to mean much. Only 36% of the results were judged relevant, which means the results from all the engines were poor. A larger study may show a real relationship between performance and trust that goes beyond marketing.

I'd rather see people search for several queries of their choosing from the same four engines and be delivered result pages free from branding and see which results they preferred. It'd be much more useful.

you might find 48 hour side project i did a few months ago interesting (but it has branding)... its SearchTheThree.com [searchthethree.com], and it basically returns the results from google, yahoo and windows live in three colums next to each other. its quite buggy atm tho, and IE just pukes and gives up with the current design:P... but i have been meaning to do something more with it, but between lack of time and work, and lack of big ideas for it, i just never got around to it...

And it's obvious that Yahoo was the better one, just look at the results [searchthethree.com]. Also MSN live never returns any hits? Something is wrong with your page? Beautiful and clean design, but is it really ok to take that background image? (It's from Vista isn't it?)

my personal choice of search engine is google i have to admit... as for the windows live, as i mentioned, its quite buggy at the moment (sowwy:$)... and the background image is part of a wallpaper that ships with mac os x:)... back in january they used as the wallpaper for the iphone during the demos and stuff... and the rest of the design is kinda based on the standby screen of the iphone (and yeah, sometimes i have a bit to much fanboi tendencies... lol)

If the intended audience of the study are search engine power users who need to know about slight differences in algorithmic performance, then yes, your study would be more "useful."

However if the intended audience is the marketing and research divisions of major search engine players, then this study goes a lot farther in saying what gets market results than yours would. It answers the question: "do we spend $1 million improving our tech or do we spend it selling our brand?"

Of course it's hardly a secret that customer satisfaction has very little to do with the freely made up mind of the customer. Its why car companies go to such lengths to provide little perks to purchasers and leasers of new cars. A few tens of thousands of miles of free oil changes tends to make the average customer a lot happier with their car than actually buying a really good car does. What makes the average geek happy with one gadget he buys, uses for a week and then leaves in the drawer, and unhappy with another? Even with smart consumers its very rarely a hard honest brass tacks look at features, performance, and price.

Agreed.A better follow-up study would be to present users with four sets of results, two of which are clearly inferior by any objective measure*, label the inferior results with "Google" and "Yahoo" and see if the users still pick "Google" and/or "Yahoo" as being the best.* "clearly inferior by any objective measure" doesn't mean that the results are horrible, just clearly inferior. (Although one could also do a study where the results are indeed horrible, lable the horrible with "Google" and see what the

Branding is about consumer trust. If they trust your company to do something consistently well, they will place a good deal of faith in that (hopefully without becoming fanboys). As a result, whenever a product buying decision comes up, they are more likely to select the branded product.

I would argue that there is a difference between the philosophy of branding and the reality. You nail the philosophy on the head... use a distinct brand to establish a 'relationship' based on trust with consumers. 'Informed' consumers operate on this wavelength.Realistically, however, a brand's ability to produce high-quality results is not essential to the maintenance or a brand's power in the marketplace. Consumers routinely prove that trustworthy results are nowhere near as important as simple familia

If I was told a tidbit of medical advice came from Dr. Dean Edell I'd have weighted that higher than if I'd been told it came from Dr. Laura. This would be based on having heard each of them dispense advice in the past.

Now if they asked people who had never used a search engine and they rated them based on the name, then I'd call it "branding". But if the people polled had past experience with one or more of the search engines, that direct experience is liable to have a much greater influence than name recognition only.

Yes. Branding is something you buy with a billion dollars a month in advertising, like M$ does. Reputation is something you build with a quality product, like everyone else does. People can tell the difference.

I can hardly get through a TV program without having to watch one of MS "switcher" adds trying to convince people to drop Google search in favor of live search. Oh wait, that was in your reality, what was I thinking.

That's called projection. You consider your own circumstances universal and project your reasoning onto others. The results are wrong because the circumstances are different. There are no TV ads in my reality, so what you were thinking was wrong.

If I was told a tidbit of medical advice came from Dr. Dean Edell I'd have weighted that higher than if I'd been told it came from Dr. Laura. This would be based on having heard each of them dispense advice in the past.

The fact that Dr. Laura is not a medical doctor wouldn't even enter into it?

There's nothing wrong with a little extra rigor to determine if common sense "knowledge" is actually true.

Yeah, but common sense and looking at people's brains aren't the only two options. If you're drawing conclusions about behavior, studies (like this one) about behavior are more appropriate than seeing whether some neuron fires when you see a Coke can. It's been directly established for decades that perceptions are affected by prejudices.

Yes, of course we knew that brands matter in decisions. What they showed was it triggers emotional reaction. Not logical decisions. Not recall of past experience with a brand. Emotions.

Are you talking about the brain researchers or the search engine study? If the latter, I don't understand how you concluded that from the article. If the former -- yes, I agree that that research is completely unrelated to what is being studied in the search engine experiment, which is why I was mystified by the author's ref

like dave said, it shows the results from the three big search engines, side by side... altho keep in mind it was a 48 hour side project of mine just, and the level of bugginess can very... windows live search for example has an ugly tendency to just not work at all with a lot of queries...:P... i've been meaning to take a serious stab at the project shortly tho when i get some free time:)

People use logic for decisions, NOT emotions. Geez everyone knows that

Seriously though, branding may seem a bit shallow and unscrupulous, but it certainly taps into how the brain works. We associate and generalize. Google good and MS bad, that type of thing. Then you throw in colors, images of happy people, etc, and you get a positive reaction. It may not be enough to surpress logical thinking, but these associations are powerful.

Google good and MS bad, that type of thing. Then you throw in colors, images of happy people, etc, and you get a positive reaction. It may not be enough to surpress logical thinking, but these associations are powerful.

So isn't it encouraging that people are able to see through the butterflies, "reach your potential" and other obvious bullshit that M$ spends a billion dollars a month on? If the results were really all about marketing M$ would have dominated the results as much as they do advertising. Y

So true, marketing is only the foot in the door. You may be able to convince people to initially look at your product, but after that no amount of marketing can fix real problems. It's fortunate that no amount of hype or image can outweigh the truth, even if some refuse to believe. Marketing is powerful, but it's on a subconscious level that is outweighed by the higher thought processes. That's the idea at least:D

So let me get this straight. 1 search result, and brand it 4 different ways. Then you take 32 people, and show each all of the results. There's no way that's enough to overcome statistical errors, biases, vague worded questions, morons, etc.

And most importantly, you taint the results by showing each person all four!

"Do you like Result 1?"
"Sure"
"Do you like Result 2?"
(thinking) "That looks familiar" (said) "I guess."
"Do you like Result 3?"
(thinking) "Okay, now that has to be the same thing" (said

Hey, wait a second... im getting confused. If they wanted to see what brand people trust the most can they just look at usage logs (google being used more than other search engines). To me a more interesting experiment would be to do this all without showing and the logos to see what engine gives back the most useful results. I don't understand why they did the experiment....

Being the most used brand does not mean you are the most trusted brand, or the highest-quality brand, or the cheapest brand; nor do any of the latter imply the former. There is a correlation, true, but you can't tell how strong that correlation is without experiments like this.
This experiment has shown that there is a disconnect between the quality of a brand's product and the trust of a brand's product (in the short term, with a low sample space). Maybe you don't find that interesting, but I do.

I remember when google first came out. Yahoo, msn, altavista, hotbot and several other sites were stronger brands than google. Google's search page was clean and wasn't cluttered with tons of stuff. It returned lots of results and it had the least amount of spam compared to the others. Nobody knew what google was but once I found it I started telling people about it.

I would say the factors for internet site success are: results, interface & word of mouth.

The tricky part about the internet is that if any of those three things change or is bested by another competitor, the flow of traffic will change.

Internet customers are not loyal to brands. They go with the flow. If all of a sudden google became obsessively cluttered or slow response or cluttered with spam, then we'd be ready for something else. And which ever service seemed to step up to the plate then the flow would change to the new place. But since we're all comfortable with 'google' right now, if there's a competitor that is offering a similar service, we have no reason to move the flow of traffic. But if google started being annoying, there are NO loyalties.

I remember when google first came out. Yahoo, msn, altavista, hotbot and several other sites were stronger brands than google. Google's search page was clean and wasn't cluttered with tons of stuff.

I heard about Google long before I used it. I stayed with AltaVista because I had learned their search syntax, so I knew how to do very targetted searches. What killed AltaVista for me was they somehow let their spidering get very out of date, and eventually, AltaVista just returned pages of broken links. Probably somewhere, some PHB, decided that money could be saved if they did less of this pesky "spidering the Internet" stuff which seemed unrelated to delivering adverts.

I just thought of an example. I've been a huge fan of google and google maps for quite some time. I absolutely hate microsoft. Somebody showed me the cool 'birds eye view' on http://maps.live.com/ [live.com] (firefox or IE required, safari doesn't work)... it blew me away. I now use this to look around quite a bit. Google tried to combat with the 360 view thing. Honestly, that 360 view thing is completely worthless. Doesn't pull off the 'birds eye view' even close.

E.g: Why does one drag around the map under the field of vision? This strangely reverses controls. It'd feel much more natural to drag the field across the map the way a brid or a plane would fly around.

The same thing goes for the buttons that indicate the direction you're looking towards: The map doesn't change its orientation (rightfully so), it keeps North on top at all times. So why do the buttons on the compass interface wander around

On the other hand, you are counted as an expert in this area which means that you are much less likely to care about brands. My friends in the fashion industry doesn't care about brands, they know what looks good and they don't need brands for reassurance.

You have the competence to judge what is a good search result and what isn't. Most average Joes and Janes does not have that competence, and will - when unsure - go for the brands they can recognize.

This is very interesting. I too would generally rank Google and Yahoo as my top search engines in terms of "quality.

A few months ago, though, I became aware of Yahoo's "paid inclusion" program. Basically you pay Yahoo money, and your company is more likely to show up in the "organic" results area of the search. Note i'm not talking about the sponsored results at the top of the page or the ones on the right side, the "paid inclusion" results are indistinguishable from normal search results. Apparently this is a well known feature in the industry but personally i was surprised to find out about it.

Technically, the ranking algorithm doesn't weigh these sites higher. PI just assures that your site is crawled often and that it is "crawled" according to the page definition feed that you provide.

I ran into this whole concept because i was surprised that the company i work for came up at the top of certain searches even though we have a lot of competitors. Even though we ARE better and more popular than many of our competitors, i was still surprised that we were at the top on some pretty specific terms. It showed a keener understanding of our site than i would have expected a crawler to have. At first i was really psyched about Yahoo's technology, but then i found out that we use paid inclusion.:-)

I'm still undecided how i feel about this program. In my mind the main results are supposed to be purely based on site content/popularity , unlike the sponsored links. When you get an advantage by paying for access to their crawler... that's no longer the case. On the other hand, this isn't THAT different from other SEO techniques which are by definition mechanical ways that you can improve your standing. Only difference is that yahoo is directly involved in it.

I think the situation will closely mirror all other historical examples in advertising. Consumers start out intensely naive and susceptible to the most ridiculous, blatant pitches imaginable. Gradually their awareness increases, to the point where they become jaded and impervious to the traditional forms. This causes advertisers to refine their methods to become more subtle (think product placement or "QualComm Park".) We start hating those. Rinse, wash, and repeat.The thought that search results could beco

That looks like a continuation of Yahoo's directory days. Back when they relied on their directory, and not search results, people would submit sites and Yahoo would review them and categorize them. After awhile the backlog got so long that Yahoo started charging $200 to guarantee a review, though category placement was still up to them. Guess they just carried the policy over to their search engine.Here's a news story from 1999 about it when it began:http://searchenginewatch.com/showPage.html?page=21 67091 [searchenginewatch.com]

Google, even in China, mentions the Tiananmen Square killings on page 3 or 4. In China, Tiananmen Square is known for a lot more things than the killings, much like the Lincoln Memorial is known for a lot more things than the Million Man March.

I like the Google brand, but I can't seem to get over the ugliness of the logo. Yahoo, on the other hand, has an appealing and very identifiable logo, but I can't stand using the search engine/directory/whatever the heck it is.

I wonder, how much does the asthetics of the logo affect the brand strength? Any thoughts?

Here's a more understandable summary that in TFA: They got Google search results on four e-commerce topics, stripped identifying HTML, and created 16 fake results pages with branded header and footers for four search engines The query results for a topic had identical content and presentation. They presented each of 32 test subjects four results pages, one for each topic. Which of the four brands each was shown and in what order the topics were presented were randomized. Subjects looked at each result one at a time. They were asked to evaluate the results, following links as they chose to and commenting out loud. Note that all results sections were identical is both content and format, with only the branded page header and footers being different.

The authors claim that Yahoo got the highest ratings (averaged over all four topic queries), 15.3% more than the overall average, compared to Google's 0.7% over average. But their table and graphs show results all over the map. Google scored 52.2% over average on the home improvement query. MSN had the highest score on the camping mexico query, Yahoo was highest on techno music, and the made up unknown search engine got the highest score on the laser removal query. That says to me that when you have four search engines and four queries to mix together in random order you need a lot more than 32 subjects to get statistically meaningful results. The paper contains no analysis of statistical significance of the results.