Study: Google “Favors” Itself Only 19% Of The Time

For the past year or so, there’s been a rising meme that Google is altering its search results in ways to favor itself over competitors. Now a new study is out showing the opposite. Google is far more likely not to show its own products in the first spot of its search results.

The survey is from Ben Edelman, an assistant professor at the Harvard Business School. Edelman is a long time Google watcher who regularly publishes interesting and detailed looks at search and advertising-related topics. Edelman has also consulted for Microsoft and is involved in a lawsuit against Google.

Edelman reaches completely different conclusions from his survey than I do, writing at the end:

By comparing results across multiple search engine, we provide prima facie evidence of bias; especially in light of the anomalous click-through rates we describe above, we can only conclude that Google intentionally places its results first.

How have we ended up with such different conclusions? Why, I love Google, of course — and he clearly hates them! Seriously, statistics can easily be turned to whatever you want them to be. I feel like Edelman is turning his study into the most negative view possible. I’m just looking to provide some balance to that.

The Study

Edelman’s study is clever (and one I’m pretty sure has been done a few years ago by others). Search for products that Google offers, and see whether Google lists it own products over those of competitors. Searches included things like:

mail

email

calendar

chat

maps

video

The study did this, for 32 different searches at Google, Yahoo and Bing, back in August 2010.

The Age Issue

Immediately, the age of this testing is a problem. Back in August, Yahoo was still providing its own results. Today, it’s powered by Bing. The study provides no conclusions about what’s happening on Yahoo, right now.

In addition, results change all the time. For some of these queries, I get different results at either Google, Bing or both versus what the study reports. In short, this study says nothing about about the current state at any of these services.

I asked Edelman about this, via email. He agreed, saying:

Your reference to the age of our data is well-taken. When we started the process, we intended to get this paper out quite a bit faster. It’s easy enough to redo the results with the newest data; we can, and perhaps we will, depending on response to this piece. But incidentally, it’s also useful to preserve results as they were a few months ago. That will certainly be important in disputes: Folks will be discussing not just what results appear now, but also what was done in the past.

Sure, but a single historic data point still doesn’t necessarily prove anything. As I emailed back:

I agree it’s interesting to know what the results were six months ago for historic reasons, but that’s also just one data point. They might have been much “worse” in terms of Google’s supposed favoring six months before that. Or much less.

The Small Sample

Edelman’s report prominently features a chart that highlights in red how Google apparently favors itself in the top three listings for various terms:

Look at all that red, which indicates when a search engine supposedly favored itself. All that red must make it true! And for Yahoo, too!

Still, drawing a conclusion from only six queries would be foolish. The report is primarily based on a list of 32 searches, which are listed on a separate page. Using the full list, it’s easy to find a chunk that doesn’t portray Google as full of red favoritism:

Even looking at the “larger” sample of 32 queries, it’s still a relatively small dataset. More important, it doesn’t correct for the popularity of the particular services.

Probability Versus Popularity

There’s a section of the report where Edelman talks about doing some regression analysis to disprove that the favoritism that he alleges is just random.

Look, just because someone says they’ve done a regression analysis doesn’t mean that something is statistically correct — even if it sounds that way. The fundamental structure of the test itself might be flawed. If so, all the analysis you do isn’t going to correct for that.

Edelman’s study assumes that any time Google puts one of its own services first, that’s somehow favoritism rather than a reflection of how important the service is in general.

For instance, using Edelman’sown data, there were five instances I found where both Google and Bing put Google’s services in the top position:

books

images

maps

translate

video

Crazy — Bing putting Google Maps first! Is it a plot by Microsoft to try and prove how unbiased that it is? Or is it perhaps a reflection that a lot of people use Google Maps — and thus link to it — which in turn can influence the search results at both search engines?

If you wanted to do a really scientific study of “favoritism,” you’d first calculated the relative popularity of each service. Then you might first try to determine if the service is being listed in the order you believe it “should” appear at. If Yahoo Mail is the most popular email service on the web, and Gmail the second, then do they get listed in that order?

Even then, I still think you have issues. Popularity doesn’t always equal relevancy. But I definitely know that, regression analysis aside, there are underlying problems that make these stats suspect.

Search Algorithms Don’t Agree

One of the biggest problems I have with the report is when Edelman writes:

Taking seriously the suggestion that top algorithmic links should in fact present the most relevant and useful results for a given search term, it is hard to see why results would vary in this way across search engines.

It’s not hard to see why search engine result differ at all. Search engine each use their own “algorithm” to cull through the pages they’ve collected from across the web, to decide which pages to rank first. The articles below explain more about this:

Google has a different algorithm than Bing. In short, Google will have a different opinion than Bing. Opinions in the search world, as with the real world, don’t always agree.

Indeed, there have been many studies over the past years that have found search results often do not agree. Another reason for this is that the core collection of documents — the “index” — that search engines search through are not exactly the same.

Google Searchers Aren’t Bing Searchers

One factor that both search engines assess is clickthrough. Bing has explicitly said that the number of clicks a listing gets is one of many factors it considers. You can imagine that something listed in the top position would be expected to get a certain percentage of clicks. If it doesn’t, that can be a sign that maybe something else should be there, because searchers are bypassing it.

In either case, clickthrough is not the most important ranking factor. It’s just one of many of them. But it can have an influence. In turn, that might help explain some of the “favoritism” that Edelman saw. If someone’s searching for “maps” on Google, they may be more likely to want Google Maps than Yahoo Maps — and vice versa.

If clickthrough is helping measure this, is that favoritism — or is that ensuring your search algorithm is working best for your particular audience?

So Why’s Yahoo Mail Second?

Edelman did try to account for the potential clickthrough factor, and it’s perhaps the most intriguing part of his report. He used two different sources to gather clickthrough rates on Google, Yahoo and Bing. For email, he found that Gmail was listed first on Google and pulled 29% of the clicks versus Yahoo coming second and getting 54%.

That’s odd. It suggests that searches on Google aren’t being well served by that search — and Edelman counts it as further proof that Google is favoring itself. Then again, it could be….

Lots of searchers were looking for Gmail and realized they could get to it another way after doing the search

Google’s results are screwy. Google, like all search engines, has screwy results for all types of things

It is intriguing, but that’s also apparently the most damning part of the clickthrough analysis that Edelman could find.

In general, he says only “sometimes” does the second results listed get more clicks than the first. Apparently, it doesn’t happen the majority of times. It would have been useful for him to have shown how many times each service listed itself first and had clickthroughs that supported that.

Hey, Who’s Ranking For “Search Engine?”

Now if you really want to crank up the conspiracy theories, let’s talk about Google’s most important product, search.

Google offers a search engine, just like it offers email, chat and other products. That search engine is still Google’s most important and profitable product, to my knowledge. So what do I get in a search for search engine on Google?

Google doesn’t list its search engine at all. Yes, Google is finally at least getting a spot as a company by having its Google Custom Search service appearing here — but that’s a completely different product that the consumer oriented search engines that it does list like Dogpile, Bing, AltaVista, Ask.com, Yahoo and even tiny Duck Duck Go.

Seriously — Google can’t list Google.com but manages to get Duck Duck Go into its top results? It makes no sense. Having watched this results set for literally years, I’m borderline believing that rather than favoring itself, Google is deliberately downgrading itself here, as a way to show the world how it doesn’t favor itself.

But Google Admits Favoritism!

At the end of the report, Edelman tries to turn a comment by Google’s Marissa Mayer — made when she was vice president of search product and user experience — as further evidence that Google is tampering with its results, as if this further confirms his findings:

At a 2007 conference, Google’s Marissa Mayer commented:

“[When] we roll[ed] out Google Finance, we did put the Google link first. It seems only fair right, we do all the work for the search page and all these other things, so we do put it first… That has actually been our policy, since then, because of Finance. So for Google Maps again, it’s the first link.”

We credit Mayer’s frank admission, and our analysis is consistent with her description of Google’s practices. But we’re struck by the divergence between her statement and Google’s public claims in every other context.

Edelman’s report is about Google’s “algorithmic” results, the “10 blue links” as they’re sometimes called — the meat of the page. Mayer wasn’t talking about algorithmic results. She was talking about Google’s OneBox units, where Google shows results from its various vertical search engines.

Edelman disagrees. From our email exchange:

Look at that last sentence: “For Google maps again, it’s the first link.” “For Google maps again” – that must mean, when a user runs a map-related searches, like “Boston map” or maybe “Boston museum.” But “it’s the first link”? There was never going to be more than one map embedded on the page. Nor, to my knowledge, has Google ever run a map rich result box that linked to multiple different map services. (In contrast, as Marissa mentions, finance rich result boxes do link to multiple finance sites.) So what does Marissa mean when she says “It’s the first link”? I think she means all the other maps sites drop to lower algorithmic search links; Google puts its map service at the top and others below. And others are indeed ordinary algorithmic results, not any kind of special rich result or onebox result.

No, Google Doesn’t “Admit” That

It’s pretty easy to know exactly what Mayer meant. You can watch her answer yourself here at 44:51 seconds into her talk. She’s talking about the first link in the list of sites that appear when you do a stock search on Google and get a OneBox result. Google puts itself first on that list:

See the first arrow? It points to a list of sites that appeared under the current stock price for Google, when I did a search for goog, the Google stock symbol. Mayer was saying that before Google Finance was launched in 2006, links there were ordered by popularity. But after it launched its own service, Google thought it was fair to list itself first on that line.

Look at the second arrow. That’s the first “algorithmic” result that Edelman’s report covers and which he says Mayer’s comment is about. It’s not about that, at all. Indeed, that first algorithmic link goes to Yahoo Finance — not Google Finance. Google Finance comes second in the algorithmic listings.

Beyond The Algo Listings

Talking about the algorithmic listings as if they are somehow independent of the rest of the search page is kind of absurd, of course. Studies have found that they tend to be the results that attract the most clicks. But the days of search results that are only 10 blue links are long gone. People might interact with OneBox and other smart answers that ALL major search engines have long provided.

Is it unfair for Google to “favor itself” by showing me images from its own image search engine rather than Bing’s, as it does here:

Or does that just make sense — that Google also has an image search engine, and it should point people to that? Certainly, Bing does the same:

To me, it just makes sense for Google or any search engine to point to its vertical search engines if it runs them. They obviously believe they have good results for their users there. For them to not to would be like complaining that the New York Times continues to run its own entertainment section rather than including the entertainment section from the Los Angeles Times.

Do you count the things that Edelman’s report considered? Whether Google lists any pages from itself in its top results, or just the top three listings, or whether Google itself first above all else?

To make sense of Edelman’s figures, I tried to keep it simple. I went to his full list of 32 searches that were conducted. I found that there were 11 searches in total where Google listed itself first above all others. I then looked to see in which of these cases that Bing also listed Google first. That happened five times, which I mentioned above. That left these six searches where Google — and only Google — “favored” itself

academic article

blog

email

finance

mail

scholarly journals

So in 6 our of 32 cases, Google would seem to have favored its own products in a way that might raise eyebrows – 19% of the time. After all, if Bing’s going to list Google first in those other cases, counting these “against” Google doesn’t seem fair.

Even in these cases, the stats still don’t reflect that competitors may be very visible. Even if Google listed itself first for “email” — however it happened — is it really anti-competitive when its competitor is prominently listed in second place? Wouldn’t it really be more of a concern if Google didn’t list its competitors at all?

Google’s Response

Shortly after I published this, I also received an unsolicited statement from Google (it will commonly send statements around to journalists, if a story or study is making the rounds. Here you go:

Mr. Edelman is a longtime paid consultant for Microsoft, so it’s no surprise that he would construct a highly biased test that his sponsor would pass and that Google would fail. Google never artificially favors our own services in our organic web search results, and we perform extensive user testing to ensure that search results are ranked in a way that provides users with the most useful answer.

Google also sent some examples that contradict the idea that Google favors itself. Among them?

search engine

book flights

directions

Hey, didn’t I mention that search engine example! Indeed, I did – along with some other things that Google is pointing out, such as taking the popularity of services into account or what searchers at particular services might prefer.

We Want Relevancy, Not Regulation

A search engine’s core algorithm results have long been seen in some quarters as “editorial” content that shouldn’t be tampered with to favor anything other than what’s best for the searcher. Not every search engine has followed this practice, of course. In my experience, Google has been the best at it, refusing to fix things by hand even when it should (see Google, Bing & Searching For The New Wikileaks Website).

I have a great fear of governments stepping in to dictate what a search engine’s listings should be. To me, that’s akin to telling a newspaper what it can report on, or trying to regulate opinions anywhere. There are no “perfect” search results, nor will you ever find a set that’s “neutral.” An algorithm, ultimately, is an opinion. Opinions are not neutral.

What we really should be concerned with isn’t whether Google (or any search engine) is being “fair” but rather if it is providing relevant answers. I can remember when Lycos favored itself so much back in the late 1990s that it was difficult to do a search that didn’t lead you back into Lycos.

Where’s Lycos today? Right. Relevancy will attract and retain users. If Google or any search engine isn’t providing relevant results, the market will likely correct things. And in fact, that’s what a recent survey found. People are far more concerned about getting more relevant results than government regulation of results. See the two articles below, for more on this:

Sponsored

http://www.linkedin.com/in/philipsegal Philip Segal

I agree with this article in the sense that I believe a lot of the recent clamor about Google’s favoritism is from businesses who are sour from losing their free Google traffic. Google bought Youtube because it was the most popular video site and they favor Youtube in the results for whatever reason, naturally based on their algorithm or otherwise, and it’s fine because it’s relevant (ie. what searchers want).

But where do you draw the line? What happens if Google keeps buying companies in each vertical, and soon owns the major players in addition to the playground?

Google wants in on Travel, much to the chagrin of everyone in the industry. Google recently bought Like.com, created Boutiques.com and is now buying Adwords ads and competing organically with the company I work for (Shopzilla), not to mention their recent favoritism of Google Product Search in the results, much to the chagrin of the entire Comparison Shopping industry. They all but killed Mapquest when they moved into maps. Etc, etc., etc.

I’m not saying I agree or disagree with any of this; a business that builds it’s core competency around organic search does not “own the right” to that traffic. But once again, where do we draw the line? Is Google a Search Engine or the owner of all online commerce?

As they continue to blur the line between their original purpose as an intermediary between Point A and Point B by… buying Point B so they can be Point A and Point B… well, once again I ask, where do we draw the line?

There isn’t yet a clear answer in the law or in our minds – this is completely new territory, and a company has never held so much power. I think we are going to find out where that line is in the coming years, and I think we’ll look back on 2010-2011 as the time when Google began overstepping its bounds.

http://rockfi.sh steveplunkett

Danny Sullivan is the greatest search engine journalist on the planet.. there.. i said it..

thanks danny.

http://www.michael-martinez.com/ Michael Martinez

And then there is the claim by Foundem that Google manually adjusted its algorithm to “whitelist” their aggregated content (in violation of Google guidelines) after intervention from the AdSense team.

Google is certainly not the only search service to promote its own content (or content on which it is being paid) — so we don’t have to hang them for that.

But where should the line be drawn? How much self-promotion should we tolerate from the top 2 search engines before we say, “Enough! Let us compete for your visitors the way we want to!”?

http://www.blindfiveyearold.com ajkohn

I’m on the fence with this one because there are so many ways to look at it.

19% isn’t exactly trivial. But there’s no real way to determine whether that occurred because of bias or on algorithmic relevancy that is specific to Google. While the CTR data is compelling – was really the only part of the analysis that made me wrinkle my brow – you simply can’t assume that it’s THE piece that should determine rank.

Then I am of two minds about whether this is really an issue or not at all. As you point out, there is still visibility for competitors – usually within one or two positions. So should we hold it against Google for being higher? Should we even if they’re putting an elbow on the scale?

There’s a line there obviously, but I can’t say I get too worked up about it.

Conversely, I am uneasy with Onebox implementations and presentations. They take up an enormous amount of real estate and are triggered and displayed in varying degrees and times of year. OneBox essentially unseats an algorithmic result.

I’m okay with Google being the first link. It’s their engine. I get it. But what about the other links in the OneBox?

The OneBox results are not always (I’d like to say frequently but should do more research before throwing that out there) aligned with the algorithmic results. For instance, you might be the third algorithmic result, but have no link within that OneBox.

So, not only does Google capture a prime position in OneBox but other sites rise or fall based on who appears in that OneBox. And it isn’t always clear how those decisions are made.

Just one example of this is to search for ‘diabetes insipidus’. There’s a Health OneBox that lists Google first, then Mayo Clinic, Medline Plus and WebMD.

The algorithmic results are quite different with Wikipedia and diabetesInsipidus.org as top results. Medline Plus (aka NIH) is actually ahead of Mayo Clinic and … WebMD isn’t in the algorithmic results at all.

That seems more of a concern than whether the top two mail providers are flip-flopped at the top of results. All in all, never a dull moment in search – and I wouldn’t have it any other way.

nicholaslatham

Sorry Danny but your argument about “So what do I get in a search for search engine on Google?” does not really work. You are on Google doing the search – there is no need to show bias by ranking Google first, as you are already there. The other searches relate to a different product (i.e. leaving search to go to mail, shopping, calendar etc) Would you rather they show a link where the searcher clicks to take them to the page they are already on??

Rather in this case to show bias, what I would do would be to downgrade the rankings of my nearest competitors and artificially boost the rankings of sites such as “Dogpile”, which are not so useful. The likelihood is that people would bounce straight back to Google.com.

Giacomo Banchero

it would be interesting to analyse the google beahviuor with the ugc from blogger, youtube, picasa…etc…

http://daggle.com/ Danny Sullivan

Philip, I think you draw the line – or at least pay more attention — to when Google is running destination sites rather than search engines.

If Google wants to offer shopping search, it just makes sense. Why wouldn’t you let a search engine offer a specialized shopping search. It’s like saying a grocery store can’t sell certain products that people would expect it to.

If you say no — that Google can’t buy or perhaps fully use verticals it has built — you’re limiting its ability to compete. Bing gets to have travel search but Google can’t? It makes no sense. It makes even less sense for companies that hope to be acquired.

When Google was prevented from doing a deal — not a purchase but just a deal — with Yahoo, suddenly the Microsoft offer for went from a value of $9 billion or more to “OK, Yahoo, you can keep 88% of what you sell.”

In the end, vertical search engines still send outbound traffic to others. Where more attention should be paid is when Google has destinations of its own. YouTube is a classic example. Are Google’s searches full of YouTube content because Google favors YouTube (as some assume) or because there’s a lot of content on YouTube (as Google contends).

AJ, the 19% is trivial, when you consider some assume it is much higher. But that number is really meaningless. I used it more to stress how difficult these numbers are in reality. Edelman can take these same stats and come out with something much higher. But the point is, to me, the fundamentals of the study are so flawed that none of the stats from it are valid.

Nicholas, it’s a perfectly valid and important question that shouldn’t get dismissed, as some do, with the “well, you’re already on Google” argument.

Know what’s a top query on Google? Searches for Google. Go hit Google Insights for search, put the number searching for either sex or Hotmail, both known as popular terms, alongside Google. Who are these crazy people searching for Google on Google?

It doesn’t really matter, however. Google’s result are supposed to provide the most relevant answer. There is simply no way that Google’s results for “search engine” are relevant when it fails to list the most popular search engine in the world — Google.

http://www.LivingInMacon.com Jonathan Rainer

Search Maps on bing.com and Bing’s maps do show first on my query. (this is the only one I tested).

Now, I have to ask: what in the world is wrong with Google maintaining a bias for certain results? Who cares? If I’m on Google and search for maps, I probably want Google’s map app. I’ve searched for Google voice before because it’s easier to search than figure what the exact URL is for it. I want Google Voice to show up first, not some random site in Russia. Chances are that if I want to find Yahoo’s email system, I’m going to go to Yahoo first and search there.

Great article, BTW. I know your point is to strike a balance against his biased analysis of the numbers, but it really bothers me that a Harvard professor is spending hours trying to figure out that Google is in business for Google, not for Yahoo and Bing. I’m a Realtor, and if someone calls me asking about a listing, I’m not going to tell them that another agent can show the property. I’m going to tell them I can show them property. It’s what I do. Google is better at mail, maps, and many things that it does (with social media sites the obvious exception). Those search terms should result in Google products ranking first.

http://www.rlmseo.com John Crenshaw

Great article. Who cares if Google is promoting their own products…on one of their own products? That’s what businesses do, and it’s a good idea…it helps make your customers aware of other opportunities to do business with you.

Why are people whining about whether Google favors itself over competitors? They are a for profit company, that’s what they’re supposed to do, and they’re obviously very careful about how they go about doing that. I think Google’s doing everything right on this front.

I’ve got to mention the reference to “google.com” not appearing in a search for “search engine.” I think the assumption there is that if you’re searching Google.com for “search engine,” you don’t want Google.com in your results…unless you’ve got some sort mental issue.

Chris Harvey

A couple interesting trends I noticed that it didn’t seem like you hit on (but I may have missed it given the late hour and my skimming some of this)

1) Bing frequently lists wikipedia as a high result. More frequently than Yahoo and Google, I should say. And there’s nothing wrong with that, per se, as people frequently will want to read a wikipedia article on whatever topic they happen to search for. But as a more relevant result than say, gmail/hotmail for email or popular video sites? I’m not sure that makes sense

2) When you exclude duplicate entries (by this i mean multiple results from the same domain taking multiple spots in the rankings) and compare google and yahoo directly, google doesn’t seem to rank itself first anymore than yahoo does, and in fact gives yahoo better standing than yahoo does google, not to mention the frequency of placement agreement (rss readers, for example)

Attend Our Conferences

Attend Marketing Land's SocialPro conference and learn fresh new strategies and tactics from some of the savviest brands and digital marketing agencies managing earned, owned and paid social media marketing campaigns across multiple platforms. Visit the SocialPro site to learn more..