Thursday, October 23, 2008

The Site

Experts-Exchange has been around for quite some time. Even today the site still gets a lions share of traffic. Alexa ranked the Experts-Exchange as one of the most visited 1500 sites for the first 6 months of 2008.

I personally stopped using the site years ago when you had to upgrade your account to a premium membership to view answers. What surprised me at the time was that the site continued to get great search result ranking. Only this week however did I finally discover what's been happening all this time.

Cloaking

How has the Experts-Exchange gotten such good search result ranking if they don't show answers to non authenticated users? Simple...they do show answers to anonymous users, just not all of them. Experts-Exchange targets specific search engine user agents and renders out a different page for them. This is a practice called cloaking.

Experts-Exchange very purposefully shows one page to the Google Bot and another page in entire to other user agents (say MSIE). This is a candid breach of the of Google quality guidelines and ideally would get you removed from the search index.

Shadiness in Action

You can taste the difference yourself by performing the following steps:

User Firefox to navigate to a Experts-Exchange question of your choice using a normal browser agent (you can inspect your user agent by looking at the request in Tamper Data. When I ran the URL out of stock Firefox the request looked like:The Response to this request is the following page (bottom of page shown).

Now use User Agent Switcher to change your user agent to Google Bot and make a request to the same URL. The request should look like:And of course you get a dramatically different page.

Free Answers

One might think that this is great, that you can get free answers from what's normally a paid service. The icing on the cake is that Google will even promote these answers to the top of their index based on content that only you the savvy user can see! Hurray for us (at least those who know how to step around the cloak).

Free answers are great right? It's worth mentioning that there's even more free answers to be had. For some URLs you'll see the answer anyways if you simply scroll to the bottom of the screen regardless of your user agent. Even if that wasn't availiable you could always try to dig the answer out of Google's page cache.

The unfortunate part is that this has been going on for years and Google doesn't really seem to care. Here's a site that's blatantly cloaking, and not terribly shy about it. There's not only Wikipedia entries that talk about this in detail, but blog entries going back at least a year describing this particular version of search engine abuse.

I'm positive that Google is aware that sites like this are showing specialized content to their crawler, but I'm blown away that they haven't done anything about it, especially given that this particular site has been doing it for quite some time.

When you see companies like this break the rules without any action from search engines, what's really there to make you follow the rules? Imagine if lots of sites started to cloak, trust me you'd get irritated in a hurry with the search results you'd get. Either way this is one problem I don't know how to fix. Either way at least we all know how to get the content we want off of the Experts-Exchange.

About Me

Tyler Holmes is a Solutions Architect working in Portland, Oregon. He lives mostly in the MS tech stack and is currently treading the waters of Communication/Collaboration and Business Intelligence with off the shelf/open source technologies.