The Moz Blog

Craigslist Blocks Search Engines

According to a thread at SERoundtable, a few million results in Google (and other search engines) from Craigslist have dropped out due to a change in the robots.txt. The change must be very recent, as I had used Google to search CL just a week before the holidays. It's an interesting strategy from the classifieds giant - If anyone knows the reason, please do tell.

8 Comments

My guess would be lag between indexing, ranking and someone finally finding a page in the SERP. By that time the classified ads were usually expired, pulled, etc.
Also I believe CL is going to start focusing more on advertising. They have a $125M per year advertising model but only charge %25 of their advertisers a fee. Now what better way of advertising and monetizing your users than by putting more clicks between them and what they are looking for. More impressions = larger ad capacity. Enough users know them to just go directly to the site.
-PK

Put this in the "don't believe everything you read" column. Danny's investigation reveals that nothing has changed and the listings are still in the major SEs. The robots.txt isn't actually blocking anything new.
Thanks emad

It was inevitable, in my opinion. I'm sure a lot of people have been spamming Craigs List to get inbound links (which were, in my opinion, beneficial for sites where Google and Yahoo! were concerned).
I have not depended on Craigs List for linkage, but I did test a couple of site links and the improvement in rankings was measurable.
By removing themselves from the search indexes, Craigs List ensures that their content will (mostly -- there are many clueless link-spammers out there) stay reasonably honest.
I'll continue to use Craigs List to announce local events and news, which is as it should be.

I'm sure this is a defensive move. With Google (Google Base) and Microsoft (Freemont) poised to enter the classified ad space they are looking for ways to maintain their #1 position. If Google or Microsoft indexed Craigslist and include the listings in their own classified site it will be tough to compete with the money and visibility that the search engines have. Craigslist must be betting that they can hang on to their users without generating new search engine traffic. It's a bold move and one that I think will help them maintain their lead for longer but will ultimately prove to hurt the site. Personally, I found Craigslist through search engines.

Hopefully we will hear it from the horse's mouth directly.
Craigslist is such a powerful site and works so well on its own. Doesn't need to be in the SE's. But lets wait and see what they have to say on the matter.
Dave

I am very glad to hear this. While not a major problem now, during the Jagger update, craigslist posts started to rise to the top for some fairly competitive key phrases. This will prevent it. If only ebay would do the same thing....