Craigslist reportedly blocks search engines to kill competitors

Craigslist seems determined to destroy sites like Craiggers and Padmapper.

A third-party site that relies on Craigslist data has accused the classified giant of changing its policies in an effort to kill it off. Craigslist has reportedly told major search engines to stop indexing its users' ads. That could leave 3taps, which operates the site Craiggers and re-publishes the data for use by third parties such as Padmapper to use, out in the cold.

"At approximately noon on Sunday August 5, Craigslist instructed all general search engines to stop indexing CL postings," says a statement on the 3taps homepage. Because 3taps had been harvesting Craigslist data from cached copies offered by Google and Bing, the move to block search engines means Craigslist is "effectively blocking 3taps and other third party use of that data from these public domain sources," 3taps claims.

"We are sorry that CL has chosen this course of action and are exploring options to restore service but may be down for an extended period of time unless we or CL change practices," the company says.

3taps seems slightly confused about when the block started. Its website states it began on Sunday, August 5, but the Verge noticed a tweet, published early Tuesday morning, that says Craigslist changed its policy on Monday. We've asked 3taps for clarification and will update when we have it.

The search engine block would be only the latest step Craigslist has taken to shut down competing services. Two weeks ago, the firm filed suit against 3taps and Padmapper, accusing them of copyright and trademark infringement. Last week, Craigslist beefed up its terms of service in an apparent effort to shore up what experts have told Ars is a flimsy legal case.

But the lawsuit would become all but irrelevant if Craigslist prevents third parties from crawling its site in the first place. That would also be a serious loss for users who prefer the more sophisticated interface of sites like Craiggers or Padmapper to Craigslist's "old and busted" look.

We've asked Craigslist for comment and will update the story if they respond.

Update: A couple of readers have pointed out that Craigslist has set the NOARCHIVE robots meta-tag, but not the NOINDEX tag. So Craigslist posts should still show up in search results, but 3taps will be deprived of the cached copies of those posts it was using to build its own database.

So these services were harvesting the data not from craigslist directly, but through archived copies of google and bing.

I really have little sympathy. They were building a business without a licensing agreement to secure the data or the business model. Craigslist is in the green on this mater. If it was important to these 3rd party vendors they should have worked out licensing with Craigslist directly.

It like complaining that you cant see movies for free because your neighbors shut their curtains.

It's sad to see tech companies act like old companies - so afraid of innovation that they stagnate. And when others try to add value, they make it hard to work with them. Looks like the stage is set for someone to take this away from them.

I really have little sympathy. They were building a business without a licensing agreement to secure the data or the business model. Craigslist is in the green on this mater. If it was important to these 3rd party vendors they should have worked out licensing with Craigslist directly.

Except until a quite recent TOS change, Craigslist didn't have copyright ownership of the contents of user posts so they have no right to license or restrict.

I like the CL format. They spare us useless graphics. CL was mobile ready before mobile happened.

The CL parasites have nothing on Facebook, which allows users to steal images from other websites. I see Facebook theft in my server logs. Facebook legal is a dead end. Unless you sue them, they don't reply.

People who aggregate without pushing the viewer to the site is a parasite. Google News when you go to the News sites it actually pushes you to the site where the content is, which gives them advertising eyes. For these companies I have no sympathy. Maybe they can work something out with Craigslist with a change in method.

Padmapper was the best way to find an apartment in Portland. Every other service I could find was terrible. I really hope this gets resolved, because you know CL isn't going to offer this functionality. They haven't upgraded their site functionality in EVER.

While Craigslist is well within their rights they either have the business acumen of a rock or they are crazy like a fox. I can't figure out which just yet.

There is, without a doubt, quite a bit of pent up demand for improved services. These parasitic services provide a tremendous opportunity for Craigslist to monetize variable levels of service at extremely low risk and cost.

But instead they choose to simply shut them out and continue to sit on their hands... Unless, this activity is simply a prelude to forcing licensing deals that are extremely favorable to CL.

My gut feeling is Craigslist just wants to stick to their free semi-agrarian utopia circa 1999, until they are eclipsed with someone with a better interface and an actual business plan.

Craigslist is turning into one of those big companies that just don't 'get it'

They are acting exactly like the huge corporations that horde 'their' data and cause people to start using other services because they become more convenient and useful.

Someone needs to start up a competing service with an included API & common sense TOS.

Craigslist are run on the whim of Craig. He doesn't give a shit about being a huge corporation. He's a total weirdo and not a huge corporation.

What's it say about huge corporations, when their behavior is indistinguishable from total weirdos?

If they're huge corporations, it says that they're probably not doing as poorly as people on Ars say. And since Ars commenters think they are better at everything than everyone else, it's important to keep in mind.

That these sites use Craigslist at all is a pretty big compliment to a service that has refused to innovate at all for years. This is a great opportunity to make some friends and grow the Criagslist user base, but it seems they would rather keep the path clear for the Next Craigslist to rise up and replace them.

I personally avoid using Craigslist because I don't like that their interface so favors recent posts that you have to break their terms of service and essentially repost every day, if not multiple times a day, to get any exposure. Minimalist is nice but broken community is not.

Craigslist is turning into one of those big companies that just don't 'get it'

They are acting exactly like the huge corporations that horde 'their' data and cause people to start using other services because they become more convenient and useful.

Someone needs to start up a competing service with an included API & common sense TOS.

Craigslist are run on the whim of Craig. He doesn't give a shit about being a huge corporation. He's a total weirdo and not a huge corporation.

What's it say about huge corporations, when their behavior is indistinguishable from total weirdos?

If they're huge corporations, it says that they're probably not doing as poorly as people on Ars the Internet say. And since Ars Internet commenters think they are better at everything than everyone else, it's important to keep in mind.

Their site, their rules. Don't like it? Use something else. If something else isn't good enough, use theor site. Dunno what all the uproar is about. I have successfully used CL for about a decade for everything from finding rescue dogs to finding a job and apartment. The site has always worked for me just fine.

So these services were harvesting the data not from craigslist directly, but through archived copies of google and bing.

I really have little sympathy. They were building a business without a licensing agreement to secure the data or the business model. Craigslist is in the green on this mater. If it was important to these 3rd party vendors they should have worked out licensing with Craigslist directly.

It like complaining that you cant see movies for free because your neighbors shut their curtains.

You might have some kind of a point if Craigslist showed ads on their site, or anything to do with pageviews. They don't.

I'm curious about what Craig's motivations are. It isn't money, and that shouldn't be viewed with incredulity.

I remember there was a Craigslist searcher about a decade ago that eliminated geographic boundaries for searches. It was very handy for finding rare cars. Craigslist shut it down because, if I recall correctly, it "ruined the entire purpose of Craigslist--to create local communities of people."

Craigslist is useful, intrusive, free, and has been for years. They can do whatever they want. Love that site.

Timothy B. Lee / Timothy covers tech policy for Ars, with a particular focus on patent and copyright law, privacy, free speech, and open government. His writing has appeared in Slate, Reason, Wired, and the New York Times.