Yahoo, MSN Censor More than Baidu

China unblocked many usually censored web sites following intense international pressure and scrutiny after having promised uncensored access during the Olympics. Five days later (August 6, 2008) I tested the search engines that Google, Yahoo! and Microsoft customize for the Chinese market as well as the leading domestic search engine Baidu. I found that all of the search engines were still censoring content that was unblocked by China. one interesting find was that Yahoo! was censoring less than all the others and Baidu (and Google) were censoring much less than Microsoft.

For purposes on comparison Google and Microsoft make a good match because both have to de-list web sites form search results while Yahoo! and Baidu index form within China and thus do not (usually) index sites already censored by China. (For more read my report on search engine comparison.)

Now over a month later things have changed. While these sites remain accessible in China some are still censored by the search engines. Google has dropped to only censoring two sites and is now censoring the least amount of content. Baidu is next with three censored sites. Microsoft remained steady, but Yahoo! has shifted from censoring the least amount of sites to the most!

The divergence between Yahoo! and Baidu is very interesting. If both crawl from within China and are subject to China’s filtering why is Yahoo! censoring so much more than Baidu? It could be that the conclusion that Yahoo! and Baidu do not de-list content is not fully accurate. If the sites are accessible in China then Yahoo! is likely de-listing the sites. Because of the suboptimal method of censorship notification employed by Yahoo! (a standard disclaimer on every page regardless of whether any of the results are censored or not) I cannot fully distinguish between sites that are de-listed and sites that have not been indexed (e.g. because China blocks them).

I’m still struck by the fact that over a month later sites that are available and uncensored in China are still censored by these search engines.