Announcing crawler improvements for Live Search

Today we’re pleased to announce several improvements in the crawler for Live Search that should significantly improve the efficiency with which we crawl and index your web sites. We are always looking for ways to help webmasters, and we hope these features take us a few more steps in the right direction.

HTTP Compression: HTTP compression allows faster transmission time by compressing static files and application responses, reducing network load between your servers and our crawler. We support the most common compression methods: gzip and deflate as defined by RFC 2616 (see sections 14.11 and 14.39). Compression is currently supported by all major browsers and search engines. Use this online tool to check your server for HTTP compression support.

The following links provide configuration information for IIS, and Apache.

Conditional Get: We support conditional get as defined by RFC 2616 (Section 14.25), generally we will not download the page unless it has changed since the last time we crawled it. As per the standard, our crawler will include the “If-Modified-Since” header & time of last download in the GET request and when available, our crawler will include the “If-None-Match” header and the ETag value in the GET request. If the content hasn’t changed the web server will respond with a 304 HTTP response.

To check if your site already supports the “If-Modified-Since” HTTP header, you can use this online tool to check your server for HTTP Conditional Get support. Alternatively, you can check using Fiddler for Internet Explorer, or Live Headers for Firefox. Each of these tools allows you to create a custom GET request and send it to your server. You’ll want to make sure that your request includes the “If-Modified-Since” header like the following simplified sample:

If you have not yet configured conditional get on your site, we would strongly encourage you to do so, as it can significantly help reduce server load as most browsers and crawlers already support this feature (e.g. IIS, Apache).

In addition to these two features there are many more improvements in performance that should help further optimize our crawling. As a result, we’ve also upgraded our user agent to reflect the changes, it is now “msnbot/1.1″. If you think you are experiencing any issues with MSNbot, or have any questions about the updates, please use our Crawler Feedback & Discussion form.

Join the conversation

I am curious how this new design affects the data usage. Are there numbers known what the reduction is in percentages?

7 years ago

Anonymous

How can the Conditional Get be done when your site is delivered dynamically by Apache/PHP using mod-rewrite functions for creating static URLs? From everything that I can tell, each page is new at the time of request. Is this the intended result that the new MSNbot 1.1 is looking for?

7 years ago

Anonymous

These improvements arrive a little bit to late but at least they’ve finally arrived. I wonder when true competition in the search market will arrive too! For instance, I again ask you guys at Live Search: when are you (for "you" I mean G, Y and M) going to finally deliver a decent image search tool? Image search is a hundred steps behind text search in any engine. The one who delivers image-based (instead of text-based) image search will most certainly change the market rules… But I don’t see any SEO company or Search Engine Blog mention image (and video, and music, and…) as of strategic importance whatsoever.

7 years ago

Anonymous

thank you very much for this article

7 years ago

Anonymous

Google and Orbitz both use gzip compression to deliver compressed versions of their pages to HTTP 1.1-compliant browsers. Google.com has been compressing for a long time. This improvement comes to late for Live Search! BTW, what’s the compression rate? Google’s typical savings on compressed text files range from 60% to 85%, depending on how redundant the code is.

7 years ago

Anonymous

Normally how long does a site or a newly website submitted will be indexed via the MSNBOT crawler? It might be an interesting topics which i wish to discuss on my website.

Thanks

David Cheong.

7 years ago

Anonymous

Thank you for your improvements!

7 years ago

Anonymous

Thanks so much for this! This is exactly what I was looking for.

7 years ago

Anonymous

Great job,

But don’t get too comfortable now, there is still a lot that has to be improved.

7 years ago

Anonymous

Thanks for your guide in Conditional Get!

Regards,

7 years ago

Anonymous

MSNBOT crawler is currently supported gzip! Thank you for your improvements!

7 years ago

Anonymous

I find that our site has a lot of page crawled by MSNBot and shows as having high ranks. However, actual searches does not show them. What could be the problem?

7 years ago

Anonymous

The latest Crawler Improvements for Live Search move is a welcome move and should offer a lot of utility and ease to the web master in their endeavour to improve their website performance on both counts

First of improving the quality of their site thereby offering convinience and quality to the visitor.

Second To improve the content of their web site by increasing feedback that they receive from Live.com

7 years ago

Anonymous

Thanks for your guide.

Regard for you!

7 years ago

Anonymous

thank you very much for this article

7 years ago

Anonymous

Still a lot of improvement to be made, but this is definitely a step towards better engines running and better results for the users.

7 years ago

Anonymous

just trying to learn this stuff on my own

7 years ago

Anonymous

This is an excellent resource. Thanks

7 years ago

Anonymous

I made a office live web site..and submited it to all seach engines.. the only one that does not let me pull it up by the company name is yours ??? WHY is this since it is a microsolf site…..my company name is [ J. Orr Eq. Sales ] …if you can help me please do….my email is [ joescarts@live.com ]

7 years ago

Anonymous

This is an excellent tool. cheers,

7 years ago

Anonymous

Excellent tool, I’m glad we have MSN LIVE!

7 years ago

Anonymous

I’m newbie …. so, I will learn more

7 years ago

Anonymous

Time to look up related items and try and work out if I can do it. thanks for the update

6 years ago

Anonymous

Thanks for the article and compression checker. We have seen considerable improvement on traffic by leveraging compression. Also look at mushit.com for image compression. Very simple and effective tool

6 years ago

Anonymous

Great tool. Really useful and adds a little bit extra to help us improve our sites.

I am trying to get my website listed on MSN and they have asked me to insert a code into my website header for verification. I have done this and repeated it several times and they still keep saying the cannot find my site. Any ideas would be welcome. It is an office Live site (.aspx)

6 years ago

Anonymous

I don’t understand why a remote server said my site had an error???? The site is up and running

6 years ago

Anonymous

why can;t i find my web address

6 years ago

Anonymous

Laptop repair website with blog and forum

6 years ago

Anonymous

how this webmaster may enchanced my web

6 years ago

Anonymous

Is there a workaround if my host HTTP Compression is not enabled? Does it affect ranking if this features is not implemented on host?

Thanks,

Srednarb Group

6 years ago

Anonymous

I haven’t seen msnbot crawl yet my site

6 years ago

Anonymous

Great info. I am always trying to learn all I can about crawler bots to improve my seo. Thanks

6 years ago

Anonymous

This is good for me as webmaster, I’m looking good crawling….

6 years ago

Anonymous

it was really nice blog and more informatics about faster crawlers.

6 years ago

Anonymous

Hey man your crawler is not working … 1 month old url submission still not crawled.

6 years ago

Anonymous

This is very useful info for webmasters – at lease I got HTTP compression working now.

Thank you

6 years ago

Anonymous

I’m always up for a little performance tuning for my WordPress based sites, or all sites for that matter, so this was a good little article to stumble across. Thanks.

6 years ago

Anonymous

my site is up and runing for last 6 month, i added it to live search 2 month ago, still live search does not recognize my site? !!!

Dear webmasters, When blacking out my setup I find that some of the text doe’s not appear on the buttons Q.Is this a flaw with Win.XP or am I missing something ?

6 years ago

Anonymous

modifications are welcome…

But i always wonder why my site receives very less organic traffic from live as compared to Google?

What is wrong with my site in view of live?

6 years ago

Anonymous

Hi

Why my site very late to crawl by MSN Crawler ?

Thanks

6 years ago

Anonymous

i cant find my homepage in live search

6 years ago

Anonymous

It only crawls 1 post on my blogs if even that. I have hundreds.

6 years ago

Anonymous

Thank you for the improvements. This are good news for our servers! Keep it going!

6 years ago

Anonymous

Great to hear that Live search team is doing something now. Hope my site: http://www.easytourchina.com will get reindexed soon. It is a 10-year-old site which disappeared from live search for a couple of years.

6 years ago

Anonymous

Live does seem to index my site but I have yet to be able to get it to verify my site….No such problem with Google and Yahoo

6 years ago

Anonymous

YOUR BUSINESS IS VERY GOOD I MUST CONFESS

6 years ago

Anonymous

Is it possible to implement HTTP compression for PHP displayed pages? I mean if the pages are pretty much static, only loaded via PHP, what command could I issue using, I would figure, the header() command, in order to provide HTTP compression from PHP?

Victor

6 years ago

Anonymous

little msnbot in my site and i want to know why

thank you for msn live seaech

6 years ago

Anonymous

thanks for this articles..it helps a lot..

6 years ago

Anonymous

I used to be on the MSN Live search, but for the last month I am not, I wonder what the problem is.

6 years ago

Anonymous

Does anyone know if sitemap.xml is supported by MSN for sure?

6 years ago

Anonymous

Thanks for this article. i using mod_deflate.

6 years ago

Anonymous

Yes sitemap.xml is now supported by MSN

6 years ago

Anonymous

What’s it exacly

{

URL generates an ETag value: "1edec-3e3073913b100"

}

6 years ago

Anonymous

Thank you for the improvements. sitemap.xml is now supported by MSN

6 years ago

Anonymous

After read this article, I know that sitemap.xml support MSN. Thank a lot for this information

6 years ago

Anonymous

My site did’nt crawl to msn help me..

6 years ago

Mr.noom

Thanks. for your articles

6 years ago

mnlpn

getting 302 error… don't understand why

6 years ago

Anonymous

This section of article was quite informative.. give a clear idea about the subject.. really it was very helpfull

MSNBOT crawler is currently supported gzip — this is a good information too.. thanks

Thanks again

Sachin

6 years ago

Quality Directory

I will be pleased when Bing crawler and indexer improves like GoogleBot and Yahoo Slurp.

6 years ago

carlosspaul

Does anyone know if you can force "If-Modified-Since" in the header by using an .htaccess file as most people including myself can't afford a server ourselves with superfast broadband access and are forced to buy hosting? I'm using a wordpress blog. Thanks for any help in advance.

6 years ago

modesto

I see that many of you have had a problem with the Bot not indexing your site. One big problem with this may be that you are submitting the site through too many submission services. This is considered spamming. I never submit a site and all of my sites are indexed within a couple days.

6 years ago

Anonymous

Interesting article

6 years ago

Anonymous

thanks for this great article

6 years ago

Anonymous

well thanks for such a informative article

6 years ago

Anonymous

Thank you very much for this article.

Regards

James

6 years ago

autotoolshop

I am happy to know that. Bing crawled 25 pages of my website over the previous one year and I wish he can work harder.

I try to referencing an official site http://www.expert-mcth.fr on bing but bing are only in english and i think for the french market that not corresponding of the french law and market that make 1 month and the crawler of bing aren't pass visiting my site for me bing have a lot of problem for referencing french site.

For info expert mcth are one the official site of Afnic the french .fr registration on Legal entities