There are always more URLs than Google can fetch, so they try to get as many as possible without destroying your website. To do this they use a relaxed crawl rate.

Google increase the URL crawl rate slowly and see if response time goes up. If your site can’t handle the crawler they will not crawl much of your site.

Google checks Robots.txt only about once per day to help keep the load off your server? Having a +1 button on your site can override robots.txt? Both these points are interesting to me.

Google sets a conservative crawl rate per server. So too many domains or URLs will reduce crawl rate per URL. If you use shared hosting, then this could easily be problematic for you. If you do not know how many other websites are on the same IP-Address as you, then you may be surprised. You can easily check this by putting your domain or IP-Address into Majestic’s neighborhood checker to see how many other websites are hosted on the same IP-Address. If one shared site on the same IP has large number of URL and it is not yours, then you could be losing crawl opportunities, just because there’s a big site that isn’t connected to you in any way on the same IP. You can’t really go complaining to Google about this. You bought the cheap hosting, and this is one of the sacrifices you made.

Google crawl more pages than those in your sitemap but it does help them decide which pages are more popular.

If a CMS has huge duplication, Google then knows, and this is how it notifies you of duplicates on GWMT. This is interesting because it is more efficient to realize a site has duplicate URLs at this point than after Google has had to analyze all the data and deduplicae on your behalf. Google then picks URLs in a chosen order. One important to choose one page in comparision to other is Change Rate of page content.

Googlebot can be blocked from accessing your server, so you need to make sure your hosts have no issues or they will think your site is down. Biggest and smallest ISPs can block Googlebot at the ISP level. Because ISPs need to protect their bandwidth, the fact that you want Google to visit your site does not necessarily mean it will be so. Firewalls at the ISP may block bots even before they see your home page. They may (more likely) start throttling bits. So if your pages are taking a long time to get indexed, this may be a factor.

Strong recommendation – set up email notifications in Web Master Tool. Setup email forwarding on webmaster tools as a priority – this is very important so you don’t miss any error messages.

Make sure your 404 page delivers a 404 status – or it will get indexed which happens a lot. Soft error pages create an issue and so Google tries hard to detect those. If they can’t, they end up crawling the soft error as a crawl slot (at the expense of another URL crawl, maybe). So if you don’t know what a soft error is, it is when an error page returns a 200 response instead of a 404 response. You can use Firefox add-on Live http header to check this.

Google has to pick the best URL and title for your content. They can change it to better match the query. They then generate a snippet and site links. Changing them improves the CTR. It’s as if you are writing a different title for each query.

If server spikes with 500 errors, Googlebot backs off. Also, firewalls etc can block the bot. This can after a few days, create a state in Google, that says the site is dead. If Googlebot gets 503 error on robots.txt they stop crawling. Be careful, if only some part of your site is offline, do not to serve a 503 on robots.txt.

For displaying result, Google needs to:Pick a URLPick Title: Usually Title Tag, sometimes changes tag based on user query. This is win win for everyone.Generate Snippet: Will create stuff on page, but strongly recommends using Rich Snippets.Generates Site-links: depends on query and result as to whether this appears. If you see a bad site-link issue (wrong link) check for canonicalisation issue.

Inspired by the Sultan's Elephant, an interactive show featuring a mechanized elephant, the massive robot looks surprisingly lifelike aside from a few nuts and bolts and some joints at the trunk and legs showing.

The 12-meter high x 8 meter wide elephant was pieced together using 45 tons of reclaimd wood and steel.

When the majestic animal goes out for its walk, it is like architecture in motion departing a steel cathedral. The 49 passengers on board embark on an amazing journey on the Ile de Nantes. Each time the pachyderm goes out, it is a unique spectacle for everyone to enjoy.

From the inside, the passengers will be able to see the moving gears that power the legs. They can make the elephant trumpet and control some its movements, thus becoming truly a part of the Machine. On the back of the Elephant, it’s like being on the 4th floor of a moving house, with a breathtaking view of the banks of the Loire River. In this time-travelling carriage, the passengers can voyage to the imaginary world of Jules Verne in the city where he was born.

About Me

DS is an IT Professional since 1984 (when he joined Computer Science at J K Institute of Applied Physics, Allahabad University, India).

He did Preliminary studies (10+2) from K V HFC Barauni (Bihar, India). He studied Computer Science from Allahabad University. He also, studied practical application of Computer Science from IERT (Intitute of Engineering and Rural Technology, Allahabad). Did some Computer Science teaching in Allahabad & Varanasi for one year in Private Computer Institutes. Developed and sold some business software through Datapro Information Technology in Varanasi.

He Joined Nirula's (Fast Food & Hotel Chain of India) in December 1991. In Nirula's he developed several software related to Hotel Management, Production Planning & Management, HR Management etc. Managed the entire IT Infrastructure from ground up. There he got developed the first eCom website of India.

He joined IndiaMART in 2003 as Manager Technical and Managed the Production Servers (Linux). Developed expertise in Apache and Qmail.

Joined Monster.com India in 2005. Built the complete Server farm Infrastructure from ground up for APAC region.