If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

Spider - The browsers are like a program and to download the web page.

Crawler – The program is automatically to follow the links are web page..

Robots - It had automated computer program can visit websites. It will guided by search engine algorithms It can combine the tasks of crawler & spider helpful of the indexing the web pages and through the search engines.

Spider Scan and judge you website content ,Crawler crawl you content and index it in there searches according to Inbound links both techniques helps google to judge importance of website in terms of ranking ,robot.txt file is that file which is included into web coding so that google not consider it in crawling or to hide page from google crawling .

i see.Spider Scan and judge you website content ,Crawler crawl you content and index it in there searches according to Inbound links both techniques helps google to judge importance of website in terms of ranking ,robot.txt file is that file which is included into web coding so that google not consider it in crawling or to hide page from google crawling .

Spiders create a text-based summary of content and an address (URL) for each webpage.
Crawler is a program that automatically follows all of the links on each web page.
Robots is An automated computer program that visit websites & perform predefined task.

Crawler – a program that automatically follows all of the links on each web page.

Robots - An automated computer program that visit websites & perform predefined task. They are guided by search engine algorithms & are able to perform different tasks instead of just one crawling task. They can combine the tasks of crawler & spider together and help in indexing and ranking of websites on any particular search engine.

*** spam signature deleted by moderator - please do not put it back ***