How To Add Custom Robot.Txt Files On Blogger And WordPress Blog

Hey guys today I’m back with a new article of Blog SEO trick. Maybe you’ve heard about the term robots.txt. What is robots.txt? is it necessary in the settings? what if I leave it alone? There are probably many other questions you have, especially if you are newbie blogger. To better understand the meaning of robots.txt, to make it more understandable, I have made this tutorial. If you want to get your blog indexed and crawl your pages fast then you should add this custom robots.txt file in your blogger blog. as well as it’s a part of search engine optimization so you must be aware of the terms.

What is Robots.txt?

Robots.txt is a text file which contains few lines of simple code. It is saved on the website or blog’s server which instruct the web crawlers to how to index and crawl your blog in the search results. That means you can restrict any web page on your blog from web crawlers so that it can’t get indexed in search engines like your blog labels page, your demo page or any other pages that are not as important to get indexed. Always remember that search crawlers scan the robots.txt file before crawling any web page.

How It Work ?

Robots.txt is a command for the search engine robots to explore or browse a pages of our blog. Robots.txt is arguably filter our blog from search engines. Let’s say robot wants to visits a Webpage URL, example, http://www.technorelate.com/about.html. Before it does so, it will check for http://www.technorelate.com/robots.txt, and then it will access the particular webpage.

Step#3. Now verify your blog. Then click on Crwal>Robot.text Tester and you’ll get the Robot.txt file code.

How To Add Custom Robot Txt File On Blogger

Step#4. After that goto your blogger blog and click on Search Preference. Then click on Custom Robot.txt and paste the code here and save it.

Paste This Code

User-agent: Mediapartners-Google

Disallow:

User-agent: *

Disallow: /search

Allow: /

Sitemap: http://www.technorelate.com/sitemap.xml

How To Add Custom Robot Txt File On Blogger

If you have migrated your blogger blog then you Can Change Your Robots.Txt File by Two ways one is using CPanel and Another One is Using Plugin. I’m Here Going to say About Plugin Method because you are coming from Blogger You Nothing Know about how to add robots.txt in Cpanel.

Step#5. Now First thing you have to do is download Yoast SEO Plugin. Now Click Yoast SEO Menu In the WordPress List. After That you need to select Tools Option in the WordPress Menu.

How To Add Custom Robot Txt File On Blogger

Step#6. Now Now You Need to Select File Editor For Updating the robots text file. Now You have to give your robots.txt in the robots.txt Editor field.

How To Add Custom Robot Txt File On Blogger

Paste this code for WordPress

User-agent: *

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

sitemap: http://www.technorelate.com/sitemap.xml

Sitemap: http://www.technorelate.com/post-sitemap.xml

Sitemap: http://www.technorelate.com/page-sitemap.xml

Sitemap: http://www.technorelate.com/deals-sitemap.xml

Sitemap: http://www.technorelate.com/hosting-sitemap.xml

User-agent: *

# disallow all files in these directories

Disallow: /cgi-bin/

Disallow: /wp-admin/

Disallow: /archives/

disallow: /*?*

Disallow: *?replytocom

Disallow: /comments/feed/

User-agent: Mediapartners-Google*

Allow: /

User-agent: Googlebot-Image

Allow: /wp-content/uploads/

User-agent: Adsbot-Google

Allow: /

User-agent: Googlebot-Mobile

Allow: /

That’s it guys you’re done all steps for SEO. Hope you liked this post. Do not forget to give your suggestions and feedback. How much have you benefited from this post Please indicate by commenting.