How To Optimize Robots.txt in WordPress For Search Engine Bots

In the last SEO post we mentioned about Robots.txt file and how it plays a major role in Search Engine Ranking. It tells the search engine bots which important part of our blog to index and what not to index. This is why you should properly configure a Robots.txt file otherwise any error on your part will result in a poor SERP result.

What is Robots.txt and why should every WP Blog use it?

A Robots.txt is a simple text file in the root directory of the domain. It contains rules for proper indexation of your website. Basically the file gives a search engine information about which parts of your site to index and which part they should not go to.

With the start of Panda Algorithim, duplicate content or low quality sites were being punished and therefore any initial or low quality content should be noindexed with the help of a robots.txt file.

Even if your website is not complete, then you can configure robots.txt to not index your website and once completed you can remove the noindex tag from your development website.

Do I Really Need a Robots.txt File?

It is certainly not necessary to have a Robots.txt file.Your website/blog will be indexed even if you do not have a Robots.txt file. WordPress itself has a virtual robots.txt file created in the root directory.

But having a physical Robots.txt file is beneficial as it provides a way for search engines to index only those portion of site which are deemed important and Bing,Google,Yandex and other high quality search engines will honor the robots.txt file. However low quality search crawlers might just disregard your robots.txt file completely.

Where to Find Robots.txt

It is very easy, you will find robots.txt file in the root of your WordPress installation. If you are still having a static file on your server then you must be having a robots.txt file in the root folder of the domain.

If you do not have a robots.txt file in your WordPress installation then using some of the plugins might help you in creating a Robots.txt file. If you yourself want to create a robots.txt file then you should just follow the syntax of robots.txt. If you have a multiple domain hosting then each of your domain directory should have it’s individual robots.txt. One main domain will not work for all the domains.

The robots.txt has its own syntax to define rules. These rules are also called “directives”.

Search Engine Bots or Spiders

Search engine bots are also known as spiders and some spiders are good while others are bad. Some follow the Robots.txt directives while others do not. Full List of Search Engine Crawlers who do not follow Robots.txt Directives and are called Bad Bots.

As per last report from InCapsula published in 2014 only 27% of search spiders are good bots while 29% of all the website traffic is done by bad bots which might be some impersonater, or hacking tools , content scrapers and spammers.

Out of the 27% traffic which good search engine bots crawl be sure not to block any one of the bot because your traffic will get down if you block any good search engine bots.

How to Create Robots.txt ?

Robots.txt is a simple text file placed in the root directory of your WordPress installation. You can create Robots.txt by following ways

Using a SEO Plugin like Yoast SEO

On Your Desktop

Using a SEO Plugin : We will be using Yoast SEO to create a robots.txt file. The default WordPress robots.txt files looks like

User-agent:*Disallow: /wp-admin/ Disallow : /wp-includes/

You can copy this and make them spider or search engine specific robots.txt. Like Googlebot-Mobile for Mobile search , Googlebot for desktop search , Googlebot-Image for Image search , Mediapartners-Google for Adsense ads etc. Just place the name of specific robots in the user agent section above.

1.Open up your Yoast SEO Options and Look for Tools and then click File Editor.

Using Desktop PC : You can open a normal text file and rename it as robots.txt. Enter the robots directives and the save the file. After saving the file upload the same file via FTP or any other means to your the WordPress root directory.

Robots.txt Sample File

Here is the sample Robots.txt file which you can use on your WordPress installation.

Share this:

Related

About Rajiv

I am a Technology Geek and founder of many websites. I am passionate about Android, Windows, and Electronics.I am currently Working as a part-time Freelancer & a full-time Blogger. If you have anything to be written just ping me.