It can take weeks until your site or parts of it get crawled and indexed, but sometimes it can take a few days, so you want to make sure that you block search engines from the start.

Search engines are constantly crawling the web, so it’s wrong to believe that you need backlinks or you need to submit your website to Bing and Google Webmaster Tools in order to get crawled and indexed.

How to block search engines

1. Enable a WordPress setting

This adds the following code into your robots.txt file:User-agent: * Disallow: /

And it also adds this one <meta name='robots' content='noindex,follow' /> in your website’s header.

Note: If you somehow already have a robots.txt file, WordPress will not replace it or modify the code in it after you enable that setting.

Now, you might have noticed the note made by WordPress: “It is up to search engines to honor this request.”.

Even with the robots.txt and meta robots tag, search engines may still crawl and index parts of your site, so this method is not always 100% safe.

2. Password protect your website through cPanel

If you password-protect your website, you can be sure that search engines won’t index it, because web crawlers can’t access content in password-protected directories.

Let’s see how you can do this…

1. Log into your cPanel and look for Password Protected Directories:

2. After you click on it, a pop-up will appear, where you have to select the document root:

3. Press Go and then select the folder where your WordPress is installed (usually public_html):

4. After that, check the box next to Password protect this directory, and give the directory a name. Press Save and then go back (there’s usually a link that sends you back) to create the user.

5. Insert a username and a strong password, and then press Add/modify authorized user.

You’re done!

3. Password-protect your website using a plugin

An easier way to password-protect your website and block search engines, is to use a plugin, such as Password Protected or Hide My Site. The second one is a bit outdated, so we’ll just take a look at the first one.

Install the Password Protected plugin and activate it. You can find it under Settings. Here’s how its settings look like:

Conclusion

If you wish to have a nice and clean start, then you should definitely use one of those methods to block search engines from crawling and indexing all the mess that exists before launching your WordPress site.

Hope you enjoyed the post! If you have anything to say or ask, please drop a comment!

Share the post to help out others

4 Comments

This is a valid reason for discouraging your WordPress site but there is another you should really consider: Your site is ment for private or restricted use. Examples; Corportate intranet, some sort of private social network, a fancy private photo sharing network. Im currently building a theme for administrating leases by real estate agents, so this come in handy.

Just after writing the post, I stumbled upon someone asking for help in a community. He has a copy of his live site, which he uses for testing. He was worrying about duplicate content, and he was right to be worried.

He said that he blocked search engines using robots.txt, but the site was still appearing. So, this is another reason to password-protect a website; if you are using a copy of your live website for testing and troubleshooting purposes.

I had learn lot of new things while reading your post. Also, it opens up my mind by the possibilities when it comes to search engines. Have you encountered spiders trying to access your website using meta robots?