Blog — September 19, 2016

Blog — September 05, 2016

How to Hide a Page from Google

By Pat Flynn on July 14, 2010

This is a short tutorial about how to hide or remove a page from Google.

Well actually, we’re not “hiding” anything from Google, we’re just telling Google not to index certain pages on our site so they don’t show up in the search engines.

Why would you want to do this?

There are some instances where you actually do not want content on your site to show up in the search engines. Let’s say, for example, you have a newsletter and it comes with a free eBook or a report. If you create a special page that has links to download your eBook, it will get picked up by search engine robots, show up on the results page for certain keywords, and people can potentially bypass your opt-in form to get the eBook that you’re giving away.

How do I know this?

Because it happened to me.

Two weeks ago, I typed in the name of my guide, eBooks the Smart Way, and the top result was the lead-in page with the opt-in form.

This is good.

However, the second resulting link was the actual page that had the links to download my eBook!

This is bad.

Although I’m not sure how many people downloaded my eBook by going straight to the download page, even if the number was zero, it’s just not good practice to have the ability to bypass the lead-in page and have access to a giveaway like this. This was definitely an act of carelessness on my part, and I should have known better.

And this is why I’m writing this post for you, so you don’t make the same mistake as me.

How to Hide a Page (or Post) From Google

Before You Publish

The best practice would be to not include the page that you don’t want to be included in the search engines from the very beginning.

In order to do this, you’re going to have to tell the search engine bots to “skip over” that page when going through your site. The easiest way to do this is to use a WordPress Plugin called Robots Meta.

This plugin has a lot of options as far as the metadata and backend stuff of your blog is concerned, but what’s nice is that on every new page and post that you create, you’ll see a little box on the side that looks like this:

If you select “noindex” the page will not be picked up by the search engines. The “follow” and “nofollow” stuff relates to the links that are on the page, which is a topic for another blog post.

If you don’t have WordPress, you can follow the more technical instructions on Online Tech Tips.

After You Publish

In my case, I had already published the page that I wanted to take out of the search engines. If you’re in a similar predicament, don’t fret – here’s what you have to do.

Note, you don’t need WordPress in order to do this.

Step 1: Follow the Steps for Before You Publish First

In order to remove the a URL from Google’s index, you must first “block” that page using the technique that I described above. If you’re on WordPress, use the Robots Meta plugin and make sure to change the setting to “noindex” and update the page.

Doing this alone will not change anything right away, so you’ll have to proceed with the following steps.

Step 2: Sign Up for Google Webmaster Tools and Verify Your Website

Go to Google Webmaster Tools and sign up for an account if you haven’t already. If you’re new, you’ll have to insert a short piece of code into the <head> section of your site, before the <body>, to verify that is indeed your site.

Step 3: Go to “Crawler Access”

Open the Site Configuration menu and click on Crawler Access:

Step 4: Click on “Remove URL” and then on “New Removal Request”

Step 5: Insert the URL (case-sensitive) that you want to remove.

Step 6: Set the “Reason” for the request to: Remove Page from Search Results and Cache and click the button for “blocked by noindex metatag” (this is what Step 1 was for). Then click “Submit Request”.

And there you have it. After that, you’ll see that the request to have that page removed was submitted, and after a day or two, you should see it in the “removed” column. Then, you can check to see if it’s in Google, and you won’t have any luck finding it.

Although you may not be able to use this tip right away, don’t forget about it in case you may have to use it in the future.

Kidding aside, this post’s a great eye opener for me. I was thinking about of what use this feature would be when I saw it on the internet. Guess I was just too closed minded, thanks Pat for opening my eyes.

Big hug to you and The Family

Patrick

Pat

Thanks for the hug! Hehe.

Glad I could help, and all the best to you and yours as well, Patrick. Cheers!

Its good for hiding any kind of “bonus” content that you want people to sign up for.

People shouldn’t go overboard though. Don’t “no index” everything when some of it could be used to actually promote your list signup. Eg, if you have autoresponders sending people to a series of pages on your blog containing a 7-part series of lessons, put a “If you’ve come here directly and want to get all of the lessons free signup here” type call to action on the page for people who drift in via search engines etc.

I like the idea of adding a call to action like that on certain pages.

Pat

Sweet tip Paul – not that I think about it, I’ve actually seen this before on other sites and used it to “get the full course” or go back and start from the beginning, if I came in during the middle of the tutorial, for example.

We learned about this the hard way with a customer. We somehow executed this on his home page and for a day or 2, his entire site was not showing up in Google. He ranks extremely high for his industry and he saw a big drop off in traffic. Safe to say, we undid the changes and thankfully everything went back to normal.

Great advice on this one though. Most people don’t know about this option.

Pat

Hey Justin – sorry to hear about the issue with the customer, but glad to hear everything is squared away. We learn as we go, don’t we! Cheers dude!

Pat,
You just saved me from having to write this type of post myself :-).

I recently stumbled onto a IM “guru’s” membership area by doing a simple google search for something else on the site.

When I contacted support they responded telling me that it didn’t happen, lol.
I emailed back screenshots of the membership area to prove that, in fact, it WAS happening and they may want to address the issue.

I have since been randomly performing this same search on the sites of fellow VA’s and am finding many whose premium items aren’t hidden.

I’ve shared a link to this post on my Favorite VA forum and have tweeted it to alert others to check their sites.

Wonderful info as always Pat.
Thanks,
~C

Pat

Christina! Thanks so much for sharing the post with your friends on the VA forums. I actually have seen quite a bit of traffic come in from that site (I visited it and saw your post – thanks!).

Thanks for the very timely post, I’m coming out with a free ebook this week, so I’m definitely going to hide my download page from google. I really enjoy your wordpress tips. Keep em coming if you have more!

Pat

Sweet – I’m glad I came out with this post when I did then!

I have a whole lot more tips to share – I can only post so fast! Hehe.

@Pat, have you considered password protecting the page that has the actual eBook? I think WordPress provides that function. If you do that even though Google indexes it, the users still need your password to get it.

Pat

Hey Owen – I don’t really like password protecting things. It’s an extra step that the customer or subscriber has to take to access the content, and I hate to have to make them do that. In some cases, it’s a must, but for this I’d rather not require a pw.

Your point is valid and another way of looking at it is that they are also willing to take the extra step to get the eBook (at least those who enter in the password) hence adding and additional means to pre-qualify your list

You have a valid point as well, Owen. But I also think that the password is a sign of something “forbidden” or “secret”, and there is nothing forbidden or a secret about a FREE e-book. So if the the reader pre-qualifies once and gets to the squeeze page, why would he have to pre-qualify another time having to type in a password? It just reduces your target group and potential return.

Hey there Pat,
While that case is clearly bad, another quick step you could take is to password protect the PDF (assuming it is a PDF). If you send an email out to the person after they purchase the eBook, you could include the password in the email.

Pat

Thanks Will, good tip, and very doable, however like I mentioned to Owen above, I don’t really like to make my customers or subscribers have to perform extra steps to access the content.

I have used this as well on sitelinks in Webmaster Tools when Google had added a page that I thought was a silly page to highlight. I added a noindex to it and now a more important page is showing in my sitelinks. You just never know what Google will do…

Pat

Ahh – great tip Richard. I think we should all look at what’s showing up in Google search for our keywords, and figure out how to manipulate them as best as we can, using white hat techniques, obviously.

Thanks for this wonderful tutorial Pat. I also have several squeeze pages on the internet, and I wasn’t aware of this. One of my squeeze pages is currently ranking number 1 with 43m+ results on “build your list fast” keywords. Of course I don’t want people to bypass my opt-in page. Good thing you came up something like this.

Thanks Pat!

Pat

Wow – awesome keyword rankings. Keep up the good work, and I’m glad I could help! Woot!

Another great way to utilize this is to block out search engines from indexing while you’re still working on the site. During development you may need to test out the website on a live server but if you forget to noindex it you’ll start showing up before you’re ready to launch.

Although your pages will be updated later it can still take time for everything to reindex on the page within search engine results.

Hi Pat,
I read this post the other day when reviewing my google reader and thought must remember that – and then today I had the very need to do this and so of course knew i could come back here and look how to do it – very useful – thanks!

Rick

When beginning your site there is a setting in WordPress under “Settings” called privacy where you specify you don’t want your site indexed – it also asks this during the install. Don’t forget to change after you are ready for you site to be indexed. Also if the page is already indexed you could move the file/directory to a new name or location and use the robot.txt file to prevent indexing under the new name/location. Might be easier than trying to get it out of the google index. Does the robots meta plugin allow you to block a whole directory or just files?

In order to prevent search engine bots from indexing your page, or from following links to your Thank You page/ a page you want to hide from search engine you will want to insert this HTML code into the head section (the area between the and tags):

When I do a google search for my website, I see some of my main menu items, some of my extra menu items, my blog, and some random posts. How can I get google search to only show my main menu items underneath the website?