Whenever I take part in a Site Review session at some conference, one of my fellow panelists will inevitably tell a webmaster something like, “You don’t have an XML sitemap. Create one and submit it to the search engines as soon as you can.” I’ve yet to have the opportunity to play devil’s advocate on that during a session, but I’m going to do it here on SBS. Because I believe that XML sitemaps are the most overrated SEO tactic ever.

XML Sitemaps: This Week’s Hot Topic

Coincidentally, XML sitemaps are the hot topic this week on the SEOweb:

On Search Engine Roundtable, Barry Schwartz wrote up a great article asking if sitemaps are important for SEO. He points to a Google Groups discussion where a Google employee touts the benefits of sitemaps. Barry’s post went hot on Sphinn.

Those are all good and interesting discussions, and it’s worth the time to read them. But I don’t agree with the “sitemaps are a great idea” crowd. So, why am I calling sitemaps the most overrated SEO tactic ever?

XML Sitemaps Don’t Solve Problems

I’ve done SEO on sites as small as 4-5 pages. I’ve done SEO on sites with 15,000,000+ pages. I’ve never once recommended the site owner create an XML sitemap and submit it to the search engines. Sitemaps don’t really solve any problems where indexing and crawlability are concerned. Let’s use a typical 100-page site as an example:

Crawlability Problems

If you have a 100-page site, and the spiders can only access 25 of your pages, fix your crawlability problems. Using a sitemap to solve crawlability issues is never a good idea. It’s like putting a band-aid on your chest after open-heart surgery. You need a lot more help than that.

Indexing Problems

If you have a 100-page site, and the search engine has decided that only 25 of your pages are strong enough to be indexed, forcing the other 75 pages on them via a sitemap isn’t going to help your cause in any way. It’s not going to improve your overall site strength or make your site profile look any better. The solution to getting those 75 pages indexed isn’t to spoon-feed them to the search engine; the solution is to make those pages better by improving the content, acquiring better/more links to them, and so forth.

No Problems?

If you have a 100-page site, and the spiders are able to crawl all 100 pages, and all 100 pages are indexed, and life is good … maybe you’re thinking a sitemap is a good complement, or something to do “just to be safe.” Why? If life is that good, you don’t need an XML sitemap. Let the spiders keep doing what they’re doing; let them crawl through your pages, let them figure out which pages deserve more frequent crawling, and which don’t. Don’t get in their way, and don’t steer them off track with a sitemap.

My attitude towards sitemaps is pretty much in line with what Barry wrote in his SER article above. He said, “I have always been a believer that well on-page optimized sites do not require or even benefit much from Google Sitemaps.” That’s exactly it.

But that’s not all. I’ve seen a case where not only did the sitemap not solve any problems, but it was also the cause of the problem.

XML Sitemaps Can Cause Problems

I’ve been helping a friend recently with a highly-trafficked and popular blog. This is a blog with great authority in its niche. It’s been around for several years, has tens of thousands of inbound links, and has all kinds of trust.

But it also had a problem: Referral traffic, specifically from Google, had dropped off the charts. Google had stopped indexing new posts. The most recent article in the index was a couple months old. We couldn’t figure out why. But the facts were clear: Google was suddenly sending a fraction of its normal referral traffic and not indexing new posts. To be frank, I’m not smart enough to know what was wrong. I still don’t know why all this happened.

My suggestion? Take down the XML sitemap. I figured we had nothing to lose. We deleted the sitemap from Webmaster Central. We pulled it off the site. We removed all links to it. It was gone. Nuked.

Results? Eight days later, here’s what I emailed to my friend:

New posts were being indexed quickly by Google, within an hour of posting.

Old posts that had fallen out of the index were back in.

Total pages indexed in Google were at the highest level since we started working together.

Referral traffic from Google was up 136% from the week before.

Granted, my experience is purely anecdotal. What happened in this case may not happen in the next case. Maybe it was pure coincidence! (I doubt it.) We’re all biased by our own experiences, and my experience is that sitemaps are completely overrated when it comes to SEO. They don’t solve problems, and can cause problems. Your mileage may vary.

Additional Resources

I mentioned that sitemaps are a hot topic this week. In addition to the links above, here are two that also question the need/value of XML sitemaps:

I’ve always felt that having one didn’t do much if anything, but having them actually hurt you doesn’t seem to make any sense to me. It’s certainly possible and I’ve heard from others the same view, but these examples always leave me wondering if it was something else that caused the drop/jump in indexing & ranking.

Hey Matt- great post. I often have to tell folks NOT to do an xml site map or to take down and existing one on a site that is having problems with indexing.

Natural crawlability is crucial. So, I simply upload the verification code and then let Google crawl the site. I can then use Webmaster Tools to see what Google finds in the crawl and work to correct it.

Many web masters think they should only put what they deem to be important pages on an xml site map, instead of all of the pages that should be indexed. Sometimes, they put absolutelyeverything on an xml site map, like pdfs that are duplicates of web pages. Others throw one up and then ignore it. Then, when pages are added or deleted, we can have a real mess.

I think an xml site map is so easy that everyone wants to do it and charge their clients for doing it, instead of rolling up their sleeves and doing the hard work, instead.

I do use sitemap on my blog and have it automated through a plugin. All my pages have been indexed till now – but still it’s early days.
Everywhere I see people recommend having a sitemap. This is a good alternative way to look at sitemaps.

MadHat – I sat and studied that site for a couple weeks, and indexing had fallen off for a couple months. And then we yank the XML sitemap and all is well. No way you’re gonna convince me it was pure coincidence.

I can’t believe it! A couple of months ago I had a simmilar experience. A pretty old website (3 years old) unexpectedly stopped being indexed by Google (for 3 weeks). No big changes on it: just content updates. I decided to remove the XML sitemap and crawling started 2 days before. Coincidence?. I don’t think so either…

A quick pointer on the problems – Sitemaps was always billed as a way to help Google find if content had been updated. If was updated, it was a signal for Google to crawl it – if not, a signal to Google to not crawl.

Therefore saving Google money on bandwidth.

In fact, I think the USP for Sitemaps was always that it saved Google bandwidth, little else.

Our sitemap currently has a “warning” basically stating that Googlebot is having issues with 301 redirects one some of our pages. I do wonder if this warning is occurring because of the sitemap, since I didn’t think 301’s were an issue for Google.

Typically, I would agree with you– however on certain larger sites with international audiences there may be a good use for XML sitemaps.

A particular client of mine (very large international organization) has a country drop down menu. Creating a country specific sitemap for country folder /us/ /can/ and submitting them to Google Webmaster Central and specifying the content for a specific country has been a tremendous help in making sure the right pages appear or the right audience. (ie– UK users don’t see US or Australian pages)

I know there are 100 different ways to deal with this same problem– but the client had other issues in stake (brand etc) which kept us from utilizing any other method. XML sitemaps worked.

Of course, your primary audience is small business owners and this won’t affect them at present

I wish you would have saved the sitemap, Matt. It’s a VERY SPECIFIC directive, and can indeed cause trouble if it is not accurate. But if accurate, it should do no harm. We could have examined it.

As noted, the sitemap is designed to save Google time and effort. If yours says “don’t bother – most of my stuff is the same…still” it may also be an indicator of value, for trendy content. Google will use what it has available. The key for us, is to make sure Google gets what we want it to get, so it makes the deductions we want it to make.

Great post Matt. I’ve always suggested to my clients that they / we create an XML sitemap and of course distribute it to all the necessary search engine hot spots. However, you’ve certainly raised several interesting points with this post. You’ve inspired me to treat each client on a case-by-case basis as far as who to suggest a sitemap for… especially when working with well established websites.

Interesting post…it is certainly one side to the sitemap discussion. The actual =/- of sitemaps can be debated but for me the benefits far out weight the negatives. The stance I would suggest taking when it comes to sitemaps would be why not? IF it took a lot of resource to build one I would maybe think twice but with all the free tools out there….

When you look at the bigger picture, the introduction of xml sitemap was for indexing and regular crawlability. What happens when such pages are not properly structured? So first things first..Build a site that can be easily followed and the issue of a sitemap might never arise. But for some sites an xml sitemap can be helpul..

I agree with that. I never used sitemaps, instead I prefer to optimize internal linking. If necessary I like to use an “onsite sitemap” on a sepreate page or in the footer, which also serves as an overview for users (even though most users don’t pay attention to footer links).

There has been so much talk about using a sitemap to help google help you, so I went with it. There are usually too many variables involved to determine the direct effect of any particular change, but I noticed an increase in indexed pages at the time.

Interesting discussion. I was worried that by not having an XML sitemap on my blogs, i was losing out on something special.

But i see now it is not a big issue. This article has alleviated some concerns. Plus i wouldn’t know what to do with crawl problems anyway. Anothing blogging technicality to look into, “crawling and indexing problems”. Maybe you can write a how-to post on this, Matt.

I’d be happy to provide you my GWT crawl results for you to do a case study on. Just let me know.

By the way, Matt, I wonder if you would consider writing more articles on those controversial subjects, such as duplicate contents (especially I have read that google would consider http://www.youdomain.com and http://yourdomain.com and are duplicate contents (or sites?)) In that case, all professional seo should be a standard procedure to choose between www. and no www. and do some sort of redirect or what best in the very very beginning when hosting a site.
I seldom see it in any recommended seo practice.

I originally thought no seo professional with logical thinking would disagree with Matt,but in fact a lot of others still said: Why not? It is a recommeneded seo practice by SE and it would take too much of the time.
I also found that a few of these kinds of seo practices are just useless, such as the Revisit tag, Google never follows this tag – it certainly has its own alogrithm to determine how frequent it would browse a site for update!

I’m calling “context” on this one. In a lot of cases, yes, they’re overrated. Nothing beats “natural crawlability” (love the phrase, btw!). BUT- If you build dynamic websites via a CMS, and your organization is adding / updating pages all the time, having a list of “This one got updated, this one got updated, this one got created” that a spider can look at and go straight to the new stuff means that it won’t waste YOUR bandwidth by crawling your entire 10,000+ page site when only 20 pages have been updated since the last crawl. It also means that those 20 pages WILL get analyzed the next time Google visits your site, in case the spider is set to only crawl, say, 1,000 site pages per visit.
I’ve noticed an honest net positive effect of xml sitemaps, in that new pages get indexed much faster using them (I have mine set to be automatically generated upon request for all my clients, so any call to the sitmap gets a completely fresh snapshot of the website).

I’m not discounting your rankings-drop experience, however. While I never experienced that, evidence does point to it being the result of an xml sitemap issue on your part. I do wonder if google crawls the sitemap instead of links on the site pages, instead of “in addition to”… Something that will hopefully be answered in the google group.

From my experience, there is plenty of good reason to run an XML sitemap. It’s not a make or break for sure, but it can be advantageous.

Of course if it is breaking something it’s worth removing, fixing the problem and then potentially putting it back, but to tell people NOT to use it for the sake of some issue that arrose andofr which you never had the real reasoning/answer I think is irresponsible.

With the XML sitemap generator for WordPress I am able to get my content into the Google index in as little as 4 minutes, so that’s not to be sniffed at.

Natural crawlability is more than just a sitemap sure, but at the same time it just like anything else in SEO. No one part in itself is make or break, but SEO is about the sum of all parts together.

We are doing an experiment. We have 20 sites of various types and size using the XML sitemap protocol in conjunction with Google Webmaster Tools and we have 20 sites not doing so.
Both groups undergo the same exact SEO tactics and operational scrutiny.
We hope to determine if the sitemaps are a waste of our time and bandwidth or viable and valuable.
We are only 11 days into the experiment. I hope to have some quantifiable results in 30-60 days.

To be honest, I am of the opinion that this whole post was written up with linkbait in mind. I have worked on many sites in my time, most of them large corporate sites, not once have I ever seen the addition of a sitemap cause a loss in traffic. Quite the opposite.

The first thing I do is always the creation and submission to Google of an xml sitemap which is automatically updated when new content is added. Why?

Because it gives an instant traffic and indexing boost. Every single time. Whether the site has been 7k pages or 150k pages, the addition of the xml sitemap has always been beneficial.

Our sitemap currently has a “warning” basically stating that Googlebot is having issues with 301 redirects one some of our pages. I do wonder if this warning is occurring because of the sitemap, since I didn’t think 301’s were an issue for Google

Hey, I know you wrote this article some time ago but I believe it is still relevant even today. I was glad to find it because it says what I have been saying for a long time so I refer it to my clients when they ask about doing an xml sitemap and look at me funny when I tell them an xml sitemap is not going to help you rank better in Google nor fix all the issues I see with your website having good “seoness”.

Too, I see “seo pros” all the time selling “sitemap submission” (and site submission) services. Good way to tell a bonafide professional from one that maybe read some txtbook on “seo” and hung a shingle.

Frisor – I’m generally of the opinion that the vast majority of sites don’t need an XML sitemap, but I believe I’m in the minority on this. The only time I’ve ever recommended use of an XML sitemap was a situation involving a website with about 15 million pages that wasn’t being crawled very deeply.

I actually made an XML sitemap after posting that question and it seems Google has been quicker at re-crawling and re-indexing the content on my site. The site is new and has a few static pages which are updated often. I’ve specified daily in the XML – perhaps that is the decisive factor and not the existance of the XML sitemap itself.