SERP Analysis – Dissecting Your SEO Competitors

It’s no secret, and it certainly doesn’t happen by accident: in 2017, ranking in major search engines such as Google has become a science, the science of knowing your competitors, of knowing your customers/visitors and of course of knowing which SEO practices to ditch and which to keep and improve.

What used to be a handful of on-page, off-page, and architecture factors has evolved into hundreds of signaling attributes fed through a complex algorithm that is constantly being refined and learning from your behavior.

With search engines becoming smarter and more sophisticated with each passing day, it has become crucial for businesses seeking the benefits of organic search engine optimization to comprehend the undertaking required to rank highly in competitive search verticals.

This post aims to step you through dissecting your organic search competitors in SERPs so you can better understand the time, money, and effort you will need to put forth to achieve your target rankings.

This post is also a piece of the larger puzzle from SEO strategy guidelines for online businesses.

The Lay of the Land – Understanding Your Competitive Landscape

In the same way a landscape architect needs to understand the elements of the property he is designing for in order to achieve the goals of the project, you need to understand how your website stacks up to your competitors, what gold (or iron) may be underneath the surface, and how your surrounding environment can impact the outcome of your campaign.

First, you need to know what you are getting yourself into. Before you go throwing your clients money at writing content and building links relationships targeting a specific set of keywords, you need to assess the foundations and maturity of your competitive market.

Think of yourself as an asset portfolio manager, because that’s what you are. In the same way fund managers need to analyze company financial data and operational performance before making investment decisions, you need to analyze your SEO competitors to inform your investment decisions.

In 2017, there are still a lot of companies that dive right into a new market and immediately start spending money on pay-per-click advertising, landing page design, and social media marketing without really identifying if and where the opportunity lies.

This whole approach of ready, fire, aim may work for some but it is the exception and not the rule.

Instead you should take steps to calculate your risk:

Analyze your competitors within your target search vertical

Identify weaknesses in either under-served segments or for under-optimized keywords

Know your costs

Project your returns

Create a realistic timeline for both implementation and break-even

Getting Started with the SEO Competitive Analysis

Once you have completed your initial keyword research, review your keywords to identify which are most relevant for your business and which have the highest volume:

1. Start by identifying your core set of target keywords, no less than 5 and no more than 10.

For example, if I were to pick the core keywords that best represent this post and have the highest monthly search volume, they would be:

2. Next, use Keyword Planning Tool to find ideas that Google identifies as semantically related. Back in the days when Google’s Contextual Targeting was in Beta I ran the above keyword list through and drilled down to the second tier, see below:

Now, similar grouping can be acquired from Google Keyword Planner, though, starting August 2016, low AdWords accounts no longer have access to exact search volumes.

3. Group your target keywords into 3 different buckets based on volume. This usually separates out head, body and long tail terms.

This gives you a sense of what kind of monthly search volume you can hope for if you are able to rank within the top 1-5 spots, with the #1 spot capturing on average 36% of clicks.

4. Optimize from tail to head.

For anyone just getting started with SEO this may be a new concept, but you may have also seen or read about ‘the long tail of search.’

Essentially what you are doing here is focusing on long-tail terms which are highly relevant to your business and also offer opportunities to build content for the larger, more competitive head terms.

In 2017, long tails have become even more important for SEOs. Let’s take a closer look at this!

Anatomy of a Keyword

With your core keywords grouped into buckets based on exact local monthly search volume, you will most likely notice that the longer the search query, i.e. the more words in the phrase, generally the lower the search volume. Voice Search and Mobile-First Index might drastically change that, but we’re not there yet.

As a general rule of thumb the more words in the query the easier it is to rank for that phrase. This is certainly not an absolute but has been the case for a majority of the terms I have worked with. *If you know of examples where this theory does not hold up, please share in the comments.

Below is a graphical representation of what is meant by the long tail of search:

As you can see above, shorter query phrases receive the most volume, these are your head terms, followed by body (usually 3-4 words) and then the tail which usually consists of keywords that contain 5+ words.

When optimizing any website digital asset for the long-term it is important to capture the low-hanging fruit to begin creating conversions as soon as possible.

This is best done by optimizing backwards; from tail to head versus the other way around.

What I mean by that is to focus on acquiring traffic for the relatively non-competitive terms and working your way up the tree, by optimizing for your long-tail terms and building toward your head terms over time.

What this allows you to do is create content that is targeted to rank for your long-tail terms in the short-term while building relevance and authority for your body, and ultimately head terms, since these are often included in the long-tail queries.

Due to Google bringing constant changes to SERP, you will discover that the number of competing pages for some keywords is no longer available. For example, some time ago I’ve learnt that there were 349 million pages competing for the #1 spot on Google for the head term SEO Blog . But for the very long-tail term SEO blog focused on user experience philadelphia there were only 12.6 million competing pages, and this phrase contained the head term. Now, if you search for “seo blog”, you will no longer see total number of competitors:

If you will encounter similar situations, I think is fair to assume that those keywords are highly competitive and nowhere near the concept of low-hanging fruit.

Looking at Competitors’ Authority and Trust

The next concept that it is important to understand is Domain Authority, often referred to as DA. This is a very handy metric created by SEOmoz that essentially assesses a domain’s potential to rank. The higher the DA and PA (Page Authority), generally the easier it is for the website to rank for a keyword.

It’s worth noting that there are SEOs who prefer using TF (Trust Flow) and CF (Citation Flow) from Majestic when checking their competitors authority and trust levels.

DA/PA and CT/TF are all third party metrics meant to somehow replace the void left by the famous Google PR (Page Rank). Obviously none of them could replicate PR real values of nowadays (as those values are being kept secret for quite a while now). It is difficult to recommend one tool over another, but for this analysis I went with Moz DA.

In the screenshot below I searched for ‘seo analysis,’ notice that seoworkers.com (#1 spot) is outranking a near exact match domain, seo-analysis.com (#9 spot), who also has more links, 65,341 vs. 41,935, most likely due to the fact that this domain has significantly higher domain authority; 65 compared to 27.

As you can see in the screenshot above, for the query ‘seo analysis’ the DA ranges from 27 to 87.

Please also note that in this example the site with the highest DA, SEOchat.com, is also ranking in the last position (#10). This is a perfect example of how there really is no one metric for determining SEO competitiveness.

Taking a closer look at the numbers in the screenshot you will notice that SEOchat.com only has 12 links coming into this page from 2 domains. Also please note that the only site outranking SEOchat.com with less page links (having only 1) is built on Google Sites, giving it a DA of 100.

DA to rule them all

There are varying opinions as to the true power of domain authority, but in my opinion, it is more or less the holy grail of SEO.

Now that you have your core set of keywords, let’s take a look at the DA of the competing websites. This is easily done in Chrome using the Mozbar extension, which will display a metric bar for each search result right underneath the meta description, as you may have noticed in the screenshot above.

Even though it is not a surefire indicator of ranking potential, I still hold that it is important to first look at competitor’s DA when starting your SEO analysis. In my opinion this will allow you to gain at least some initial visibility into your ranking potential.

Ranking potential is an assessment of where you can realistically hope to get your rankings to; if the top 5 websites have a DA of 90+ and your website is at 20, it is not realistic to think that you will be able to grab one of the top 5 spots in no time. Again this is not always the case, but tends to generally be a safe standard of procedure.

Data Mining – Release the Scrapers

In order to conduct the analysis we are going to need to go out and grab some data; what we are going to focus on for now are the basic elements that contribute to domain authority.

For the purposes of this analysis I am going to pick a body keyword and look at the top 10 ranking websites.

Compile these in Excel for analysis, which should look something like this:

Next we will look at the types of links pointing to these websites. You can do this through Open Site Explorer report (MOZ), Site Explorer (Ahrefs), Backlink Checker (Majestic), Backlink Audit Tool (SEMrush).

This needs to be done for each ranking URL, but for the sake of this post, I am going to pick only one. Instead of choosing the big brands, Search Engine Watch and Raven Tools, I’m going to use Web Confs; www.webconfs.com/how-to-analyze-your-competitors-article-39.php

Now we need to generate the Open Site Explorer report, which will look something like this:

I’ve downloaded the CSV and upload into Link Detective. This returned a breakdown of all of the links by type, and look something like this:

Link Detective is not available anymore, but there is no need for it in 2017, as many of the tools listed above (Ahrefs, Majestic, etc) already provides you with the relevant data & charts.

This type of report are helpful because they show how your SEO competitors are building their links, and whether or not they are using spam tactics to manipulate their link profile.

From the above report we can quickly identify some actionable opportunities:

15% are from article directories and 5% are from link lists. These are not hard to trump with more authoritative links.

The next thing I recommend is to take a look at what I refer to as the domains link diversity ratio or LDR, a very simple equation for identifying how diverse the website’s backlink profile really is.

This produces a score between 1 and n, with 1 being a perfect score representative of a website where every inbound link is from a unique domain, which is not really possible outside of some imaginary SEO vacuum.

What you are looking for are the outliers; the highs and lows. High scoring websites can be really high, sometimes to the tune of 257.14, which would be a website with 18,000 indexed links coming from only 70 websites. The lower your LDR the better.

Again this is quick and dirty, but can provide some quick visibility into the legitimacy of the links. Often time’s sites with a very high link diversity ratio are buying links, using link networks, or creating microsites to engage in link schemes.

Based on the data from OSE, the link diversity ratio for the webconfs.com competitive analysis URL is 1,775.36 (19,529 links coming from 11 linking root domains) and 156.28 at the top-level domain (1,802,896 links from 11,536 domains)

Now I personally have nothing against webconfs.com, but their link profile for this page is relatively weak. *Hint* if you operate an established SEO Blog (minimum DA ? 30) this is traffic for the taking.

To take link analysis one step further you should look at a more meaningful measure of the legitimacy of a website’s links by looking at the number of linking C-Blocks, or the number of websites hosted on the same IP address.

This is most useful if you have identified a handful of competing domains that you believe may be interconnected and managed by the same company.

Competitor Analysis and Opportunity Identification

To use an old saying, now we are getting down to the meat and potatoes.

So what do you do with all of the data you have now gathered and some of the surface opportunities you have begun to identify?

Start mapping out a plan for action.

The easiest places to start are obviously scooping links out from your competitors that are either broken, of low-quality, or from the same C-Block.

As for the more competitive terms in your buckets, the ones that will really drive qualified traffic to your website, you need to take a more scientific approach, and here is what I mean.

Usurp The SERP – Planning the Takeover

The example I have provided is very small, to the extent that the nitty gritty analysis could be done manually without taking excessive amounts of time.

However, if you needed to identify tens of thousands of keywords to invest in as part of an enterprise SEO strategy, this would require a more scalable approach to evaluating individual keyword opportunities.

This can be achieved through designing an opportunity evaluation model.

Please let me preface by saying this is not a complete model, it is just an approach to analyzing competitive SEO drivers on a large-scale. However, if refined to project revenues within an order of magnitude, it can become a powerful business driver.

Due to intellectual property rights I can only share some of the model. I believe it is enough to get you thinking and moving in the right direction.

Monetization metric (M)- Measure of approximate revenue per conversion, for eCommerce this may be average order size, for display advertising this would be eCPM, or earnings per 1,000 pageviews.

Discount Rate (dr) – Rate at which the heuristic is discounted to adjust the relative significance within the model.

For DA and PA I used the average of the top n sites. N will vary in size based on your confidence interval requirements.

The model is then built on a spreadsheet so each of the heuristics can be driven by the assumptions, i.e. cost per content, average cost per link, SERP CTR, avg revenue per page or per conversion, etc. And the assumptions can be adjusted as historical data is collected.

What this allows you to do is built a large-scale model for evaluating each keyword, projecting the cost, time, and potential return based on using discount rates to adjust the relative importance of the varying competitive factors; DA, PA, and links. And placing this into a model that discounts each of these heuristics accordingly.

I realize this may be hard to visualize as it is not common in the SEO industry to talk about automating keyword analysis for thousands of keywords.

As you design your own models that are representative of your business, make sure you take the time to define your success criteria. In other words, set S.M.A.R.T. goals before you start your campaigns. This will allow you to watch the needle as it moves, and understand which direction it’s moving in.

It is not going to be an overnight success. It may take months of adjusting your assumptions and discount rates before it starts projecting accurate costs and returns.

Lastly, this does not make sense for everyone. Spending the time and energy to build an analysis model like this really only make sense if you are working on a project that requires you to look at thousands of new keywords each month.

Otherwise manual analysis and really getting a feel for your search vertical will probably give you a better sense of where to focus your optimization efforts then spending months tuning a formula.

Wrapping Up

I apologize for speeding through what is probably the most captivating part of this post, the opportunity model, but this was intentional. This concept is an important and technical approach to enterprise SEO which is too complex to properly explore within this post.

It is important however to keep in mind that starting 2017, Google’s shifts and tweaks on ranking algorithms are happening more often. What you’ve learn from an SEO competitor analysis may no longer be true after a couple of months. Competitor analysis in SEO is great for getting valuable insights, but it won’t give you a remarkable boost unless you also provide unique value to your visitors. So keep an eye on your competitors, while constantly working on ways to improve UX on your site.

In the meantime I hope have provided you with a straightforward path to begin identifying and analyzing your SEO competitors. You can also check the post in which I talk about Google search engine optimization with an website optimizer and you can expect a dedicated post on how to design and build keyword opportunity models in the near future.

As always any thoughts, reactions, or feedback is welcome in the comments. I do my best to respond to each one.

Hey Bryon – Thanks for the comment. You’re absolutely right. Making sure your link profile is built in similar fashion to the top ranking websites is generally a sound approach.

This also may provide some opportunity, if for example the link profile for a URL ranking for a specific term is comprised of gamed junk links a bit of authority link building could move the needle on rankings relatively quickly.

It’s almost hard NOT to leave a comment of praise after reading this. Definitely miss reading more articles/posts like this on a daily basis.

“This is best done by optimizing backwards; from tail to head versus the other way around.”
Cannot tell you how important I believe targeting your keywords like this is. Precisely this. Valuable time and traffic can be lost when you focus your full attention on head terms. Especially when those tail keywords contain your head terms.

Not sure why so many people turn their own heads away from them. Silly.

Really insightful post Nick. I think the whole area of forecasting is something that I haven’t seen any SEO software address. Many of the major tools help you identify where you are and your competitors but don’t connect the dots well enough in helping plan your SEO campaign. Looking forward to your newsletter

Brian – I couldn’t agree more. It is very intimidating at first to deal with so many unknown variables and assumptions. I think the hardest part is really just setting the assumptions, building the relationships into the formula, and letting the model dictate your strategy.

At the same time this is most rewarding part. As long as you are constantly refining with actual data, using the formula to inform your investment decisions becomes very rewarding as is begins to approach accuracy. Cheers!

Michael – I feel your pain. More times than not the younger people I meet who are just getting started either in-house or as an intern focus so much of their attention on head terms, and then they either get frustrated or scaled back from optimization after they are not able to produce any significant head traffic in the first few months. I’m happy to hear other people are taking this approach.

Thanks Nick. I like what you mention about being realistic in what your goals are and going after the right keywords. It is very important to go after the right keywords and know how to find keywords. This is a great comprehensive guide on link analysis.

Hey Tariehk, thanks for reading. It happens more often than you might think… so many times when I was working at an agency we would go talk to clients who would either have a handful of arbitrary terms chosen that they wanted to rank for OR they had picked a handful of insanely competitive head terms and decided ‘these are the keywords we need to focus on.’

I almosr forgot to mention that you also missed the most important aspect i.e. social signals or social reach of that particular page. Checking out number of RTS, Facebook likes and G+ scored by page does help in analyzing page a bit. Of course, it would be again difficult to see which social signal has maximum impact, but one can easily ignore Facebook likes/share.

Ricky – Thanks for the comment. I don’t know if I would go as far as to say social signals are the most important signaling factor for search engine authority but I do believe they play a role, especially as it pertains to indexing and the timeliness of a content’s topic. For example if the post was about the Olympics,provided coverage of an event, and was being tweeted in high volume I could see this playing a role in short-term rankings for closely related queries. Thanks for reading!

I’m glad you liked it! I have been working on the the first email and it’s almost ready.

I will be providing instructions on setting up a basic evaluation model and then going into the steps I took to determine the discount rates for my market. The approach can be taken by anyone to begin to refine the accuracy of the models predictions.

This is a brilliant and up to date example of competitor research. Found your article on inbound.org. Best thing I have done all day. Now about to go through the rest of your blog looking for golden nuggets.

Danielle – Thank you so much for the compliment, I’m thrilled to hear you found the information valuable. I checked out doublespark.co.uk also and you guys do some really nice design and development work… I also see you provide SEO services; you may want to check SEOLeads.org which will be launching in private beta any day now 🙂 Cheers!

I wish I had found this article earlier. In the last few months I went through tons and this is by far the best post on seo, whether you are a beginner or advanced. I think one thing which might change is the ‘DA’ metric from MOZ, since pagerank is a huge factor and I don’t think it will exist anymore.