How to do Keyword and Intent Research in a 100% (not provided) Environment

On September 23, Google announced that all search referral data will not be passed, at least for organic searches. SiteTuners is examining what that means for online marketing professionals.

There are no two ways about it - the looming 100% (not provided) is tough for online marketers. It’s tough for SEOs- they will have fewer tools to find opportunities for search term growth, to dissect brand versus non-brand search terms, and to tie search terms to traffic growth. That last bit particularly hurts, because losing visibility into how search terms drive people into the web site affects SEO budgets.

SEOs aren’t the only ones hurting, though. Conversion analysts lost a good channel for determining intent, and a good diagnostic tool for figuring out which pages to start optimizing on a limited budget.

What did you lose?

Search engine optimization and conversion rate optimization have been on a long path towards convergence, a fact highlighted by losses on both sides when (not provided) started its rise two years ago. For conversion analysts, not seeing most or all of the keywords people type into Google makes diagnostics tricky. The losses you are most likely to feel include the following:

1.Determining search intent for a popular page

Say you have a page that gets a lot of referrals from Google. One of the first things you’d be able to do is bust out WebTrends, Omniture, or Google Analytics and look at the search terms heading to that page, and then inferring intent.

Are the searches for the brand or for non-branded terms? This can help you determine what stage in the funnel they are in, and give you ideas about what to optimize and test. (i.e. do not try and optimize the “buy now” for early stage visitors)

If the visitors are coming in from non-branded terms, are they bouncing, or are they sticking around? This can give you ideas about the content tweaks for people who are just finding out about your brand.

In the near future, all clickstream tools will be able to tell you is that the visitor came from Google, and that the keyword is (not provided). This will make the old way of finding this data impossible.

2.Finding key pages to test

Suppose you’ve just been granted your testing budget, and you’re trying to make the most of limited resources. The first thing to determine is what pages to start with - your job is to find the pages that will drive the most impact for the same conversion lift.

To find this, you could run a weighted sort on the search terms and the bounce rate. That will give you the search terms people use most, combined with the least amount of engagement. If you improve click-through rates for those pages, you will impact the business more.

Unfortunately, that only works if you can see the actual search terms, and the search terms have been getting less and less visible.

Can you get it back?

The short answer is no. No, you are not getting the data back, at least in the same shape and form.

The long answer is that online, you have multiple ways of tackling different problems, with varying degrees of precision and difficulty.

Intent research is still possible. Keyword research is still possible. Finding the right pages to test is still possible. But the tools have changed, and the analysis has gotten much more difficult. What (not provided) does is make it tougher to prove your mettle.

What can you do?

The first thing to think about is goals. Why are you using search terms as part of the conversion data set? Are you after visitor intent? Are you trying to assess acquisition success? Are you finding key pages to optimize? The second thing to think about is not what you have lost - it’s what tools you have to get the job done.

Determining Intent

If you are trying to determine WHAT people are looking for, or WHY people come to your web site, search referrals were never the only game in town. With search referrals from Google gone, you’re going to rely on these even more:

1.Voice of Customer. Tools like OpinionLab, ForeSee, and iPerceptions are great for determining why people come to the web site, and how successful they are at finding what they need. You can directly ask the WHATs and the WHYs, so there’s less room to misinterpret data. You can find the tasks people are failing most, or the most broken tasks that drive revenue, and optimize those areas.

All the advantages said, these are qualitative tools, so they are based on sampling, and the top-tier tools tend to be pricey. If you are looking to do this on budget, you might want to take a look at Google Consumer Surveys.

2.Internal site search. The optimal time to set up tracking for internal site search is years ago. Still, if you haven’t enabled site search tracking yet, you should do it now - this is as close as you get to visitors telling you what they are searching for, what their intent is, all within your clickstream tool.

If you are trying to assess how successful your acquisition efforts are at the keyword level, the tools mentioned so far wouldn’t help you much.

3.Google Webmaster Tools. The data’s not perfect, you only get the top 2,000, and click-throughs from search terms are not exact equivalent of visits, but you can run an analysis of which search terms are driving the most traffic to your site via Webmaster Tools. You can mark up the non-branded terms with stars, so you can easily filter for just the terms driving new people to the web site (i.e. not the ones who already know you and are looking for your brand). From there, you can study where those searches go, assess the bounce rate for the main pages that generate search traffic, and test away.

4.Google AdWords (even when you’re not advertising). Even if you’re not planning on running a PPC campaign, there are now several good reasons to maintain a Google AdWords account. First, the Google keyword tool is gone, and the replacement, Keyword Planner, is within the AdWords interface. Second, you can now compare organic and paid terms within the interface. The data helps you see how successful your acquisitions are, what terms you are generally successful with, what terms and topics you should have content for, and what the global maxima is for the topics within your industry.

It goes without saying, but now is a good time to make sure that your accounts for Google Webmaster Tools, Google Analytics, and Google AdWords are tied to each other.

Finding Pages to Test

If you are trying to determine what to optimize and test, it could pay to have a content-centric, rather than search-centric, view.

1.High-traffic low performers. Instead of running a weighted sort on your search terms, you can run a weighted sort on your high-traffic pages. To diagnose which part of the site could use the most tuning, you need to find pages that generate a lot of traffic that ALSO have a high bounce rate - if you’re using Google Analytics, use the bounce rate to sort the pages, then select “Weighted.”

If you need to determine whether people go there for top of the funnel information or bottom of the funnel transactions, you can cross-reference the top non-branded keywords and where they go from within Google Webmaster Tools.

2.Broken tasks. If you’re running a Voice of Customer tool, you’re likely to know which task people are failing at the most. Without relying on keyword information, you’d know where the site has major usability issues and mismatched content - use tasks and task success rates to figure out which parts of the site are broken, and fix them.

Losing organic search referrals is an issue - no doubt about it. But the smartest marketers thrive in spite of - or partially due to - Penguin and Panda. They thrive in the age of social and mobile. And they will ultimately survive (not provided) - provided they are smart about it. If you are an online marketer, now is the time to adapt, use the tactics above, continue developing new strategies, and separate yourself from the rest of the pack.