Excellent Analytics Tip #12: Unsuspected Correlations Are Sweet!

This post could also have been titled "Tracking Radio Ads" or "Measuring Online Impact of Offline Marketing" or "Success in a Nonline World". It will touch on all of those.

But the title is what it is because the most lovely part of this story was how tracking with a one dimensional mindset (or in a silo) means that you will end up missing so much of the picture. And it is a story about what it means to be a Analyst in this day and age.

[This post is dedicated to my friend Nick: You are a sweetie! Thanks for everything.]

A delightful company, let's call them Market Fire Extinguishers, located at www . marketfireextinguishers . com (there is no such site as of today), would like to grow their business. They have tried lots of nice things online (primarly affiliate marketing). It worked ok.

Then they came upon a radical, for the web, idea: Run radio ads around the country!

This is getting easier to do than ever with many leading companies getting into the space and lowering the barriers to entry. Even I can run a 50 city radio ad for Web Analytics: An Hour A Day.

They ran campaigns across the US, in 50+ DMAs each that had their own set of cities.

The call to action was primarily driving people online, via a easy to remember vanity url (redirect) www . eztz . com (I am making that up here to protect the client). The audio ad also mentioned a toll free 800 number that listeners could call and purchase the product.

Using the vanity was smart, for tracking purposes (hurray!), keeping it easy to remember was even smarter. Driving people to the site was a business requirement because picking up the phone is expensive.

The first "management" level reporting was extremely simple, and visually appealing (after all pretty sells!). It was an attempt at answering the question: "we have all these ads running in 500 plus cities, which of those are effective at driving traffic to our website?".

Here is the baseline picture, before the campaign. . . .

And here is the picture that shows the impact of the radio ads. . . .

Sweet lord that is impact! Get me the champagne!

The nice "chicken pox" geo report is a good visualization in this case because it quickly shows the top 100 cities that sent in the traffic. The before and after makes a nice story in of itself in terms of showing impact of the campaign.

It is also easy, as you can imagine, to dig deeper into the data and analyze all 500 plus cities that sent traffic. You will surely go in and look at the audio ad costs in each city, the number of listeners in each (from a source like Arbitron), the number of resulting Visits and Absolute Unique Visitors etc.

Being a fan of the Trinity Strategy you will surely dive into understanding outcomes (lead forms submitted, orders placed, samples requested and what not). Then you can pick which cities were ROI positive, which were not etc etc.

One of these days it will be easy for you to do much of the above analysis in your web analytics tool.

One graph you will surely create will look like this one. . . .

A correlation of two trends, the brown is the radio ad impressions that you achieved during the course of this campaign. In blue is the resulting traffic to the website.

Nice. Clear correlation between the ad showing up and the traffic spiking on the site, the ebb and flow also match nicely though after every drop in impressions you see residual impact on the site traffic (the blue line drops less dramatically). All very wonderful.

Most people will leave it at this when it comes to measuring success for their campaigns (be they email or audio or tv or SEM/PPC or direct marketing etc). On the web, delightfully I might add, it is rare that a butterfly flutters its wing here and a tree falls in the amazon. (Ok I have been unwell for a few days now and am out of my metaphors juice!)

You take action in one channel / medium and it will surely have a impact in other channels / mediums.

"Unsuspected correlations."

So when you analyze your own campaigns / valiant traffic driving efforts, correlate other sets of data you have to see if there are hidden correlations that could help you understand the story better.

In our story the Analyst Ninja did exactly that and added a line to the graph showing the Traffic from Branded Search for the same time duration. . . .

Lovely!

Not only did the radio campaigns drive people to the site using the easy to remember vanity url, eztz . com, but our lovely radio audience, brainwashed as they are like the rest of us, also ended up typing in queries into the search engine and arriving at our site.

Not a flash in the pan, but a consistent trends, mirroring the ebbs and flows of the radio ads.

This was a surprise because that vanity is not that hard to remember yet people used the terms mentioned in the ad (company name, product name, other brand or category terms) and used a search engine to get to the site.

Lesson: People behave in ways that they are used to and many of 'em won't do what you want them to do!

Unsuspected correlation in this case raised the amount of ROI the audio campaign could claim.

But our brave Analyst was not one yet. She tried one more thing. . . .

Direct traffic in light blue.

Again a clear trend in the impact of the radio ads on the Direct Traffic (your web analytics tool could be calling this "bookmarks" or "type in" or "none" etc). Technically it is people who have "no referrers" in their session.

In this case it was people who, again this is normal on the web, heard Market Fire Extinguishers and typed in www . marketfireextinguishers .com and got to the site. And that is a hard url to spell!

Another unsuspecting correlation.

In the end the impact of the radio campaigns on this particular website was significantly more than original imagined.

There are interesting implications of the above when it comes to your next media buy and the kind of customer behaviour that will impact it.

Net Net: Next time you are asked to produce a ROI analysis perhaps you'll think of this example and ask yourself if you have correlated enough data streams and looked hard enough to paint the complete picture.

Important Web Analyst Skill Observation:

There is one other facet of this lesson that was important to me. Lots of us get so entrenched in numbers and analysis and Omniture / WebTrends / Google Analytics / IndexTools etc that we often lose touch with the outside world.

If you want to be a Magnificent Analyst spend 50% of your time with the above two. Anyone can learn to press buttons or extract data into excel. And thousands are learning that every day. What will set you apart will be your superior knowledge of the marketing and customer experience ecosystem in which we all exist.

Less than 10% of the Analysts I meet are proficient in those two things, almost everyone is proficient in the numerous web analytics tools.

It might seem obvious in hindsight to do the above analysis, I assure you that it was not. The person who did this was less a "web analyst" and more a "online marketer", the 10% I mention above.

Be that.

Important Observation #2: Correlation does not imply causation.

Correlations are a very advanced statistical technique that I am using in perhaps its most humble and lame manner above. (That had to be said!)

More importantly it is important to realize that Correlation does not imply causality.

In our case above we controlled for other externals factors (no other campaigns running, no weird seasonality carp – notice the campaigns ran after Christmas etc). This was to ensure that we were not seeing patterns where none should exist.

It is important to internalize this.

Another example. Here are number of posts I have written in the last few months and the number of RSS feed subscribers in each.

The causality is such a good point, Avinash. I have learned to be very careful in the ways that I publish data as too many times my correlations have nearly been turned into doctrine as people see what they want to see. Mark makes a good point about the tidy report, as well. I have seen that when things line up very well people are ready to close the case and take action. I think it happens because all too many times we load our analysis consumers down with caveats and such to explain fuzziness in the data, and when things do line up in a picture-perfect sense they recognize it as perfect clarity.

The data is all real data, untouched in any way, which is what made this whole thing so fascinating to me.

The items seem way too lined up than they should be partly because the graph was "squished" to fit into 480 pixels, the width of my blog. It makes things appear more correlated than they are. That was one of the reason I pointed out the lags in the blue lines, to illustrated that the drop off in traffic was not as steep as the drop off in the radio ads! You'll notice that is the case on almost every data point.

The second point is very true and something that threw me off as well. My personal analysis of the situation was: The product is rather unique, it is something we would die to have ("you have been waiting all your life to get rid of that pain, call now for the cure that is God sent", yes slight exaggeration :). The promos were targeted to be "drive time", that worked well to get traffic when people got to work.

For this business Radio turned out to be a great answer to improve brand awareness and drive people online. It is a function of many things including the youth of the company.

-Avinash.
PS: Hopefully your Analyst won't come running to you just with this graph but rather the one that plots Outcomes (because this just shows that people showed up!). Then She/He could exclaim: "Radio works! Look we made $15 million, net!!" Then bonuses all around. :)

Great post Avinash. Audio has been a key part of our strategy for some time now but tracking the online impact has been quite tough. We were doing some of what you suggest here like track traffic by city. But I did not realize all the other impact that our radio campaigns might be having.

As always you have opened my eyes to something new. I am off to run my correlations now.

I don't have a link, right now, but I remember that Anil Batra once had a post about a negative correlation between the number of posts he wrote and the number of subscribers he got.

He concluded that if there's a bigger pause between two of his posts, people might feel more tempted to subscribe, so they don't have to come to the site every (other) day if there'll only be a post about every two weeks.

I hope I'm not misquoting Anil and I know it's more of a theory than hard evidence, but it sounds like he might be right about it.

Maybe some other people on here have blogs with a high enough traffic number and have witnessed the same thing?

Ken : I suspect that mileage that each business will get might differ when it comes to ROI from their radio ads.

In places where I have privilege to do the analysis the numbers have been all over the place, depending on the outcome and type of business. But it is not unusual to see conversion from radio to be between a dollar to five dollars.

Though for some businesses it could very well be the $143 you mention.

As always YMMV and experimentation is the way to go, try it, measure it, rinse and repeat if it makes sense. :)

Crazy coincidence but, here in Perth (Australia), they just ran an ad for, funnily enough, advertising on the radio.

The point was to show how much a website can get from a radio ad, so the "website guy" takes a picture with his camera of the "radio guy" interviewing him, then says, "I'm posting this on a new site… called… http://www.thatradiobloke.com"

Sure enough, that URL takes you to radioadvertising.com.au or sommat, but then it pops up with a huge closeup of some crusty dude in a radio booth.

It was classic, and got a giggle out of me. If I could afford it, I'd totally advertise on the radio. And I'd make it FUNNY.

Avinash, I saw this on sphinn and sphunn it immediately (the submitter was a trusted source and the destination url was even more trusted ;) ). And boy am I glad I did!

First, this is interesting because I spoke with another excellent online marketer, Mikkel deMib Svendsen. His take was that offline does NOT drive traffic offline, let alone convert. And for support he pointed at the dotcom bubble, which is a pretty good piece of evidence imho. That said, he may have been thinking more about billboard ads. Any experience with those? Also, what's your experience been like with offline promotion generally, as far as getting visits online?

Second, I'm enjoying your book and the relation to your site; the former allows me to better understand your blog :).

Besides that, I hope you get better and suggest you don't blog for another month or two – you'll quadruple subscribers ;D!

Avinash, this is one post really worth reading. There are numerous articles and commentary about analytics but there are so few who really get deep enough and present it in such a practical way. You Rock!

One think I'd expect to see in the charts, is that the radio ads should have a longer effect i.e rather than just getting visits during the campaign, I'd expect to see a general rise even after the campaign ended (hence the quality of the campaign for a longer term can also be measured).

Secondly, a question: Is there a chance you can share some numbers such as conversion rate or cost per visit from the radio ads converted as visitors? i.e. if you invested $100K in radio, and the difference in traffic (organic, branded, direct) was 100K visitors then radio ad visits would cost 1$ each… I think this maybe be a number a lot of us could appreciate as a benchmark (even without the exact theme/vertical).

Another great post. What so amazing to me is that media today is bought and measured in silos while from the consumer's point of view there is no difference. And in this case, offline drove traffic online.

I can envision some web team giving each other high-5's for a successful online campaign oblivious to the fact that their traditional media counterparts just spent 10x budget which drove more people to the site.

I also would think TV, PR and Event Marketing would have the same effect to a website.

Anyways great post. You sure are lucky to work with smart guys like Nick

A client read this blog and now wants a report similar to the graph above, but instead have "leads, campaigns, and tactics" instead of "direct visits, visits to extz.com, brand search".
How do I track what my client wants and provide a report on that. I'm scratching my head on this issue.

Winn : First off tell you client that I love her/him for reading the blog, and more importantly for wanting to take action!! :)

The graph is in good old excel so really not a big deal.

Perhaps you are struggling with how to get the data for "leads, campaigns, and tactics".

Three things can be awesome:

1) It is just a matter of ensuring that you are tagging all the sources that you want to track. So make sure that your affiliate or banner or paid search or email or other campaigns have url's pointing to your site that are encoded with the right tracking.

This post on the Google Analytics Blog shows one example of how to semi automate campaign tagging with the URL Builder:

2) Make sure your pages are tagged properly with the web analytics code. You will need this, for example, to get a accurate count of leads / goals / conversions.

3) Good old segmentation! What your client is asking you to do is essentially segment the traffic and outcomes and layer that on the graph (which is exactly what I have done in the graphs in the post). Use the segmentation capabilities of your tool to pull out the data for segments you are interested in.

Great post. I have a question. How/where did you get the baseline numbers to make the 1st Geomap chart from? Was there an existing campaign going on that they captured these from or did the numbers come from elsewhere like a visitor geography report showing existing traffic levels from these locations?

How much faith can you put into visitor geography reports? It seems like some of the data shows the location of internet service providers used by users instead of users themselves. This can be misleading.

Do you have any suggestions for creating Geocharts in terms of software?

Alice : Second question first – the geo graphs are from good old Google Analytics (the "chicken pox" reports). Most web analytics have the ability to make nice Geo reports. I am unaware of a dedicated software (but that is just a gap in my knowledge).

For the first question I am going to cheat. Rick just asked a question via email about Geo data and its accuracy (especially that Virgina often shows up high).

Here is my reply and I think it answers your question:

Rick,

You are right, you'll see Virgina high because of AOL and often Georgia (Atlanta) high because of Earthlink.

Two things are changing the impact of those problems.

Both AOL and Earthlink are a lot less used and powerful than they used to be, hence the problem is reduced now. But if your customer demographic skews AOL/Earthlink then of course the impact will still be there.

Most web analytics tools are now using more sophisticated ip/geo location to identify the right visitor source. Recently I read some where that it is now 80% accurate (which is very high I think).

Besides if you compare the trends then you can still see impact from other cities.

I'm interested in learning more about your book- I noticed your blogs started in 2006- is the book up to date with all of the new Google stuff brought out this year(2008)? If so- I'm ready to order to help my online bath body company soar! you seem to really understand this stuff incredibly clear:-)

Web Analytics: An Hour A Day was released 10 months ago and is a book that is about the business and technology and strategy of web analytics. It takes a fresh and deep look at what it means to be successful at web analytics, independent of what tool you might be using and what company you might be working at. Please check out the description at amazon for more: http://www.snipurl.com/wahour

If you want a Google Analytics then might I recommend these two excellent books:

Just a quick question…we're looking at the online impact of TV spots at certain times of the day.

Google Analytics can give me hourly sessions, but would you know how time is being rounded to the nearest hour? Like if a TV spot aired at 2:30P, would I look for a spike in Google Analytics at 2P? Or 3P?

April: This is an easy problem to solve. Just use the Google Analytics API.

From my friend Justin: "Use the dimension ga:Minute when requesting data. It's also a good idea to also request ga:Hour, so you know which hour the minutes belong to :) And if you're trying to extract data from more than multiple days then you should also request the ga:day dimension.﻿"

Terrific piece, but one word of clarification — it is true that correlation does not imply causality. But that's the only part of the old maxim that most people remember, and that often distorts the bigger truth.

The reality is that science would be nowhere without correlation, and it is not causality's poor cousin or redheaded stepchild.

It's also true that if you take an action with the intent of causing a particular reaction, as marketers often do, then when there is a strong time-lagged correlation between those two things, that can be correctly seen as something pretty damn close to contributive causality.

Furthermore, in business, the level of accuracy necessary to make a good decision re "cause and effect" is nowhere near the level of accuracy needed in other applications.

[…] Avinash Kaushik, one of my favorite bloggers and Web Analytic Guru, who also happens to consult Google with regards to Web Analytics, wrote a post a few weeks ago analyzing the correlations between radio ads and web site traffic, with deep insights into web site traffic, branded vs. general searches and direct visits. It now makes even much more sense. Google will now enable advertisers who run audio ads, to analyze their effect on their web site, with integrated data from the Google Audio Ads panel and get insights from the cross-channel advertising not only on the online but also offline. […]

[…] I discovered what web analytics represent to a website just 4 months ago. I saw the book “web analytics on hour a day” from Avisnash Kaushik in a library and the “About the Author” page mentioned his very popular blog on web analytics. I had no idea who was Avisnash and I was also really unaware of Web Analytics. I went to the blog and the article Excellent Analytics Tip #12: Unsuspected Correlations Are Sweet! was just written. I do not know if you read that post already, but all I know is the impact it actually had on me (nothing emotional – don’t worry :-) ). I was so blown away by all the post itself and the correlation you have between metrics. It was just… huge! […]

[…]
Most notably radio significantly increases online traffic, according to the RAB, allocating 10% of a media budget into radio boosts brand browsing online by 52%. The power of radio to boost online browsing is highlighted in this case study by web analytics guru Avinash Kaushik.
[…]