Basically, when conducting an A/B experiment in Website Optimizer, the original referrer data is lost because of the JavaScript redirect to the test versions. This means that the traffic source will be wrong (it will say direct instead of the true traffic source).

This issue isn’t just a problem with website optimizer. Any landing page that uses JavaScript to redirect will loose the original referrer data. For example if your site provides easy to remember URLs like www.mysite.com/freeshipping and then uses JavaScript to redirect users to a different URL.

I wrote in my previous blog post that I wished Google Analytics provided an easy way to over ride the referrer value (and not use the document.referrer value).

It seems I spoke to soon.

Google Analytics DOES have a way to overwrite the referrer data.

It just doesn’t seem to be documented anywhere 🙂

I planned on writing my own hack to override the referrer value in Google Analytics. I was doing some reverse engineering of the ga.js code and found this interesting tidbit:

a._setReferrerOverride=function(b){a.yb=b}

I’m still running some tests, but based on my initial results, the _setReferrerOverride function provides a simple way to manually set the referrer value.

So now, we can actually capture the original referrer even when the visitor is redirected. Here’s how:
1 – Store the original referrer data in a cookie named realreferrer right before we redirect
2 – On the test pages, look for the realreferrer cookie. If it exists use it to over ride the referrer value in Google Analytics and then delete the cookie (so it won’t mess up page views that weren’t a result of a redirect from the original version).

Here’s the code I’m currently using in my tests.
Add this on the original page before the redirect (the GWO control script):

When running an A/B experiment Google Website Optimizer appends any parameters from the original page when redirecting to one of the test pages.

For example, the original page is index.php and the test page is index2.php.
If a user clicks on index.php?name=fred they will be redirected to index2.php?name=fred

So what happens if the test page URL already has a query parameter in it?

For example, the original page is index.php and the test page URL is index2.php?section=b
What will happen if the user clicks on index.php?section=d ?

About seven months ago I had a client with this exact scenario so I tested it.
The user was directed from index.php?section=d to index2.php?section=b&section=d
I found a work-around and didn’t think about it after that.

A few weeks ago I found out that Google was indeed aware of this “issue” and fixed the GWO code 🙂
Now GWO will merge the parameters from the two URLs, not just append them. The parameters defined during setup for the test page URL will get precedence over the parameters from the URL the user clicked on.

For example, the original page is index.php and the test page URL is index2.php?section=b
What will now happen if the user clicks on index.php?section=d&cat=22 ?

They’ll be redirected to index.php?section=b&cat=22

This doesn’t effect 99.99% of all A/B tests, but if it does effect you, Google’s fix is a life saver 🙂

If the web analytics code is after the redirect on page.html (which is what Google tells you to do) then data based on the HTTP referer will be incorrect (traffic source, keywords, etc).

If you REALLY want to capture the HTTP referer related data you can put the analytics code before the redirect. You’ll now be able to capture the original HTTP referer data, but you’ve also just inflated the page views.

I’ve solved this issue for some in-house code that uses the referer data by simply storing the HTTP referer value in a cookie right before the redirect, and then using the cookie value on the test page.

Unfortunately Google Analytics doesn’t do this.

Here’s a feature request for the Google Analytics team:

Add a way pass in the HTTP refer data manually.

I’m sure Website Optimizer isn’t the only tool using JavaScript redirects from a landing page.

I just found out today that www.prusak.com has been down for about 48 hours 🙁

I’ve been with my hosting provider – www.DreamHost.com – since 1999 and they’ve been rock solid so I never had any reason to setup any site monitoring services.

Also, I use Google apps for prusak.com email so that was working fine.

After 6 hours I finally got a reply to my urgent help ticket (they don’t have a number you can call for phone support) stating that:

Your domain is not pointed to our nameservers. Due to that, we cannot
update your DNS information if/when we need to change IPs (such as in the
cases of failing over pf services, moving apache instances, or
circumventing DDOS attacks).

Due to the sheer number of sites we host (700,000+) its not particuarlly
plausable to email our customers these changes, as <1% of our customer
base actually manage their own IPs.

Give me a break!

1 – It’s trivial to check which customers manage their own IPs
2 – When you do need to change IPs (which really shouldn’t be that often) I’m guessing only a fraction of your 700,000 sites are affected.

On a side note:
– I changed my DNS to dreamhost (so this won’t happen again)
– I signed up for a few free web site monitoring services. The only one which kept on sending me email while the site was down is BasicState.com (I figure I own them a free link). I’m not crazy about the user interface, but it works.

“What is the best page to start optimizing?”
This question comes up quite often in conversion rate optimization.

It’s an excellent question that really deserves a detailed (and lengthy) answer, but today I want to tackle a specific misconception about choosing a page that I’ve seen many people make (and even some books).

The misconception is that “It’s always better to test pages that have more traffic“.

Here’s a simple example with some numbers to explain my point.

Lets say you have an ecommerce site with these stats:

10,000 visitors a month (which land on the homepage)

3,000 visitors a month who reach your product detail page (the one with the add to cart button)

600 visitors a month buy (convert)

This means you have a 6% conversion rate for homepage visitors and a 20% conversion rate for visitors who reach your product detail page.

All things being equal, do you start testing on your homepage or the product page?
Lets run the numbers and see.

Assuming you run a simple test with just two variations (the control/original and a single test version) and that you’ll be able to get a 15% (relative) increase in conversions from your changes, here are the results:

Your site gets 2,000 visitors a week.
If you’re doing a simple A/B split with just two versions (original and test A) then each one gets 1,000 visitors a week.

Your current conversion rate is 2% * 1,000 visitors = 20 conversions a week for the original version.
You are hoping to get a 50% improvement for a 3% conversion rate * 1,000 visitors = 30 conversions a week for the test version.

Here’s what it will look like. The fields in yellow are the fields you enter.

The confidence calculator shows an 84% confidence level.
Not bad, but your boss says she wants at least a 95% confidence before declaring a winner.

You’ll see the confidence level for twice the traffic (two weeks worth of data) is 95%.
This means that if you get the same conversion rates and double the traffic, you’ll have a 95% confidence level.