When analysing Website Optimiser test data, we've identified a serious need for more fine grained data to get really crisp results. For example, Website Optimiser will tell you how many visits have occurred to each of your experiment variations and how many visits have converted. The nature of the conversion may change of course as might the secondary and tertiary effects of tests. How can we see whether a test of the on-site search functionality has increased ecommerce conversion? Can we see whether customers engage better with the site if they see more featured products? Website Optimiser data won't necessarily answer these questions out of the box. We've identified a couple of techniques that will help though.

The test(s)

Babyetc.co.uk was about to conduct a couple of tests. I'll gloss over the tests here because this post is more about the how rather than the what'. In a nut shell - are the featured products adding value and can search or newsletter signup perform better sitting next to the logo?

Newsletter signup in the header with no featured products:

Search in the header with featured products:

The tests ran for a couple of weeks and the first result comes in:

No great surprise - search usage is improved by nearly 40%. The other test is still running though. So, were searchers buying more? Are featured products engaging customers? Are test subjects spending longer on the site? Are these changes adding value to the bottom line? We'll consider event tracking first of all:

Google Analytics Event Tracking

Event tracking is currently a beta feature. You'll need to request it to be added to your Google Analytics account. It's a much sweeter mechanism for recording single events than virtual page views especially if you don't want your content report data soiled.

First of all, get your test set up and running normally. Now, you want to record an event firing for each variation and potentially for each conversion. We'll cover variation tracking - goal tracking is a lot easier!

You'll need to find out your experiment number - you can find this in the WO control script provided (it's the variable k):

Bingo - WO variation views are tracked as events, simples! So, how does this look in GA? Head over to your reports, go into Content -> Event Tracking:

In the overview you'll see something like this:

Not that interesting...drill into actions though and you start to see real data treasure. Here's the ecommerce tab:

Row 0 refers to the original, row 1 is the search box in the header and row 2 is the newsletter signup.

We can see that AOV is down - no big deal - not statistically significant. We can also see that conversion rate and overall revenue are up and highly significant. So, the test confirmed more searches were happening but beyond that, we know that searchers convert 2-3 times better than non-searchers so we would hypothesise that revenue and conversion would increase with search usage. This data confirms the hypothesis and tells us by how much!

Now we can drop this data out and run it through SPSS (enterprise scale stats tool) to run more numbers through at our leisure to gain further insights: not something you can do with WO reports.

superSetVar

User defined variables are a really neat and powerful segmentation mechanism that we can use to provide a facility that allows segmentation of data based on partiular events firing. For many cases, segmentation based on events will not be useful or interesting but coupling event tracking with WO tests does make segmentation worth while. Say I wanted to see on-site search usage stats, ecommerce performance and site engagement data in one custom report based on a user seeing a certain test variation - now segmentation becomes useful. You don't have to use event tracking with user defined variables - I did in order to explore the technique. Say for example you've extracted the variation number as in the event tracking example. Now we can use superSetVar from Lunamaterics to set a user defined value using a similar naming convention to the event tracking:

As you can see, we are a Rails shop - feel free to substitute the syntax of your choice to get the right uAcct value in place.

Right now, Website Optimiser has not suggested as clear winner for the featured products test. I wonder how things are shaping up?

If we were to head over to Visitors -> User Defined and take a look at the ecommerce tab we would see the following interesting report:

'featured-test=0' is the original where feature products are shown. 'featured-test=1' is the test where featured products are not shown.

Now, create a custom segment:

And apply this to the report then go look at the ecommerce tab:

You can probably see how you can now look at any report in GA to see how your tests in WO are affecting metrics across your site.

Now, I have to say that these techniques are still in development - if you see any potential issues, it would be great to get your feedback. Likewise, if you use any techniques here, do so at your own risk. When trying out new techniques, do so in safe profiles - not your main data repositories!

I implemented an event tracker and a superSetvar call in a rather dumb way and ended up destroying a days worth of bounce rate data! Hopefully you have been interested by this missive, perhaps even inspired but certainly warned!

We're refactoring quite a bit at the moment - being a Rails house we are making plenty of use of helpers to clean up the templates. Also, launching more/new experiments gets cleaner and easier as we progress.

Have you considered the impact this will have on bounce rate? Running an event when the experiment runs without user-interaction will push bounce rate down/eliminate it for anyone who is using the test.

Caleb, Yes! An excellent point, thanks for bringing this up. As with any experimental technique, I would always suggest pushing the data through an experimental profile. I'm glad we did this for this very reason!

Using this technique in future where an event is fired without user interaction, we would definitely use another profile.

Moneyspyder

MoneySpyder: develops and hosts state-of-the-art e-commerce sites and continuously improves the customer experience using web analytics, split-testing and regular site enhancements. We think of it more as a partnership rather than simply a service.

E-Commerce Services: leading edge technology and a guaranteed commitment of time and expertise, week on week, to improve your e-commerce site.