Monthly Archives: January 2010

When working as a web analyst I have discovered that you are more than just a guy measuring your visitors and pulling out the stats. To be really good at your work I think one needs several skills among them understanding what affects your conversion. I constantly try to get involved in all parts of the business, from SEO to PPC to CRM and any other aspect that apply to the business model. I must say this is not an easy task but I think it gives me a lot of pleasure sniffing on other people’s area of expertise. What surprises me every time is that something that looks very simple for the outside might be very complicated when you dig deeper. Anyhow, the other day I was thinking of an easy way to display what affects our conversion and I came up with the following picture (I have removed a few factors but you still get the picture).

I think that displaying your most important factors that affect your conversion in this way makes it easier for everyone in the company to understand how you can improve. The hard part is to identify these factors, hence the argument that you have to know your company´s business model very well.

Each of the factors identified here can have several sub factors. For example, looking at your entry page you have a design factor, how you present your text and what you really say (the value proposition).

So after finishing the drawing I realized two things. The first is that it gives you a very easy tool for communicating with various stake holders. The second thing was that it really showed the complexity of business optimization or in my case the complexity in my own role as a web analyst and conversion specialist. It was actually a bit of an aha-moment since to truly optimize our business I had to be that annoying Sherlock Holmes-type and not just mind my own business.

So, I would truly suggest you to make such a map show it to your various stake holders and ask them what they think. Even if they don’t like it, I think it will bring you a step closer to becoming that black belt web analytics expert.

A year ago when I changed jobs I was faced with the joyful task of being part of the startup of our A/B testing program. Since we have several sites under our wings the methodology was very important because we wanted to draw conclusions from our tests to be applied to new websites and in some sense create a kind of “theory” on website design and content.

With 150 A/B tests and counting our methodology has brought us a step closer to understanding what affects a webpage conversion rate. This knowledge has made our life a bit easier since we can apply the findings on new projects with the comfort of knowing what conversion rate we can expect. Without going any deeper into the core of the methodology I thought I would give you my approach on A/B testing.

The below points are a guide of what you should think of when doing A/B split testing and are based on my own experience. Please feel free to comment on them if you don’t agree.

1. Be prepared that your testing will bring up new question that will need testing. So you are basically in a testing process rather than just one test.

2. Think of a concept you want to test. Is this concept aligned with what users want or is it based on your own cravings? Very good inputs for testing are user surveys and focus groups.

3. Look how much traffic you have on the page(s) that you will be testing on. Large amount of traffic allows you to do smaller changes but still see an effect.

4. If you have large amount of traffic consider doing A/B/C/n testing. This will allow you to cut down your testing time considerably.

5. Analyze where the traffic to the testing page(s) comes from (I mean URLs). If a larger part of the traffic is from Google look at the keywords which show user intentions. Try also to understand how this page fits in the user path. Ask the following question: Is this page critical for users to convert?

6. Make a wireframe of the page you want to test. Make your various versions stand out from the original page. If they are too similar you will probably not reach statistical confidence, hence not be able to draw conclusions.

7. This is a no brainer but I need to say it, design the page so it fits the rest of the site.

8. Communicate clearly to your coding guy (fronted) what the success metric is. Usually it is conversion or/and money but it could also be reaching a certain page or downloading something. News site for example have more page views per user as a success metric.

9. It is very common in the beginning of the A/B testing experience that you don’t track everything on your testing page, which is a mistake. As the saying goes, it is better to be safe than sorry, so you better put more tracking elements on a testing page than you thought of. This will avoid you to redo the test later.

10. If you are working with segments, which you should, consider to divide your traffic into those segments. The segments could for example be based on entry keywords or previous page behavior.

11. Front end code your testing page(s) and setup your testing tool. Direct 50% of the traffic to one version. If you have very large amount of traffic you can do the test on only a fraction of your traffic. For example I heard that Google only tests on 10 of their traffic. As testing tools you can use Google Optimizer or any other tool that fits the requirements but remember to do QA so everything runs smoothly when you launch your test.

12. Launch the test. Keep an eye on it the first couple of hours so everything seems all right.

13. Let the test run until you reach at least 90% confidence. Usually the confidence level is shown in the tool.

14. When you reach confidence stop the test and do the analysis. Example of questions you should ask is; What do the numbers show? Did we reach our targets? Did we answer our hypothesis? Is this the best we can do or can we improve the numbers even more? What other questions type of questions did the test arise?

15. Document your results and conclusion in an easy way so you can go back whenever you need to. Documenting is important if you do many A/B tests since you probably want to use your knowledge further down the road.

16. At this point I must put in a warning since many people think that one test will be enough to get the lift you are looking for. Most of the times your results will be rather surprising which will more or less force you to do more tests. This is a good thing since the more you learn about your users the better conversion rate you will have.

17. As a final point, do not forgett to make the A/B testing process fun. If you can involve your designer and coder in the beginning of the process it will make your testing people and your whole organization much smarter in the long run.

I have worked as a web analyst for a while but I havent seen a post adressing the issue how to monitor a redesign release so I thought I would give you my view on how to monitor the immediate impacts of a redesign.

At the end of this post you will find a link to the excel document.

So, over the past six months we worked hard to make this really cool and useful redesign of our website. With a few hundred thousand of visits each day we had to get it right. The day of the launch came and within the hour I got the first question that would be repeated over the week “How does it perform?”. I looked at my web analytics package and could not see any immediate changes. Of course any of you out there that have had experience with redesign and in general hourly web analytics data know that the conversion rate, the visits, the bounce rate and such varies during the day and no immediate patterns are shown during such a short time.

Fast forward about 48 hours. The first numbers are showing up in my excel sheet and we could relax a bit. Everything looked good. Now let me get to the point, I composed an excel document to follow the effects of the redesign that I want to share with you. You can download the document at the end of the post.

On the top of the page I have collected the four most important metrics and I look at the averages per day. This way I can quickly see if the first day of the redesign looks fishy or not.

Monitor at least seven days, if you see major fluctuations do it for at least two weeks.

The second data set, the graphs, is the ones I like most since they are very simple to distribute and even the most novice can understand them with just a glance of an eye. The important part here is to calculate the standard deviation to have an upper and lower level, to understand if you should celebrate or investigate.

In the excel sheet you will also find the bounce rate. Of course if you want to have more metrics you can just create them, but these three are the most important according to my opinion.

Below you can see an example of the data I use to create the graphs.

I am not saying that this is the only way to do it, but I think it will be good for most cases. Of course I would never do a site wide redesign without testing it first, more on that another time.

My first post a second time. I mean I used to blogg in the beginning of days but stopped due to drought in the idea department but now I am back! I want to give something back to the community of e-business and more specifically the web analytics world and the e-business development, therefore I will blog about stuff like web analytics and e-business. I will try to be as personal as possible and keep it short and concice.