Have A Testing Solution

You need to do lots of split (aka A/B) testing and you may need to customize it

Therefore:

Choose testing tool(s) you can use right away and improve later.

we used vanity out of the box for a long time.

we use vanity for page level (e.g. homepage), feature level (e.g. presence or absence of “compare two store”) and within-a-page tests (e.g. presence or absence of list of produce on a family preferences page)

Google has tools – as do other services, but they’re more focused on page-level testing. We wanted to be able to choose the granularity.

We recently modified vanity to let us do splits other than 50%/50% as we wanted to use it to test with a smaller / controlled sample

However, since vanity only measures one metrics per test, and since we often care about two or more metrics – we augment the reporting by combining vanity data with our event logs (HAVE METRICS ON EVERYTHING) to form our own views of the experiment results