Re: Calculating stat. significance for revenue and the choice of using RPV over AOV

RE: Revenue Statistical Significance - See this note from our resident statistician, Leo: (I know its titled RPV but is relevant for total revenue)

We're also planning on publishing a white paper on the statistics of calculating significance for continuous variable metrics like Revenue.

How does Stats Engine work with a revenue per visitor goal?

Stats Engine works as intended on revenue per visitor goals. You can look at your results at any time you want and get an accurate assessment of your error rates on winners and losers, as well as confidence intervals on the average revenue per visitor.

In fact, your estimates should be more reliable, sooner, because we now correct for the inherent skewness in calculations based on revenue. One of the changes we made with Stats Engine is we now compute skew corrected test statistics for revenue (or any other goal that can potentially take on many values). Significance values are adjusted by a correction factor which estimates skewness from your currently running experiment. Not only does this make results reliable with considerably fewer visitors, but it also results in a more powerful test (when looking at historical revenue tests, the number of conclusive results jumped by a factor of 1.5).

Another feature of skew corrections is the resulting confidence intervals are no longer symmetric, but naturally adapt in the direction of the skew.

RE: RPV over AOV -

I can't speak to whether RPV or AOV is a more valuable metric; they accomplish different things, and for some customers I've spoken to, RPV is exactly what they've been looking for.

If you'd like to use Optimizely to report on AOV, my colleague @Brett offered the following advice:

You could create dimensions that only includes people if they make a purchase that is greater than zero dollars; this effectively removes the people who don't make a purchase from the segment.

Then, when looking at that segment, the revenue metric would be the equivalent of looking at "average order value" instead of "average visitor value".

You can also employ an integration from Optimizely another analytics tool, such as an e-commerce database or a web analytics platform like Adobe or Google.

Re: Calculating stat. significance for revenue and the choice of using RPV over AOV

A couple things to consider when selecting AOV or Revenue Per Visitor/User/Customer:

Your comparison assumes that all observations areindependent and identically distributed (iid). This means each observation shares nothing in common other than being drawn from the same distribution. Here's an example of why this might not hold true when using AOV: It is a reasonableassumption that a company will have repeat orders from customers. If today a customer purchases a video game console it is also a reasonableassumption that sometime in the futurethey might purchase accessories (like a controller), video games, etc. As such, when accounting for AOV you have to be selective, such as only using a customer's first AOV for analysis, excluding any subsequent order's AOV (from same customer). The problem here is, this is valuable information! Obviously you'd want to know the full impact of how an experiment/variation influenced purchase behavior. This is often why Revenue Per Visitor/User/Customer is used.

On Revenue Per Visitor/User/Customer, it is important that you allow the same data collection period for all visitors to ensure you collect all data required for the comparison. As an example, if you were to end an experiment after four weeks’ time, visitors who entered the experiment Week 1 would have had four full weeks of purchase history tied to them, whereas visitors who entered the experiment Week 4 would only have a single weeks’ worth of purchase history tied to them. To account for this, if you chose to stop your experiment after Week 4, you'd want to data collection to continue through Week 8 to allow equal time for RPV to build for all individuals included in the experiment. In this example, you would not count in your analysis any new visitors to the experiment after Week 4, and not attribute any transaction data from Week 5 and beyond to customer who entered your experiment Week 1 (or from Week 6 and beyond to a customer who entered in Week 2, and so on).

I think your question mostly is meant to figure out not the statistics, but simply tracking events for continous variables. You can actually accomplish that today by toggling the results page to 'totals'. You can read how here.