A/B Test Analysis

There are two ways you can see detailed analytics on your A/B tests to determine their effectiveness on your users:

Click Analytics in the A/B Testing section;

Or for more detailed reporting, you can look at your A/B testing results as an event in your Funnel and Retention reports.

Analytics in A/B Testing

Click Analytics in A/B Testing to see basic analytics on your test, including the conversion rate to a particular event from your experiment and its variants:

The "Analytics" section will also show you the number of users participating in each variant as a baseline for looking at conversions.

View A/B Test Results Within Mixpanel Reports

When you add an experiment in A/B Testing, Mixpanel automatically tracks an event, property, and super property associated with that experiment so that you can use Retention and Funnels for a more in-depth analysis of your test. For example, you can build a Funnel with multiple steps after the Experiment Started event or use Retention to see how the A/B test affects the long-term behavior of your users:

Experiment Started event, which fires only at the point the experiment is delivered to the user’s device.

Experiment Id property, which fires along with the Experiment Started event and lets you know which experiment the user saw.

Experiments property, which stores a list of all experiments and variants a user has been exposed to as a super property.

Use the Experiment Started event and corresponding Experiment Id property in your reports if you want to understand and analyze an active experiment.

Use the Experiments property and segment by the specific experiment if you want to understand the behavior of users exposed to an experiment previously. Note that if you filter a report by an Experiments property you will be looking at people exposed to that experiment at some point in time, but not necessarily recently, which is why as a best practice, you should always use Experiment Started and Experiment Id to analyze an active experiment.

Note:

The Experiments property is not visible on user profiles, though you can filter for it in Explore.

You can use the Experiment Started event to target notifications, but you cannot target notifications based on the Experiments property.

Our Chance to Beat Original values are calculated with a one-sided z-test with a 95% Confidence Interval.