Statistical Validity

Split has two options for you to use to determine which alternative is the best.

The first option (default on the dashboard) uses a z test (n>30) for the difference between your control and alternative conversion rates to calculate statistical significance. This test will tell you whether an alternative is better or worse than your control, but it will not distinguish between which alternative is the best in an experiment with multiple alternatives. Split will only tell you if your experiment is 90%, 95%, or 99% significant, and this test only works if you have more than 30 participants and 5 conversions for each branch.

As per this blog post on the pitfalls of A/B testing, it is highly recommended that you determine your requisite sample size for each branch before running the experiment. Otherwise, you'll have an increased rate of false positives (experiments which show a significant effect where really there is none).

The second option uses simulations from a beta distribution to determine the probability that the given alternative is the winner compared to all other alternatives. You can view these probabilities by clicking on the drop-down menu labeled "Confidence." This option should be used when the experiment has more than just 1 control and 1 alternative. It can also be used for a simple, 2-alternative A/B test.

Extras

Weighted alternatives

Perhaps you only want to show an alternative to 10% of your visitors because it is very experimental or not yet fully load tested.

To do this you can pass a weight with each alternative in the following ways:

This will only show the new alternative to visitors 1 in 10 times, the default weight for an alternative is 1.

Overriding alternatives

For development and testing, you may wish to force your app to always return an alternative.
You can do this by passing it as a parameter in the url.

If you have an experiment called button_color with alternatives called red and blue used on your homepage, a url such as:

http://myawesomesite.com?button_color=red

will always have red buttons. This won't be stored in your session or count towards to results, unless you set the store_override configuration option.

In the event you want to disable all tests without having to know the individual experiment names, add a SPLIT_DISABLE query parameter.

http://myawesomesite.com?SPLIT_DISABLE=trues

It is not required to send SPLIT_DISABLE=false to activate Split.

Starting experiments manually

By default new AB tests will be active right after deployment. In case you would like to start new test a while after
the deploy, you can do it by setting the start_manually configuration option to true.

After choosing this option tests won't be started right after deploy, but after pressing the Start button in Split admin dashboard.

Reset after completion

When a user completes a test their session is reset so that they may start the test again in the future.

To stop this behaviour you can pass the following option to the finished method:

finished('experiment_name',:reset=>false)

The user will then always see the alternative they started with.

Multiple experiments at once

By default Split will avoid users participating in multiple experiments at once. This means you are less likely to skew results by adding in more variation to your tests.

To stop this behaviour and allow users to participate in multiple experiments at once enable the allow_multiple_experiments config option like so:

Split.configuredo|config|config.allow_multiple_experiments=trueend

Experiment Persistence

Split comes with three built-in persistence adapters for storing users and the alternatives they've been given for each experiment.

By default Split will store the tests for each user in the session.

You can optionally configure Split to use a cookie, Redis, or any custom adapter of your choosing.

Cookies

Split.configuredo|config|config.persistence=:cookieend

Note: Using cookies depends on ActionDispatch::Cookies or any identical API

Redis

Using Redis will allow ab_users to persist across sessions or machines.

Experiment Hooks

You can assign a proc that will be called when an experiment is reset or deleted. You can use these hooks to call methods within your application to keep data related to experiments in sync with Split.

You can even use Devise or any other Warden-based authentication method to authorize users. Just replace mount Split::Dashboard, :at => 'split' in config/routes.rb with the following:
ruby
match "/split" => Split::Dashboard, :anchor => false, :via => [:get, :post], :constraints => lambda { |request|
request.env['warden'].authenticated? # are we authenticated?
request.env['warden'].authenticate! # authenticate if not already
# or even check any other condition such as request.env['warden'].user.is_admin?
}

Screenshot

Configuration

You can override the default configuration options of Split like so:

Split.configuredo|config|config.db_failover=true# handle redis errors gracefully
config.db_failover_on_db_error=proc{|error|Rails.logger.error(error.message)}config.allow_multiple_experiments=trueconfig.enabled=trueconfig.persistence=Split::Persistence::SessionAdapter#config.start_manually = false ## new test will have to be started manually from the admin panel. default false
config.include_rails_helper=trueend

You can set different Redis host via environment variable REDIS_URL.

Filtering

In most scenarios you don't want to have AB-Testing enabled for web spiders, robots or special groups of users.
Split provides functionality to filter this based on a predefined, extensible list of bots, IP-lists or custom exclude logic.

Split.configuredo|config|# bot config
config.robot_regex=/my_custom_robot_regex/# or
config.bots['newbot']="Description for bot with 'newbot' user agent, which will be added to config.robot_regex for exclusion"# IP config
config.ignore_ip_addresses<<'81.19.48.130'# or regex: /81\.19\.48\.[0-9]+/
# or provide your own filter functionality, the default is proc{ |request| is_robot? || is_ignored_ip_address? }
config.ignore_filter=proc{|request|CustomExcludeLogic.excludes?(request)}end

Experiment configuration

Instead of providing the experiment options inline, you can store them
in a hash. This hash can control your experiment's alternatives, weights,
algorithm and if the experiment resets once finished:

Metrics

You might wish to track generic metrics, such as conversions, and use
those to complete multiple different experiments without adding more to
your code. You can use the configuration hash to do this, thanks to
the :metric option.

NOTE: This does not mean that a single experiment can have/complete progressive goals.

Good Example: Test if listing Plan A first result in more conversions to Plan A (goal: "plana_conversion") or Plan B (goal: "planb_conversion").

Bad Example: Test if button color increases conversion rate through multiple steps of a funnel. THIS WILL NOT WORK.

DB failover solution

Due to the fact that Redis has no automatic failover mechanism, it's
possible to switch on the db_failover config option, so that ab_test
and finished will not crash in case of a db failure. ab_test always
delivers alternative A (the first one) in that case.

It's also possible to set a db_failover_on_db_error callback (proc)
for example to log these errors via Rails.logger.

Redis

You may want to change the Redis host and port Split connects to, or
set various other options at startup.

Split has a redis setter which can be given a string or a Redis
object. This means if you're already using Redis in your app, Split
can re-use the existing connection.

String: Split.redis = 'localhost:6379'

Redis: Split.redis = $redis

For our rails app we have a config/initializers/split.rb file where
we load config/split.yml by hand and set the Redis information
appropriately.

Outside of a Web Session

Alternatively, you can access the underlying Metric, Trial, Experiment and Alternative objects to
conduct experiments that are not tied to a web session.

# create a new experiment
experiment=Split::Experiment.find_or_create('color','red','blue')# create a new trial
trial=Split::Trial.new(:experiment=>experiment)# run trial
trial.choose!# get the result, returns either red or blue
trial.alternative.name# if the goal has been achieved, increment the successful completions for this alternative.
ifgoal_acheived?trial.complete!end

Algorithms

By default, Split ships with Split::Algorithms::WeightedSample that randomly selects from possible alternatives for a traditional a/b test.
It is possible to specify static weights to favor certain alternatives.

Split::Algorithms::Whiplash is an implementation of a multi-armed bandit algorithm.
This algorithm will automatically weight the alternatives based on their relative performance,
choosing the better-performing ones more often as trials are completed.

Users may also write their own algorithms. The default algorithm may be specified globally in the configuration file, or on a per experiment basis using the experiments hash of the configuration file.

To change the algorithm globally for all experiments, use the following in your initializer: