Search

Split Testing

It's nice to have many people visit your Web site. It's even better
when people don't just come to your site, but also enjoy your content.
But, best of all is when visitors to your site do what you would
like them to do—sign up for your newsletter, register for your SaaS
application or buy one of your products.

The rate at which visitors become customers is called the "conversion
rate", and it's probably the top priority for Web-based businesses.
If you can convert 10% of your visitors into customers,
you're doing twice as well as the person who can convert 5% of
visitors into customers.

What leads people to convert more often? That's a question to which I'm
still learning the answer, as is an entire industry of Internet
marketers and "conversion optimization" experts. The thing is, it's
often difficult to know what will work. Should you use a green
button or a red one? Should your headline be a question or a
statement?

One of the most popular, and effective, ways to check the
effectiveness of your copy is to do "split testing", sometimes known
as "A/B testing". Although I heard about A/B testing years ago, I have
only recently actually started to apply it.

In this article, I describe a simple way to run split tests on your Web
pages. The examples I show here are written in Ruby, but
there are A/B testing libraries for many different languages. There
also are third-party businesses that can help you with A/B testing,
either on your own Web site or on your mailing lists, lead pages and
other products.

It's All about Conversion

The most important thing to keep in mind when you're doing split
testing is that you're trying to get users to do something. What that
something is depends on your site. The goal at Amazon is to get you to
buy things. The goal at Google is to get you to click on ads. And
the goal for a mailing-list subscription form is to get people to sign
up.

Once visitors have achieved your goal, they have been
"converted" into
customers. The goal is to increase the number of such conversions—giving you more customers, subscribers or users of your system. The
key insight with split testing is that by changing the text, graphics
and even layout of your page, the number of conversions will change as
well. The question is, which of your various ideas will work best?

Split testing uses the scientific method, backed by simple statistics,
to try to answer that question. In science, you can test a hypothesis
by using a control and an experiment. Split testing works the same
way. You take your existing text and an alternate text, and
display them to your users. One of them probably will work better
than the other. You analyze the numbers to understand which worked
better, and then you use the best one. Then you start all over
again, trying to find another improvement via split testing.

In order for split testing to work, you need to do several
things:

Define what counts as a conversion. This often is described in terms
of the user arriving at a particular page on the site, such as a
"thank you" for shopping that appears after a successful sale.

Define the control and alternate texts.

Wait for enough users to see both.

Analyze the numbers.

Introducing Split

Regular readers of this column know that I tend to use Ruby on Rails
for most of my Web development work. Rails isn't the only Web
framework written in Ruby, although it certainly is the most widely
used. All modern Ruby-based Web frameworks use "Rack" to communicate
with the HTTP server software that invokes them. This is analogous to
the Python world's WSGI. The idea in both of these cases is that the
HTTP server doesn't need to know much about the Ruby or Python
application (or framework) and vice versa. This means libraries
that know how to work with Rack can operate with any framework, not
just with Rails. In the specific case of Ruby, Rack-sensitive gems
thus can work with Rails, Sinatra or anything else that follows the
Rack API.

The Split gem, written by Andrew Nesbitt (with a number of
contributors), is one such package of code, able to provide
split-testing execution and analysis to any Rack-compatible
application server. It is based on a number of earlier split-testing
gems, including the well-known A/Bingo gem by Patrick Mackenzie. The
idea behind Split is you identify your conversion target,
automatically try two or more variations on users, collect statistics
on which of them actually managed to achieve more conversions, and
then start to test things again.

Split takes care of much of this work for you, allowing you to
concentrate on the portion of your site you want to change,
rather than the mechanics of setting up experiments and reporting
their results. Split also implements a number of features that ensure
the statistical power of your results. For example, it will
involve each participant in only a single experiment at a time, and it will
compare results only after at least 30 people have seen it. These
options can be changed, of course, but the defaults are more than good
enough for most basic tests that you'll want to run.

You install split as you would any other Ruby gem:

gem install split -V

Note that I use rvm, the Ruby version manager, which allows me to
work with multiple versions of Ruby at any given time. One side
effect of using rvm is that gems are installed in my own home
directory, rather than on the system. As a result, I don't need to
preface the gem command with
sudo; depending on your system
configuration, you might need to do so.

After installing the gem, you also will need to install and start
Redis, an extremely fast and popular key-value store. I have used
Redis in a number of projects through the years, and I never cease to be
amazed by its utility for caching. In the case of Split, Redis is
used to keep track of the experiments you are running. You can start
Redis by executing:

redis-server

Running a Split Test

Now that you have installed the gem and Redis, you're ready to perform
some experiments. Let's create a simple Web site using Sinatra, in
which the page displays a link. While Sinatra has a reputation for
being simple and for allowing you to create a single-page
application, I've generally gone for a configuration that uses several
files: the application itself, a Rack configuration file (setup.ru)
and a Gemfile to list all the Ruby gems I want to use (via the
"Bundler" management gem). These are shown in Listings 1, 2 and 3.

Listing 3. Gemfile, Used by Bundler

Let's review them briefly. The setup.ru file is the way that Rack runs
your application. If you want to run the application without any
external HTTP server, you can use the rackup command:

rackup config.ru

Rack then will run an HTTP server (defaulting to the built-in WEBrick
server that comes with Ruby), acting as the glue between the server
and your application. The config.ru basically bootstraps your
application, loading the gems listed in the Gemfile (thanks to
Bundler), then loading your application and the special code needed to
run the Split dashboard.

Then you do something interesting. Rather than simply running your
application, you tell Rack that it should route requests to
different places. Anything starting with a / should go to
LinkSplit.new, meaning a new instance of the Sinatra application. But,
anything starting with /split will be routed to a completely separate
Sinatra application, Split::Dashboard.new, part of the Split gem.

The Sinatra application defines two different URLs, both of which are
available via HTTP GET requests. The first, /foo, displays the
contents of the foo.erb file (located in views/foo.erb), as shown in
Listing 4. While this file seems, on the surface, to be no different
from any other ERb (embedded Ruby) document, it contains the telltale
sign of a split test:

in linksplit.rb does several things at once: it defines a split test
(the first parameter), and it provides the two alternatives that you
are interested in testing. In this example, you can see that you're
testing whether "Click on me!" (the control) converts more frequently
than "Click here!" (the experiment). You then pass a block to the
ab_test method. The block can contain any text you want, and it can be as
long or short as you want. The key thing to realize is that the block
parameter (click_text in this case) will contain one of the two text
strings.

The above split test, thus, will compare the efficacy of two different
links. But, you easily could switch the class of an HTML tag (thus
giving different styles, including colors and fonts), a different
location, the addition of a picture or anything else.

Once this is in place, the Split gem will produce an experiment,
displaying one of these text strings to each of your users.
Typically, you'll want to show them equally. There are ways to change
the ratios, so that you're showing your experimental text to only a
small proportion of your users.

However, showing these different texts isn't quite enough. You also
need to be able to tell Split when visitors have
"converted"—that is,
when they have achieved the goal that you set out. In this particular
case, you want to know which text is more effective at getting
users to click on the link. So, you should report a conversion back
to Split when the user has clicked on the link. In other words, in
the Sinatra application's handler for the "/bar" method, you invoke the
"finished" method, passing it the name of the experiment you want to
indicate was finished:

finished('click_text')

Once you have put a split test in place, you can sit back and wait for
users to come to your site. Some will get the first test, and some
will get the second. How do you know which was more effective? By
using the Split dashboard, which you connected to /split when you set up
routing for Rack. It will show the percentage of users who responded
to each of your experimental texts, so you can see which was more
effective. Split also will tell you the confidence that it has in the
results of this test. As a general rule, the more people you test, the
more confident you can be in the results. But, statistics also show
that even a (surprisingly) small sample size can give you
interesting and meaningful results.

The dashboard is useful in several ways. First, it allows you
to look at your various experiments, seeing how many people are
viewing each of your experimental texts, and how many did and didn't
complete the test. It also allows you, after examining the data, to
use one of the two experimental texts permanently. This is useful in
several ways. It means that a nonprogrammer can control and decide on
each text. But, it also means that you don't have to go back into the
code and remove your experiment when it is over. You can take some
time, letting the Split system handle it for you. You also can reset
the statistics, allowing you to try new ideas or throw out old ones.

Conclusion

Split testing is a powerful tool for helping ensure that your goals
are met—whether those goals are selling something on-line or just
getting people to sign up for your mailing list. The Split gem works
with any Rack-powered Web site, and it makes it extremely easy to
implement and then act on your optimization experiments. I encourage
you to try some experiments on your own Web applications and see if
you can find ways to get your users to convert into customers. If so,
congratulations! You've discovered that modern, on-line marketing
requires an understanding of programming as much as an understanding
of the customer.

Resources

The Split gem described here is at
https://github.com/andrew/split. As indicated in this
article, Split works
with any Rack-based Web application, which basically means Ruby on
Rails and Sinatra.

Patrick Mackenzie, the author of the A/Bingo gem,
has written extensively on the subject of conversion optimization.
You can read his articles on the subject at
http://www.kalzumeus.com. In particular, look for the
"conversion
optimization" heading on his "greatest hits" page.

Reuven M. Lerner, a longtime Web developer, offers training and consulting services in Python, Git, PostgreSQL and data science. He has written two programming ebooks (Practice Makes Python and Practice Makes Regexp) and publishes a free weekly newsletter for programmers, at http://lerner.co.il/newsletter. Reuven tweets at @reuvenmlerner and lives in Modi’in, Israel, with his wife and three children.