DON’T MISS OUT

How to Choose the Right Testing Software For Your Business

If you’ve spent any amount of time on The Daily Egg or similarly focused blogs, you know that thorough testing is paramount to your web business’ success.

Every customer base is unique, and winning marketing victories depends on understanding the users visiting your website each and every day. You might be intuitive (or lucky) enough to correctly read your customers on any given campaign, but over the long run, success WILL depend on data analysis.

Despite the hundreds of voices claiming that marketing is an art, you’ve heard of Neil Patel, Rand Fishkin, Avinash Kaushik, Danny Sullivan, and Brian Clark for one reason: They understand marketing is a science.

Unfortunately for the the small business owner, data analysis is decidedly technical. Simply reading a few blogs won’t be enough to turn data into dollars. You’ll need a handful of powerful tools at your disposable.

Fortunately, there are an abundance of such tools available.
If you have a marketing team ready to pounce on this, these programs will provide an effective setup for efficient testing. If the burden falls on you, the business owner, there will be a slight learning curve. But don’t worry, each of the programs listed here are highly successful, fully supported, and come with a dedicated team invested in making your experience flawless.

Keep reading for our review of six A/B testing software.

1. Google Analytics

As the only free option on our list, Google Analytics is the baseline of all Internet data analysis. It offers users the rare opportunity to draw directly from Google’s extensive data pool. If you’ve never used Google Analytics before, or if you’ve wandered around the dashboard a few times without really grasping its potential, here’s a great place to start.

Once you’ve learned the basics, it’s time to use this new-found data to drive revenue for your business. This requires actionable steps, and you can begin here.

So what about A/B testing?

Analytics offers a feature called “Content Experiments,” which allows users to split-test page variants within the program. It’s a useful feature with advanced reporting options but unfortunately lacks a visual editor, posing a major stumbling block for many would-be users.

The main drawback to Google Analytics is the steep learning curve.

There’s a wealth of information there, but you’ll need to be a pretty proficient analyst to take advantage. Advanced use of Google Analytics should be an aspiration of every business owner, but it’s definitely not the easiest place to start.

2. Optimizely

Optimizely is a great first stop for business owners wanting to start testing. Installation is remarkably simple, and the WYSIWYG interface is highly intuitive, even for newbies. With plans starting at $17, Optimizely is a very cost-efficient entry point.

But don’t mistake simplicity and budget pricing for an inferior product. Optimizely’s testing software is good. Very good. With split, multivariate, and mobile testing options, you’ll be teaching data anlysis by the time you’ve outgrown this product’s utility.

Plus, as your company grows and you become more familiar with testing, advanced features at the higher price point allow for continued use without needing to learn completely new software.

If you’re starting completely from scratch, Optimizely might not be your best option. You’ll need passable landing pages to test in the first place, and you are entirely responsible for creating your own A/B tests. This is a pretty big prerequisite for someone whose marketing knowledge consists of nothing more than a few blog post reads.

The Pros

Cost-efficient premium option for entry-level users

Easy, intuitive interface

Effective A/B, multivariate, and mobile-focused testing options

Solid basic offering with quality advanced features at higher price point

The Cons

Lacks outside data integration

No algorithm alternatives for judging results, such as bandit or adaptive testing

2nd Opinion

“To start, Optimizely is very competitively priced and dead simple to set-up and install, so the bar to start getting a return on your investment starts off low. From there, it’s up to you. What ideas do you have to improve your digital business? Optimizely is a tool that will help you test and measure them, and quantifiably tell you if they work or not. Your return will be as good as your testing ideas.”

3. Visual Website Optimizer

Visual Website Optimizer (VWO) is a full-featured testing suite that’s achieved tremendous popularity in the last two years. Overall, it’s very similar to Optmizely but with a few key differences.

At $49 per month for the starting price point, it’s not exactly the “Why not?” pricing of Optimizely, but you can expect a much wider feature set for the money. VWO boasts a simple, easy-to-use interface and comes with an impressive 7 different testing options:

A/B Test

Multivariate Test

Split URL Test

Targeting

Usability Test

Conversion Tracking

Heatmap and Clicks

The software also includes a handful of idea-generation tools. This is a unique selling point for business owners unsure of where to start in creating and running A/B tests.

Overall, VWO’s benefit to your company depends on your needs. If you need something dirt cheap to get started, or something very advanced, Optimizely is probably the way to go. If you’re needing beginner- to intermediate-level testing options at a traffic level greater than 1,000 visitors per month, VWO is the perfect tool for your business.

The Pros

Feature-rich for the price – full feature set available at all price points

Intuitive, easy-to-use interface

Idea-generation tools

The Cons

Lacks same level of real-time support options compared to competitors

Lacks multi-page testing options

Hit or miss online documentation

2nd Opinion

“Overall, Visual Website Optimizer has the slight edge for casual A/B testers, but its more expensive than the basic version of Optimizely. For serious A/B testers, I definitely recommend Optimizely – more comprehensive offering, even though you have to pay extra for premium features.”

4. Unbounce

Unbounce has carved out a niche for itself by focusing solely on landing pages. That’s it. Just landing pages. And they’ve managed to make themselves somewhat of an authority on the topic as a result.

Unbounce’s unique distinction is that they offer 80+ pre-designed, optimized landing page templates. This is a fantastic resource for business owners starting from scratch or who have seen little success with their own designs. Each template is engineered by the Unbounce team for the purpose of conversions and can be quickly customized for your company’s use.

In addition to templates, an Unbounce subscription will allow you to perform A/B testing on your landing page. You’ll be given a few metrics for interpreting the data, and that’s it. Other than a few periphery extras for the higher-priced options, the full Unbounce experience consists of landing page templates and A/B split testing.

If you’re a true marketing rookie and want results fast, this is probably your best bet. Few other programs cater to ground-zero quite like Unbounce, and you’ll be able to publish conversion-optimized landing pages in a matter of hours. To sum things up, Unbounce is limited in its goals but accomplishes those goals admirably.

The Pros

Landing page templates provide great start-from-scratch resource

Solid split testing capablities

Simple, easy-to-use interface

Designed to easily integrate with other marketing and testing software

The Cons

Limited to simple A/B split testing

Relatively high cost for a limited feature set

2nd Opinion

When creating a new landing page, Unbounce is our company’s go-to for A/B testing conversions. It is quick, gets the job done, and the analytics are actionable. We can’t live without its confidence score rating, which tells you how likely a good conversion rate is, or whether it is mere luck.

5. KISSmetrics

Unless you’ve been living in a box for the last few years, you’ve heard of KISSmetrics. This popular testing tool focuses on the human side of data. Reports and testing data are connected to real people, who are tracked across their interactions with your website.

At $150 per month to start (with full-year commitment), KISSmetrics is definitely a premium service, but it comes with the advanced abilities you’d expect at that price point. You won’t find software offering deeper or more diverse sets of data. KISSmetrics lets you keep a pulse on the entire conversion funnel, from start to finish, and identify where and how customers are disconnecting from your website.

If you are serious about data, KISSmetrics is one of the best tools around. Its ability to connect data to actual users might also be a great alternative for business owners struggling to wrap their heads around more traditional reporting styles.

The Pros

Highly flexible data reporting options

Simple, intuitive interface

Can trace data to real people

Excels at funnels with extremely accurate funnel-based data

Integrates well with complementary software

The Cons

Lower powered A/B testing compared to other testing software

High cost – won’t be affordable for many small businesses

More advanced data reporting requires a higher learning curve

2nd Opinion

“I use other split-testing tools, such as Optimizely, but KISSmetrics is a must if you’re serious about getting accurate results. The problem with many of the tools out there is they don’t measure the impact of the test throughout your entire funnel. That’s why I love KISSmetrics; it will show you exactly what the impact is for a test at every step of the buying process (not just the next step).”

6. Crazy Egg

Of course, our list wouldn’t be complete without the home team’s offering. Crazy Egg specializes in click-tracking. The goal of all testing is to understand user behavior, and Crazy Egg’s software allows business owners to identify exactly what users are doing when they visit your website.

If you’re a visual learner, Crazy Egg will be your go-to source for data. The six different available reports frame data in an easily digestible way that is intuitive and provides immediate insight into user activity, regardless of your analytical skills.

If you’re at least capable of building a landing page, Crazy Egg is the ultimate entry-level option. At $9 per month, it is the cheapest testing software you’ll find anywhere.

While it is decidedly limited in scope and test variety, you’ll find it much easier to use than Google Analytics and most other options on this list. It is, perhaps, the only offering on this list that allows business owners to immediately read and understand test data with zero context for web analytics.

The Pros

Highly visual data – enjoyable to view

The benchmark of interface simplicity

Extremely low learning curve with intuitive reporting

Cheapest pricing you’ll find for testing software

The Cons

Limited feature sets and testing options

Limited utility for dynamic pages

2nd Opinion

“It’s a great tool considering how inexpensive it is. If used correctly and you have a plan for tracking your websites, this tool can make a world of a difference.”

24 COMMENTS

Comment Policy

Please join the conversation! We like long and thoughtful communication.
Abrupt comments and gibberish will not be approved. Please, only use your real name, not your business name or keywords. We rarely allow links in your comment.
Finally, please use your favorite personal social media profile for the website field.

SPEAK YOUR MIND

In order to take customers onboard your modification team and to minimize the chances of revenue and customer loss, A/B testing has become the sure shot way for various business organizations. A/B testing not only helps in smooth transition of the changes but also helps in improving the user experience and conversion rates of a website. Some of the tools that can help in this direction are Google Analytics, Unbounce, Optimizely and such others. I would also like to mention Mocking Fish as it is lower in cost, provides more accurate real time results and is easy to implement.

Thanks for this detailed review. Would love to add 3MinuteOptimizer (disclaimer: I’m the founder of the business) which focus on exclusively on setting up split URL tests fast. Hope we could make it to the list.

You also forget such tool as a Changeagain.me
It’s the easiest A/B testing toll based on Google Experiments API, so it’s fully integrated with your account in Google Analytics. So all the data is gathered with maximum accuracy.

Hey Jacob! Really appreciate you mentioning Unbounce in this list. Have to give props to the other 5 you chose to highlight here 🙂 All do a great job in their specialty. I also liked the “second opinion” quotes—something different that I haven’t really seen done before in a top tools type of post that adds credibility. Thanks!

Google Content Experiments can also be used to run automated A/B tests. From a high level, the way this approach works is to start showing variations that work better more often, automatically. It’s a much better solution to the question of ‘What works better: A or B?”

In the most basic example of an A/B test, you have a variation A and a variation B each shown to 50% of your user base. By definition, this approach will be sending half of your users to a worse performing version during the entire duration of the test!

The automated approach is based on a bandit algorithm that updates the percentage of your user base which is shown a given variation dynamically. With each new piece of data that you collect on the conversion rate and statistical confidence of the variations in a test, it adjusts the percentages automatically so that better performing variations are promoted and worse performers are pruned away.

This leads to:

1) faster results, because your directing test resources (i.e.: users and their data) to validate what you actually care about (i.e.: confidence in the best variation’s performance)

2) a higher average conversion rate during the test itself, because relatively more users are being sent to the better performing variation automatically, and

3) less time and effort required to actively manage your experiments.

Though the math behind this approach is slightly more complex than a traditional A/B test, it’s worth talking about for audiences that are interested in making data-driven decisions because of how much better the results are that it produces.

Great comment Zac – I’m convinced minimizing regret is the way to go, but I think there are limitations/concerns with existing offerings – Auto-optimization sounds great in theory, but the reality is it isn’t as simple.

For example, most of the available tools haven’t implemented contextually aware auto-optimization methods, so you ultimately end up creating a lot more work implementing the same experiment for many different user segments (a time consuming pain point), or you settle for, “what’s the best option for most”.

Separately, overlapping experiments with an auto-optimization setup, where one experiments better-performing arm can inadvertently skew (positively) the results of another experiments inferior arm, is a real problem, and most tools available don’t offer a solution to it.

There are also issues with traffic quality and returning customer bias (through things like the recency/novelty effect) in-advertently skewing results of one arm, when it in fact might be the inferior one – in this situation, you would in-advertently be maximizing, not minimizing regret, and depending on the method of your “stopping point criteria”, might lead to wrong conclusions being made.

A frequentist approach *typically* doesn’t suffer the above issues, since you are typically doing a 50/50 split of traffic, and can then segment results. There are, of course, ways to address these issues in an auto-optimization setup, but as of yet I haven’t really found a tool that does so.