Do your tests measure for success?

A common disappointment among companies deploying testing and optimization technology stems from tests that do not yield the radical improvements expected. Working with our customers, we've seen that sometimes even the most dramatic design changes produce no significant differences in their click-through or conversion rates. While that can be surprising, much of its concomitant disappointment can be eliminated by defining upfront what success or failure means to you.

For example, success in testing can be measured many different ways, depending on your goals and your organization. There is no one optimal way to measure testing success:

For some, success is a dramatic increase in a revenue-based metric, knowing that most senior stakeholders will respond to incremental revenue.

For others, success is a small increase in key visitor engagement metrics, knowing that a series of small gains eventually adds up.

For still others, success is a reduction in the number of problems present throughout the site, knowing that reducing these barriers improves usability.

For some, especially those with an increasingly dated site, success is simply being able to deploy a new look without negatively affectting existing key performance indicators.

And here's where it gets interesting. Sometimes our customers think that because they have not experienced what they consider to be success, they have failed. But that's not necessarily the case, because testing powers a continual learning process about your visitors and customers.

For example, if a particular image fails to increase conversion rates, you have learned that your audience does not respond to that particular image. In this context, there is no such thing as a failure in testing, only a failure to achieve the specific defined objective.

We know that not every test can yield incremental millions of dollars in revenue for your business. Some tests will fail to produce the change desired; others will yield results but not for your key performance indicators; and still others will simply fail to produce statistically relevant differences. Even so, there are no "failures" in testing other than a failure to carefully design your tests and a failure to carefully consider what you've learned. If you have learned what doesn't work, congratulations -- that's considered a success when testing.

Measuring testingOne way to measure your testing program is to integrate test data with your web analytics. There is a very natural relationship between testing platforms and analytics applications -- one is designed to drive improvements on your site and the other is designed to help quantify the value of those improvements. The combination of testing and analytics should support both optimization and an incremental learning process about your visitors and customers.

The basic integration of testing and measurement systems involves exchanging data about test participation, either through an after-the-fact bulk data loading or through real-time tag transformation. The most fundamental and important data to pass is some kind of test identifier -- whatever value your testing application is using to keep track of the test(s) in which your visitors are actively participating.

More importantly, you need to ensure that the data are flowing into the analytics application and can be used to create appropriate metrics and visitor segments necessary for deeper analysis of the test. Simply having an ID to indicate participation in "any test" is not enough. We recommend passing data that will allow identification of the visitor or session level for each test being run. Ideally, your analytics platform will allow you to load test metadata to increase the granularity against which analysis can be performed.

If you're not able to get this level of granularity, not to worry -- you will still be able to benefit from testing. But keep this practice in mind as you upgrade and evolve your measurement technology, and always look for opportunities to dig deeper into your test results through analytics.

Done well, this type of integration allows the measurement team to create segments, build key performance indicators, and drill down into the activity of individual visitors based on test participation. You'll be able to evaluate your testing efforts over multiple criteria and be able to evaluate test participant behavior over multiple sessions -- what better way to measure success or failure?

Kim Ann King is an award-winning marketing executive and has been a B2B software marketer for more than nearly three decades. She currently serves as the Chief Marketing Officer of web and mobile optimization firm SiteSpect. Her first book,...