A/B Testing with Webrat

It all began a year ago. We wanted to introduce a new navigation bar, but gradually, gaining feedback (and confidence) that it was the right upgrade for our users. My idea at that time was a random cookie which would show 20% of our users the new navigation while giving them a separate google analytics code for tracking their visit. And it worked well. Within a few weeks we were able to increase the cookie up to 100% and then simply throw it away.

GUI Based A/B testing in a custom CMS

But the idea remained. We now had a successful prototype for A/B testing. The next hurdle was introducing it into the CMS. Implementing a new page in the code for every hairbrained, well-intentioned GUI change is not exactly exciting stuff for web developers to do. But, this is exactly what our half-assed solution required. A product manager added a new A/B test to the backlog for a developer to implement. How about introducing /page/A/ and /page/B/ params to the CMS system?

Development was fairly straightforward. A single new table in the DB to maintain the new A/B mapping, and a few day’s development later, the product management and editorial staff were able to create their own A/B tests!

The next question was how to ensure we served A/B pages as requested, not to mention the myriad of edge cases (B page requests for pages that only existed in A form, etc.). We needed acceptance tests.

Webrat saves the day (again)

As our backend system is behind an SSL authentication mechanism, Mechanize traps the 401 code and we authenticate with our own do_visit (patch is here). Our build server is a Debian Lenny using Ruby 1.8.7 and Webrat 0.6.0.

Now that we have stable access to the CMS, we can write a webrat test for our A/B testing functionality. Let’s create a B page:

And that’s it. Of course, you need to do some extra testing for the boundary cases, but I’ll leave this as an excercise for the reader 😉 …
Have any webrat tricks you’d like to share with us? Let us know in the comments!