HotelTonight A/B Tests and Finds Surprising Results

Apptimize

Feb 11, 2015

HotelTonight skyrocketed up the app lists in the last several years to take a leading position in the mobile travel market primarily because they are fast moving company with a laser focus on user experience and design. At Apptimize, we often hear that mobile app teams struggle with the fine line between design led UX and data driven conversion optimization. The HotelTonight team has figured out how data (qualitative and quantitative) and design aesthetic can work nicely together to create a cohesive strategy. Last week, I visited HotelTonight’s beautiful office in downtown SF to chat with Audrey Tsang, their Director of Product, about how they have integrated A/B testing to their culture.

Apptimize: How has A/B testing affected the way you think about user experience?

Audrey: Without measuring, you don’t know how you’re doing, right? Without measuring, you can’t improve. So A/B testing is a great way to do that. Our inventory changes a lot day to day, market-to-market. There’s all kinds of things that impact the user’s experience and in order to control for that, we have to be able to test simultaneously. So really A/B testing helps us bring down the noise so we can draw out the insights.

A/B testing has allowed us to question the assumptions that we make about our user experience. It’s made us revisit some assumptions that we had held dear. Some exciting “oh…” moments have been where we really thought that something would make a difference and what we ended up finding out was that it doesn’t make any difference at all.

Apptimize: Can you tell us an experiment that really surprised you?

Audrey: Some of the hardest to change parts are the ones that say, take the most rebuilding to do. And we had one experiment where we really wanted to question that. It would have cut out a big step in our checkout flow. And we’ve always relied on that step before. What A/B testing allowed us to do was take a risk and test an alternative way that reduced friction. We could evaluate whether or not it was worth it, and then do the real hard work of fixing the architecture to match that change if it proved worthwhile.

A/B testing has been instrumental in helping us refine and iterate on our core product.

Apptimize: With all the experiments that you want to run, how do you manage and prioritize everyone’s ideas?

Audrey: We keep an experiment backlog that anyone in the company can contribute to. We then take it and scope it out, which involves back of the envelope projections on impact to the user problem that we’re trying to solve or the hypothesis we’re making. From there, we can prioritize the backlog for further investigation before building and running the experiment.

Before A/B testing, we still had a feature backlog. We had something called a war room process where, every quarter, we decide on the features and the strategic initiatives we’re going to focus on for the quarter. That process has a similar democratic approach to it where we solicit ideas from the whole company and this is to give a cross-functional view on what needs to be changed and what can be improved upon.

With A/B testing, we can iterate faster by creating experiments as opposed to full blown features. So while those same ideas have kind of lived on in both lists throughout the history of the company, we’ve been able to modify how we work a little bit with the experiments. We’ve been able to move faster, iterate faster, and take a few more risks.

More articles you might be interested in:

HotelTonight is the leading mobile-only solution for last minute hotel bookings, with over 20 million app downloads across 1,700 markets in 36 countries. THE CHALLENGE Even though HotelTonight is far ahead of its competition in the steps and time needed...

How HotelTonight Stays Ahead in the Watch Game

While there are a bunch of novel options for the new Apple Watch, HotelTonight is one company that’s in the lead pack. Not only have they produced an awesome app to launch on day one, they’ve already discovered and pre-emptively...

HotelTonight A/B Tests with Apptimize

HotelTonight, a booming mobile-first travel company, is doing things that their competitors can’t do. This is due to their focus on customer experience, desire for innovation, and attention to data. In this video, Director of Product, Audrey Tsang, talks about...

Schedule A Customized Demo

Get in touch with us for a detailed walk-through of Apptimize.

About Apptimize

Apptimize empowers product teams to efficiently run A/B tests, rollout and manage new features, and deliver personalized user experiences. Based in San Francisco, Apptimize is backed by US Venture Partners, Google Ventures, Y Combinator, and others.