5 tips for A/B testing on Android versus iOS

Apptimize

Feb 10, 2014

If you have an Android and iOS app, your two versions are probably similar but not exactly the same. Your users are probably similar but not exactly the same. So make sure you don’t test on one platform and assume the results are the same for both. Here are some ideas of things to test that have yielded different results for some of our customers.

What to test: Think about testing multiple revenue strategies on both platforms. You might get different results. For example, if you’re an ecommerce app, try featuring items of different price tiers on your home page. Variant A might be featuring all items with a low price point, variant B at middle price point, and variant C at a premium.

Test different engagement strategies

Studies also show that iOS users tend to spend more time in-app than Android users. This is especially true of iPad of users. I’m sure there’s some correlation here with the amount of money spent that I mentioned above.

What to test: Try different user flows that require different amounts of engagement. For example, if you’re testing a game with in-app purchases, try showing the first purchasable item to the user at different points in the game experience.

Account for different out of app experiences

Nuances in the way an OS does things is very important to the overall user experience. The fact that Android has 3 to 4 buttons (back, home, menu, and sometimes search) on the the phone versus iOS’s singular home button makes the navigation of any given app very different. The way users interact with notifications on iOS versus Android is also very different. iOS tends to take users from a notification into the app while Android allows the user to do many more tasks without launching the app. There also just many more combinations of app client/OS version/hardware on Android than on iOS.

What to test: Experiment with different user flows. A user might hit the back button to go back to a previous page on Android, but iOS users don’t have that option. You might want to test providing a back button to iOS users. Alternatively, going back may be an action you don’t want users to take, so you can try to test variants to reduce the click of a back button on Android.

See if differences in performance matter

There are a handful of different iOS devices, but there are many, many more Android device with varying quality of hardware and versions of software. Something that works fast enough on iOS may not work fast enough on Android. You can find what crashes your app with some pre-release testing, but you can’t really effectively evaluate the trade-off of performance versus feature without A/B testing.

What to test: A/B test your features before doing a full launch. You might have a feature that causes your app to have a slight delay in the user experience. It might only cause a 0.2 sec. delay in iOS which users may tolerate given the benefit of the feature, but it might cause anywhere from a 0.2 to 2 second delay on Android devices which may turn out to be a deal breaker for users.

Test more on Android because it’s easier

With our Visual Apptimizer, we’ve eliminated the need to resubmit your iOS app into the Apple app store every time you want to make a design, copy, or layout change to your app, but any test involving a change to your app’s binary will always require new approval. Getting a new version of your app approved by Google is easier and takes less time than doing so for Apple.

What to test: When it comes to large flow redesigns that necessitate code redevelopment, try different things on Android first because the Google Play store approval is easier to get through. Gain learnings there, and then refine your theories on the two platforms separately.

More articles you might be interested in:

Optimize Marketing Campaigns in Real-time for Native iOS and Android

A/B Testing Example – Retale, a location-based app that aggregates retail circulars, ran a Black Friday campaign in the weeks leading up to November 28, 2014 that increased their product click-through rates in the busiest shopping week of the year. While...

Apptimize Brings Codeless, Real-time A/B Testing to Android Apps

San Francisco, CA – Three months after announcing a $2.1M seed funding round and the iOS version of the Visual Apptimizer, a what-you-see-is-what-you-get visual editor for creating A/B testing variants, Apptimize is launching the Visual Apptimizer for Android. Not only...

Android Visual Apptimizer Beta

iOS customers have been using our Visual Apptimizer, a what-you-see-is-what-you-get visual real-time editor, since we launched it in January. Today, we’re announcing the beta of our Android Visual Apptimizer. Now, Android A/B testers can also enjoy: A/B testing without having...

Schedule A Customized Demo

Get in touch with us for a detailed walk-through of Apptimize.

About Apptimize

Apptimize empowers product teams to efficiently run A/B tests, rollout and manage new features, and deliver personalized user experiences. Based in San Francisco, Apptimize is backed by US Venture Partners, Google Ventures, Y Combinator, and others.