Defining Tracking Events On the Fly: Charity Miles Case Study

Apptimize

Jul 30, 2015

The App

Charity Miles is an innovative app with a fresh take on running. By partnering with different corporate sponsors and charities, they help change the world as well as people’s lives through exercise. Emphasizing action over words, Charity Miles sponsors donate money to charities their users choose for every mile they walk, run, or bike with the app. Together with their network, they make the world a better place by telling stories through actions.

The Problem

As with any other app, one of the hardest things to do is to create effective growth campaigns that help your product and message spread. One of the ways that Charity Miles did this was by creating a sharing feature that allows users to post about their achievements and money earned. The team wanted to try out a few different methods to see how they could increase shares, and reach more potential users.

The A/B Test

Charity Miles decided to test a different call to action wanted to see if tweaking the messaging would increase the number of shares, and help get their message to a wider audience.

While the old call to action got the job done, they wanted to see if having more humorous and quirky copy would do a better job of converting. They used the Visual Apptimizer to quickly change the text and launch their experiment.

Surprise Complications

A little while after the experiment deployed, the team noticed something peculiar: the key conversion events for button presses weren’t being triggered, even though users were seeing the variants. Concerned, they contacted Apptimize to help triage the situation. Together, they discovered a small mistake. Two lines of code used in their proprietary analytics platform to track the clicks had been left out of the latest release.

Every once in a while, stuff like this happens. You just have to suck it up and push it out on the next release. Or do you?

Instead of calling it a day and redeploying to the App Store for a future release, the teams worked together to figure out a potential solution.

The Fix

Working together, the teams created a clever solution to track the data, without having to re-deploy new code. Instead of coding a solution, they defined new metrics for both the Facebook and Twitter share buttons using the Visual Apptimizer. Then, they simply copied and re-launched the experiment, and started seeing results trickle in within minutes.

Even with a minor setback, Charity Miles was able to get the experiment relaunched and back on track within 30 minutes, something that would be unfathomable without the tool. As this doesn’t qualify for an expedited review through the App Store, even if Charity Miles redeployed the code immediately, they would have had to wait approximately 9 days just for their review to be approved.

Results

The team saved more than a week in lost time by using the Visual Apptimizer. When you’re on a mission and growing, that time makes all the difference in the world. As for the results? The test results showed small decreases for both Facebook and Twitter shares. However, neither of the differences were statistically significant, suggesting that this change in language didn’t have a large impact on user behavior.

Even though the test didn’t produce statistically significant results, the team has still learned that this messaging would not have produced the desired results. Furthermore, they’ve been able to gain a better understanding of their user base, as well as more clues as to what types of changes would have a more significant impact for their fans.

Using testing, Charity Miles was able to avoid using up valuable development resources to measure the impact of a simple messaging change. Instead, they spent a fraction of the time that it would have taken with a regular dev cycle, and got solid data in return.

More articles you might be interested in:

Announcing Mobilize by Apptimize 2017

We are excited to announce Mobilize by Apptimize 2017! Mobilize — a conference focused on mobile testing and data-driven product development — will take place on Monday, October 16th at the iconic Westin St. Francis in San Francisco. Building off...

What You Need to Know About Facebook’s New Mobile Logins

Last week at F8, Facebook announced their changes to social logins. Over the next few months, they’ll be testing the new logins with select apps, and opening release to all devs later this year. Below are the key takeaways you...

Automatically Import Events from Mixpanel, Flurry, GA, and More

Mobile A/B testing really has never been easier. Apptimize now automatically imports events from your existing analytics API without any work whatsoever. New A/B tests can now use the events you already track in Mixpanel (or Flurry, Google Analytics… or...

Schedule A Customized Demo

Get in touch with us for a detailed walk-through of Apptimize.

About Apptimize

Apptimize empowers product teams to efficiently run A/B tests, rollout and manage new features, and deliver personalized user experiences. Based in San Francisco, Apptimize is backed by US Venture Partners, Google Ventures, Y Combinator, and others.