7 Testing Tips to Ready your App for Launch

Over the past year we have helped several of our clients (e.g. QuizUp, a news reader app, and several others that we can’t name) achieve launch success. Launch success, as we define it, is shipping a high quality app with minimum or no user complaints, which is receiving many positive reviews, and ideally has exceeded the expectations of the executive team. Today, we’d like to share some of the best practices that we have seen being applied for these successful launches, focusing on the development side. We hope this is a valuable collection for startups, founders, and CTOs or their first QA hire.

As a growing startup ourselves, we are always continuing to learn, so if there are other strategies that worked for you, we would be happy to hear from you. Tweet us!

1) General: Feedback Loops

No matter what the app, a primary need is feedback, and not just from your development team or other staffers. Relying on developers and staffers exclusively to determine the quality of your application design is the equivalent of asking your mother whether the sweater your grandmother knitted for you looks good. In grandmother’s presence, no less. In other words, no developer wants to admit their ideas make little sense, and no designers want to admit (without prompting) that combining teal and orange was a bad idea. Sometimes the team will want to please the stakeholder a little too much, and will let bad ideas filter into the app design.

In app development, a poorly executed feature will lead to a bad app review. We covered the dangers of a poor app reviews in our previous article, 77% will not Download a Retail App Rated Below 3 Stars. This is why it is so important to involve outside opinions as soon as possible.

The feedback can start as early as sharing prototypes with trusted developer friends or potential clients and colleagues outside the company. Involving trusted outside parties early is one way to get the desired (and required) feedback in a controlled manner.

2) Facebook Group for Early Adopters

Another solution is to create an app-centric team facebook group, and post development news, prototype information, surveys, and any other feedback options you can think of to help in development. By limiting the users to trusted colleagues, long-time clients (if applicable) and blog or mailing list subscribers, you can gather a fairly wide spectrum of opinions without necessarily announcing your idea (or at least, how the idea will be accomplished) to the world prematurely. You can also be reasonably sure to obtain the opinions of power users, those who know app development, or the industry on which the app is based, and will provide the quality of feedback you need to develop the best app possible. The benefits of finding power users to test your ideas or to provide general feedback were discussed in our previous article, Successful Mobile Teams.

Early feedback from power users is invaluable

Feedback should be provided throughout all the above stages, save perhaps the initial core requirements stages. Brainstorming within the team can form the initial basis for app requirements and design. This is because in the initial stages, too many participants can prove disruptive to the process. However, the results of these initial sessions can and should be validated by survey results and power user feedback obtained shortly thereafter.

Establishing this focus group early is a key to success, allowing you to validate features, survey potential users, and to provide feedback on the application functionality. Your team may be able to lock down a great deal of UI questions and desired functionality through prototype sharing before coding begins. While this may be an unrealistic expectation, the desirability of such a well-defined app is undeniable. If looking to create a feedback group early in the prototype stages, a strong web application to consider is InvisionApp, a prototyping tool with strong group creation, design sharing and user interface prototyping features.

Even if not gathered in time for the first run of prototyping, such feedback groups will quickly validate your work, or get a wayward app back on track by pointing out any flaws or user experience issues.

3) Beta Users

When the app is further along, it is important to select a strong group of beta users. Determining your beta users can be simplified greatly by drawing from an existing pool of power users from within the above mentioned Facebook or Invisionapp focus groups. Scrum and Agile teams tend to operate with the principle of a minimum viable product, or MVP. When the actual app is in development, and MVPs are being delivered, you want your beta users already selected, and a tool to rollout iterations to these testers. These testers can be invaluable to a startup company, serving almost as a free QA team.

Two of the most commonly used platforms for Beta Testing are HockeyApp and Testflight. Of the two platforms, Hockeyapp is the most flexible, supporting development on the Android, iOS, OSX,Windows, and Windows Phone platforms. HockeyApp was acquired by Microsoft in December, 2014, but is still dedicated to supporting all the above platforms. Testflight, on the other hand is now iOS specific, and now owned by Apple. It requires a subscription to iTunes Connect (Apple’s content distribution platform) to function. Regardless, as many iOS developers can attest, Testflight is a very strong sharing platform for iOS apps-in-development.

For those considering HockeyApp and Testflight for iOS development, the following pros and cons (relevant to iOS testing) may be of assistance:

Hockeyapp

Pros:

No need for an approval process

Quick upload of new versions while maintaining access to old versions

Cons:

Requires a bit more work to initially set up each beta tester (need to have their device registered with HockeyApp so that you can register their UDID with a provisioning profile and then a new archived build needs to be uploaded to HockeyApp containing that updated provisioning profile)

Another strong recent entry into the beta distribution market is Crashlytics Beta, supporting Android and iOS. Launched in May of 2014, this entry has made some headway, although they are better known for their crash reporting. One big selling point of Crashlytics Beta is that the service is free. In addition, it can be integrated into your build scripts. Take your Android app for instance. You can integrate Crashlytics into Android Studio and deliver builds to all your beta testers in a second. Another great thing about the platform is that your users will receive notification from the Crashlytics app directly on their devices. Your new build (and instructions) will be waiting for them there. For the folks who don’t want to share their data with twitter (since crashlytics is part of Twitter) you might want to check out crittercism.

If you are developing for Android only, you can also consider Google Play’s native beta app testing. It is fairly robust, and makes sharing dead simple, by leveraging Google+ communities.

Staged Rollouts Dashboard from Google

Regardless of the solution used for beta distribution, it is very important to gain feedback from actual users prior to app launch. Many look and feel issues will only be discovered by users with the time to navigate and explore the app on their own devices. Beta distribution can also help in ensuring that a greater number of device models are tested.

4) Testing for Device Fragmentation

Ensuring that your application (whatever the platform) is thoroughly tested is key to a successful launch. Ideally, you want to ensure it is tested on a multitude of devices, and not just those available to the immediate team. In many small startups, a great challenge faced is that of a limited number of devices, whether iOS or Android. In some small companies, testing is limited to two or three device models, and android or iOS simulators, such as Genymotion. While these may be able to simulate OS fragmentation, rarely do they adequately simulate the differences in hardware found between devices. With the number of different manufacturers creating phones and tablets, particularly for Android, this can be a serious problem in implementing a stable, reliable app.

Testmunk dashboard identifying issues

This is where an automated testing platform such as Testmunk can prove exceptionally useful. As mentioned above, Beta distribution can somewhat mitigate testing shortfalls by ensuring that the app is run on a multitude of different devices. However, this is somewhat of a crapshoot, as there is no formalized testing until a bug is reported and identified. Beta users test the app organically, according to how they would expect the app to function. If written correctly, automated tests can quickly do what would take a QA team of 10 a few days in but a few hours.

Typically, automated testing platforms expect you to supply the devices. In other words, you can run a multitude of very thorough tests, but only on the devices you have on hand to test. Testmunk solves both of these problems, by offering a battery of test devices over the cloud, as well as the testing platform upon which to run your tests. This can help put smaller organizations on an even footing with larger rivals, by ensuring that little to no advantage is realized by the larger organization in having a dozen more devices than Average Joe Software. Testmunk offers other advantages as well, in that you can visually confirm the results in screenshots taken of each step, and review log files from the dashboard, making troubleshooting a relative breeze.

Regardless of whether you use Testmunk, automated testing is highly recommended prior to launch. It can ensure your app is thoroughly tested on the platforms your organization deems key to success, and can speed development greatly. Manual testing is still necessary of course, as there is no substitute when it comes to testing user interface and experience, but automation and Testmunk can help a great deal.

5) Trial Launch

When you feel ready to launch, but are concerned about the reception, or have niggling doubts about last minute bugs, you may wish to perform a limited launch, or staged rollout first. Technically, your beta distribution can be considered a limited launch, but this is not what we are talking about. A staged rollout for an app typically means providing a percentage of users a new feature of an app, or limiting your rollout to a specific region, state or country.

Google’s developer console supports staged rollouts, allowing you to limit the percentage of users able to download your app. A staged rollout allows you to obtain feedback from an unbiased sample of users, as well as to limit the impact of server load. A staged rollout, however, allows users to rate and review, so it is important to keep an eye out for poor ratings, in order to catch and fix issues prior to the larger release.

In iOS deployment, however, it is not possible to launch to a specific percentage of users, or a state or region of the US. Testflight mitigates this somewhat for iOS developers, by supporting larger distributions of betas, up to 1000 users (by contrast, HockeyApp only supports 100). There are also some third party tools, such as Taplytics, that allow you to perform staged rollouts similar to Google. Until this is natively supported, iOS developers can be considered somewhat at a disadvantage when it comes to obtaining real-world, unbiased feedback from staged rollouts.

A video from Google I/O about staged rollouts

6) A Geographic Test Market

As mentioned above, iOS developers have limited options when it comes to staged rollouts. However, there is a workaround. A strategy that is commonly used by larger companies who aim to either publish a big worldwide release or a big PR push in the U.S is to first pretest the app in a secondary, less important market. In order to get feedback from a larger number of unbiased users, the app developer can deploy in Singapore, as an example, rather than its target market of the US.

This strategy, known as a geographical rollout, is employed fairly often. Game developers often use Asian countries, possibly due to large populations of tech-savvy individuals. I also know of a social messaging app that used Mexico before having a public launch in the USA. Typically, if targeting a release in the US, however, it is more common to do geographical rollouts in Australia, Canada, or New Zealand. It is important to pick a country that has a market similarity to your target audience.

Secondary markets (blue) and primary market (red)

Geographical rollouts offer the development team an opportunity not only to vet the app one last time, but also the support team, any potential back-end issues or simply any features that are not being received well.

7) Support and Documentation Testing

There are a number of pre-launch considerations from a PR / marketing perspective. While this article is primarily concerned with the technical aspects of app launch, there are a few instances in which these concerns overlap. One of these is support and documentation.

Having support available at app launch is a key to success. For example, if your app has features that could require a learning curve, a support team can mitigate any poor reviews due to misunderstood features. Similarly, it is important to have prepared adequate documentation, and help both within the app, and on your website.

If doing geographical rollouts to test response to your application, you would be remiss if you did not also strongly test your support services and documentation. This is a prime opportunity, as your support solution, whatever it may be, can answer the questions of real-time and new users, not just those that have gone through prototyping, beta testing, and already know what your app is about. You need to know any shortcomings in your support process, support documentation, and possibly support training, prior to release.

If you are not rolling out geographically, this is still important to consider. One possibility is to keep a pool of beta ‘licenses’ handy, and not to distribute them until shortly before launch, to a group of ‘virgin’ users. If an iOS app, this is easier to do with Testflight, which allows for 1000 test users. If you keep 200 of those potential test users untouched until this pre-launch stage, you can simulate a ‘go-live’ scenario. For HockeyApp this is less feasible, due to a hard limit of 100 users for iOS ad-hoc distribution. However, there is no such limit on Android.

Another consideration is timing of your app release. While this is typically promoted as a public relations/marketing concern, it is also a very real technical/logistical concern as well. It is all well and good to do a geographical rollout for testing/feedback purposes. It is quite another to do it accidentally. If preparing an international launch, (especially if your primary target market is the US) it is important to recognize the limits of some of the default options within the Play Store and App Store. For example, setting a launch date ahead of time via iTunes Connect’s availability feature can be tempting. Unfortunately this means that New Zealand users will obtain the app 12 hours before your target audience. If your primary campaign and launch date information is based on a US time-frame, this can mean your app is live overseas before your support is.

Support and documentation are key considerations for app success, in particular if your app has many features, or advanced features. Support must be online the moment your app is, in order to ensure the best possible experience, and the best possible reviews.

In Summary…

Having implemented the above suggestions, your launch has every chance to be considered a success. However, the job does not end with a successful launch. Consistent and repeated effort is required to sustain momentum and build on this initial success. Hopefully the work done in the initial stages will provide the lessons needed to succeed after launch as well. If your launch is not as much a success as you’d like after this guidance, well… there’s always marketing. But that is a topic for another article!

About the author:Martin Poschenrieder has been working in the mobile industry for most of the past decade. He began his career as an intern for one of the few German handset manufacturers, years before Android and iPhone were launched. After involvement with several app projects, he soon realized that one of the biggest pain-points in development was mobile app testing. In order to ease this pain, he started Testmunk. Testmunk is based in Silicon Valley, and provides automated app testing over the cloud.Follow Martin on twitter

Hi Martin. Thank you for the suggestions and the step by step explanation. Very useful information for product development start up companies. Even though we are not in to product development but it is worth readable and worth sharing it for those who need it ….