A Beginner’s Guide to Automated Mobile App Testing

A step-by-step tutorial for people completely new to mobile development, manual QA testers and automation engineers who are new to mobile.

This is a step-by-step tutorial in mobile app automation software. It is tailored for people new to mobile automation. This tutorial you will teach you how to create and execute your very own automated mobile app testing, even if you have little to no mobile development experience. It has been written in light of the best practices we at testmunk have derived from working with clients introducing automated testing to their organizations.

An introduction to the tools required for mobile automation

Certain steps of this tutorial differ for iOS and Android – make sure you follow the correct process for your app.

This tutorial also frequently refers to sections of the testmunk documentation – directing you when to use this documentation.

Introduction to automated mobile app testing

First, it’s important to understand that mobile apps are made up of elements with specific names and functionalities. An example of such an element is the app’s ‘log in’ button. Once that element is pressed, it will take you to the log in page of the app.

A quick way to get started is to use one of our sample apps, either for Android or iOS, with predefined steps. Here’s an example of how we would write an automated test script for this app:

We write a test that instructs the phone to perform the following steps: type an email address into the “Email” field, type a password into the “Password” field and press the “SIGN IN” button element.

Then we check whether the app has done what’s expected. E.g. ‘Then I should see “Hello World”.’

This basic test fills out the email and password fields, presses log in, and checks whether the desired element has appeared. If “Hello world” does appear, then the test passes and the program moves on to the next test. For anything other than “Hello world”, the test fails.

Android vs. iOS

The process of writing and running an automated test depends on the platform you’re using. From here this tutorial branches into separate steps for Android and iOS.

It might also be the case that you will need a little help from your iOS or Android developer – maybe buy them a coffee.

1.a) Prerequisites – Android

Download the latest SDK from the Android website. After your download copy and paste the sdk and eclipse folders into your Applications folder.
For your Android SDK to work, you will need to configure the bash profile on your computer. Instructions on how to do this can be found in the testmuk documentation here.

Source code

Most companies use GitHub to store their source code. Either get access to GitHub, or ask your Android developer to simply create a .zip file of the app source code for you.

Before we get started with automating tests, we first want to make sure that the Android app works. Plug in your Android device and open the command line, then navigate to the folder in which your app is stored and type: ‘adb devices’. This returns your device’s unique code (see below).

It is important to ensure that USB debugging is enabled on the Android device you’re using. For Nexus and Sumsung devices, this can be done in the settings. Go to Settings > About device > Build number and tap build number 7 times. Then go to ‘Developer options’ in the main settings menu and enable USB debugging. For other devices, this process varies, so check your manufacturer’s documentation.

1.b) Prerequisites – iOS

Xcode is the development environment used to create iOS apps. Xcode is needed to integrate the Calabash test framework and build an .ipa app file to test.

What is Calabash? Calabash is a framework that allows you to write automated mobile application tests for iOS and Android. It provides APIs for mimicking input to the devices, and reading its output.

An .ipa file is the app file itself; you probably have 30 or 40 apps on your iPhone, and all of them are .ipa files.

Developer certificate / Provisioning profile

Acquire a valid developer certificate in order to build an .ipa file. You can download one from the Apple Dev Center, or you can ask your iOS developer. You will also need a provisioning profile to build your .ipa file on a device.

Before we get started with automating tests, we first want to make sure that everything works. Open the Xcode project in Xcode and let it run on the simulator. Guidance on running an app on the Xcode simulator can be found here under the ‘Simulator’ heading.

If everything runs like it should on the simulator, it’s time to check if your app works on a real device. Plug your iOS device into your Mac and open Xcode and your xcodeproj again. Guidance on running an app locally on a device can be found here under the ‘Device’ heading.

Hopefully everything worked! Now we are coming to the fun part: automating your functional tests.

2.a) Installing the Calabash test framework – Android

For this part of the tutorial, you need to head over to our docs page and follow the steps outlined in the Android ‘Installation’ section. We recommend that you download our sample application or use your own source code. Don’t go onto the ‘Preparing testcases’ section just yet.

2.b) Installing the Calabash test framework – iOS

For this part of the tutorial, you need to head over to our docs page and follow the steps outlined in the iOS ‘Installation’ section. Don’t go onto the ‘Preparing testcases’ section just yet.

If your app uses some custom configurations you may need to ask your iOS developer what to select when setting up your app to be tested. At this point it might be a good idea to use our sample app.

3. Write your first test

In the sample app folder go to TMSample/features and you’ll see a file called my_first.feature. Open it (using a text editor such as Sublime Text or Rubymine). This is the file in which you write your testcases and is specific to the Calabash framework. The features file is for testing only, and has no effect on the app itself.

So let’s create some tests by editing this features file. The following examples are taken from the testmunk iOS sample app.

Structure of the features file

The structure of a features file is as follows:

A “Feature:” line, is a general title for your test suite. Please don’t delete this section or else your test will not work. You can customize it with your own description, though. For example: ‘Feature: Testsuite v1 for myapp.’

A “Scenario:,” which is the title for your testcase

The teststeps to be executed.

Note that each testcase should end in an assertion. “Then I should see …” or “Then I should not see…”. This checks that everything is how it should be in your app. More assertions can be found in the predefined steps.

Now write your own scenario

The best way to do this is to use the testmunk sample teststeps as a template. Go ahead and copy and paste an existing scenario at the bottom of the features file and start editing the actions and target elements in your app. Check out the example below.

But how do we target a specific element in the app? See the testmunk documentation for details on how to use the Android UI Automator Viewer to find the names of the elements you want to target. When running the commands, make sure you’re in the folder that contains your app’s source code.

For iOS, we use the accessibility inspector. See here in the testmunk documentation for details on how to use the accessibility inspector to find the names of the elements you want to target. Don’t worry about the ‘Calabash console’ section just yet. Take a look at this short video tutorial on using the accessibility inspector.

Example: In the testmunk sample app, you want to check that given a user is on the “Plain Table” page, and they press the “Home” button, they are taken to the Home screen. To test this, we would write the following teststeps in the features file:

In the assertion, we have chosen to check for the element “Segmented Control” as we know this is on the Home screen. However, we could choose any other element that exists on the home screen to check if we’ve returned there.

Note that this example is for the testmunk iOS sample app.

Your final version of the features file could look something like this. This testcase is written for the table view in the iOS testmunk sample app.

IMPORTANT: Remember to write your teststeps from the point of opening the app for the first time as we’ll be using the command ‘RESET_BETWEEN_SCENARIOS=1’ which uninstalls and reinstalls the app between each test. We recommend doing this to ensure that testcases are executed independently.

So you should now have a features file that includes the scenario you have written.

Note: the syntax for Android is slightly different than that of iOS. Check the Calabash github documentation for Android predefined steps.

4.a) Run your first test – Android

Once you’ve written your testcase, go ahead and run the features file. This section explains exactly how to do that.

If you have an Android device to hand, connect it via USB to your computer and follow the section on ‘Running on your local device’ in the documentation. Again, ensure that USB debugging is enabled on your Android device.

IMPORTANT: The docs will tell you to run ‘calabash-android run sample.apk –verbose’. Alternatively, you can run ‘RESET_BETWEEN_SCENARIOS=1 calabash-android run sample.apk –verbose’ (without the quotes). We recommend doing this to ensure that testcases are executed independently. Don’t go onto the ‘Running on multiple Android devices’ section just yet.

Once the test is running, you should see an output like this in your terminal (left) whilst the testcases are executing on your device or emulator (right).

4.b) Run your first test – iOS

You may also want to test locally on an iOS device. An explanation of how to do that can be found in our docs under the ‘Running locally on a device’ sub-section. Don’t go onto the ‘Running on multiple iOS devices’ section just yet.

5. Testing on multiple devices

Most people don’t have an arsenal of devices laying around that they can test on to that ensure that users on all devices, operating systems and versions of operating systems have a flawless experience on an app. But testmunk does.

We have a large suite of devices you can run your tests on. Once running your tests on the suite, you’ll be able to see screenshots of each teststep on every single device you’ve tested on (see below).

A free testmunk account allows you to test on a real device via the testmunk dashboard. You don’t need to have the device in your possession – the real iOS devices are located in the testmunk lab.

6. Use your testmunk account to run tests on devices in the cloud

Create a new app or use an existing app from the dropdown menu on the top right of the toolbar

Create a New Testrun using the button on the left hand side of the toolbar

Drag and drop the debugging .apk (Android) or .ipa (iOS) file that includes the calabash framework. FOr iOS, see below for how to export an .ipa file

Exporting an .ipa file – iOS only

To export the IPA file for your app, open your Xcode project, make sure to select the ”…-tm” scheme and “iOS Device” as your target device. Then, in the title bar and go to Product > Archive. In the Archives window that pops up, press the Distribute… button, select Save for Enterprise or Ad Hoc Deployment, choose the Provisioning Profile you sign your app with, and export the file. Leave the Save for Enterprise Distribution checkbox unchecked.

More info on exporting .ipa files can be found in our documentation under the ‘Hint’ section.

Upload your testcases using the Testcase button in the toolbar. You must upload a zipped (.zip) features file.

Select the devices you want to test on by using the icon on the far right side of the toolbar, selecting ‘account settings’ and then the ‘devices’ tab.

Within a few minutes, you’ll see the results on your testmunk account dashboard (see below).

If you see a testcase that contains an error, click on ‘More’ to discover what that error is with the aid of screenshots from each teststep. You can also download the logs for the test.

We recommend you run your tests locally first on a device or simulator. When you’re happy with the runs, upload the app and features.zip file to testmunk and execute on multiple devices.

7. Final note on custom step definitions

The predefined steps that you will find on the Calabash Wiki pages for Android and iOS and also in our testmunk docs cover a wide range of use cases. As you get more advanced, however, you’ll need to write tests with complexity beyond the scope of the predefined steps. This guide (iOS) explains the process of creating these custom test steps and provides useful tips and hints for doing so. Note that the Android syntax will differ slightly.

8. Further resources

Calabash Google groups for Android and for iOS. If you’re stuck on something, chances are someone else has been stuck on it too and may have asked for a solution in one of these groups. Or if not, you can go ahead and ask the question. Just ensure you follow their forum guidelines before you post.