Android User Interface Testing with AWS Device Farm – Part 1

I often get asked about automated testing of user interfaces of Android applications. Amazon released a service on AWS called Device Farm to support user interface (UI) testing, this seems to be a good time to take a look.

UI testing can be immensely frustrating due to it’s inherent fragility. It often seems that as soon as you have a stable test, along comes some very minor change that causes all your hard work to need updating. For this reason, many applications have a suite of automatic tests to check logic but rely on the skill of a human tester to visually check the output. In the world of Android however, there is an ever-pressing need to be able to test on as many devices as possible due to the wide variation in screen layouts and device capability.

Apps should of course still be structured such that as much of the code as possible can be written test-first without complex test environments and with a minimum of mock-objects. Unfortunately, that still leaves the UI to test. Given the huge number of variations of the Android platform out there, it has become imperative to implement some form of automated user interface testing that can be run on multiple platforms to improve confidence that users will be happy when the next revisions hits the streets.

Amazon have recently added Device Farm to their suite of AWS service offerings. Device Farm allows applications and their associated tests to be uploaded onto the Device Farm service where the tests will then be run on a wide range of real devices. Last time I looked, this allowed me to run tests on over 90 devices in one go. Amazon being Amazon, I have every expectation that the list of devices will grow and grow.

Just a few of the 90+ Android devices available to test against

Device Farm supports tests implemented using Appium on JUnit and TestNG, Calabash, Instrumentation and UI Automator. This is great news if you are already using one of these frameworks. For this post, I want to take a quick look at creating Instrumentation tests using a tool called Robotium Recorder.

Robotium Recorder is a great tool in that it allows UI test scripts to be created by recording user actions. In addition, it allows screenshots to be recorded (which can then be accessed on Amazon Device Farm) and simple graphically generated assertions testing content of the UI.

Creating a test with Robotium recorder:

The great thing about using Robotium Recorder is that you simply start the recorder (it works with both Android Studio and Eclipse ADT). You then select either the Android module or the APK you wish to test and start recording a test.

Once Robotinium Recorder has performed it’s initialization, the app under test will start on the device you have chosen to run it on (I used an AVD). Then, as you perform actions in the application, Robotinium Recorder captures them.

Generate a Test Class

Once you are happy with the recorded steps (as the screenshot shows, you can delete steps). You save the recording. At this point, Robotinium generates the source-code for a test case which extends android.test.ActivityInstrumentationTestCase2.

The actual test steps are in the testRun() method of the generated code and are really quite easy to follow:

Run the Tests on Device Farm

Once you have the generated test code, you can add code in to your test to check that actions performed by the “user” were actually interacting with the underlying logic. You can do this either through the Robotium API (the solo object in the generated code) or by adding code to validate the state of the underlying application.

Once you’ve proved that your test works (simply run the generated code as a JUnit test in the normal way) you are ready to test on the AWS Device Farm. Your project will have two Android modules, one being the application under test and the other the test code.

You need to build both modules so that the two apk files can be uploaded to Device Farm. I’ll take a look at that in part 2 of this post.