Android Testing Tutorial: Unit Testing like a True Green Droid

As experienced app developers, as the applications we develop mature, we get a gut feeling that it is the time to start testing. Business rules often imply that the system has to provide stability throughout different releases. We also ideally want to automate the build process and publish the application automatically. For this, we need Adnroid testing tools in place to guarantee that the build is working as expected.

Tests can provide the extra level of confidence about things we build. It is difficult (if not impossible) to build a perfect, bug-free product. Therefore, our goal will be to improve our odds of succeeding on the market by setting up a test suite that will quickly spot newly introduced bugs in our application.

When it comes to Android, and the various mobile platforms in general, app testing can be a challenge. Implementing unit tests and following principles of test-driven development or similar can often feel unintuitive, at the least. Nonetheless, testing is important, and shouldn’t be taken for granted or ignored. David, Kent and Martin have discussed the benefits and pitfalls of testing in a series of conversations between themselves compiled in an article titled “Is TDD dead?”. You can also find the actual video conversations there and get more insight if testing fits your development process and to which extent could you incorporate it, starting now.

In this Android testing tutorial I will walk you through unit and acceptance, regression testing on Android. We will focus on the abstraction of the unit of tests on Android, followed by examples of acceptance testing, with the focus on making the process as fast and simple as possible to shorten developer-QA feedback cycles.

Should I Read It?

This tutorial will explore the different possibilities when it comes to testing Android applications. Developers or project managers who want to better understand the current testing possibilities of the Android platform can decide using this tutorial if they want to take any of the approaches mentioned in this article. However, this is no silver bullet, as the discussion involved in such a topic inherently varies from product to product along with deadlines, codebase quality of code, level of coupling of the system, developer’s preference in architecture design, projected lifespan of the feature to test, etc.

Thinking in Units: Android Testing

Ideally, we want to test one logical unit/component of an architecture independently. This way we can guarantee that our component works properly for the set of inputs that we expect. The dependencies can be mocked, which will enable us to write tests that execute fast. Furthermore, we will be able to simulate different system states based on the supplied input to the test, covering exotic cases in the process.

The goal of Android unit testing is to isolate each part of the program and show that the individual parts are correct. A unit test provides a strict, written contract that the piece of code must satisfy. As a result, it affords several benefits. —Wikipedia

Robolectric

Robolectric is an Android unit testing framework that allows you to run tests inside the JVM on your development workstation. Robolectric rewrites Android SDK classes as they’re being loaded and makes it possible for them to run on a regular JVM, resulting in fast test times. Furthermore, it handles inflation of views, resource loading, and more stuff that’s implemented in native C code on Android devices, making the need for emulators and physical devices to run automated tests obsolete.

Mockito

Mockito is a mocking framework that enables us to write clean tests in java. It simplifies the process of creating test doubles (mocks), which are used to replace the original dependencies of a component/module used in production. A StackOverflow answer discusses about the differences between mocks and stubs in fairly simple terms that you can read to learn more.

// you can mock concrete classes, not only interfaces
LinkedList mockedList = mock(LinkedList.class);
// stubbing appears before the actual execution
when(mockedList.get(0)).thenReturn("first");
// the following prints "first"
System.out.println(mockedList.get(0));
// the following prints "null" because get(999) was not stubbed
System.out.println(mockedList.get(999));

Now, we know that we can specify action-reaction pairs that define what happens once we execute a specific action on the mocked object/component. Therefore, we can mock entire modules of our application, and for each test case make the mocked module react in a different way. The different ways will reflect the possible states of the tested component and mocked component pair.

Unit Testing

In this section, we will assume MVP (Model View Presenter) architecture. The activities and fragments are the views, models being the repository layer for calls to the database or remote services, and presenter being the “brain” that binds all these together implementing specific logic to control views, models, and the flow of data through the application.

Abstracting Components

Mocking views and Models

In this Android testing example, we will mock views, models, and repository components, and we will unit test the presenter. This is one of the smallest tests, targeting a single component in the architecture. Furthermore, we will use method stubbing for setting up a proper, testable chain of reactions:

Mocking the Global Networking Layer with MockWebServer

It is often convenient to be able to mock the global networking layer. MockWebServer allows us to queue responses for specific requests that we execute in our tests. This gives us the chance to simulate obscure responses that we expect from the server, but are not straightforward to reproduce. It allows us to ensure full coverage while writing little additional code.

Custom Test Doubles

You can write your own model or respoistory component and inject it into the test by providing a different module to the object graph using Dagger (http://square.github.io/dagger/). We have the option to check if the view state was updated properly based on the data supplied by the mocked model component:

Running Tests

Android Studio

You can easily right click a test class, method, or whole test package and run the tests from the options dialog in the IDE.

Terminal

Running Android app tests from the terminal creates reports for the tested classes in the “build” folder of the target module. Even more, if you plan to setup an automated build process, you will use the terminal approach.
With Gradle, you can run all debug flavored tests by executing the following:

gradle testDebug

Accessing Source Set “test” from Android Studio Version

Version 1.1 of Android Studio and the Android Gradle plugin brings support for unit testing your code. You can learn more by reading their excellent documentation on it. The feature is experimental, but also a great inclusion since you can now easily switch between your unit tests and instrumentation test source sets from the IDE. It behaves in the same way as if you would switch flavours in the IDE.

Easing the Process

Writing Android app tests may not be as fun as developing the original application. Hence, some tips on how to ease the process of writing tests and avoiding common problems while setting up the project will help a long way.

AssertJ Android

AssertJ Android, as you may have guessed from the name, is a set of helper functions that is built with Android in mind. It is an extension to the popular library AssertJ. Functionality provided by AssertJ Android ranges from simple assertions, such as “assertThat(view).isGone()”, to things as complex as:

Running tests that require Robolectric from the terminal can introduce new challenges. For example, you may see exceptions like “Theme not set”. If the tests are executing properly from the IDE, but not from the terminal, you may be trying to run it from a path in the terminal where the specified manifest path can not be resolved. The hard-coded config value for the manifest path may not be pointing to the right location from the point of execution of the command. This can be solved through the use of custom runners:

Robolectric Dependencies to Support Libraries

Another interesting testing problem is that Robolectric isn’t able to reference support libraries properly. The solution is to add a “project.properties” file to the module where the tests are. For example, for the Support-v4 and AppCompat libraries the file should contain:

Acceptance/Regression Testing

Acceptance/Regression testing automates part of the final step of testing on a real, 100% percent Android environment. We do not use mocked Android OS classes at this level - the tests are run on real devices and emulators.

These circumstances make the process much more unstable due to the variety of physical devices, emulator configurations, device states, and feature sets of each device. Furthermore, it is highly dependent on the version of the operating system and the screen size of the phone to decide how the content will be displayed.

It is a bit complex to create the right test that passes on a wide range of devices, but as always you should dream big yet start small. Creation of tests with Robotium is an iterative process. With a few tricks, it can be simplified a lot.

Robotium

Robotium is an open source Android test automation framework that has been in existence since January 2010. It is worth mentioning that Robotium is a paid solution, but comes with a fair free trial.

To speed up the process of writing Robotium tests, we will move away from manual test writing to test recording. The trade-off is between quality of code and speed. If you are making heavy changes to your user interface, you will benefit a lot from the test recording approach and being able to record new tests quickly.

Since Testdroid Recorder is an Eclipse plugin and we are referring to Android Studio throughout this article, it would ideally be a reason of concern. However, in this case it is not a problem, as you can use the plugin directly with an APK and record the tests against it.

Once you create the tests, you can copy and paste them in Android Studio, together with any dependency that Testdroid recorder requires, and you are set to go.
The recorded test would look something like the class below:

If you look closely, you will notice how much of the code is rather straight-forward.

When recording tests, don’t go scarce on “wait” statements. Wait for dialogs to appear, activities to appear, texts to appear. This will guarantee that the activity and the view hierarchy are ready to be interacted with when you perform the action on the current screen. At the same time, take screenshots. Automated tests are usually unattended, and screenshots are one of the ways you get to see what actually happened during those tests.

Whether tests pass or fail, reports are your best friend. You can find them under the build directory “module/build/outputs/reports”:

In theory, the QA team could record tests and optimize them. By putting effort in a standardized model for optimizing test cases, it could be done. When you normally record tests, you always have to tweak a couple of things to make it work flawlessly.

Finally, to run these tests from Android Studio, you can select them and run as you would run unit tests. From the terminal, it is a one-liner:

gradle connectedAndroidTest

Performance of Testing

Android unit testing with Robolectric is extremely fast, because it runs directly within the JVM on your machine. Compared to that, acceptance testing on emulators and physical devices is a lot slower. Depending on the size of flows that you are testing, it can take anywhere from a few seconds to a few minutes per test case. The acceptance test phase should be used as part of an automated build process on a continuous integration server.

The Takeaway

There are a variety of Android testing tools available, and as the ecosystem matures, the process of setting up a testable environment and writing tests will become easier. There are still more challenges to tackle, and with a wide community of developers working on daily problems, there is a lot of room for constructive discussions and fast feedback.

Use the approaches described in this Android testing tutorial to guide you in tackling the challenges ahead of you. If and when you run into problems, check back with this article or the references linked within for solutions to known problems.

In a future post, we will discuss parallelization, build automation, continuous integration, Github/BitBucket hooks, artifact versioning, and the best practices for managing massive mobile application projects in greater depth.

About the author

As a software engineer and a mobile enthusiast, Marko has invested time and resources into perfecting the development process of building mobile apps. With experience building products iteratively, he has developed analytical skills, wireframing and prototyping techniques, and coding best practices to develop quality software and, more importantly, the software that the client really wants. Android and iOS are his playgrounds of choice. [click to continue...]

Comments

tibbbi

nice one, but I soo prefer Espresso with Hamcrest notation instead of Robotium. You dont have to use all the sleeps and waits with Espresso, as it runs the next thing always only if the UI thread is idle. Got any experience with it?

José Naves Moura Neto

Excellent article, Marko. It covers not only modern Android testing but also teaches us different kinds of tests.
Kudos!

Marko Vitas

Thanks @josenaves:disqus, glad you like it!

Marko Vitas

Hey @tibbbi:disqus, I agree with you on the Espresso points. Personally, I don't have real-life, production experience with it. Glad to hear you find it much more stable and easy to write tests. As you said, it has synchronisation mechanisms in place so that it detects that the UI is in a "ready state" for interaction, which is great. Definitely, will take a closer look for the next testing iteration. Thanks!

Chrystopher Chabert

Have you also looked into PowerMockito ? It provides the necessary support to mock final classes / methods. Works like a charm.

Marko Vitas

@chrystopherchabert:disqus, I checked it out conceptually. It adds the extra flexibility for more extreme mocking cases.
As stated here http://stackoverflow.com/a/6386065/2018294, if you are using legacy code that you can't change, then it offers a great option for you to mock parts that you don't have access to. On the other hand, if you are doing the same mocking on your own codebase, maybe it is time to refactor. Nevertheless, I totally agree that it provides great flexibility, which can help a lot if you don't have resources to refactor. Thanks for pointing it out.

hisham2007

great article

Eduard Hasanaj

Dear all.
So far I have understood almost everything that I have read about android (tutorials, doc). Now I am seriously struggling to understand testing. I want to be able to test my app using predefined test cases in a json files. I want to loop through them.

As a software engineer and a mobile enthusiast, Marko has invested time and resources into perfecting the development process of building mobile apps. With experience building products iteratively, he has developed analytical skills, wireframing and prototyping techniques, and coding best practices to develop quality software and, more importantly, the software that the client really wants. Android and iOS are his playgrounds of choice.