Testing Runtime Permissions: Lessons Learned

Recently I've been working on adding the Android M Runtime Permissions support to the project I'm on at the moment. As an Android user, I find runtime permissions awesome. Permission model was not very user-friendly before Marshmallow: every time you were trying to install an app, you were presented with a long list of permissions that you have to grant to that app without even seeing it in action! Most times those apps won't have any malicious intentions and will require certain permissions for some secondary features, but it still looks nasty. As an Android M user, you won't be forced to make the decision upfront: the apps are now required to ask for permissions at run time and provide enough context on why the permission is needed. Even if the user decides to deny the permission, this will only result in her not being able to use a certain feature - but not the whole app. Overall, this makes complete sense to me, and I enjoy being in control of the apps I use on my Android M device.

However, things are not that rosy for us developers, or, better said, it's a pain. There are many possible scenarios that you have to keep in mind. Additionally, I can't say I like the official APIs for working with runtime permissions: they're quite verbose, and the callbacks just force your code to be scattered around and lose in readability.

Another problem that I was aware of after reading Handling Android runtime permissions in UI tests is that adding support for runtime permissions will inevitably break our Espresso tests. My team relies on automated UI tests to catch regressions in the most crucial user flows, and it's worked fine for us so far. Turns out, Espresso just doesn't care about runtime permissions, as it doesn't provide any means of dealing with permission dialogs. The following won't work:

onView(withText("ALLOW")).perform(click());

The reason behind this is that the permission dialog is an Activity from a different package, and Espresso can't access it. Luckily, there is a solution - the UiAutomator framework. UiAutomator is more flexible than Espresso: it doesn't limit your tests to your application, allowing you to navigate anywhere on the device and interact with any UI elements. Still, unlike Espresso, it doesn't do any internal synchronizations leading to more flaky tests, and its APIs are not as simple and intuitive as the Espresso ones. We'll use the combination of both to write UI tests for a simple application that requires the access to Contacts.

The demo app

First thing that our app wants to do is to load a list of contacts from the device's Contacts database. As soon as the app is opened, it will ask the permission to access Contacts:

If the user taps Allow, we'll proceed and load the contacts from the database:

We'd also like to handle possible edge cases, such as if the user taps on Deny button, therefore not granting us the permission to access Contacts. In this scenario we'd like to show some short rationale and provide a button that will help user grant the required permission if she changes her mind:

Another possible scenario is that the user checks the Never ask again check mark and taps Deny, and then at a later point decides she still needs this functionality. This sounds like a real edge case, but we want to be prepared. And since we can't bring the permission dialog back anymore, the only way to fix it is to help the user navigate to the application Settings and toggle the switch for the Contacts permission inside the Permissions screen. We'll need our longer rationale for this:

Alright, now let's see how we can use the combination of Espresso and UiAutomator to implement the test cases for these scenarios. For the complete code example please visit the repo page on GitHub.

Setting up UiAutomator

UiAutomator comes with the local Maven repository that is a part of the Android SDK, so you can simply add this line to the build.gradle file:

As the error message suggests, this can be fixed by adding the tools:overrideLibrary element to the Manifest, but the trick is that it should end up inside a Manifest file under the androidTest directory, so just create one with the following contents:

Test case A: Request permission dialog is displayed

First, let's write a test that will simply verify that the request dialog is shown as soon as we open the application. Remember, we can't do this with Espresso, since it doesn't have access to any UI that's outside our package. Therefore we'll use UiAutomator, and we'll create a helper method that looks like this:

UiDevice is an abstraction that UiAutomator uses to interact with the test device: in addition to the findObject() method, that helps us get hold of the UI components on the screen, it has a bunch of methods such as pressBack(), pressHome() and others. UiSelector is a class that helps us build a query which we can pass to findObject() to locate the UI component we're interested in.

At the end of the test we'd also want to dismiss the permission dialog to avoid having it on the screen when the next test starts. To Deny the permission, we'll use the following helper method:

This method uses the same approach to locate the Deny button, and then invokes click() to interact with it. Note that the method will throw a UiObjectNotFoundException if it can't find the button on the screen.

Having these methods, we'll be able to create the following test method:

Note that the method name starts with a. I'm doing this here to control the order in which the test methods will be invoked by the test runner. By the way, don't expect JUnit to figure this out automatically, if you need a certain order for your test methods, add the following annotation to the test class:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)

I felt reluctant to impose the strict ordering for the tests, since I believe it's always better if the tests are independent. But in order to have a suite of independent tests we need to implement the cleanup properly, bringing the application state to initial after every test. In this particular example this would mean that we have to make sure all permissions are revoked before each test starts. The problem with this approach is that the application process is killed when you revoke a permission. The test runner doesn't expect this kind of behavior, therefore the tests will fail to run to completion. Hence, we'll have to resort to the strict ordering in this test suite.

If the permission is already granted, we'll just load the contacts, otherwise we'll request the permission. We'll also keep a flag called isPermissionAlreadyDenied to avoid requesting the permission immediately after Deny was clicked, since the closing of the permission dialog will trigger onResume().

Test case C: Long rationale is shown if the permission is Denied permanently

By "denied permanently" I mean that the user has checked the Never ask again check mark, thus preventing us from displaying permission dialogs in the future. The following test verifies that we're showing a longer rationale that contains the instructions on how to grant the permission from the Settings screen:

Additionally, we're including some cleanup code that will prepare the setting for our next test. We need to navigate to the Settings screen, then open the Permissions page and grant the permission by toggling the switch. The logic of opening the Settings screen should be implemented in the Activity code, so the test code can just tap on the Grant permission button, and then use the following methods:

Unfortunately, the Android API doesn't provide a clear way to identify the fact that Never ask again was checked, therefore we'll have to rely on the behavior of shouldShowRequestPermissionRationale(), which returns false in this case.

The click handler code for the Grant permission button looks like this:

Since we've previously granted the permission, the code for loading the contacts should fire, and we'll have the list populated.

Conclusion

In the short example above we've managed to implement a few test cases to verify correct user interactions with the permission dialog. Unfortunately, Espresso doesn't allow us to interact with system dialogs, so we used UiAutomator, which is perfectly compatible with Espresso. The tests helped us implement the user flow correctly, and to make sure we're handling the edge cases properly.

Check out the complete source code for this example on GitHub. The UiAutomatorUtils class is a collection of reusable helper methods for the scenarios described in this example, feel free to copy it into your project.

If you're testing runtime permissions in your own project, I'd love to hear about the approach that works best for you. Feel free to share some thoughts and get in touch.