There are many automation tools out there in the market. Both open source and commercial. We made an attempt to try some well-known open source tools to find out if they are useful to adapt for mobile app testing. We have documented their features and the limitations as we tried them to test some of the android apps from Google Play store. We are yet to discover their value on iOS if they are supporting iOS. Calabash is one more open source tool which we would be trying sooner. Going forward we will try out commercial tools if we feel they are worth giving a try. We will share our thoughts as we do it. Please feel free to give a constructive feedback.

Monkey Runner by Android:

There are three classes in package com.android.monkeyrunner: MonkeyDevice, MonkeyImage and MonkeyRunner.

The MonkeyDevice, represents a device or an emulator. The class includes methods to perform various UI and user actions on device. The MonkeyImage represents a screen shot image of the device or an emulator. MonkeRunner: This includes methods to establish connection between device and monkey runner programs and managing the execution of the monkey runner program. Individually they support set of operations as explained below to enable automation.

The methods supported are drag gesture, touch, key press, type equivalent to consecutive key presses, wake up the screen, reboot device, launch an activity, install a package, uninstall a package, broadcast intents as if it’s generated by app, Execute a shell command, Take screen shot which is a monkey image. Read the device property variables such as current package and class of activity, device name, screen height, screen width, capture the screen shot, write it file or compare with the other image etc.

The touch and drag methods need the co-ordinates on the screen to be specified, which limits the developing the scripts which are scalable and can easily maintained. However there are plugins developed by third party but they seem to be in primitive stage.

Provide methods to enable interactive execution of the script like prompt for inputting the data, prompt list of choices.

Supports only Jython as scripting language.

Supports only android apps and doesn’t have support for UI locators. UI interactions are co-ordinate based. All these limitations make monkey runner not a good choice for automation.

Supports data driven testing by executing the parameterized editing of the .mt scripts with default values. The data is part of a CSV file and specified as source for the script.

Supports verification commands by checking the value of the component matches the argument by absolute value, by wildcard, by regex expression and by image

Supports think time and time out in milli seconds as the delay for each command as well as global think time and time out if not specified for any command

Supports command to wait for an element optionally with a time out (default 10s). Also has waitfornot which is inverse of waitfor

We can call .mt scripts within .mt scripts using the monkey talk language.

Can edit test suites .mts which invoke multiple tests which are test scripts. Before invoking they call a setup script and post execution a tear down script with optional CSV file as input or other arguments. This is a GUI based editing. The tests are in line with the JUNIT tests and the results are pare part of the junit panel and an xml file.

Supports JAVA API’s to write the tests as pure java standard junit tests with the support of monkeytalk-java-all-in-one.jar in its class path. Can integrate with junit runners such as eclipse, ant and maven to clean, compile, run and report the results of the junit test code. The reports are generated as xml/html files.

Supports gestures such as tap, move, drag and swipe (Note the gestures recording didn’t work when tested with android app, not very reliable I would say).can capture screenshots.

Customer Support lacks big time at least for open source community

When instrumented the layout is screwed by the monkey talk. To verify view the layout elements and their mapping in layout using the uiatomatorviewer tool with app running on and without instrumentation.

Cannot launch the app. Only upon launching the agent establishes connection.

Agent is not very reliable in terms of establishing connection when the app in background is brought to foreground i.e. resumed.

Cannot test for scenarios where the app under test launches another app which is not instrumented. Cannot test with the devices pre-installed apps.

Does not support key presses.

No mention of pinch and zoom or multi touch gesture support

Can invoke native shell commands in the scripts.

The support isn’t good for the queries on forum. The product bug fixing is slow.

Has good support for non-critical and good to have features such as CI, scripting and recording tests. But some of the core features to enable testing such as gestures support don’t seem be stable with record and playback. It does too much instrumentation.

Does not support Windows mobile apps

Appium by Sauce Labs:

Appium is a server and the client implements Mobile JSON wire protocol. The clients can implemented as webdriver compatible binding for Java, Python, Ruby, Perl, C#.

Implements subset of selenium based API’s and then the custom API’s compatible to both android and iOS. However there are some platform specific as well as emulator/simulator only API’s.

The iOS app’s can be automated with appium only on Mac-OS X. However the appium for android can used on windows and Linux. Check for the corresponding versions of the tools and software for compatibility.

It looks like java binding supports most number of the API’s. I tried python bindings to try the automation for some of the apps but java has more methods. There are other scripting language support such as ruby, C#, python, java script etc.

It has support for vast number of actions. single and multi-touch gestures like tap, flick, pinch, zoom, move, drag and drop, swipe, long press by element id or co-ordinates, shake, Has key press support, can start activities, install an app, remove an app, launch an app, close an app, lock screen, check for locked screen, hide keyboard, reset app (stop and launch?) check if an app already installed, open notifications, force the app to back ground.

Support various class of locators by name and by accessibility id, locators specific to android (uiautomator) or iOS (uiautomation).

Supports Selendroid with limited features, for Android 2.3 and above for native or hybrid apps, above android 4.2(API level 17) uiatomator support is available. However some of the features such as zoom/pinch are available starting API level 18.

Supports pull and push file interfaces.

We can invoke shell commands as well in client program. At least I could do it in python for android.

Web apps are supported with safari on iOS and browser on android.

In case of hybrid app’s there is an interface to switch the context to any of the web view and native view specifying the context id. Can identify multiple contexts

Works across apps without instrumentation.

Has fairly good support with its discussion forums (involves mostly appium users and sauce labs support engineers) and documentation on-line.

The product is being improved with new features and being stabilized at good speed.

Does not support record test case feature.

Does not support Windows mobile apps.

What features make a good mobile automation tool?

1) Support for most well-known mobile OS platforms such as android, iOS and Windows. Other one being blackberry.
2) Support for most famous versions of the OS’s (Android, iOS and Windows are most important).
3) Supports record and playback feature.
4) Ability to export the recorded actions as standard test scripts (preferably junit tests) and able to be edited by the engineer. Ex: testing login feature with data sources.
5) Supports invoking all kinds of UI objects (or views or controls) and their discovery by various search criterion (by name, by type, by class name, by id etc.).
6) Supports all kinds of gestures such as swipe, pinch, zoom, flick, shake, scroll and patterns.
7) Does not need any instrumentation of the apk, thereby testing the app as it is.
8) Does not need to provide apk source as in case of testing the pre-installed apps on mobile.
9) Ability to record or write the test scripts for one platform (OS and its version) and run the same scripts on other platform with minimal changes.
10) Provides supports for the continuous integration for agile development and testing.
11) Supports verify (or assert) during test and generate test report at the end of each test.
12) Supports native, hybrid and web apps.
13) This is not really a feature automation tool should support. But, it’s good to have. The sauce labs mobile cloud is compatible with the appium test scripts. Xmarin cloud supports calabash for android and iOS. Calabash is an open source automation framework supporting ruby binding. If the customer needs us to test with various devices to address device fragmentation and we do not have those devices in our organization then we need to have an account for mobile cloud. With appium or calabash we know which cloud to choose if it has those devices which we are targeting to test on.
14) Support object based discovery of UI elements rather than image based to easily maintain and scale test scripts.
15) Support key presses and have support for things such as forcing the app to background and bringing it back to foreground, install app, remove app, launch app, terminate app, wake up the screen, lock and unlock screen etc.
16) Supports various well known programming/scripting languages such as java, python. C#, ruby etc. to write automated scripts for various test cases.

Ashish Mathur

Thanks Ashish for your comment Its great to see you commenting on our blog. We will amaze you with more cool work soon. Thanks again. Lets catch-up somewhere for a drink or coffee. My number is 9880952643 and e-mail st [at] testinsane [dot] com Cheers!

Shailja

August 27, 2015 at 12:20 pm

Hi Santhosh, so does Appium directly works on device with a hybrid app? Last time we tried, it wasnt supporting test running on device. Recently we believe they launched a feature to run test on device with native apps, not sure if they support Hybrid apps.

Shailaja, The appium works for hybrid apps as well. When our test script for appium intends to interact with the webview of the app, it needs to discover the webview context to switch to and use the driver.context() api to switch. Once in web view context, use api’s which you would normally use when using selenium for web. The appium treats both emulator and real devices same with few exceptions. Android SDK provides only emulators. Have not come across any simulators for android platform. Anytime, emulators are the best choices compared to simulators. For iOS apps only simulator is supported on mac devices.In comparison to real devices the emulator won’t be a good choice as it’s slow and crash occasionally.
Regards, Sandeep Tuppad

Shailja Sehgal

September 2, 2015 at 10:32 am

Thanks Sandeep for your thoughts, my question was more for what kind of test should we automated on emulator vs a device.

And general assumption(or rather perception) is that automation run faster on emulator than device and the devices are not supposed to carry and execute loads of test and usually gets heated with constant execution in case of CI environments where you run test multiple times in a day.
So in such cases what would you suggest.

Shailaja, it’s a perception that the automation tests run faster on emulator, with exceptions. if we have an android emulator running on x86 machine for ARM architecture, the the app’s byte code is converted to the arm machine code by the dalvik virtual machine. However the host machine is x86 based. So the arm instructions are translated to the equivalent x86 instructions to run. The translation across the CPU architecture is always inefficient and might slow down. Also there are other programs(or processes) which are running on the x86 machine and the emulator is competing with them for the CPU cycles. However if we have a very very powerful x86 machine(lot of RAM and powerful multi core with higher clock speed, may be other parameters) and we configure the emulator program to run with higher priority then we may cross the performance offered by the real android device with arm architecture. Nowadays the mobile devices are powered with multi core CPU’s and adequately high RAM I have observed the app’s execute faster on the real android device than emulator and there is no translation of the instruction across architecture as the instructions are run natively i.e supporting arm architecture. There are other limitations as well with the emulator. What if the app is depending on the sensors and mobile carrier services such as making phone call or sending and receiving a SMS and it’s not uncommon. I would use the emulators when I want to test the app, how it scales up on the device with some of the screen sizes, pixel density and the android OS version, which is not very widely supported on the devices in market(provided I don’t want to invest on that device and there is no serious difference in the behavior of the app on emulator and the real device(s) the app is targeted for).

Shailja

September 2, 2015 at 9:56 am

Hi , any thoughts if multi threaded execution i.e multi-simulator launch/execution of automated test is possible in Appium. E.g i launch one simulator for iPod and one for Android and do some kind of parallel execution using Appium, since it sits on top of selenium and multithreading it supported on selenium grid, we are thinking it may work but need to try out.

All we need to do is launch multiple appium servers and create a client driver instance for each appium server. The appium server can be launched for multiple devices with differing configuration but same app under test and we can distribute the tests to different servers either in custom implementation or selenium grid(I believe it’s possible to create a grid with appium server as nodes, though I have not given a try).

Good to know this much detailed information. I think for the person who is new to mobile automation . For him Appium would be a better option considering its a generic tool for both android and ios and also has Java language which people know mostly.

Thanks Gaurav. I am glad to know that you landed here and thank you so much for your comment. It does motivate us to do more. Keep loving us and we amaze you with our work for the testing community across the globe. Cheers! (~Santhosh Tuppad)

malathy m

Yoav Geva

February 4, 2016 at 2:00 pm

More information on Monkey runner: there is java implementation called chimpchat, which give Ui locators if needed for more capability, Monkey runner does not test only 1 app but all of android applications, The monkeyImage class give you Ability to check if part the UI looks the same in all devices, ability that does not exist in other tools, cause it compare by Image and not by selector.