Open test automation frameworks are constantly evolving, and at least one new test framework emerges every year and becomes the favorite one among developers. That puts more challenges on the development and testing/QA teams to make long-term planning, since investing in test automation yields results over a longer period of time and test assets cannot be ported to new frameworks frequently without losing a significant amount of the investments.

Appium has been the first choice among many centralized QA and testing organizations since it has its merits. The cross-platform support on a high level allows mobile tests to leverage common parts on both Android and iOS, even though a lot of platform-specific code still needs to be developed. It also allows you to use any of the most popular languages to write the tests. But on the downside, as a multilayered framework (a framework on top of native frameworks) it has performance issues not unknown to many test engineers.

Native test frameworks like XCode XCTest/XCUITest for iOS and Android Instrumentation (for example Espresso) are developed and maintained by the platform vendors Apple and Google. The organizations can be assured of the longevity and evolution of these tools but the drawback is that the test assets created with those tools are specific to the platform, with zero portability over to the other.

The scope of the analysis

We wanted to understand the characteristics of various test frameworks with productivity in mind, and analyzed test run data of our Public Cloud from the last couple of months with a total of 224,958 device tests and a total of 27.9 million test cases. That gives a good statistical significance for this analysis.

Client-Side Execution – The ordinary test execution where tests are running on the tester’s machine (client) remotely and the Appium server (Server) running on Bitbar Cloud communicates with the devices and communicates with the client over HTTP.

Server-Side Execution – Users upload their app and test files to Bitbar Cloud where we manage the execution, scheduling, and parallelization as well as test result processing. The Appium client and server are both running in the same logical processing unit bringing unprecedented performance to Appium testing.

Calabash

Findings

Test suite duration

With these data, we analyzed the data from three dimensions – average test duration, the number of concurrent devices used and the popularity of test frameworks.

The test suite duration on average is 4.0 minutes with 1.4-minute standard deviation. There is no significant difference between the frameworks except that the duration of Appium Android Client-Side Execution was on average 6.9 minutes. It is worth noticing that on average Appium server-side execution for Android takes 3.3 minutes, which is 52% faster test execution time compared to remote client-side execution. On iOS side the difference between client-side and server-side execution is smaller, only 11%.

How Appium Client-Side works: Every command requires a round-trip from the client residing locally on the developers/testers desk to the device in the cloud data center (in our case in San Jose, CA or Poland, Europe), and back. Every command. That creates significant latency and increases test execution times dramatically. This can be seen in the average test duration for Appium client tests.

XCTests and XCUITest are used for slightly different purposes. XCTest can be used to write small unit tests and tests for user interface elements, and it can even be used to execute performance testing on different parts of your application whereas XCUITests is used to develop UI tests that reflect users’ interaction with the application. Due to the different nature of the tests, the average test durations are much longer when using XCUITests (on avg. 5.4 minutes) vs. 4.1 minutes for XCTests. UI tests are always slower to execute.

The number of concurrent devices used

We also analyzed how our public cloud customers use concurrency in their testing. On Bitbar Cloud we have no concurrency limitations, which allow users to test faster and get results more quickly for every build.

Android Instrumentation tests are run on 13.2 devices concurrently and iOS XCTests on 22 iOS devices concurrently. It is worth noting that Appium client tests are always on 1 device/test.

The popularity of test frameworks

On Android, the standard instrumentation is the most popular framework. 71% of all test sessions on real Android devices executed Instrumentation tests.

On iOS, the share is more evenly spread: Appium (server-side execution) and XCTest were used on 42% and 34% of the device usage. Calabash is becoming obsolete due to discontinued support and development.