Android App Performance Test Scenarios

With end-to-end performance testing, Android developers gain a true understanding of how their apps will behave.

Android application performance tests need to follow an end-to-end application delivery model that tests for all the varying scenarios on each component involved. That way, android app developers can gain a true understanding of how their Android apps will perform on a given device as the end users would perceive them.

In a recent DevX article, we proposed such an end-to-end approach for mobile application performance testing. In this article we validate that approach for Android developers by will walking through an Android application performance test using a sample banking app.

Android App Performance Test Components

The Android app performance test involves the following components and tools.

Android Application

The sample Android app is a hybrid application for processing banking transactions. Hybrid mobile applications are installed on mobile devices just like native mobile apps but users can access them via either an installed app or the device browser. The aim is to test this application for performance and find different performance viewpoints.

Server Component

The server component (see details below) handles the business logic for the banking transactions initiated from either a mobile thick/thin client or a desktop Web browser.

Operating System: Windows Server 2000

JDK: 1.6

Web Server: Apache Tomcat

Database Server: MySQL

Network Emulator

A network emulator recreates a wide variety of network conditions such as latency, jitter, packet loss/error/reordering, and bandwidth restrictions, so that developers can simulate environments such as Wide Area Networks (WANs), Wireless LANs, GPRS, 3G, 2G, IP over Radio / Radio over IP (RoIP), Satellite or MPLS networks. The network emulator determines the traffic load generated by the application under different network conditions. We used Infosys WindTunnel, an in-house tool developed by Infosys Labs, for this performance test.

Load Generator

A load generator tool generates load on the application under test, and it will measure the load times. It allows configuration changes to ramp up the load, increase concurrent users, increase transactions per second, enable parameterization of the URLs, and name each URL. We used JMeter to recreate the load from different delivery modes for the sample application.

Android Device

We used both a high-end and a low-end device for testing to because performance can vary and we wanted to prove the importance of variability in device configurations on the same platform.

Performance Monitor

Perfmon is a performance monitoring tool that is mapped with windows. The required performance counters are configured before running the test. After you start the configured data collector set, it automatically collects the results data during the scenario run.

Android Debug Bridge

The Android Debug Bridge (adb), a tool that comes with the Android SDK, allows us to control and interface with our Android device. We used various adb commands to monitor the sample hybrid Android app.

Dalvik Debug Monitor Server

DDMS is an Android debugging tool that provides port-forwarding services, screen capture on the device, thread and heap information on the device, logcat, process, and radio state information, incoming call and SMS spoofing, location data spoofing, and more.

DDMS is integrated into Eclipse and is also shipped in the tools/ directory of the SDK. To launch the tool, follow these instructions.

From Eclipse: Click Window > Open Perspective > Other... > DDMS.

From the command line: Type ddms (or ./ddms on Mac/Linux) from the tools/ directory.

To set up representative Android app performance testing scenarios, follow the instructions for each of the following settings.

Workload Type

Script the necessary flows for different workload types using the load generation tool for the server.

Thick client — Install the hybrid apk file in the emulator. Record the necessary flows using JMeter.

Workload Mix

Load the server with the different workload scenarios:

100% from the thin client

100% from the thick client

Combination of both thick and thin client (For example, 40% load from the thin client and 60% load from the thick client)

Network types

Identify the different network profiles for which you will test. Set up the network emulation tool for the following profiles.

Airtel* Average 3G

Airtel* Best 3G

Airtel* Worst 3G

Airtel* Average 2G

Airtel* Best 2G

Airtel* Worst 2G

Airtel is an Indian telecom operator. Note that the speed emulated for the profiles do not represent the actual values, but they should during actual tests.

Device types

Install the Android hybrid apk in both the high-end and low-end devices.

High-end: Samsung Galaxy S

Low-end: Samsung Galaxy Y

During your actual performance tests, be sure to recreate all the scenarios applicable to the test application's context.

Android App Performance Test Execution

Here are the steps for actually executing the Android app performance test.

Load the server: The server was loaded with the scenarios identified in the previous section using JMeter. The server was monitored for performance while loaded.

Select a network type and a device: With the server loaded, the identified network profiles were recreated using the network emulator tool. In the tool, we edited the bandwidth, jitter, packet drop parameters based on our requirements such as 3G or 2G. The device was configured to use the emulated network. The POC MAC address of the device and the server were configured in the network emulator tool to filter the relevant information.

Access the application on a real device: The hybrid application was installed on the device to be tested. The Android command for installing the application (apk) is:

adb install <apk>

Once installed, the transaction was carried out on the real device with server and network conditions emulated. In addition, the transaction was also carried out on the on-device browser.

Monitor the components involved: The components were monitored during the transaction execution on the device.

These steps were repeated to cover all the test scenarios mentioned earlier. The following data was captured on the device using adb.

Server

Emulator

Device

CPU %

√

√

√

Memory Info

√

√

√

Processor Info

√

√

√

Network Info

NA

√

√

Response Time

NA

√

√

Android App Performance Test Results

Here are a few sample results that can be obtained by testing. Note that these results are only the tip of the iceberg of what you can do with end-to-end performance testing.

Device Performance

The graph below is the device result which is monitored by the adb commands. We observed no huge difference in PSS, memory allocated, heap size or used memory with the increase in load.

Network Performance Comparison

The graph below presents the network speed for 3G and 2G networks. Load is generated via the load runner tool. We observed that the 2G response time rose higher than that of the 3G as we increased the load.

As you can see, extensive analysis can be performed with the end-to-end data available for the different components involved in mobile application delivery. Such analysis would not be possible if the device were tested in isolation with no control over server/network conditions.

Conclusion

This article presented a proof of concept for end-to-end mobile application performance testing using a sample Android app. It detailed every component to be tested, the method for testing as well as the challenges involved.

When adopting this approach, be sure your Android application performance tests are under these conditions: