Mobile Testing – The Next Generation Testing

Test Automation of Mobile Apps

In 1973 when Martin Cooper made his the first mobile phone conversation to his Rival in Bell Labs, he never envisioned the fact that he was stepping the stone of a technology revolution in 21st century. The latest form of that $3995 costs 2.5 pound weighed device now changed life of this planet tremendously in every aspects including entertainment, social, technical and business.

International telecommunication Union in 2012 reported that there were billion mobile phone customers worldwide. Advent of mobile web in 1995, multimedia advancement and introduction of smart phones with improved processing capability changed the face of this technology landscape abruptly.

Quality always matters

Starting from snake game app in Nokia there we 80,000 app were published in Apple App store and Google play. Estimates shows that app downloads in 2013 will range from 56 to 86 billion. By 2017-18, there could be 200 billion downloads from different app download centers.

Being said all these promising statistics, quality of the applications floating in the market is always a matter of concern. A recent study from Compuware shows that the app down pour may stop any time! 79% of the users may reject the app if it’s failed consecutively two times.62 % of the user’s experiencing catastrophic issues like app crash or freeze. 47 % of the active users are experiencing performance issues. With the heavy explosion of smartphone models and usage of mobiles data changed this mere commutation device as powerful business tool. A small defect in the system may lead to a huge financial loss and credibility of enterprises. An app going to market is literally going to public hand with diversity in usage pattern and technical know-how. An error prone app will simply reject with very few downloads. There comes the importance of independent verification and validation mobiles apps going to stores. Variety of mobile applications exists in market like web applications, platform applications, SMS, Mobile websites. So all need to be tested to assure quality. Because of such a boom in industry, it is not surprising that mobile application testing spins up.

Quality is never an accident; it is always the result of intelligent effort.

John Ruskin

Purpose

This white paper explores industry best practices on mobile testing and demonstrates effective ways of managing mobile application quality through test automation. As with conventional applications, there is an increased need to automate mobile applications testing to improve scale and efficiency. A well-orchestrated automation strategy enables hardware manufacturers and software developers to reduce effort and accelerate time to market.

Mobile Test Automation Challenges

Because mobile testing must be performed on many different combinations of devices, browsers, and operating systems, it is time-consuming –and costly– to do all of the testing manually.

Test automation can be used to reduce the time and costs associated with testing. In addition, test automation can be used to increase the productivity of the test team.

But key call out, automation testing is not intended to replace manual testing it is to reduce the effort/time to market for the product.

There has to be a clear goal/objective defined before investing in mobile apps testing. Because it has its own challenge like tools support, personnel learning curve and infrastructure support as well.

It is not practical to try to automate everything, especially for mobile devices. However, leveraging Common off the Shelf (COTS) tools can greatly benefit the automation process.

Mobile applications can be broadly classified into three different types based on the mobile development technology employed to create them. These are native mobile application, web mobile applications and hybrid applications.

Mobile Testing Challenges for Native Applications

To many, “mobile apps” have become synonymous with native applications (and hybrid applications). Commonly downloaded from an App store, they offer the user a unique experience that maximizes the capabilities of the device & operating systems for which they are developed. The app download is often controlled by the gate-keeping app store, with mechanisms in place to charge potential consumers. While native applications can provide a rich experience to the user—and possibly a lucrative one for the developer—they also add some complexity to the lives of those tasked with testing them. Testing needs to ascertain that the app can be successfully downloaded to the device, executed on the device and interact with the supporting back content infrastructure. When updates are made you need to be sure that the application can to be pushed out to and accepted by the end user. There’s a misperception that successful testing of app functionality on one device provides assurance across all others of the same operating system.

Mobile Testing Challenges for Web Applications

Like the Web itself, a mobile Web application is viewable by users around the world. Even if you’re initially targeting only users in a single country or on a single network, it helps to understand the global dynamic. When we test mobile Web applications we encounter several challenges presented by the nature of the global, mobile Web. As we understand the nature of each challenge, we can explore different technology options to manage issues and mitigate risk. Coming up with the right solutions for your requires an assessment of the advantages and disadvantages inherent in each of the testing options available to you and determining the technology that best suits your testing requirements. The test automation of mobile apps is challenged by devices, network, and scripting.

Generic challenges include:-

Device/platform diversity

Hardware complexity

Application complexity

Carrier network issues

Compressed development schedules technical challenges:-

Large number of test cases

API level test execution

Verification of UI control

Handset specific features.

The following are some of the reasons that make effective Test Automation of Mobile Apps challenging:

A mobile device is much more restricted compared to desktops – the underlying OS typically sandboxes each application and allows very limited inter process access, unless a phone is ‘rooted’.

Controlling the UI navigation of a mobile device is harder – in addition to lack of control mentioned in the previous point, mobile UIX response times are harder to predict than desktop equivalents and hence makes screen grab based predictions of pass/fail harder.

Mobile devices, by definition are not statically located entities – depending on where the device is, the network may introduce many challenges that completely break down a prior tested use-case.

The proliferation of different screen sizes and form factors make UI based testing more challenging as well.

Mobile Testing Industry Opportunities

In the growing mobile market, the need for specialized mobile testing is growing. The following three opportunities contribute almost 80% of total market demand.

Types of testing that can be performed in mobileweb, native/hybrid app includes

Usability testing

Functional testing – manual

UE testing

Regression testing – automation

Accessibility and security testing

Performance testing

Mobile Automation Testing

Functional Testing

Mobile Performance Testing

Mobile Testing Strategy

All mobile testing centers are around the emulators, simulators, tools and devices.

Emulators

An emulator is a hardware replica of an actual mobile device. The emulator simulates the mobile device and enables the tester to test a mobile app on a PC without having to test it on an actual mobile device.

There are three types of emulator:

Device emulators

Operating System (OS) emulators

Browser Emulator

Advantages of using emulators are the following:

One of the greatest advantages of testing with emulators is that an emulator will let you know exactly what is happening “behind” the device LCD, allowing a tester to do a debug and actually open up a screen to see what is happening.

The tester can also provide the developer with screenshots, high-level information, and data messages.

Emulators tend to be cost-effective because most of them are freeware.

Because the virtual device (emulator) is in control of its own software stack, testers can collect important information on “each component” of a content page.
Some examples of this information include inner text and page redirect URLs.

Many types of content compatibility tests – such a s verifying image sizes or identifying broken links – can be accurately and quickly performed.

Simulators

Simulators mimic the software of the device for testing purposes. Testers typically use the native browser of a PC for mobile browser simulation. To get a native browser to simulate a mobile browser, testers change the “User Agent” settings in the native browsers. This approach is typically used for automated functional testing.

With simulators, testing can be done quickly and easily without the use of emulators. In addition, simulators are cost-effective because they can be used without purchasing any new software. For Firefox browsers simulating iPhone and Android, automation is possible with tools like Quick Test Professional.

But this is mainly from the functional rather than look and feel perspective and is primarily used for functional automation testing. To reduce manual effort/cost, one of the popular ways is to go for functional automation.

For mobile web, this is the way automation can be successfully done and can greatly reduce effort.

Real Devices

By definition, devices are physical resources which need to be managed. Unlike simulators, where additional “devices” can be either additional software installed or simple configuration, real devices are something physical to own.

Conclusion

Test automation is a booming technology which has to be learnt as its importance is understood by the market trend which is tilting more towards smart phones and tablets usage replacing laptops and desktops.

So for performing quick testing in a competitive market where multiple platform and devices support has to be tested for any app in very short span of time, automation is the only way which can save us in achieving the quality goals.

Also, Read our previous post on User Experience where we list down some top three reasons when User Experience becomes User Exploitation.