Testing

Do you have a testing strategy to make sure that every one of your Mobile App users has a great experience every time they use the App while also balancing the trade-offs between cost, quality and time-to-market?

Expert verification and validation of mobile apps across various devices and OS platforms

Our experienced team of mobile testing analysts are skilled and excel at visualizing all the varied real world parameters that an app must function within and work towards making sure it does.

Boston Technology Corporation offers end to end mobile application testing services that take into account parameters like devices, networks, bandwidth, navigation, user experience and more. Our services include

The mobile device eco-system is extremely fragmented. Thousands of different devices, multitudes of screen resolutions, variations in different operating systems (especially in Android world), large geographic coverage and plethora of network operators, all make testing a mobile App in an time and cost effective manner a real challenge. And we haven’t even begun to talk about ‘Internet of Things’ and Wearable devices.

An effective mobile testing strategy should focus on two factors – reducing testing costs and improving ‘Time to Market’ for the App. Some of the key aspects to consider when formulating a testing strategy include:

Type of functional and non-functional tests to include in testing plan

Choosing the most important devices to test on based on expected usage pattern, target audience and market share of the devices

Choosing the right automation methods and tool(s) for different types of test coverage

Functional testing verifies that the Mobile Application meets the functional requirements of the App. This testing step is critical to ensure that the App functionality built to meet the stated business requirements and objectives works correctly as designed. Functional testing leverages automation tools but many of the testing steps require manual testing to effectively mimic App user behaviour.

Functional testing typically follows these process steps:

Review testing requirements and decide on a testing strategy that outlines type of functional testing, testing mode and expected outcomes

Prepare test cases based on functional requirements of the App

Execute test cases and document outcomes

Compare actual and expected outcomes and generate test reports

User Interface testing:

User Interface (UI) testing ensures that your Mobile Application returns the correct UI output in response to a sequence of user actions on a device, such as entering keyboard input or pressing toolbars, menus, dialogs, images, and other UI controls. UI testing also includes testing for:

Colour palette

UI consistency

Graphic Display

Integration Testing:

Integration testing tests inner working of different App components as well as integration of the App with external applications as such as SalesForce, SharePoint, Facebook, Twitter and any other Enterprise or social media applications. This testing typically starts at the low-level component level and continues until all hierarchical components till the highest level are tested. Such detailed bottom-up testing facilitates efficient error detection in the multitude of internal and external integrations within your mobile app.

System testing (End-to-End testing):

Once all the App components are integrated and integration testing is completed successfully, the App as a whole is tested rigorously to ensure that it meets Quality Standards. This type of testing is performed by a specialized testing team. System testing is critically important because:

It is the first time in the testing process where the application is tested as a fully integrated functional App

The application is tested thoroughly to verify that it meets the functional and technical specifications

In this step, the application is tested in an environment which is very close to the production environment where the application will be deployed hence closely mimicking the ‘real-life’ experience of the App

System Testing enables us to test, verify and validate both the business requirements as well as the Application Architecture

Regression Testing (Automation/Manual):

Whenever a change in an App is made, it is quite possible that other areas within the application have been affected by this change. The intent of Regression testing is to ensure that a change, such as a bug fix did not result in another fault being introduced in the application. Regression testing is so important because:

It helps minimize the gaps in testing when changes are being made in an application as a result of integration, UI or system testing

It helps mitigates risks that an unintended effect on the App functionality was caused by an unrelated change

User Acceptance Testing

During UAT, the mobile app is tested for user acceptability to validate the end to end business flow of the application. This testing is executed by UAT users in a production-like environment comprising various devices and OS as well as web services and backend database servers. UAT is performed after System Testing is successfully completed and all or most of the major defects have been fixed.

Non-Functional Testing covers testing mobile applications for usability, performance, scalability, security as well as compatibility on different device and OS platforms. This testing is typically done after the application has been tested for functional requirements and provides insight into the production readiness of the application. Non-functional testing can be effectively carried out by using testing tools unlike functional testing which is largely manual. For this type of testing to be effective, it is important to create specific measurable test objectives.

Usability Testing:

Application usability can be described as how effectively users can use, learn, or control the App. Some questions to answer when determine usability are:

Is there a UI metaphor being used to help users adapt? (for example, the ‘Mobile’ is a metaphor)

Are the most common operations streamlined to be performed quickly?

Can new users quickly adapt to the Application without help? (Is it intuitive?)

Do validation and error messages make sense and help users navigate through the Application effectively and easily?

Security Testing:

Security testing confirms that the application is secure from internal or external security threats. It is a process to ensure that the App protects critical data and maintains functionality as intended. It measures how effectively the application can resist unauthorized attempts at usage or behaviour modification, while still providing service to legitimate users. Few examples of items covered in security testing are:

Is the App secure from hackers who may attempt to hack into it without proper authorization

Role based security to provide authentication for using specific App functionality (if used in the App)

User, group and role administration

Encryption of communication between the App and any third party application integrations as well as back end database systems

Performance testing:

Performance refers to the degree to which an application completes its designated functions within given time and device constraints. In Android and iOS applications, performance testing aims to verify performance goals, such as execution time, responsiveness, memory usage, memory leakage, power consumption, and network usage. For performance testing to be truly effective and reflective, it’s important to benchmark performance requirements against similar Apps and use those benchmarks as inputs for testing measurements.

Helps gain an understanding of the performance of individual pages/URLs/Actions taken in the application

Helps detect any memory leaks in the Application that may cause degradation in performance of the App as well as the device

Localization testing:

Localization is a process of adapting internationalized software for a specific region or language by adding specific localized components and translating text. Time-zones, date formats, language, currency are some of the factors important for effective localization of any application. Localization testing ensures that these factors change seamlessly and in many cases without any user interaction or intervention when the same application is used in different regions of the world.

App Installation and Up-gradation testing

App Installation and Up-gradation testing ensures that any App user is able to Install and Upgrade the App without any glitches. Testing App updates is critically important before release candidates are submitted to the app stores.

When a user upgrades an App, most functionality and features will probably work and serious issues or bugs are unlikely to occur if the new version of the App was tested thoroughly. However some updates may fail if the upgraded App references some of the old (cached) file in the app bundle that didn’t get reset during the upgrade process. New version of the App may require some configuration changes to these files that were stored on user devices when the app was originally installed. The App may work in such a way that updates to those files will not run until the next time the app is cold-started (user has to apply fresh install as the database is not ‘emptied’ and reset during the upgrade process).

It’s important that the installation and upgrade process is tested thoroughly for expected behaviour on all currently supported OS releases and devices.

Compatibility Testing

Compatibility testing is a ‘must-do’ testing step for mobile Apps. iOS Apps can be tested on just a handful of devices (different version of iphone and ipad) but the Android device market is very fragmented so compatibility testing on all supported Android devices becomes critical. The purpose of compatibility testing is to ensure an app’s key functions behave as expected on a specific device, OS and network. This testing is fairly quick and can be planned well in advance. Some of the steps in this testing are:

Check that the App works on all the versions that are in-between the specified version and the higher/latest version in the market

Check that the App works as expected on all the specified Android devices

Check that the application works with different networks (for example, 2G, 3G, Wi-Fi etc.)

Check that the application works in offline mode for the functionality supported in the off-line mode

Testing Strategy

The mobile device eco-system is extremely fragmented. Thousands of different devices, multitudes of screen resolutions, variations in different operating systems (especially in Android world), large geographic coverage and plethora of network operators, all make testing a mobile App in an time and cost effective manner a real challenge. And we haven’t even begun to talk about ‘Internet of Things’ and Wearable devices.

An effective mobile testing strategy should focus on two factors – reducing testing costs and improving ‘Time to Market’ for the App. Some of the key aspects to consider when formulating a testing strategy include:

Type of functional and non-functional tests to include in testing plan

Choosing the most important devices to test on based on expected usage pattern, target audience and market share of the devices

Choosing the right automation methods and tool(s) for different types of test coverage

Functional Testing

Functional testing verifies that the Mobile Application meets the functional requirements of the App. This testing step is critical to ensure that the App functionality built to meet the stated business requirements and objectives works correctly as designed. Functional testing leverages automation tools but many of the testing steps require manual testing to effectively mimic App user behaviour.

Functional testing typically follows these process steps:

Review testing requirements and decide on a testing strategy that outlines type of functional testing, testing mode and expected outcomes

Prepare test cases based on functional requirements of the App

Execute test cases and document outcomes

Compare actual and expected outcomes and generate test reports

User Interface Testing:

User Interface (UI) testing ensures that your Mobile Application returns the correct UI output in response to a sequence of user actions on a device, such as entering keyboard input or pressing toolbars, menus, dialogs, images, and other UI controls. UI testing also includes testing for:

Colour palette

UI consistency

Graphic Display

Integration Testing:

Integration testing tests inner working of different App components as well as integration of the App with external applications as such as SalesForce, SharePoint, Facebook, Twitter and any other Enterprise or social media applications. This testing typically starts at the low-level component level and continues until all hierarchical components till the highest level are tested. Such detailed bottom-up testing facilitates efficient error detection in the multitude of internal and external integrations within your mobile app.

System testing (End-to-End testing):

Once all the App components are integrated and integration testing is completed successfully, the App as a whole is tested rigorously to ensure that it meets Quality Standards. This type of testing is performed by a specialized testing team. System testing is critically important because:

It is the first time in the testing process where the application is tested as a fully integrated functional App

The application is tested thoroughly to verify that it meets the functional and technical specifications

In this step, the application is tested in an environment which is very close to the production environment where the application will be deployed hence closely mimicking the ‘real-life’ experience of the App

System Testing enables us to test, verify and validate both the business requirements as well as the Application Architecture

Regression Testing (Automation/Manual):

Whenever a change in an App is made, it is quite possible that other areas within the application have been affected by this change. The intent of Regression testing is to ensure that a change, such as a bug fix did not result in another fault being introduced in the application. Regression testing is so important because:

It helps minimize the gaps in testing when changes are being made in an application as a result of integration, UI or system testing

It helps mitigates risks that an unintended effect on the App functionality was caused by an unrelated change

User Acceptance Testing

During UAT, the mobile app is tested for user acceptability to validate the end to end business flow of the application. This testing is executed by UAT users in a production-like environment comprising various devices and OS as well as web services and backend database servers. UAT is performed after System Testing is successfully completed and all or most of the major defects have been fixed.

Non-Functional Testing

Non-Functional Testing covers testing mobile applications for usability, performance, scalability, security as well as compatibility on different device and OS platforms. This testing is typically done after the application has been tested for functional requirements and provides insight into the production readiness of the application. Non-functional testing can be effectively carried out by using testing tools unlike functional testing which is largely manual. For this type of testing to be effective, it is important to create specific measurable test objectives.

Usability Testing:

Application usability can be described as how effectively users can use, learn, or control the App. Some questions to answer when determine usability are:

Is there a UI metaphor being used to help users adapt? (for example, the ‘Mobile’ is a metaphor)

Are the most common operations streamlined to be performed quickly?

Can new users quickly adapt to the Application without help? (Is it intuitive?)

Do validation and error messages make sense and help users navigate through the Application effectively and easily?

Security Testing:

Security testing confirms that the application is secure from internal or external security threats. It is a process to ensure that the App protects critical data and maintains functionality as intended. It measures how effectively the application can resist unauthorized attempts at usage or behaviour modification, while still providing service to legitimate users. Few examples of items covered in security testing are:

Is the App secure from hackers who may attempt to hack into it without proper authorization

Role based security to provide authentication for using specific App functionality (if used in the App)

User, group and role administration

Encryption of communication between the App and any third party application integrations as well as back end database systems

Performance testing:

Performance refers to the degree to which an application completes its designated functions within given time and device constraints. In Android and iOS applications, performance testing aims to verify performance goals, such as execution time, responsiveness, memory usage, memory leakage, power consumption, and network usage. For performance testing to be truly effective and reflective, it’s important to benchmark performance requirements against similar Apps and use those benchmarks as inputs for testing measurements.

Helps gain an understanding of the performance of individual pages/URLs/Actions taken in the application

Helps detect any memory leaks in the Application that may cause degradation in performance of the App as well as the device

Localization testing:

Localization is a process of adapting internationalized software for a specific region or language by adding specific localized components and translating text. Time-zones, date formats, language, currency are some of the factors important for effective localization of any application. Localization testing ensures that these factors change seamlessly and in many cases without any user interaction or intervention when the same application is used in different regions of the world.

App Installation and Up-gradation testing

App Installation and Up-gradation testing ensures that any App user is able to Install and Upgrade the App without any glitches. Testing App updates is critically important before release candidates are submitted to the app stores.

When a user upgrades an App, most functionality and features will probably work and serious issues or bugs are unlikely to occur if the new version of the App was tested thoroughly. However some updates may fail if the upgraded App references some of the old (cached) file in the app bundle that didn’t get reset during the upgrade process. New version of the App may require some configuration changes to these files that were stored on user devices when the app was originally installed. The App may work in such a way that updates to those files will not run until the next time the app is cold-started (user has to apply fresh install as the database is not ‘emptied’ and reset during the upgrade process).

It’s important that the installation and upgrade process is tested thoroughly for expected behaviour on all currently supported OS releases and devices.

Compatibility Testing

Compatibility testing is a ‘must-do’ testing step for mobile Apps. iOS Apps can be tested on just a handful of devices (different version of iphone and ipad) but the Android device market is very fragmented so compatibility testing on all supported Android devices becomes critical. The purpose of compatibility testing is to ensure an app’s key functions behave as expected on a specific device, OS and network. This testing is fairly quick and can be planned well in advance. Some of the steps in this testing are:

Check that the App works on all the versions that are in-between the specified version and the higher/latest version in the market

Check that the App works as expected on all the specified Android devices

Check that the application works with different networks (for example, 2G, 3G, Wi-Fi etc.)

Check that the application works in offline mode for the functionality supported in the off-line mode

Related Resources

The Ultimate Cheat Sheet On Mobile App Testing

Ask any mobile app developer about mobile app testing and you’ll get an earful. You might even wonder what the fuss is all about. After all, it’s about ensuring quality, functionality and performance. Right! But when it comes to those small devices we cannot live without, things are much more complicated than that.

Here is a cheat sheet on app testing that will come handy while planning a mobile app development project.