Transcription

1 Whitepaper Effective Test Management can help you to launch mobile payments faster, smarter and cheaper sqs.com Ensuring Quality Assurance is implemented in complex systems Introduction The primary role of test management is to ensure that a quality assurance and testing programme is conducted efficiently and effectively. In principle, the role of test management for m-payment systems is the same as that for traditional systems, in practice however, there are important differences that can mean the difference between success and failure. The many combinations of mobile devices, operating systems and infrastructure create a logistical nightmare and not all organisations will have capacity to test all possible permutations. A 2013 global survey of 1,500 executives 1 found that over 56 % of respondents cited the lack of specialised methods and processes as the biggest barrier to mobile testing in their organisation. By balancing the variables to be tested against risk, an m-payment system test strategy can ensure that sufficient testing will be conducted within reasonable timescales and budget levels. Further, a testing programme for mobile succeeds by focussing on the unique characteristics of mobile devices. Considerations such as battery consumption, network hand-offs between mobile and Wi-Fi networks, required levels of security as well as user interface and App Store delivery need to be considered within the overall test strategy. This paper presents a pragmatic, risk-based approach to testing m-payments systems including issues such as organisation structure, development methodology and project risk. Author: Sven Euteneuer Global Head of Technical Quality SQS Software Quality Systems AG, Germany Published: March 2014 SQS the world s leading specialist in software quality

2 1. The mobile quality challenge There are three primary considerations to be addressed in mobile application testing programmes: Complexity: the complexity of the software, hardware and infrastructure involved Quantity: the number and variety of system configurations to be tested Characteristics: the characteristics of the system that need to be tested (SQS uses the term quality fingerprint to describe this) 1.1. Complexity of m-payment systems Studies have shown that there is often a strong correlation between the complexity of software and the number of defects in that software 2. Despite their apparent simplicity, mobile applications are often more complex than their desktop or webbased counterparts while offering fewer features and functions to end users. To understand the complexity of m-payments systems it is necessary to look at the system as a whole, not just the app. Outside the device, links to back-end systems and network connectivity add a layer of complexity, while inside the device, hardware, software and operating system variations result in mobile systems that are typically much more complex than traditional software development projects. This complexity makes effective test management and testing practices in m-payments crucial. The issue of complexity and, by implication, quality, applies throughout the whole life of the system, not just the initial release. Research by the Rochester Institute of Technology 3, which used lines of code (LOC) as a measure of complexity, highlighted how mobile applications are not only more complex than traditional equivalents, but also become more complex more rapidly when new versions are released. The research showed that mobile applications demonstrated a rapidly accelerating growth in complexity as new versions of the applications were released, while desktop equivalents tended to towards a steady growth in complexity Quantity and variety of testing configurations Complexity of the payment system itself is not the only issue, the sheer number and variety of mobile platforms upon which mobile applications must operate presents a major quality management challenge. Test strategies for m-payment systems need to support testing on multiple mobile devices across multiple platforms, but they must do so pragmatically. Consider: fully testing 420 device models over 29 Android versions would require 12,180 combinations to be tested for each test case. Even with test automation, which can dramatically reduce the effort required, is testing all 12,180 combinations just for Android worthwhile? Ensuring a secure and reliable customer experience against the backdrop of a technically complex and diverse mobile environment is one of the key challenges when planning quality assurance activity Characteristics of m-payment system quality the quality fingerprint To many the notion of quality is somewhat ethereal, yet to define a test concept for m-payments, it needs to be operationalised, i.e. there must be a way to measure it and determine when the desired degree of quality is achieved. Quality models are used to accomplish this and the ISO/IEC model is used as the de-facto standard in the IT industry. This model uses quality attributes to subdivide the general notion of quality into more tangible sub-characteristics (Figure 1, next page). As for every system, the specific combination and prioritisation of these characteristics varies and can be thought of as a kind of fingerprint. Just as fingerprints differ between virtually any two people, so too does, the specific combination of quality characteristics for an IT system the quality fingerprint. Page 2

4 In traditional software development, risk-based planning can be applied to determine the degree to which functionality should be tested which features and functions to prioritise and which aspects of the system can be simply smoke tested. However, when applied to mobile systems, the risk-based approach needs to be broadened to include platforms, operating systems and versions for each device the application will run on. Additionally, it needs to consider all test tasks that arise from the broader scope of the mobile quality fingerprint. This broader approach to risk-based testing facilitates the pragmatic allocation of people and infrastructure, helping to deliver a product that has been sufficiently quality assured while at the same time keeping testing costs from skyrocketing. Multiple Distribution Channels Dozens of Target Platforms Hundreds of Devices Vendor Figure 3: Diversity of platforms, versions and devices in m-payments systems Live Example SQS strategic solution reduces testing effort by 80 % for a major mobile payments platform SQS proven experience in m-payments gave a leading financial services organisation the confidence to appoint them as a strategic partner for its complex multi-device retail mobile banking services. Developing a test strategy and environments was challenging as testing needed to include six major operating system versions across both ios and Android devices. SQS built a test infrastructure to support an offshore testing model, enabling the team to execute tests remotely without the need for on-shore resource. SQS implemented test automation to increase the pace and quality of testing: the designed environments and infrastructure enabled the client to execute each test cycle of 2,000 test scripts in just 30 person-days, a saving of 80 % compared to equivalent manual testing. Page 4

5 2. Test organisation and development methodology Risk can help determine where resources are focused, but risk-based testing of itself is not enough. Speed, agility and efficiency in test management and development practices are key to meeting the challenge of delivering systems in the dynamic and complex mobile marketplace. Achieving the requisite speed, agility and efficiency, means underpinning testing practices with a systematic, processbased testing approach. With such an approach in place, test cases and other assets are created efficiently and the rate at which those assets are produced also rises. Re-using test assets also makes automation simpler an important benefit when dealing with regression testing across mobile devices and platforms. Adopting a systematic or disciplined approach to test management does not remove flexibility from the quality assurance process. Regardless of whether Agile or Waterfall methods are employed, test management needs to be tailored so that it provides the flexibility required to fit in with the development methodology adopted. Best practices such as early error detection and the shift left of testing (i.e. testing tasks being performed earlier in the development lifecycle) are typically, though not exclusively, seen in Agile development projects. Bringing these practices into an m-payments development project will increase effectiveness and ultimately reduce cost. Page 5

6 3. Test effectiveness and efficiency Project and programme owners sometimes regard testing as a burden that slows delivery. However efficient and effective testing not only supports product delivery, but also accelerates the delivery of a quality product. The testing team s ability to deliver effective and efficient testing that will increase the speed of product delivery is largely determined by the tools, processes and strategy adopted in short, the test management approach Doing the right thing: testing effectively Test managers need to think differently about mobile testing compared to traditional software testing projects: methods and processes need to be modified to make testing more effective without huge increases in budgets. But what needs to change in test management for m-payments? Risk needs to be used much more comprehensively in prioritising testing effort. By expanding the concept of risk-based testing, quality assurance effort can be laser-focused to mitigate the biggest risks and so maximise the testing budget. Minimising product risk within the available testing budget means prioritising effectively. How? By extending risk-based testing to include the unique characteristics of m-payment systems. Applying risk management at all levels enables test and development teams to allocate resources effectively. The chart below illustrates how risk and priority drive the effort allocated to testing, for example, while critical functional test cases will be executed on every relevant device at least once, test cases with a lower than moderate risk and priority might only be executed once on a single device. Less frequently used device configurations can be risk assessed by conducting smoke tests through crowd-based testing services while higher priority requirements, such as security, can be addressed by in-depth security testing. Risk Each test case is executed on every relevant device at least once. A B C 1 All All All 2 All All Once Each test case is executed at least once on one device. Priority 3 All Once Once Figure 4: Prioritising test effort by risk and priority Page 6

7 3.2. Testing more by testing efficiently Prioritising testing tasks helps to reduce the overall volume of testing work. Nevertheless, there is still a clear need to conduct testing efficiently and make best use of available resource while still achieving the desired quality assurance outcomes. There are several strategies to improve testing efficiency - two of the most effective approaches for m-payments are testing earlier in the development cycle and using both test tools and automation frameworks Testing earlier: shifting test activities left reduces cost and effort Shifting testing left introduces test activities earlier in the development lifecycle. Opportunity cost, re-test effort and correction effort are all typically much higher the closer a project or an agile iteration is to go-live. Identifying and eliminating a defect early is almost always cheaper and faster than addressing that same defect later. The benefit of shifting left can be realised across all types of testing, from functional requirements that describe what the product does to non-functional requirements that define qualities such as security, performance and scalability. An example of the value of shifting test activity left can be found when addressing one of the top considerations in m-payments for both end-users and providers security. Tasks such as threat modelling, static analysis and dynamic analysis can often be conducted early in the project and help weed out conceptual as well as technical flaws that would have turned into real-life vulnerabilities. Carrying out security-testing tasks earlier in the lifecycle helps to minimise the risk of unexpected vulnerabilities being discovered late in the project and incurring the corresponding cost of rework and/or delay. Conducting testing activity earlier in the lifecycle does not necessarily mean that testers need to execute test cases against software delivered by the development team. Techniques such as document review, developer testing or using analysis tools are all capable of identifying errors early, well before the start of classic testing activity Tools and automation increase speed and reliability A key target for test automation is testing the functionality of the user interface (the GUI) where the speed and reliability of automated testing can yield significant benefits, particularly when testing multiple devices and configurations. Other tasks that can be automated include: Test environment provisioning Test data provisioning Security testing Reliability testing However, the return on investment when implementing automation for these tasks should be examined carefully before proceeding. Test environments that support multiple device and operating systems can require a significant investment. It is important to assess the benefits of mobile test tools and mobile test automation products against the total cost of ownership. Page 7

8 4. Conclusion SQS 2012 iqnite survey found that testing accounts for a significant portion of project budget 4, while work at the University of Cambridge found that on average, developers spend 50 % of their time finding and fixing bugs 5. Ensuring quality can be a costly undertaking; the key is to conduct quality assurance and testing effectively and efficiently. Why do we spend so much time and effort on testing and quality? Not just for the sake of creating a quality product per se, but because creating a quality product makes sound financial sense. For m-payments, where the risk of financial and reputation loss is much higher than with most other systems, creating high quality software saves money: it reduces risk by ensuring that the end product is mature, reliable, cheaper to maintain and fit for purpose. No organisation wants to be subjected to the public and customer relations disaster of a failed m-payments system and effective test management will help to minimise the risk of such an event ever occurring. SQS has been providing test management and test execution services to corporates across the globe for over 30 years. With a track record of innovation in quality assurance, SQS consultants are leading the drive towards more efficient and effective testing of new technologies such as m-payments through techniques such as better risk management, test automation, environment management and security testing. Page 8

9 5. References 1) World Quality Report , 2) Empirical analysis of CK metrics for object-oriented design complexity: implications for software defects, Subramanyam. R, Krishnan, M.S., ) The evolution of mobile apps: an exploratory study by Zhang, Sagar & Shihab, ) Up to 30 % of project budget in 69 % of new projects: SQS iqnite survey ) SQS Software Quality Systems AG, Cologne All rights, in particular the rights to distribution, duplication, translation, reprint and reproduction by photomechanical or similar means, by photocopy, microfilm or other electronic processes, as well as the storage in data processing systems, even in the form of extracts, are reserved to SQS Software Quality Systems AG. Irrespective of the care taken in preparing the text, graphics and programming sequences, no responsibility is taken for the correctness of the information in this publication. All liability of the contributors, the editors, the editorial office or the publisher for any possible inaccuracies and their consequences is expressly excluded. The common names, trade names, goods descriptions etc. mentioned in this publication may be registered brands or trademarks, even if this is not specifically stated, and as such may be subject to statutory provisions. SQS Software Quality Systems AG Phone: Fax: Page 9

Evaluation of the Iceland State Financial and Human Resource System REPORT OF THE INDIVIDUAL EVALUATOR Annex 2 SYSTEM AND SOFTWARE QUALITY This paper lists the properties used in the two main models in

Whitepaper How will Security Testing help to reduce risks and build customer confidence in mobile payments sqs.com An insight to successful strategies beating the challenges of complex systems Introduction

GO LIVE, ON TIME, ON BUDGET HOW TO OPTIMISE SAP Implementations AND UPGRADES THE PROBLEM IT leaders are familiar with demands-juggling ; a skill that reflects the success of IT. Business software systems

Automated testing for Mobility New age applications require New age Mobility solutions Executive Summary Today, mobile phone has transformed from its former role as a mere medium of communication to that

12 Top Tips to Create a Great Mobile App Let s be honest, there are a lot of ordinary apps out there and in an increasingly noisy marketplace, another ordinary app will just disappear from view. Nick Black,

SOLUTION BRIEF CA Wily Application Performance Management May 2010 how can I deliver better services to my customers and grow revenue? we can With the right solution, you can be certain that you are providing

RUN THE RIGHT RACE Keep pace with quickening release cycles. Discover automation with the human touch. CHOOSE A TEST TO RUN BELOW 26032015 FUNCTIONAL TESTING With Borland everyone from business analysts

White paper Demystifying mobile testing solutions Executive Summary It is no-brainer that the number of enterprise and consumer mobile applications has grown exponentially in recent years. Companies must

Latest Trends in Testing Ajay K Chhokra Introduction Software Testing is the last phase in software development lifecycle which has high impact on the quality of the final product delivered to the customer.

Whitepaper Project Management Office sqs.com PMO as a strategic success factor for project-based organisations Abstract Project-management-based organisations with either large or numerous projects can

How do you manage the growing complexity of software development? Is your software development organization as responsive to your business needs as it could be? Borland Core SDP enables your IT organization

Unit 7: Metric for Process and Product 7.1 Software Measurement Measurement is the process by which numbers or symbols are assigned to the attributes of entities in the real world in such a way as to define

Service Virtualization: Reduce the time and cost to develop and test modern, composite applications Business white paper Table of contents Why you need service virtualization 3 The challenges of composite

Systems Analysis and Design in a Changing World, Fourth Edition Learning Objectives Describe implementation and support activities Choose an appropriate approach to program development Describe various

Automated Testing Best Practices This document includes best practices to consider before implementing automated software testing. These best practices are strategic and are applicable regardless of the

March 2008 Grant Halverson CEO, GFG Group Regional Processing Models The search for successful regional and global IT processing models has been a major focus of the last fifteen years across banks, insurance

THE BUSINESS CASE FOR BUSINESS CONTINUITY MANAGEMENT SOFTWARE When it comes to building a business continuity management (BCM) program that s complete, current, and compliant, there is no substitute for

Why do current systems fail? Standish Group found that 51% of projects failed 31% were partially successful Main causes were poor user requirements: 13.1% Incomplete requirements 12.4% Lack of user involvement

The Importance of Continuous Integration for Quality Assurance Teams Without proper implementation, a continuous integration system will go from a competitive advantage for a software quality assurance

Quality Assurance Services Quality Assurance Sasken focuses on the ISV, Retail and Insurance verticals in the Quality Assurance space. Along with our ecosystem partners, we support our customers in cloud

White paper: Unlocking the potential of load testing to maximise ROI and reduce risk. Executive Summary Load testing can be used in a range of business scenarios to deliver numerous benefits. At its core,

How to address top problems in test data management Data reuse, sub-setting and masking Business white paper Table of contents Why you need test data management... 3 The challenges of preparing and managing

MANAGING DIGITAL CONTINUITY Project Name Digital Continuity Project DRAFT FOR CONSULTATION Date: November 2009 Page 1 of 56 Contents Introduction... 4 What is this Guidance about?... 4 Who is this guidance

Software Testing Rajat Kumar Bal Introduction In India itself, Software industry growth has been phenomenal. IT field has enormously grown in the past 50 years. IT industry in India is expected to touch

Whitepaper Building a Successful Testing Partnership for Outsourced Agile Delivery sqs.com Introduction The growing drive for an acceleration in the pace and quality of IT change is fuelling a global increase

Accelerating software testing effectiveness using Agile methodologies.. How can testing be completed faster, and more efficiently, within short iterations? The Problem It is a painful paradox that while

Mobile application testing for the enterprise Accenture brings together deep knowledge of the enterprise, expertise in mobile technologies and strong end-to-end testing practices to help all enterprises

What s new in the HP Functional Testing 11.5 suite Ronit Soen, product marketing John Jeremiah, product marketing Today s agenda A new world order for applications impact on QA HP s response announcement

101-301 Guide to Mobile Testing Perfecto Mobile & Toronto Association of System and Software Eran Kinsbruner & Joe Larizza 2014 What To Do? Great News Your first Mobile Project has arrived! You have been

Adopting Agile Testing A Borland Agile Testing White Paper August 2012 Executive Summary More and more companies are adopting Agile methods as a flexible way to introduce new software products. An important

Service Virtualization A faster, more efficient and less costly way to develop and test enterprise-class applications As cloud and mobile computing gain rapid acceptance, IT departments are expected to

Pathways to Digital Growth Course Outlines IT Service Management This course will help individuals understand the disciplines and processes that help service management staff to deliver and support quality

A Guide To Evaluating Desktop Virtualization Solutions T A K E A W A Y G U I D E Table of e s What To Look For In Evaluating Desktop Virtualization... 3 Achieving the Optimal Benefits of Desktop Virtualization...

BELATRIX SOFTWARE Mobile QA & Testing Best Practices Summary: Software testing of mobile applications uses a lot of the same QA & Testing practices typically used for other software applications. However,

www.anite.com Experts in wireless device and infrastructure test solutions Anite is now part of Keysight Technologies When customers expect more from mobile devices we help you deliver Anite is the mobile

c o n t e n t s t r a t e g y MCEETYA A u s t r a l i a N e w Z e a l a n d Online curriculum content is used in all schools in Australia and New Zealand to increase learning effectiveness and to support

Rapid Bottleneck Identification TM A Better Way to Load Test WHITEPAPER You re getting ready to launch or upgrade a critical Web application. Quality is crucial, but time is short. How can you make the

Test management best practices Introduction Purpose Few people can argue against the need for improved quality in software development. Users of technology that utilizes software have come to expect various

PROGRAMME OVERVIEW: G-CLOUD APPLICATIONS STORE FOR GOVERNMENT DATA CENTRE CONSOLIDATION 1. Introduction This document has been written for all those interested in the future approach for delivering ICT

The Cadence Partnership Service Definition About Cadence The Cadence Partnership is an independent management consultancy, specialising in working with a wide range of organisations, solving complex issues

Levels of Software Testing There are different levels during the process of Testing. In this chapter a brief description is provided about these levels. Levels of testing include the different methodologies

Numara Track-It! 10.5 Product Family Overview www.numarasoftware.com Introduction The Numara Track-It! family of IT Management solutions gives you the ability to immediately take control of your Help Desk

WHITE PAPER IT Knowledge Exchange Series MEASURING PRE-PRODUCTION APPLICATION, SYSTEM AND PERFORMANCE VOLUME STRESS TESTING WITH TEAMQUEST A white paper on how to enhance your testing discipline Contents

etri White Paper Fundamentals of Performance Testing The Increasing Need for Proper Performance Testing due to Increasing Software Complexity in the Enterprise There have been two significant changes in

Asset Management in the ITSM Industry Prepared by Daniel Wood Head of Research Service Desk Institute Sponsored by Declaration We believe the information in this document to be accurate, relevant and truthful

Automated Business Process Discovery & Validation The Worksoft Suite Worksoft Differentiators The industry s only platform for automated business process discovery & validation A track record of success,

Proactive Performance Management for Enterprise Databases Abstract DBAs today need to do more than react to performance issues; they must be proactive in their database management activities. Proactive

To ensure the functioning of the site, we use cookies. We share information about your activities on the site with our partners and Google partners: social networks and companies engaged in advertising and web analytics. For more information, see the Privacy Policy and Google Privacy &amp Terms.
Your consent to our cookies if you continue to use this website.