5 Introduction Over the past few years, the crowdsourcing paradigm has evolved from its humble and isolated custom initiatives to major success stories, such as Wikipedia and Amazon Mechanical Turk. In the field of information technology, crowdsourced software testing has witnessed increased interest and adoption, especially among startups and smaller companies. However, adoption has been quite slow among larger businesses, largely due the paucity of published business literature and use cases. To help facilitate crowdtesting by large businesses, we shall discuss some important issues such as make-or-buy decisions, building an internal testing community or partnering with external agencies, application suitability, governance, tester quality and will conclude by providing some guidelines for successful crowdtesting engagements. IT landscape today In recent years, Information Technology (IT) has assumed such strategic importance that many companies have already made substantial investments in ERP, CRM and other business applications, demonstrating the critical nature of enterprise IT. These companies also have consumer-facing components, such as websites and mobile applications that should not only have good design, but also deliver uninterrupted functionality. It is almost impossible to predict the users of such systems and their tastes a priori. Crowdsourced software testing helps achieve a high probability of consumer acceptance of software modules by ensuring that the software is defect-free to a very high degree. Fig. 1 Crowdsourcing trends since 2007 With the growing trend in cloud computing, BYOD, etc., IT services are delivered over multiple channels and consumed by diverse stakeholders, whose IT systems may come in different configurations. 1 Truly robust software testing is time consuming and ensuring that every possible permutation and combination of features, localizations and platforms works as intended is nearly impossible. For example, if a web application does not render in a particular browser or a particular software tool fails to deliver a critical functionality, it may lead to disruption in business. Through crowdtesting, companies could effectively eliminate the problems associated with in-house software testing team missing critical elements during the internal testing phases. 1 Real-World On-Demand Apps Testing: 5

6 Fig. 2 Regional interest in crowdsourcing According to Paul Herzlich, a software-testing analyst at Ovum, an independent IT industry research institution, If you are testing software that all kinds of strangers are going to use, then why not use a bunch of strangers to test it. Thus, crowdtesting offers a simple, elegant and cost-effective solution to what would otherwise be an intractable quandary. What is crowdsourced software testing? Crowdtesting is a software testing methodology that leverages a community of external expert software testers with diverse backgrounds and demographics from all across the globe. It differs from the traditional approach in that testing is carried out by a larger number of testers from different places, rather than by a limited number of in-house testing professionals. These communities range from a few hundred to several thousand testers globally. 6

7 Fig. 3 Crowdtesting VS traditional in-house methods Crowdtesting subjects the application to a set of realistic scenarios, loads and user paths, which cannot be replicated by an internal testing team. Knowing an application s performance under real-world conditions in advance makes refactoring easier and cheaper. Such testing also increases reliability and decreases the time required to develop and deploy. Usually, almost all noticeable bugs or issues are detected and fixed before a full-fledged deployment. Exploratory and Enterprise Crowdtesting Presently, much of the focus is on exploratory crowdtesting services, where testers analyze software for issues, bugs or defects based on generic guidelines and test cases. With free hand exploratory testing, the outcomes cannot be quantitatively predicted as it is difficult to qualify the range, area and number of possible defects in advance. Testers usually identify issues in the most obvious places, but highly skilled testers may uncover defects hidden under layers of the software. Exploratory testing is most suited to test less mission critical applications with a large number of users to ensure reproducibility of a set of issues under different configurations or to simulate a typical load on an application (Kaner 2008). Enterprise crowdtesting puts more rigor and structure into the software testing process, while simultaneously retaining the spirit and agility of exploratory testing. Thus, enterprise crowdtesting incorporates the best practices and concepts of both informal and formal software testing methodologies. The degree of professionalism involved in enterprise crowdsourced software testing is higher. 7

8 Fig. 4 Enterprise crowdtesting In this approach, testers are scrutinized and deployed based on comprehensive selection criteria. The software testing process involves clearly identified scope and is guided by extensive guidelines, tailored to fit the needs of the project. The team has clearly identified roles and responsibilities; a single community point of contact for the client organization; formalized reports and reporting structures; and extensive documentation and project tracking, etc. Furthermore, enterprise crowdtesting entails formal processes built over existing software testing methodologies, such as Agile and SCRUM. As a result of such alignment, the client organization experiences a smooth synchronization between their internal development and external software-testing cycles. This allows the client to reduce project management overheads and maximize the benefits from both teams. What applications are good candidates for crowdsourced testing? Crowdsourced testing methodology is most suitable for applications that are user-centric. It is frequently used in mobile applications and game development projects by releasing cheap or free versions of products in beta stage. Large corporations would also benefit from crowdsourced testing by simulating a large user base to understand usage patterns and improve on the feedback, while ensuring their applications run smoothly on a number of different devices, operating systems, browsers and language versions. 8

9 Fig. 5 Crowdtesting ROI apps with high exposure to market For example, Microsoft released the beta version of its Office 2010 productivity suite, which was downloaded and tested by 9 million people who provided 2 million valuable comments and insights, resulting in substantial product improvements. As can be seen from the following matrix, applications with a high Defect Exposure Factor i.e. criticality as a measure of immediate customer exposure post release, are good candidates for crowdtesting. Fig. 6 Application criticality and defect exposure factor 9

10 How does it work? Most crowdsourced testing companies provide the platform and the project management framework, including governance and legal structures, for the testing cycles. The crowdtesting company also allocates a qualified Project Manager, who oversees the testing process. On the consumer side of crowdsourced software testing services, the clients specify the type and battery of tests they wish to have performed; the skillset and demographic information of testers; and the types and configuration of devices the software must be tested on, etc. The client company could also consult the crowdtesting company to ensure testing adequacy. Fig. 7 Crowsourced testing process 10

11 On the supplier side, the testers complete a profile, indicating their background and qualifications, the devices and platforms they have access to and other relevant details. Testers may sometimes be required to participate in assessments or trial projects as a pre-condition for community membership. Before a project starts, testers are provided with detailed test plans, sample scenarios, tools, scripts and instructions. During execution, the testers document their observations and are rated based on the volume and quality of their reports, which ties directly with their compensation and incentive structures. The community combines aspects of collaboration and competition, as members work to find solutions to the stated problem. Forums facilitate knowledge management, networking and discussion of bugs or relevant issues, while rating systems allow for recognition of a job well done, which helps participants to gain credibility and improved career opportunities. Crowdtesting should be considered as additional, complementary testing prior to production release rather than a replacement of an organization s internal testing team. It is most effective in catching residual UI and configuration defects prior to production release. Onsite, outsourced & crowdtesting Onsite testing is the established practice of deploying an internal team for software testing purposes. This approach has limits to its scalability and is ineffective in reproducing realistic usage situations. In outsourced software testing, the process is rendered by an external service provider, on a per-project or agreement basis. Outsourced software testing has the obvious cost benefits of highly skilled resources at a fraction of costs through labor arbitrage. In crowdtesting, the testers voluntarily test software, motivated by their inherent drive to detect software defects, establish their standing in a community and gain peer recognition, etc., despite the risk of not getting paid if no issues are detected. This self-organizing feature, shared community goals and intrinsic motivation are strong forces for crowdtesting s success. What to choose? The onsite team, due to its proximity to customers, delivers value by focusing on technological and business challenges requiring immediate attention. In an outsourced testing project, the testing team(s) could be either captive or non-captive to the outsourcing company. In a captive model, the testing team is part of the company and performs software testing as an ongoing activity from which, the captive team gains experience and efficiency levels through an enhanced understanding of the most common internal applications. 11

12 Non-captive teams are external to the company and work with several clients on multiple projects. From these engagements, non-captive teams gather experience and expertise across several areas, domains, functions industries and technologies. The client company could leverage such diversity in projects that require thinking outside the box and special testing efforts (Brabham 2008). A crowdtesting team s laser focus, single-minded efforts, inherent motivation and scalability deliver tremendous value by detecting issues before go-live. The crowd s knowledge is diffused among testers, but the collective testing knowledge base is larger than that within any company. Given the complexities of modern systems, an optimum testing organization should comprise of onsite, outsourced and crowdsourced testing teams in the right proportion. Converging on the right proportion is an element subject to experience and experimentation. Build a community or establish partnerships? While some companies, such as Netflix and Amazon, have built their own crowd capabilities internally, this is not an activity that most companies could achieve on their own or makes tactical business sense. Crowdtesting is a complex activity requiring deep technical, functional and business knowledge. So, unless the work is so strategic or highly confidential to the business, companies should look for a crowdtesting company that delivers all the necessary ingredients to deploy the crowd almost immediately. These ingredients include experienced testers, devices, configurable architectures, participation mechanisms, ground rules, engagement methodologies, legal constructs, support mechanisms, etc. Security & Governance More often than not, confidential customer information is exposed to the crowd during testing cycles. Crowdtesting firms have realized the need for security and confidentiality; hence they have implemented several layers of screening, filters, governance mechanisms, obfuscation tools, data management frameworks, etc., to assure clients that their data and information is processed with the utmost confidence. For projects requiring high security, a pre-screened list of white hat test engineers that have a longstanding credibility and professional reputation are proposed and usually accepted. Test data management ensures security by obfuscating sensitive information or creating test only data. This helps minimize the impact of potential security breaches during large-scale testing. Testers are also required to sign a general Non-Disclosure Agreement (NDA) when they join the community, which forbids them from discussing their customers, products, etc., both offline and online on social networks, personal blogs and elsewhere outside the private testing platform. Customers also have the freedom to choose their testers from the pool and furnish the chosen ones with customized NDAs, which testers must sign before viewing the customer s project. 12

13 Furthermore, standardized communication channels, access restrictions, etc. help secure data and gain confidence in testing efforts, which results in a seamless transition across the various stages of the crowdtesting project lifecycle. Ensuring the quality of testers To ensure the quality of testers, formal and informal tests are periodically administered to individuals based on their experience, available material and concepts mastered. The testers and project managers are continually monitored to ensure both quality and integrity. Through constant filtering, applicants without formal training and significant professional experience are eliminated. Increasingly, crowdtesting companies are also sponsoring professional certifications for experienced testers, so their standing and responsibilities grow in proportion to their contributions. To avoid potential conflict of interest among testers, performance-based compensation mechanisms are used to ensure that compensation is commensurate with their level of engagement and actual contribution to software quality improvement. This means that testers who detect a larger number of less impactful defects are compensated less than those that detect a few, high-impact defects. These mechanisms also help prevent any flame wars, jockeying, etc. within the community and increase testers focus, ownership, loyalty and affiliation to the community, thereby creating a congenial and respectful working environment. Choosing a crowdtesting company Choosing the right testing company is of course by far the most important stage of the crowdtesting process. It is essential to perform adequate due diligence, extensive project evaluation and understand the capabilities and scale of the testing company, etc. before selecting a crowdtesting platform. In general, a proven and experienced crowdtesting platform will makes the operational activities much easier, as there is usually less maintenance and governance required during the operational stages because the testing company has already addressed many of the relevant issues from past experiences with other clients. Therefore, it is important to evaluate crowdtesting platforms on at least the following criteria to ensure that the platform contains at the minimum functionality and security required: 1. It is important to ascertain the crowdtesting company s level of experience, delivery capability, skill diversity and global outreach to accommodate the evolving requirements of enterprise customers. 2. A credible pedigree demonstrated through organic growth and experience with both corporate and mid-scale companies. It is relatively easy for a small crowdtesting platform to offer mobile application and website testing services, but very few companies have the maturity to test 13

14 enterprise applications and provide turnkey testing services. 3. Financial stability built over a period of time is particularly important if client companies aspire to undertake long-term projects without worrying about the service provider exiting the market. 4. Many crowdtesting platforms operate merely as matchmaking services between clients and testers rather than providing a managed service. These platforms do not assure the number of software testers, amount of testing work done by individual testers, the quality of testers or test results, nor any reliable outcomes. Such platforms are best avoided. 5. An assiduous review of the Terms and Conditions is mandatory to understand their scope of services and SLAs provided. 6. A sourcing hierarchy of constraints and criteria document should be used for evaluating platform operators for every major crowdtesting project. This document should address various elements, such as: Price Structures Quality, Safety, Security and Confidentiality Professional Services Portfolio Capabilities and Processes Delivery Methodology Service Infrastructure Compensation Structures Social Responsibility Convenience and Scalability Functionality and Quality of the Platform and Tools Risk Management Structures Summary Having discussed the various elements of crowdtesting, it could easily be inferred that large enterprises could expect substantial cost savings and scalability by partnering with established and experienced crowdtesting companies. Crowdtesting tools, processes, frameworks and methodologies have matured to such a high degree that crowdtesting could easily be incorporated into the normal development lifecycles at organizations. Crowdtesting decreases the likelihood of human error by diversifying software testing activities to a much larger team. By subjecting the application under realistic conditions, actual performance data and execution paths could be traced, which could be plowed back into re-engineering or improvements, leading to further quality improvements. Besides these direct benefits, large companies could also experience faster time to market, free up resources to focus on core operations, gain access to a large pool of talents and increase flexibility, etc. Getting started is easy and those organizations that dare to take the plunge are assured of the benefits in both the short and the long run. 14

15 About the author Dieter Speidel (53) is founder and CEO of PASS Group, a leading Swiss provider of software and system test services. PASS Group is owner and home of passbrains.com, a global platform for on-demand crowdtesting services, with offices in Zurich, Berlin, Boston, Belgrade, Bangalore and Mumbai. Being active as an entrepreneur in the IT outsourcing, software development and testing business since more than 30 years, Dieter Speidel owns strong expertise with global delivery models, including near-/offshoring and crowdsourcing. Since 2011, he specialised further in the application of crowdsourcing models for IT related services and developed the passbrains platform and community for crowdtesting and knowledge services. PASS Group, owner of passbrains.com, is a leading global provider of On-Demand Managed Application Testing Services. Their delivery centers are located across the U.S., Europe and India. PASS offers a unique service model which integrates the strengths of crowdsourcing and outsourcing solutions to optimize the software testing life cycle to a wide range of different industries. PASS operates its own platform and community for crowdsourced application testing (www.passbrains.com). Through passbrains, customers can get their web-, mobile- and enterprise applications tested on-demand by selected testing professionals and users with specific demographics, test configurations and mobile devices. The passbrains community counts thousands of registered testers with a wide range of skills and domain expertise from more than 100 countries. Visit us online at and www. passbrains.com. Copyright 2014 PASS Group AG PASS Switzerland International Headquarter Dufourstrasse 91 CH-8008 Zurich Phone: PASS India Vaishnavi Badri, #48 5th Main, Jayamahal Extension Bangalore Phone: PASS USA West Olympic Blvd, Suite 200, Los Angeles, CA Phone:

25 March 2014 The Magazine for Professional Testers CROWD TESTING By Dieter Speidel, Mayank Mittal & Mithun Sridharan Enterprise Crowd Testing The information technology world is shifting from one driven

Top Ten Reasons to Transition Your IT Sandbox Environments to the Cloud WHITE PAPER BROUGHT TO YOU BY SKYTAP 2 Top Ten Reasons to Transition Your IT Sandbox Environments to the Cloud Contents Executive

Openbravo for Partners A pure channel organization devoted to Partner success # Devoted to Partner Success Openbravo for Partners are especially designed to accelerate partners practice towards business

A P P L I C A T I O N S A WHITE PAPER SERIES COMPANIES ARE SEEKING NEW WAYS TO STREAMLINE PROCESSES, REALIZE FURTHER COST REDUCTIONS AND INCREASE TIME-TO-MARKET. MANY ORGANIZATIONS ARE CONSIDERING BUSINESS

Digital Innovation Series Executive Checklist to Transitioning Processes Building a Flexible Model The Digital Innovation Series traces the trends that are driving the digital-innovation imperative that

Product or Service what are headline here 5 questions every customer asks about CA Services? agility made possible question 1 why should we use CA Services? answer Meet the CA Services experts on LinkedIn

Transform the economics of IT, unlock the business value "Avanade has become a trusted partner. They understand our business and they continue to offer advice and solutions that keep our systems in tune

US ONSHORING OFFERS SUPERIOR EFFECTIVENESS OVER OFFSHORE FOR CRM IMPLEMENTATIONS Whitepaper Eagle Creek Software Services March 2015 Introduction CRM services are shifting from a focus on point solution

Data Sheet Cisco Optimization s Optimize Your Solution using Cisco Expertise and Leading Practices Optimizing Your Business Architecture Today, enabling business innovation and agility is about being able

Sonata Managed Application Lifecycle Services Leveraging IT to Deliver Growth-Centric Business Transformation Make IT an Enabler of Your Business with the Right Partner In today s complex and ever-changing

Differentiate your business with a cloud contact center A guide to selecting a partner that will enhance the customer experience An Ovum White Paper Sponsored by Cisco Systems, Inc. Publication Date: September

Financial Services the way we see it Cloud Computing in Banking What banks need to know when considering a move to the cloud Contents 1 Overview 3 2 Why Cloud Computing for Banks? 4 2.1 Cost Savings and

A DSG WHITE PAPER New to the Cloud? DSG s Guide to Cloud Accelerated Business Copyright 2014. All Rights Reserved. Demand Solutions Group. The inherent simplicity and costeffectiveness of Cloud computing

WHITEPAPER An ECM Journey Abstract Over the last few years, Enterprise Content Management (ECM) has evolved multifold. This paper describes the past, current and future state of ECM, and talks about the

Openbravo Subscription and Recurring Billing Managing a Subscription-based Business and How a Technology Giant Did It 1 Presenter Jon Setuain Senior Consultant at Openbravo 2 Trusted Vendor World leader

Five Things to Consider in an Enterprise CRM Evaluation An Oracle White Paper July 2010 Five Things to Consider in an Enterprise CRM Evaluation EXECUTIVE OVERVIEW When it comes to Customer Relationship

Cloud CRM Scalable solutions for enterprise deployment Simplicity in a complex world Finding, attracting, winning and retaining customers is the lifeblood of every business. But building a scalable, integrated

White paper Demystifying mobile testing solutions Executive Summary It is no-brainer that the number of enterprise and consumer mobile applications has grown exponentially in recent years. Companies must

Swiftly Deploy Private and Hybrid Clouds with a Single Pane of Glass View into Cloud Infrastructure Enable Fast, Easy, and Robust Cloud Computing with RightScale and Eucalyptus Overview As organizations

The Cloud... Your Way IT as a Service (ITaaS) shifts the focus to business productivity 707 E. MAIN STREET SUITE 1425 RICHMOND, VA 23219 888.342.1204 The Proxios Cloud ITaaS IT as a Service Proxios brings

The Massachusetts Open Cloud (MOC) October 11, 2012 Abstract The Massachusetts open cloud is a new non-profit open public cloud that will be hosted (primarily) at the MGHPCC data center. Its mission is

Top ten reasons to transition your IT lab environments to the cloud WHITE PAPER BROUGHT TO YOU BY SKYTAP 2 Top ten reasons to transition your IT lab environments to the cloud Contents The Promise of Cloud

Meeting the needs of Healthcare Healthcare: quality care today, and a healthcare system for tomorrow Like all advanced healthcare systems the NHS is faced with growing pressure from rising expectations

END TO END DATA CENTRE SOLUTIONS COMPANY PROFILE About M 2 TD M2 TD is a wholly black Owned IT Consulting Business. M 2 TD is a provider of data center consulting and managed services. In a rapidly changing

A P P L I C A T I O N S A WHITE PAPER SERIES SYNTEL, A U.S.-BASED IT SERVICE PROVIDER WITH AN EXTENSIVE GLOBAL DELIVERY SERVICE, SUGGESTS SPECIFIC BEST PRACTICES FOR REDUCING COSTS AND IMPROVING BUSINESS

Standardization in the Outsourcing Industry November 2010 Outsourcing provides rapid business transformation and cost reductions through labor arbitrage and consolidation of business processes spread across

Realize More Success with Software-plus-Services Cloud-based software from Microsoft Dynamics ERP Cloud computing is Internet-based development and use of computer technology. Large central data centers

BUYER S GUIDE IT Managed Services Buyer s Guide for SAP customers May 2012 flexible service delivery Moving to the beat of IT innovation with SAP Managed Services to control costs and harmonise IT landscapes.

View Point Lifting the Fog on Cloud There s a massive Cloud build-up on the horizon and the forecast promises a rain of benefits for the enterprise. Cloud is no more a buzzword. The enabling power of the

Accenture & NetSuite Gray background is only to allow visibility of all elements on page. Delete as needed. Delivering High Performance Turn off NOTES in Layers menu to Through the Cloud eliminate these

HP Software Professional Services Increase the value of IT with HP s end-to-end consulting Brochure Our IT Service Management project with HP has an 18-month payback and now has given us a better understanding

Picasso Recommendation Mission The School of Dentistry for the University at Buffalo (UB) uses a custom application called Picasso to manage their clinic operations. Developed by two engineers in 1997,

Endpoint Management and Mobility Solutions from Symantec Adapting traditional IT operations for new end-user environments During the past few years the traditional concept of end-user computing has drastically

WHITE PAPER Aspire's Approach to Test Automation by Ujjawal Bagaria, Aspire Systems Automation has been seen as the long term solution for cost reduction of manual testing across the globe. A successfully

ITIL 2015 and Beyond: Six Trends Driving Transformational Change in IT Best-Practices The Landscape of IT Services Provision is Changing! Customers and users are demanding a higher level of customer service

The Rise of Social Media in Financial Services Balancing Risk and Reward SOCIAL MEDIA About Infosys Many of the world s most successful organizations rely on Infosys to deliver measurable business value.

3 NREN and its Users The NREN s core activities are in providing network and associated services to its user community that usually comprises: Higher education institutions and possibly other levels of

How do you manage the growing complexity of software development? Is your software development organization as responsive to your business needs as it could be? Borland Core SDP enables your IT organization

Best Practices for Implementing an IP/SIP Contact Center Transform Customer Service Beyond the Contact Center and Improve Satisfaction Levels Table of contents Introduction:...1 SIP is the foundation of

Know Us The SRM group is four decades old multi-million dollar business house currently operational in 15 cities worldwide. SRM group has made its presence felt in education, training, Electronics, Technology,