Transcription

2 Three Myths of Automated Testing 1. Automated testing always results in improved software quality 2. Every application development project or testing team can use automated testing tools 3. Automated testing is an all or nothing proposition

3 Three Realities of Automated Testing 1. Automated testing requires a higher initial investment but can yield a higher ROI 2. Skills and training are required to be successful with any automated testing tool 3. Every company doing automated testing still does some amount of manual testing

4 Testing Fact Industry surveys indicate that 75% of all functional testing is still done manually

5 Question #1: Why do most companies still rely so heavily on Manual Testing?

6 Why Manual Testing? Time: Testing teams may not have time to investigate alternatives to manual testing, learn how to use tools, and/or build & maintain scripts Application Complexity: Certain applications may be too complex and not suitable for automated testing Skillset: Some testers (business analysts, etc.) may lack skills needed to be successful with test automation tools Cost: Organizations may not own automated testing tools and may lack the budget to invest in tools Job Security: Testers / QA organizations may be comfortable and experienced with manual testing and feel threatened by automation Awareness: Organizations may lack the awareness that viable automated testing alternatives exist

7 Question #2: When is Manual Testing a better alternative than Automated Testing?

8 When is Manual Testing a Good Idea? Subjective Validation: For application functions that must be validated subjectively by humans such as usability or look-and-feel, manual testing may be the only option New/Changing Functionality: For new application functions that are still being developed and evolving / changing frequently, creating automated scripts may be a waste of time Strategic Development: For strategic application functions that you want testers to pay specific attention to, hands-on manual testing may be a better alternative Complex Functionality: For application functions that are extremely complex, test automation may be a major challenge (time & cost investment outweighs the benefit)

9 Question #3: When is Automated Testing a good alternative to Manual Testing?

10 When is Automated Testing a Good Idea? Regression Testing: For re-testing preexisting application functions that are being carried forward to new versions (usually the majority, unless app is brand new) Smoke Testing: For getting a quick high-level assessment on the quality of a build and making go / no-go decision on deeper testing Static & Repetitive Tests: For automating testing tasks that are repetitive and relatively unchanging from one test cycle to the next Data Driven Testing: For testing application functions where the same functions needs to be validated with lots of different inputs & large data sets (i.e. login, search) Load & Performance Testing: No viable manual alternative exists

16 Alex s Bio Quality Assurance Manager for Synchronoss Technologies, Inc. a 15 year veteran of the software testing industry, extensive experience with sales force automation and transaction management systems. In both my current and most recent prior positions, I started the Quality Assurance Departments and built them from the ground up.

20 When do we evaluate automation vs. manual testing New Function test case manual test works / passes release create automated script for regression Evaluating functions for automation is the responsibility of the entire project team and needs to happen during the entire SDLC. Includes automated unit tests, nightly builds and scripts with or without Functional Testing for Web Applications (formerly etester) that can validate build files, DB, configurations and GUI

21 When do we evaluate automation vs. manual testing Reviewing Requirements Can this function be automated? What, if anything, does development need to do? When writing test cases Can this function be automated? If yes, ensure test case is written to ease scripting (step by step) While Testing Ensure the test case is clear and precise Are the results predictable? Does the test have to be run many time to get the desired result?

22 When do we evaluate automation vs. manual testing What to think about Positives Will productivity increase? Will test coverage increase? Will test accuracy increase? Is this a large data input? Is this GUI intensive? Negatives Requires human intervention Requires 3rd party system Test has unpredictable results How often will this function change?

23 Automation: What works and what doesn t Lessons learned Automate functions that are 80% stable and unchanging Work with development on unique names for controls and fields Don t overlook bulk data loads for the support of regression testing Remember to include script maintenance in estimates Make scripts as generic as possible Use control file for URL, User IDs, and Passwords Upper management will always feel that regression should be 100% automated. This perception must be managed while setting proper expectations of what can be accomplished.

24 Automation with Functional Testing for Web Applications (formerly e-tester) Key needs All controls and fields have unique names Test harness Stable environment controlled only by QA Established application Don t forget about data loads we increased productivity of manual regression testing by 28% by automation the loading of data used for regression testing. Script maintenance must be included in your estimates for testing going forward. Functions will change and scripts will need to be updated.

25 Automation with Functional Testing for Web Applications (formerly e-tester) Script management Each project has a dedicated etester desktop Those that work automation, work with the project team to ensure they are working on areas of the application that are stable, somewhat unchanging and will improve productivity. Come up with, and stick to a naming convention for scripts Smoke testing can be done quickly and more reliably Can have scripts run overnight, review results in the morning and advise development of issues faster

26 Automation ROI Items to think about when calculating your ROI Tool Investment Learning Curve For Tool For Application Employee Job Satisfaction You will Get Overnight testing Testing reports ed to you More test coverage in same or less time More repeatable tests Faster test coverage You will not get All this immediately Expectations and implementation must be well managed

27 Summary Do Use this as a guideline and change it to fit your process Work to control expectations of automation Use the QAZone (now in OTN) for Tips, Trick and Information Don t Try and automate without support from development Over estimate what can be automated Let someone else set the expectations of automation

What s new in the HP Functional Testing 11.5 suite Ronit Soen, product marketing John Jeremiah, product marketing Today s agenda A new world order for applications impact on QA HP s response announcement

The Importance of Continuous Integration for Quality Assurance Teams Without proper implementation, a continuous integration system will go from a competitive advantage for a software quality assurance

WHITE PAPER Aspire's Approach to Test Automation by Ujjawal Bagaria, Aspire Systems Automation has been seen as the long term solution for cost reduction of manual testing across the globe. A successfully

Real Application Testing Fred Louis Oracle Enterprise Architect The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated

Q SQA Labs Value Assured QUALITY ASSURANCE TESTING TOOLS QUALITY ASSURANCE TESTING TOOLS Quality Assurance refers to the steps taken to make sure that a company s products or services are of sufficiently

Copyright www.agileload.com 1 INTRODUCTION Performance testing is a complex activity where dozens of factors contribute to its success and effective usage of all those factors is necessary to get the accurate

White Paper The Ten Features Your Web Application Monitoring Software Must Have Executive Summary It s hard to find an important business application that doesn t have a web-based version available and

Load Testing an Internet Site Course Overview Course Title Load Testing and Performance Tuning Methods for Optimizing Web Site Performance Description Maximizing Web site performance is essential for those

Testing, What is it Good For? Absolutely Everything! An overview of software testing and why it s an essential step in building a good product Beth Schechner Elementool The content of this ebook is provided

STLC-Software Testing Life Cycle SDLC Software Testing Lifecycle Software Testing Life Cycle (STLC) defines the steps/ stages/ phases in testing of software. However, there is no fixed standard STLC in

Process A Whitepaper Copyright 2006. Technologies Pvt. Ltd. All Rights Reserved. is a registered trademark of, Inc. All other trademarks are owned by the respective owners. Proprietary Table of Contents

Datamaker for Skytap Provide full-sized environments filled with up-to-date test data in minutes Is your testing constrained by environments and data? As applications have become more complex, provisioning

2010-02-01 To Be (Automated) or Not To Be (Manual): A Dilemma for Small Development Shops? Automated software testing has long been an integral part of big software development organizations but is often

Silk Performer LOAD TESTING The key to keeping business applications running 2 Load testing Executive Summary Every business depends on applications to automate its core business processes. These applications

Collaborating for Quality in Agile Application Development From Beginning to End + 1 Agenda Application Development Challenges Meeting the Challenge in the Enterprise End-to-End HP/CollabNet Solution Agile

The ROI of Test Automation by Michael Kelly www.michaeldkelly.com Introduction With the exception of my first project team out of college, in every project team since, I ve had to explain either what automated

Top Ten Reasons to Transition Your IT Sandbox Environments to the Cloud WHITE PAPER BROUGHT TO YOU BY SKYTAP 2 Top Ten Reasons to Transition Your IT Sandbox Environments to the Cloud Contents Executive

White paper Demystifying mobile testing solutions Executive Summary It is no-brainer that the number of enterprise and consumer mobile applications has grown exponentially in recent years. Companies must

Bringing Value to the Organization with Performance Testing Michael Lawler NueVista Group 1 Today s Agenda Explore the benefits of a properly performed performance test Understand the basic elements of

Writing The Business Case for Automated Software Testing and Test Management Tools How to successfully research, plan and present a convincing business case that will justify the budget and resources you

IBM RATIONAL PERFORMANCE TESTER Today, a major portion of newly developed enterprise applications is based on Internet connectivity of a geographically distributed work force that all need on-line access

etri White Paper Fundamentals of Performance Testing The Increasing Need for Proper Performance Testing due to Increasing Software Complexity in the Enterprise There have been two significant changes in

Lifecycle Performance Management (dynatrace) Description Compuware dynatrace is the new generation of application performance management. Its patented PurePath Technology provides the world s most accurate

Service Virtualization: Reduce the time and cost to develop and test modern, composite applications Business white paper Table of contents Why you need service virtualization 3 The challenges of composite

QAI /QAAM 2011 Conference Proven Practices For Managing and Testing IT Projects Co-Presented by Mr. Bill Rinko-Gay and Dr. Constantin Stanca 9/28/2011 Format This presentation is a journey When Bill and

Expert Decision Support Automation for Technical Support & Field Service Organizations Excel at anything Expertise without limits Is Search really the future of Support? Expanding Search With Automated

ecommerce and Retail Rainforest QA enables ecommerce companies to test highly visual user interfaces and customer experience and flow. ecommerce lives and dies by a great user experience, which in turn

"#$%&'&()%*+,-)).'',(+-$(/#0%,%-)%.1$/2-$(/#.3%)$4&-+$(+% 55567%8(-,-8)6+/2 "7%+1$(9%:122-&; CIO s are under pressure to understand how to deliver IT projects that enable business growth and innovation,

Technical white paper A closer look at HP LoadRunner software Table of contents Sizing up the system 2 The limits of manual testing 2 A new take on testing: the HP LoadRunner solution 3 The HP LoadRunner

Rapid Bottleneck Identification A Better Way to do Load Testing An Oracle White Paper June 2009 Rapid Bottleneck Identification A Better Way to do Load Testing. RBI combines a comprehensive understanding

Numara Track-It! 10.5 Product Family Overview www.numarasoftware.com Introduction The Numara Track-It! family of IT Management solutions gives you the ability to immediately take control of your Help Desk

APPLICATION OF SERVER VIRTUALIZATION IN PLATFORM TESTING Application testing remains a complex endeavor as Development and QA managers need to focus on delivering projects on schedule, controlling costs,

IBM Software Group Operationalizing Application Security & Compliance 2007 IBM Corporation What is the cost of a defect? 80% of development costs are spent identifying and correcting defects! During the

Using WebLOAD to Monitor Your Production Environment Your pre launch performance test scripts can be reused for post launch monitoring to verify application performance. This reuse can save time, money

TestTrack Test Case Management Quick Start Guide This guide is provided to help you get started with TestTrack test case management and answer common questions about working with test cases and test runs.

ESTABLISHING A MEASUREMENT PROGRAM The most important rule is to Understand that software measurement is a means to an end, not an end in itself Three key reasons for Software Measurement Understanding