Transcription

2 Defect Prevention versus Defect Detection Defect Prevention Techniques that are designed to find defects before the artifact has been developed Defect Detection Techniques that are designed to find defects after the artifact under test has been developed IISP,

3 Why is Defect Prevention so important? The cost factor for discovering and fixing defects in later phases versus early phases can be up to 100:1 IISP,

4 Stage Containment in a typical IT organization, most defects are injected into a system in its early stages IISP,

5 Stage Containment in a typical IT organization, most defects are injected into a system in its early stages and fixed in its later stages IISP,

7 Quality Control Activities within the development process to detect the introduction of flaws Test planning and execution Quality control measures a product against the existence of an attribute Determines whether the product conforms to a standard or procedure (also known as compliance checking). Proactive approach focused on defect detection Examples: Writing and executing test cases and scripts Participating in verification and validation activities Reporting defects to identify opportunities for process improvement IIST,

9 Risk Based Testing Defect Prevention Techniques Business Value Risk-based testing is a structured method for reducing testing effort based on component risk. It is a proven practice in testing which relies on strong business and IT participation to deliver real business value. As applied to the development lifecycle, risk management provides a means to focus testing efforts where they will make the greatest difference. Accenture's unique utilization of Risk-Based Testing together with our proprietary assets in this space will make it easier to engage the business units early in the lifecycle, especially for applications where requirements have not been documented thoroughly. Objectives of Risk-Based Testing Minimize threats to the solution delivery objectives. Provide a systematic approach for: Evaluating and improving the quality of requirements; Improve both test effectiveness and efficiency; Monitoring and reporting progress in reducing risk; Analyze R1 I S K Adjusting risk reduction actions (e.g. test scope) based upon the effectiveness of prior actions or additional knowledge. Gain consensus on high risk requirements R A T I O 0 Risk Ratio Analysis Design Build Test Activities Risk Ratio Test Impl. IISP,

11 Risk Assessment Analysis Risk Factor Distribution It is important to understand the primary factors driving the assessed risk level. Impact Probability Quantifies the true impact of each risk, also referred to as the level of damage potentially caused by the change. This is an estimate of the overall scale of impact if a defect related to the change were found in production. 5 = Critical 1 = Marginal This is an assessment of the likelihood of an error being delivered into production at a point in time. 5 = Very Likely 1 = Unlikely Indicates the relative control which can be exerted on the probability of the risk occurring in production. Level of Control 5 = Total Control 1 = No Control Indicates the Confidence level of the Risk evaluated. In a typical situation, as we understand and get more clarity on requirements confidence level increases. Confidence Level 5 = Very High 1 = Very Low IIST,

17 Defect Prevention Value of Reviews Provide early quality assessments (verification) of the work products Detecting dependencies and inconsistencies in software models, such as links Improved maintainability of code and design Colleagues will see what you can t Catch bugs earlier It s cheaper to prevent a defect than to repair one later in the life cycle Identification of defects not easily found by dynamic testing IISP,

18 Defect Prevention Value of Reviews (cont.) Early warning about suspicious aspects of the code or design (high complexity measures) Opportunities for process improvements Prevention of defects, if lessons are learned in development Reduce overall project time scales Potential opportunities for communication and learning IISP,

20 Defect Detection Techniques - Functional Functional: Functional testing is based on requirements and functionality. Functional testing covers how well the system executes the functions and features. Functional testing is not based on any knowledge of internal software design or code. Functional testing covers the front-end functions, as well as the back-end operations such as security and how upgrades affect the system. Generally functional testing is often done toward the end of the development cycle. It is recommended to be started earlier such as individual components and processes. Black Box, Acceptance, Closed box and Integration testing are functional testing techniques. IISP,

21 Defect Detection Techniques Structural Structural testing is typically based on the architecture of the system Aspects such as a calling hierarchy, data flow diagram, design specification, etc. Structural (white-box) testing may be performed at all test levels System, system integration, or acceptance testing levels (e.g., to business models or menu structures) Structural techniques are best used after specificationbased techniques IISP,

22 Defect Detection Techniques Structural Structural testing is primarily about coverage Coverage is the degree to which a structure has been exercised by the set of tests Typically it is expressed as a percentage of the items being covered More on coverage later Tools can be used at all test levels to measure the code coverage of elements, such as statements or decisions Tools are especially useful at component and component integration testing IISP,

23 Black Box Versus White Box Black box testing focuses on functional testing, not based on any knowledge of internal software design or code, where as White box focuses on knowledge of the internal logic of an application's code. Black box testing is based on requirements and functionality, where as White box testing is based on coverage of code statements, branches, paths and conditions. Equivalence Partitioning, Boundary Analysis and Error Guessing are successful techniques for managing the amount of input data for Black box testing. Statement Coverage, Decision Coverage, Condition Coverage, Decision/Condition Coverage and Multiple Condition Coverage are five important techniques for White box testing. IISP,

24 Static Versus Dynamic Static testing is performed using the software documentation. The code is not executing during static testing where as Dynamic testing requires the code to be in an executable state to perform the tests. Most verification techniques are static tests where as validation tests are dynamic tests. Feasibility Reviews and Requirements Reviews are examples of Static testing. Unit, Integrated, System and User Acceptance testing are examples of Dynamic testing. IISP,

26 Key Metrics that drive Process Improvement Defect Discovery Phase Phase of the SDLC in which the defect was found Defect Discovery Method Testing method that found the defect Defect Origination Phase Phase of the SDLC in which the defect was originated or first introduced into the product Root Cause Primary reason why the defect occurred Defect Resolution Type How the defect was resolved Defect Leakage Reason Root cause to determine method defect escaped the testing phase IISP,

31 Effective Execution (Quality Assurance) Goal: Effectively execute all of our projects (delivering high-quality software, on budget and on schedule) Questions: Are defects detected early in the SDLC? Where are we discovering defects? How are we discovering our defects? Are defects escaping to later test phases? Metrics: Defect detection by discovery phase Defect detection by discovery method IIST,

33 QA Metrics Defects by Discovery Phase Key Observations/Recommendations 82% of defects are found in Testing Too much pressure on the test team to find defects More defects must be found upstream Recommend more unit testing and/or tracking unit test defects. IIST,

37 QA Metrics Defects by Discovery Method Key Observations/Recommendations Very low % of automated versus manual tests How much are you investing in automated tests? Are you getting your expected ROI on automation? How are developers finding such a high % of defects in formal testing? Understand how/what type of testing the developers are doing to find these defects Incorporate these findings into the formal testing process 15% (140 Defects) were found but no method was recorded This is valuable data that we are missing Remove the option to select None from your test management tool IIST,

38 Effective Execution (Development) Goal: Questions: Metrics: Effectively execute all of our projects (delivering high-quality software, on budget and on schedule) How effective is our inspection process? Where are injecting defects into our products? How are we resolving defects in our products? How much time are we spending resolving defects? What are the primary root causes for our defects? Defect detection from Fagan tools Defect detection by injection phase Defect resolution type Total resolution time/total Verify time Defect root cause IIST,

40 Development Metrics Defects by Origination Phase Key Observations/Recommendations Note the high % of defects found in Test. Is this accurate? 42% of defects originated in Build This is a large number of coding defects Are developers unit testing? Recommend that coding standards, tracking unit test defects and peer reviews 9% of defects originated in Analyze? Is this accurate? 20% of defects are not accounted for IIST,

42 Development Metrics Defects by Resolution Key Observations/Recommendations 13% defects were closed As Designed Is there a disconnect between the specifications and the application? Why are so many defects invalid? 8% defects were closed as Duplicate Are testers not communicated with each other or searching for outstanding defects prior to entering their own? 7% defects were closed as Not Reproducible Why are so many defects non reproducible? Are there environment or data variables? Is the tester doing something abnormal during the execution of the test? Is the developer trying to reproduce it in a different environment? IIST,

43 Understanding the Cost of Poor Quality (CoPQ) Cost of Poor Quality: The cost of rework associated with fixing any defects in the product. This includes support and maintenance costs associated with a product. Cost of Quality: The regularly schedule cost for testing a product. This includes unit testing as well as the cost for each test phase. REWORK!! IIST,

44 Hours Rework by Resolution Rework by Resolution Change Implemented As Designed Not Reproducible Obsolete Duplicate Resolved in Another Project (None) Change Not Justifiable Posted to Wrong Project IIST,

45 Rework by Resolution Key Observations/Recommendations Over 10K hours were spent fixing defects Is this acceptable? Do you want a further breakdown of this? (see next slide) 2K hours were spent investigating defects that were invalid due to the application working as designed Is this acceptable? IIST,

49 Root Cause Frequency Key Observations/Recommendations Almost 200 defects (CRs) are caused due to Logic errors This project is having serious issues with their design logic Are requirements clear? Are these inexperienced developers? Do they truly understand how the system is built? Over 170 defects are due to UNKNOWN reasons Unacceptable We have no idea how to fix these issues because the root cause is unknown IIST,

Note: This material is for Evaluators reference only. Caters to answers of CSTE Mock Test - Part I paper. 1. A branch is (Ans: d) a. An unconditional transfer of control from any statement to any other

ISTQB Certified Tester Foundation Level Version 2015 American Copyright Notice This document may be copied in its entirety, or extracts made, if the source is acknowledged. #1 When test cases are designed

Effective Test Management Practices Dr. Magdy Hanna Chairman International Institute for Software Testing mhanna@testinginstitute.com http:// Principles-1 What is most frustrating in your role as a test

Software Testing Rajat Kumar Bal Introduction In India itself, Software industry growth has been phenomenal. IT field has enormously grown in the past 50 years. IT industry in India is expected to touch

Increasing Business Efficiency and Agility for ATGbased ecommerce Systems This case study follows a Tier 1 retailer migrating to an ATG-based ecommerce platform and upgrading its software development process

Software Testing Interview Questions 1. What s the Software Testing? A set of activities conducted with the intent of finding errors in software. 2.What is Acceptance Testing? Testing conducted to enable

Latest Trends in Testing Ajay K Chhokra Introduction Software Testing is the last phase in software development lifecycle which has high impact on the quality of the final product delivered to the customer.

Software testing cmsc435-1 Objectives To discuss the distinctions between validation testing and defect testing To describe the principles of system and component testing To describe strategies for generating

EXHIBIT L Application Development Processes Optum Development Methodology Development Overview Figure 1: Development process flow The Development phase consists of activities that include the building,

QA Roles and Responsibilities There are various roles on projects, some people may play more than one role. You should always check with your organizations testing methodology on what your role(s) are.

Effective Management of Static Analysis Vulnerabilities and Defects Introduction According to a recent industry study, companies are increasingly expanding their development testing efforts to lower their

WHITE PAPER Integrated Quality Approach Building Quality into the by Venkatesh Veerachamy How do you detect bugs in the early stage, remove risks in product release and reduce reworking costs? Integrated

RFP 1. Applications IT Architect Analyzes and designs the architecture for software applications and enhancements, including the appropriate application of frameworks and design patterns and the interrelationships

Custom Software Development Approach Our approach to custom software development combines benefits from several standard development process models. We tend to have a well-defined, predictable and highly

Certified Software Quality Engineer (CSQE) Body of Knowledge The topics in this Body of Knowledge include additional detail in the form of subtext explanations and the cognitive level at which the questions

A Testing Exercise: (From Glenford Myers: The Art of Software Testing) A program reads three integer values from a card. The three values are interpreted as representing the lengths of the sides of a triangle.

The document in this file is adapted from the IEEE standards for Software Project Management Plans, 1058-1998, which conforms to the requirements of ISO standard 12207 Software Life Cycle Processes. Tailor

CUT COSTS, NOT PROJECTS Understanding and Managing Software Development Costs A WEBINAR for State of Washington Agencies Critical Logic, Inc. July 9 2009 Starting at 3pm, Pacific Daylight Time Critical

etri White Paper Fundamentals of Performance Testing The Increasing Need for Proper Performance Testing due to Increasing Software Complexity in the Enterprise There have been two significant changes in

Development*Process*for*Secure* So2ware Development Processes (Lecture outline) Emphasis on building secure software as opposed to building security software Major methodologies Microsoft's Security Development

Audit Checklist The following provides a detailed checklist to assist the audit team in reviewing the health of a project. Relevance (at this time) How relevant is this attribute to this project or audit?

Note: This material is for Evaluators reference only. Caters to answers of CSTE Mock Test - Part III paper. 1. Independence is important in testing is mostly due to the fact that (Ans: C) a. Developers

QAI /QAAM 2011 Conference Proven Practices For Managing and Testing IT Projects Co-Presented by Mr. Bill Rinko-Gay and Dr. Constantin Stanca 9/28/2011 Format This presentation is a journey When Bill and

Release management Release management process is a software engineering process intended to oversee the development, testing, deployment and support of software releases. A release is usually a named collection

ITIL A guide to service asset and configuration management The goal of service asset and configuration management The goals of configuration management are to: Support many of the ITIL processes by providing

Tonight s Speaker Life of a Tester at Microsoft Urvashi Tyagi Software Test Manager, Microsoft You will learn about what a software tester does at Microsoft, how the role interfaces with program managers

Test Engineering Foundation Course Outline General Description This course provides test engineers and test managers with the essential ideas, processes, tools and skills they need in order to set themselves

Using TechExcel s DevSuite to Achieve FDA Software Validation Compliance For Medical Software Device Development The FDA requires medical software development teams to comply with its standards for software

White Paper Requirements-Based Testing: Encourage Collaboration Through Traceability Executive Summary It is a well-documented fact that incomplete, poorly written or poorly communicated requirements are

Defect Tracking Best Practices Abstract: Whether an organization is developing a new system or maintaining an existing system, implementing best practices in the defect tracking and management processes

REQUIREMENT DRIVEN TESTING Test Plan for Project name Requirement Driven Testing [Pick the date] [Type the abstract of the document here. The abstract is typically a short summary of the contents of the

Unit 7: Metric for Process and Product 7.1 Software Measurement Measurement is the process by which numbers or symbols are assigned to the attributes of entities in the real world in such a way as to define

Building Security into the Software Life Cycle A Business Case Marco M. Morana Senior Consultant Foundstone Professional Services, a Division of McAfee Outline» Glossary» What is at risk, what we do about

Shorten your 11i Upgrade and Patching Cycles with Automated Testing Rod Lehman Senior Director of Product Marketing Can You Make an Informed Go-Live Decision? Go / No-go? Go Will the application work as

PHASE 6: DEVELOPMENT PHASE The Phase features a key step in the project: system construction. The previous phases lay the foundation for system development; the following phases ensure that the product

A Guide to Project Management The purpose of this website is to provide a job aid for project management procedures using the IPECC project management model. The information will be provided in a summarized

THE ONLY BOOK CAN SIMPLY LEARN SOFTWARE TESTING! Page 1 Contents ABOUT THE AUTHOR... 3 1. Introduction To Software Testing... 4 2. What is Software Quality Assurance?... 7 3. What Is Software Testing?...

Test What You ve Built About Your Presenter IBM i Professional for 16 Years. Primary Focus is IBM i Engineering / Programming Well Versed in 2E. Well Versed in RPG (All Flavors) Well Versed in CM Products

1.0 Introduction In today s fast-paced lifestyle with programmers churning out code in order to make impending deadlines, it is imperative that management receives the appropriate information to make project

INL/EXT-09-17022 Rev. 2 Independent Verification and Validation of SAPHIRE 8 Software Project Plan March 2010 The INL is a U.S. Department of Energy National Laboratory operated by Battelle Energy Alliance

Software Application Control and SDLC Albert J. Marcella, Jr., Ph.D., CISA, CISM 1 The most effective way to achieve secure software is for its development life cycle processes to rigorously conform to

FSW QA Testing Levels Definitions 1. Overview This document is used to help determine the amount and quality of testing (or its scope) that is planned for or has been performed on a project. This analysis

Sample Exam ISTQB Agile Tester 2014 Foundation Level Extension Version 1.0 Copyright Notice This document may be copied in its entirety, or extracts made, if the source is acknowledged. Table of Contents

IAEA-TECDOC-1328 Solutions for cost effective assessment of software based instrumentation and control systems in nuclear power plants Report prepared within the framework of the Technical Working Group

The Role and Development of an Enterprise Architect: A Devil s Advocate Perspective May 2009 Robert S. Ellinger Ph.D. Enterprise Architect The Devil s Advocate Thesis 2 The Problems There is little respect

These slides are distributed under the Creative Commons License. In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform