The Cost of Software Quality

Testing can be considered an investment. A software organization-whether an in-house IT shop, market-driven shrink-wrap software vendor, or Internet ASP-chooses to forego spending money on new projects or additional features in order to fund the test team. What's the return on that investment (ROI)? Cost of quality analysis provides one way to quantify ROI.

Testing can be considered an investment. A software organization-whether an in-house IT shop, market-driven shrink-wrap software vendor, or Internet ASP-chooses to forego spending money on new projects or additional features in order to fund the test team. What's the return on that investment (ROI)? Cost of quality analysis provides one way to quantify ROI.

IntroductionThe process of negotiating a software testing budget can be painful. Some project managers view testing as a necessary evil that occurs at the end of the project. In these people's minds, testing costs too much, takes too long, doesn't help them build the product, and can create hostility between the test team and the rest of the development organization. No wonder people who view testing this way spend as little as possible on it.

Other project managers, though, are inclined to spend more on testing. Why? Smart software managers understand that testing is an investment in quality. Out of the overall project budget, the project managers set aside some money for assessing the system and resolving the bugs that the testers find. Smart test managers have learned how to manage that investment wisely. In such circumstances, the test investment produces a positive return, fits within the overall project schedule, has quantifiable findings, and is seen as a definite contributor to the project. (For some ideas on helping to make your managers—and yourself—smarter see James Bullock’s article, "Calculating the Value of Testing," and Rob Sabourin’s article, "At Your Service," both available here at www.stickyminds.com.)

What Does Quality Cost? The title of Phil Crosby's book says it all: Quality Is Free . Why is quality free? Like Crosby and J.M. Juran, Jim Campenella, in Principles of Quality Costs, illustrates a technique of analyzing the costs of quality start by breaking down these costs as follows:

Cquality=Cconformance+Cnonconformance

Conformance costs include prevention costs and appraisal costs. Prevention costs include money spent on quality assurance—tasks like training, requirements and code reviews, and other activities that promote good software. Appraisal costs include money spent on planning test activities, developing test cases and data, and executing those test cases once.

Nonconformance costs come in two flavors, internal failures and external failures. The costs of internal failure include all expenses that arise when test cases fail the first time they’re run, as they often do. A programmer incurs a cost of internal failure while debugging problems found during her own unit and component testing.

Once we get into formal testing in an independent test team, the costs of internal failure increase. Think through the process: The tester researches and reports the failure, the programmer finds and fixes the fault, the release engineer produces a new release, the system administration team installs that release in the test environment, and the tester retests the new release to confirm the fix and to check for regression.

The costs of external failure are those incurred when, rather than a tester finding a bug, the customer does. These costs will be even higher than those associated with either kind of internal failure, programmer-found or tester-found. In these cases, not only does the same process described for tester-found bugs occur, but you also incur the technical support overhead and the more expensive process of releasing a fix to the field rather than to the test lab. In addition, consider the intangible costs: Angry customers, damage to the company image, lost business, and maybe even lawsuits.

Two observations lay the foundation for the enlightened view of testing as an investment. First, like any cost equation in business, we will want to minimize the

Pages

About the author

Rex Black

Rex Black is President and Principal Consultant of RBCS, Inc., a consultancy that provides testing experts worldwide, serving clients such as Bank One, Cisco, Hitachi, IMG, and Schlumberger in consulting, training, and hands-on implementation. He has written Managing the Testing Process, Critical Testing Processes, and numerous articles, along with presenting papers and keynote speeches at international conferences.