A widely cited study for the National Institute of Standards & Technology (NIST) reports that inadequate testing methods and tools annually cost the U.S. economy between $22.2 and $59.5 billion, with roughly half of these costs borne by software developers in the form of extra testing and half by software users in the form of failure avoidance and mitigation efforts. The same study notes that between 25 and 90 percent of software development budgets are often spent on testing. This posting, the first in a two-part series, highlights results of an analysis that documents problems that commonly occur during testing. Specifically, this series of posts identifies and describes 77 testing problems organized into 14 categories, lists potential symptoms by which each can be recognized, potential negative consequences, potential causes, and makes recommendations for preventing them or mitigating their effects.

Common Problems with Testing

Despite the huge investment in testing mentioned above, recent data from Capers Jones shows that the different types of testing are relatively ineffective. In particular, testing typically only identifies from one-fourth to one-half of defects, while other verification methods, such as inspections, are typically more effective s. Inadequate testing is one of the main reasons why software is typically delivered with approximately 2 to 7 defects per thousand lines of code (KLOC). While this may seem like a negligible amount, the result is that major software-reliant systems are being delivered and placed into operation with hundreds or even thousands of residual defects. If software vulnerabilities (such as the CWE/SAN Top 25 Most Dangerous Software Errors) are counted as security defects, the rates are even more troubling.

Clearly, there are major problems with the efficiency and effectiveness of testing as it is currently performed in practice. In the course of three decades of developing systems and software—as well my involvement in numerous independent technical assessments of development projects—I have identified and analyzed testing-related problems that other engineers, managers, and I have observed to commonly occur during testing. I also solicited feedback from various LinkedIn groups (such as Bug Free: Discussions in Software Testing, Software Testing and Quality Assurance) and the International Council on Systems Engineering (INCOSE). As of March 2013, I have received and incorporated feedback from 29 reviewers in 10 countries. While the resulting framework of problems can apply to both software and systems testing, it emphasizes software because that is where the most of the testing problems occur.

The large number of testing problems necessitated that they be categorized. At the top level, these problems were organized into two groups

general testing problems that are not specific to any type of testing, but apply to all different types of testing

The remainder of this post will focus on general testing problems, which can be divided into eight categories:

Test planning and scheduling problems often occur when there is no separate test plan, but rather highly incomplete and superficial summaries in other planning documents. Test plans are often ignored once they are written, and test case descriptions are often mistaken for overall test plans. The schedule of testing is often inadequate for the amount of testing that should be performed, especially when testing is primarily manual. Significant testing is often postponed until too late in the development process, especially on projects using traditional sequential development cycles.

Stakeholder involvement and commitment problems include having the wrong testing mindset (that the purpose of testing is to show that the software works instead of finding defects), having unrealistic testing expectations (that testing will find all of the significant defects), and having stakeholders who are inadequate committed to and supporting of the testing effort.

Management-related testing problems involve the impact of inadequate management. For example, management can fail to supply adequate test resources or place inappropriate external pressures one testing. There may be inadequate test-related risk management or test metrics. Testing lessons learned are far too often ignored, so the same problems are repeated project after project.

Test process problems often occur when testing and engineering processes are poorly integrated. Organizations sometimes take a “one-size-fits-all” approach taken to testing, regardless of the specific needs of the project. Testing may not be adequately prioritized so that functional testing, black-box system testing, or white-box unit and integration testing may be overemphasized. Testing of components, subsystems, or the system may begin before they are sufficiently mature for testing. Other problems include inadequate test evaluations and inadequate test maintenance.

Test tools and environments problems include an over-reliance on manual testing or COTS testing tools. Often, there are an insufficient number of test environments. Some of the test environments may also have poor quality (excessive defects) or insufficient fidelity to the actual system being tested. Moreover, the system and software under test may behave differently during testing than during operation. Other common problems are that tests were not delivered or the test software, test data, and test environments were not under sufficient configuration control.

Requirements-related testing problems are related to the requirements that should be driving testing. Often, the requirements are ambiguous, missing, incomplete, incorrect, or unstable. Lower-level requirements may be improperly derived from their higher-level sources. Likewise, verification methods may be unspecified and the tracing between requirements and tests may be lacking.

Exacerbating these problems is the fact that too often research focuses on the defects identified through testing, but does not address problems that exist in an organization’s planned testing process or its implementation of that process. Not surprisingly, a 2010 survey found that 58 percent of respondents blamed the testing process and infrastructure for their most recently identified major software defects.

Addressing General Testing Problems

There are clearly many problems with the way software and software-reliant systems are tested, as discussed above. Moreover, these general testing problems are not getting significantly better, despite greater attention to test-driven development and continuous integration in the commercial software industry, which only address a few of the identified testing problems. While few projects will experience all of these problems, many projects exhibit several of them. Similarly, while these testing problems do not guarantee the software will contain an excessive number of residual defects, these problems definitely pose serious risks that must be managed.

For each testing problem described above, therefore, I have documented several types of information useful for understanding the problem and implementing a solution. As an example of the results of our analysis, the testing problem “Wrong Testing Mindset” has been documented with the following information

Description: Some of the testers and other testing stakeholders have the wrong testing mindset.

Potential symptoms: Some testers and other testing stakeholders assume the system/software works. Testers assume or are told that their job is to verify or “prove” that the system/software works. Testing is being used to demonstrate that the system/software works properly rather than to determine where and how it fails. Only normal (“sunny day” or “happy path”) behavior is being tested. There is little or no testing of exceptional or fault/failure tolerant (“rainy day”) behavior. There is no testing of input data (e.g., range testing of the handling of invalid input values). Test input includes only middle-of-the-road values rather than boundary values and corner cases.

Potential consequences: A high probability exists that the delivered system or software will contain significant residual defects related to abnormal behavior (e.g., exceptional use case paths), and these defects will unnecessarily reduce its reliability and robustness (e.g., error, fault, and failure tolerance). Customer representatives, managers, and developers will obtain a false sense of security that the system functions properly.

Potential causes: Testers were taught or explicitly told that their job is to verify or “prove” the system/software works. Developers typically conduct their own unit-level (i.e., lowest level) testing. With small, cross functional (e.g., agile) teams, it is becoming more common for developers to also conduct integration and subsystem testing. This scenario presents a “conflict of interest” for developers who are asked to build software that works and then attempt to show their software does not work. This problem is especially prevalent with small, cross-functional development organizations/teams that “cannot afford” independent, specially trained testers. There was insufficient schedule allocated for testing so there is only sufficient time to test normal behavior (e.g., use case paths). The organizational culture is “success oriented,” therefore looking “too hard” for problems is implicitly discouraged. Management gave the testers the strong impression that they do not want to hear any bad news (i.e., that there are any significant defects being found in the system).

Recommendations: Explicitly state in the project test plan that the primary goal of testing is to find defects by causing the system to fail (i.e., to break the system) rather than to demonstrate that there are no defects (i.e., to show that it works). Provide test training that emphasizes the proper testing mindset. In addition to test cases that verify all nominal behavior, emphasize looking for defects where they are most likely to hide (e.g., boundary values and corner cases).

This analysis of commonly occurring testing problems—and recommended solutions—can be used as training materials to better learn how to avoid, identify, and understand testing problems and mitigate them. Like anti-patterns, these problem categories can be used to improve communication between testers and testing stakeholders. This list can also be used to categorize problem types for metrics collection. Finally, they can be used as a checklist when

producing test plans and related documentations

evaluating contractor proposals

evaluating test plans and related documentation (quality control)

evaluating as-performed test process (quality assurance)

identifying test-related risks and their mitigation approaches

The next post in this series will explore test type-specific problems, as well as future work including using an industry survey to determine the problems that are most common and which problems cause the most trouble.

Share this

venkata maddula murty Says:
Apr 7, 2013 at 11:47 PM
Good article. But now a days lot of equipment are coming as Touch Screen based.This new technology testing is done in which ways and what are the problems involved in those kind of instruments.As a end user i have seen1) Alignment Failure2) Migrated software ( From Old Equipment to new Equipment when they were replace the funcationality is tested properly in time in right way or not we dont know some time u will see defects lately )Details of experience will be provided upon request.

venkata maddula murty Says:
Apr 15, 2013 at 11:28 PM
Excellent. I had one experience which i want to share with you.I saw Scientific Games Lottery Terminals Linux based machines were working in good conditions.They were replaced by Windows Embedded base Touch screen Terminals.The funny thing is if u press one number the next number was coming.Of course later this problem was resolved but i didnt understand how come without testing they introduced at agent level.Several customers were given wrong tickets and complaints. The process of Testing where and how it was done is the important issue which i want to inform all of you.The case is Migration where it should be more perfect rather it was worst than the old ones.

Donald Firesmith Says:
Apr 16, 2013 at 11:35 AM
Touch screen systems include several quite different components that can be the source of test failures: the software application(s), middleware, OS, SW drivers for the touch screen HW, and the actual touch screen HW. Testing the SW (especially output) is comparatively unchanged, while testing the HW is a newer and more interesting problem. As far as user input testing of the HW is concerned, you are interested in knowing if the screen correctly detects the touch, if it accurately identifies the coordinates of the touch, and if it makes this information available to the SW. HW defects could reside in any of the relevant HW components (e.g., touch screen, controller, cables, power supply), in their interfaces, in surface dirt or oils on the screen, and even in your test "equipment" (e.g., dirty fingers) and how the TE might be misused (e.g., touching with finger nail rather than finger tip). Testing the software becomes all the more interesting when several hardware components interact; for example, smart phone accelerometers to detect orientation are used to determine how the touch position is to be interpreted relative to the display.

You are right about one thing. It is embarrassing when testing is insufficient to ensure that porting does not significantly decrease system quality.

Jerry Gill Says:
May 18, 2014 at 2:12 PM
Mr. Firesmith, I work in testing and I am currently working on the final portion of my graduate degree. I read this post and would like to use the information in it as part of my work. My problem, I'm sure you may be aware, is that it is a blog and I may have difficulty in defending its use. If you have some advice on how to successfully defend its use or can direct me to another link that expresses your points in a peer reviewed article, I would greatly appreciate it. I have witnessed or been a part of every situation you described in this post and would love to reference it as part of my document. Thank you in advance for any assistance you can provide.

At the same website on the publications and presentations webpage, you can also download several presentations I have given on the topic.

Ram Says:
Jan 22, 2015 at 7:27 PM
The issue in now a days is with the damn technology it is pretty hard to cross check your application especially in touch Devices.I have an experience Web application behaving in Iphone5 doesn't behaves identically in I phone 6.If it is working fine with I devices then the problem arises with samsung models, so I felt because of extensive and wide range of Mobile/Touch devices it is pretty hard to develop or to test any software which is 100% or atleast 95% compatabile with all Touch and Mobile devices.

Apart from these some times procedures also spoil the skill set of Testers like in a company I have worked we use to spent 20% of our testing time n effort with JIRA in updating ticket, adding labels adding categories to exact sprint etc etc.

New Methodologies like Agile introduced flexibility flavor in SDLC process but comprehensive documentation like No Test CASES, TEST SCENARIOS are also causing problem.

The thing I have never understood is I know Automation and used couple of tools as well but I felt Automation is time taking and not effective like Manual Bug hunting.Though it depends on the Programming skillset of a Tester, but if you have hardcore Coding skills then Why Testing? not as a developer.

I believe and Love Bug Hunting using various smart tools and Add-on's which saves my time and gives intense results for me.

Add Comment

Name (required)

Mail (will not be published) (required)

Website

Subscribe to this comment thread to be notified of new comments on this blog post.

Please enter the text you see in the picture:Leave this field empty: Remember my information