Reduce the Cost of Robustness Testing with Polyspace Static Analysis

By Ram Cherukuri

The previous post highlighted the benefits of leveraging Polyspace® static analysis to help optimize and reduce the length of the testing phase of the verification cycle. This post will discuss the inefficiencies of robustness testing and introduce how to address those challenges.

What Are Robustness Tests and What Is Their Purpose?

Robustness tests are the set of tests designed to verify the correctness of your software under all conditions, particularly unexpected conditions. It is important to note that it is not only near impossible to test each behavior of your program, but also impractical as this can be an extremely expensive and time consuming process.

Different Approaches to Robustness Testing

There are several approaches to choose from when testing your software:

Test as much as possible to increase coverage. This will ensure that all paths in the code are reached by at least one test and there is a chance a few potential issues can be resolved.

Test for unexpected inputs. This enables you to search for errant behavior on edge cases and other cases as needed.

The two approaches are useful, but can be expensive in terms of time and cost.

Each node represents a robustness test case. Green nodes are test cases that pass while red nodes are cases that identify an issue. The gray area represents the untested run-time space.

Static Code Analysis

Static code analysis is used to detect anomalies or unexpected error conditions. Static analysis tools, such as Polyspace Bug Finder™, can help you detect a wide range of defects including numerical, data and control flow, concurrency, and security vulnerabilities. It is difficult to design tests to detect some of these issues, because of the nondeterministic behavior of the code and even harder to debug.

The defects are highlighted in the source code and include traceback information to help identify the cause and source of the defect. This further reduces the cost of tracking defects during the debugging phases of your test cycle. It is important to note that this approach is not exhaustive, as these tools can generate false negatives.

The circled nodes represent the new defects found through static analysis.

You can use sound formal methods based analysis to verify the correctness of software for all conditions. Tools such as Polyspace Code Prover™ can verify every possible combinations of inputs and parameters with each decision path. In addition to being exhaustive, the time needed to set up and run the analysis is significantly less than designing the individual test cases. Code that is proven 95% green (i.e. 95% of the checks are proven safe) significantly reduces the amount of verification effort and time necessary. Developers or quality engineers can now focus only on the 5% of the code that is not proven safe.

The green area signifies the run-time space that was analyzed and proven safe. The circled nodes represent defects missed by earlier testing.

One aerospace customer working on DO 178 B compliance uses Polyspace Code Prover to verify and prove absence of arithmetic overflows as required by the DO 178 standard. Manually verifying each instance of an overflow would otherwise be a laborious and time consuming process.

In conclusion, you can leverage the Polyspace static analysis solution to improve the overall quality of your software, while also reducing your testing cycle and the cost of the development process.