Real Users of Model-based Testing

I think’ve finally found a good visual metaphor for model-based testing. In Aliens, Ellen Ripley uses the power lifter suit to battle the Alien Queen. Model-based testing is like the power lifter suit. It has to be driven with intelligence and skill, but it’s your only hope against the super bugs that like to inhabit today’s huge and complex apps.

The survey produced many data points, which the report presents in three dozen graphs. For a more intuitive sense of what present-day MBT users are doing, I created a hypothetical profile, using only the most frequent survey responses. Here’s that composite profile of a typical MBT user.

I work for a large business organization. We follow an agile process to develop the software we test. The system under test is embedded in another product sold to our customers. It is programmed in C++, runs on dedicated computers, is networked, and uses a Windows OS.

We felt MBT could help reduce testing cost/work and bug escapes. We use a commercial MBT tool that we’ve integrated with our design repository and test harness.

At present, we’re using MBT in a pilot project to test version 1.0 of our product. We taught ourselves to develop test models and use the tool by reading documentation and experimenting. It took us each about 100 hours of this self-study to become minimally proficient with the tool.

We use MBT to test system-scope functionality. The tool generates code and data for an adapter that abstracts the SUT APIs and then drives the APIs and evaluates results from the APIs. We measure requirements coverage for our test runs. The output of this tool is not integrated with our build.

We’ve had some difficulty developing test models. Sometimes our model “blows up” (combinatorial runaway) and we’ve found that our MBT test suites are not effective for certain kinds of bugs. However, this is manageable and not any worse than what we expected. With respect to other first generation MBT problems (updating, integration, and inadequate oracle), either we have not seen these problems or they’re not obstacles.

In all, we think MBT has been moderately effective so far. For the most part, our expectations for both challenges and improvement have been met. MBT has reduced bug escapes about 60%, testing costs by 15%, and testing time by 30%. Among our co-workers, developers and managers are neutral about MBT but other testers view it as effective. Our users/customers are not aware of MBT. Overall, we are very likely to continue using MBT.