The limits of AI testing

Thursday, December 13, 2018 6:52:00 PM

Artificial intelligence (AI, or more accurately called machine learning) is all over public discussions about software testing. It seems like the next big thing! Many imply that it will take over the practice of automating the SUT for quality measurement and management.

Will it?

There are stories of AI techniques helping find software bugs. I think AI will be great for finding bugs, within 5-10 years. Maybe it will replace model-based testing techniques.

I suspect that the recognition that AI will potentially be great at finding bugs, combined with the misunderstanding from Myers’ 1979 assertion that “test is all about finding bugs,” contribute to the prediction that AI will replace repeatable automation for software quality.

As far as I can tell, AI will never be a great contributor to the movement to “shift left” because it will never be especially fast, or directed, or traceable, or even repeatable. It might be able to create useful test cases, though, that could then be vetted and used for build validation, although whether it will persist data from the app for behavior and performance is undetermined. (Persisting detailed behavior data, as MetaAutomation does, to speed up communications around the team, enable detailed analysis, and obviate the need to reproduce most failures, could be done I think for AI-generated test cases.)

AI will be at least a powerful supplementary way to find app bugs, which is good thing because finding bugs takes a diversity of techniques. But it won’t replace the need for quality automation that is at least partly repeatable.

MetaAutomation will still be the fastest and most trustworthy way to ship software faster and at higher quality.

It is risky to predict ten years out, but at least MetaAutomation is the only complete way to get data from the SUT (on behavior and perf), and AI needs that data to be effective, so for foreword compatibility and flexibility, it’s better to get started soon. See the Hierarchical Steps and Atomic Check patterns for more information.

“Quality automation” is a new term I introduced that I use to replace “test automation.” I did not do this to annoy people or drive a claim into the fertile ground of semantics! I did it for some … more

One would think that people who make automation to drive the system under test (SUT) would be eager to preserve the quality knowledge of how the SUT is driven and how it responds, how fast, etc. For … more