Wednesday, 4 March 2015

Automation Assessment

I have heard, in many organisations, speed being cited as a primary benefit of automation. In most cases the use of tools will make things quicker. But faster doesn't necessarily mean good. It's important to assess automation against other criteria.

The development team, not only the testers, should regularly reflect upon and discuss their automation. One approach to this conversation is to have each team member indicate on a scale whether they agree or disagree with the following statements:

I understand what is being checked by our automationI feel confident that a failure indicates an important problem in the applicationI do not manually repeat those checks that are automatedI find it easy to diagnose the cause of a failure

The collated answers will help to uncover whether people understand and agree with coverage, whether they believe the chosen implementation of automated checks is robust, and whether they are capable of investigating problems that are discovered.

Automation may execute quickly, but without assessing its value it may be quickly doing nothing useful.

Your last sentence pointed the whole thing out. Test Automation is for 'execution' of those automated tests that were built by a human. All a machine does is take what it is told and runs it. No AI here folks. Automation is a way of 'automating' the process of executing tests (or checks as some so stubbornly like to call them, whatever).

By leverage a machine you get efficiency gains that relate back improved performance, or perceived performance. The true benefit is to get to the stage where you have multiple machines running tests in parallel. I like to call this the "Illusion of Speed". The total time to run all tests is the same but by dividing up and running sets of tests in parallel you get a perceived compression of time. So if a set of automated tests takes 8 hours to run on one machine then if you had 8 machines and can run all the tests independently (no interdependencies) then you can divide them up across the machines and make it appear that it all ran in an hour.