Help, our automation is taking over!

I have always been a fan of testing and, in particular, automated testing. Recently, as my team moved to being “by the book” Agile, we made automated testing not only the way of QA life, but a part of the whole process – starting with how the initial requirements are defined, code is developed and ending with, yes, testing (deployment to come).

As testing now permeates the team’s existence, it creates an impression that a lot of what we do is furthering automation rather than building the product, possibly sinking a lot of time into it (hence the title). A question was asked: “Isn’t this approach actually more costly than what we used to do?” This article is the answer, with as many concrete numbers as I can dig up.

As far as the QA process, what is the main difference between what we used to do and what we do now?

Previously, QA was a standalone part of the process. QA would get the spec and the application increments and would do their best to write test cases and match the spec against the application. They would attempt to automate as much as time permits, experience showed that time did not permit much.

Now QA is the responsibility of everyone. The process starts by defining requirements in terms of user stories and then immediately translating them into BDD scenarios1. The spec, as a document, no longer exists!

The complete process proceeds as follows:

During iteration planning each story that deserves to be automated is identified and a separate task for automation is created

Immediately after the sprint planning the Product Owner (PO), either with the help of QA or on her own, creates BDD scenarios that describe how each feature should operate. This is often done even for the stories that are not going to be automated

QA takes the scenarios from the PO, augments them with its own scenarios and commits the whole thing to the repository

As the developers complete the stories, they either ask QA to implement the automated tests based on the created scenarios or do so themselves

By the end of the sprint QA is able to run the tests both from this and past sprints to validate the functionality of the application

In the appendix to this post I included the broken down estimate of the amount of time it previously took (to create the spec, answer questions and tests) vs. the time it takes now. Assuming I did not fudge the numbers the conclusion appears to be that the amount of time is about the same (actually less with the new process, but I am willing to attribute this to a roundoff error).

This seems to address the concern of the cost of adding automated testing into the process. It, however, does not nearly show all the other benefits we gained as a group. The greatest one, by far, is the supplementation of the spec by BDD scenarios. Before the QA group was often confused when dealing with (many) versions of the spec — not any more! In addition, writing out BDDs causes the PO to think as QA and specify a lot more “hidden” requirements from the start. It also allows QA to “see” the product even before something is written and ask PO necessary questions very early on.

Developers similarly benefit from seeing BDD scenarios as they also no longer deal with the spec. They are also able to discuss the product not only with the PO but also with QA (although I am still waiting for this to become a more frequent occurrence).

Finally, and almost for free, we get an automated test suite to help us with CI and regression. As we develop more and more tests, our regression time goes down, while it always used to increase before. QA are now much less stressed and tired than they used to be.

Do I recommend automation and BDD as a means to implement it? Yes, very much so. Will it be easy? Not at all, putting it on an existing project can be quite painful both technically and culturally. Implementing it successfully, though, will benefit your team for years to come.

Appendix

Below is the time estimate of our process, before and after.

Notes:

The figures are derived for the same set of requirements, enough to keep the team busy for about two weeks 2

The figures are provided for the parts of the process that have changed

In “Before” QA is mostly manual with some automation

In “After” QA works on writing automated tests, manual testing is done only when automation is not possible

Estimates for the actual Dev and QA are not included as this part of the process did not change (we are still coding and testing)

All the numbers are derived by multiplying by the number of people involved

Time differences are not always translated directly. Even if the total amount of time between “Before” and “After” might be going down, it is possible that the saving go mostly to the PO, for example, while an individual developer might be spending more time on some part of the process.

Before – QA is done manually after Dev is finished

Process step

Duration (min)

Duration (max)

PO creates the spec

2 days

3 days

Dev managers does the initial review and provides comments to PO

0.25 days

0.5 days

The review cycle is repeated at least 3 times, 1-2 hours per person per cycle

0.5 days

1 day

Dev manager takes the final result and breaks it up into tasks

1 day

1 day

Tasks and the spec are delivered to the dev/qa team, each person spends at least 2-3 hours reading the spec and understanding the tasks

QA performs regression at the end of the sprint, currently a day and decreasing

2 day

2 day

Total

10 days

13 days

Footnotes

1 BDD stands for Behavior-Driven Development. More information can be found here.

2 This is the most dangerous assumption in the whole post. Are we really doing as much now as we did before? If we are actually doing less, then the whole thing breaks! It is not possible to really test this without redoing one of our older projects, so I am going on my intuitive feel. It seems, that in terms of real output we are doing at least as much, if not more than before. The only “real” proof I have is that so far no one had asked “And this is all you did?” after seeing the results of a sprint.