The Second System Effect in Software Test Automation

Matt Heusser blogged today on generating new ideas. In his entry, he cites from Weinberg’s Becoming a Technical Leader. Weinberg lists up the three obstacles to innovation – one of the three parts of his leadership model. The three obstacles to innovation are self-blindness, the No-Problem syndrome, and the single-solution belief. Continuing he explains the three keys to innovation based on this: Learn from your errors, learn from others, and copulate together two great ideas to form an even greater idea. Matt also mentions my struggle to find an article where a great software test automation framework became shelfware just because it couldn’t stand the technical challenge – or maybe just the human aspect of it? So far, we just found opinions on software test automation projects having failed, but no real hard data.

So, to overcome this lack of available material, let me write about my experience with software test automation, the problems we had, and how we overcame them. Thereby, I will not only enable myself to learn from my own errors, but also provide a system you may copy yourself, or which you might combine with another great idea that you have established in your company. In fact, most of this shouldn’t surprise you. During the last year, I also presented our approach on a test conference – here‘s the paper I wrote on this.

The Second System Effect

Frederick Brooks defines the Second System Effect in his popular book The Mythical Man-Month. Let me quote from page 55 of my copy:

An architect’s first work is apt to be spare and clean. He knows he doesn’t know what he’s doing, so he does it carefully and with great restraint.
[…]
This second is the most dangerous system a man ever designs.
[…]
The general tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one.

I think Brooks’ words are very clear. But what has all of this to do with software test automation? Well, on my first project, we were using heavily shell scripts in order to execute functions, that would execute the system under test, and verify outputs and the like. Just after two years and a major struggle with this approach, I had found out about our problem: The Second System Effect in Software Test Automation.

Elegantly our system was started by our test scripts, whenever an account was put into the system, it was noticed down in a flat text file, so that later test scripts could read the information about the available account. We also noted down anything about the account in regards to different subsystems, which we simulated in our test automation harness. Do you see the second system? Yes, it’s just right there. We had build the application using shell scripts. This was our second system. Of course, the point where we couldn’t keep up with our developers, where the ones where they had clear technological advantages. So, this set us out to the major struggle. After two years, we were faced with the first customer, and at that time we were asked to test composed services. While this is an easy thing to do in most programming languages, it became a mess for our shell script system.

So, to define the Second System Effect in Software Test Automation, it’s the effect you get, when you build a second system with your test automation mechanisms. You may not immediately notice that you did, so you should better keep your light on.

Smells

Here are some smells you should keep an eye on. Does a change in the development code cost you a lot in your automation code? Well, maybe then you could have a second system. Does a change in the business logic affect several places of your automation code? Doesn’t smell too good. Do changes in unrelated parts of your application affect the outcome of your tests? Do you have high test maintenance costs? Do you have fragile and brittle tests? All these symptoms might point you to the second system effect.

The cure

But what can you do to avoid the Second System Effect? First of all, treat software test automation as a software development activity – and an important one. You will likely need software developers for this work. Bring testers and test automation developers closely together, so they agree easily on possible future changes. But it’s also vital to bring the application developers, the testers and the automation code developers close together. On an Agile team you will most likely find developers working out the automation code themselves. The closer you can get all these parties to collaborate, the less-likely it becomes that your automation code will suffer from the Second System Effect. Focus on the business intents of the users and customers, and avoid building the same system twice. Otherwise you will get twice the maintenance work.

Post navigation

2 thoughts on “The Second System Effect in Software Test Automation”

Thank you for putting a name to a problem which I have noticed on some of my past projects.

From my own experience, it is often the case that test automation systems can work just fine, up until the days and weeks before a major release, then suddenly when last minute changes are being made to the system the tests stop working due to changes in the product code which bring it out of line with the test automation code. Of course, the developers who maintain the test automation code are too busy on production code at this point to be able to fix it.

I want to know if you have any other suggestions for “The Cure” to these problems? For example, does keeping the production code and test code in the same place help to keep the projects in sync? If they are part of the same project, then it is easier to refactor the test code based on changes to the production code. Or if the production and test code are developed separately, how do we stick to the principle of low coupling and high cohesion to ensure that changes to one code-base won’t necessarily require a change in the other.

So far, I have just made experiences with clear separation of the production code basis and the automation code basis. Since these are side-effects from the collaboration point of view, I don’t have experience worth to compare. The outcome on the other hand is always the same: disconnected understanding on how the system should behave, with even less focus on what the user actually needs. If you get your developers and your automation code developers busy on something irrelevant, the user pays the bill.

I’m glad you liked the name I picked. From my perception it expresses the intent best. Personally I hope that testers can start talking about good and bad test automation in some near future.