by Elnur Abdurrakhimov

Automated Tests as a Competitive Advantage

Jun 14, 2015

Most of us have heard of at least one project that at a certain point became so complex and unmaintainable that people involved in it developed a fear of touching the code and the project slowly stagnated to death and got replaced by more agile competitors.

A less dramatic and more often heard story is a very buggy product where a fix for each bug introduces two new ones and users become so unhappy with that that they switch to a competing and less buggy alternative.

A variation of the last example is a product that is buggy in a very hard to notice way. It usually happens in a case of complex domain logic. Domain rules might be so complex that developers have never got them right in the first place. It might take months or even years for a user to notice that something is off, investigate it and find out that they were relying on a product that was lying to them all this time. Not a very pleasant moment…

I’m sure you can relate to at least one project developing in one of the scenarios mentioned above.

What all these scenarios have in common is the lack of automated tests or improper implementation of them.

Now imagine two similar and competing products being developed by different companies at the same time. The first one — let’s call it Solid — is being driven by tests and executable specifications, while the second — let’s call it Whinners — is being tested manually.

As you should know by now, testing is not a choice, so the difference between these two projects is not that one is being tested and the other is not; they both are being tested but Solid’s tests are automated, while Whinners is being tested by someone manually.

Since both projects started at the same time and developers of Solid took their time to set up all the testing infrastructure right from the start and Whinners did not, Whinners released their first version a week or two earlier than Solid.

Whinners had a couple of bugs, but that was not a big deal, because the product has been just a month into development and was still pretty small. And there were not many users anyway. The team clicked and typed through the whole application, found those bugs and fixed them. They introduced a bug or two while doing that, but they were not fixed immediately because the team didn’t expect them to appear and didn’t want to click and type through the whole app again — they had to be churning new features out.

The first release of Solid came out a week or two later. It had a single non-critical bug. It got reported by a user, testers reproduced it, and developers added a couple of tests to prove that the bug indeed existed and then fixed it. Turned out that fixing that bug introduced a new one, but it got caught by their test suite and hence fixed immediately.

After the initial release, both teams decided to release new versions of their product every 2 weeks. Agile, you know…

For several iterations, Whinners was leading in the number of features. That lead to them having more customers than Solid. But it was not all that shiny for Whinners.

After around 6 months into development, Whinners was releasing into production with more and more bugs. Customers were not very happy about that and started complaining. Some even started writing negative posts in their blogs.

To counteract that, stakeholders of Whinners went to their developers and demanded to fix existing bugs and stop producing so many new ones. Developers defended stating that it was not possible for them to test the whole app after each change. If they did that, they’d have almost no time to implement new features.

In response to that, stakeholders decided to hire a lot of testers. In the end, Whinners had twice as many testers as developers.

Testers did good. They caught a lot of bugs and produced detailed ways of reproducing them. That helped developers enormously and they started fixing bugs one by one. The only problem was that fixing each bug almost always introduced new ones.

That lead to constant back and forth between developers and testers. Developers were being paid for implementing new features, and testers were being paid for catching bugs. But the problem was that it was developers who had to fix the bugs. And since that lead to less and less features being implemented, developers started feeling insecure about their jobs and became defensive against the testing team.

As time went buy, less and less time of each two-week iteration was spent on implementing features. Instead, more and more time was being spent on fixing bugs and new bugs that came into existence after fixing those initial bugs.

Not only there was less time in each iteration to spend on implementing features, but bug fixing constantly spilled onto next iterations and made them unpredictable. No one could tell how much time would be spent on fixing bugs.

In response to that, the team has decided to dedicate a whole iteration on fixing bugs every now and then. Specifically, they decided that each 4th iteration would be spent on fixing bugs only.

In the meantime, Solid was doing well. Instead of having a fast start and an eventual slowdown, they maintained the same rate of production of new features.

They had a couple of bugs here and there, but since before fixing each bug they wrote at least one test to prove that it existed and was actually fixed afterwards, no bug that was fixed once got ever reintroduced again.

They had a couple of testers doing exploratory testing and coming up with new ways to break the app. Just a couple of testers was enough because they never had to click and type through the whole product after each change. Instead they just came up with some crazy ways to break the app and passed those to developers for them to write more tests and handle those edge cases.

Even though the pace of new development of Solid was slower than of Whinners initially, at this point not only they did just catch up, but Solid actually went ahead. It’s not the development of Solid somehow magically became faster, no; it’s Whinners that slowed down so significantly fighting their bugs that they became slower than Solid.

A lot of initial users of Whinners got so tired of the never ending supply of bugs that seemed to be never fixed, they started looking for less buggy alternatives and found Solid. Solid felt so much more solid to them, that not only they switched to it themselves, but they also wrote a lot of blog posts around the Web about them finally being happy with a much more solid and less buggy alternative to Whinners.

Needless to say, that lead to the fact that Solid started getting more new customers than Whinners. It didn’t mean Whinners went out of business, but they were not leading in the market anymore. Nevertheless, they still had enough customers for their stakeholders to scuba diving relentlessly and buying expensive sport cars.

6 more months into development of both products, Solid was still churning out new features at the same not-so-fast and not-so-slow pace.

Whinners, on the other hand, had slowed down to almost a halt. They almost stopped adding new features and most of their iterations became bug fixing ones.

Solid was getting more marketshare, but Whinners was still in business and made enough money to keep the product afloat.

After some time, Whinners got rid of most critical bugs they had, but they also developed fear to not touch the code to avoid adding new bugs. It became Solid’s little ugly brother that was still in business and making money but slowly dying off. Without adding new features and adapting to the market, they just couldn’t survive in this fast-paced industry.

Even though stakeholders of Whinners realized they were not leading in the market anymore, it was still good enough money for them, so they decided to optimize their costs and maximize their profits. They fired most of their development and testing teams and left just a couple of maintenance developers and a tester to be able to fix a critical bug or two in case they showed up.

Not that Agile anymore, but still in business…

And then… Then suddenly something unexpected happened. Another bubble burst and the market had to adapt. The old ways of doing things didn’t work anymore.

Sure, 80% of the market both Solid and Whinners served to was the same. But the other 20% of it had changed and changed dramatically. That meant that 20% of software had to be rewritten to adapt to the new reality of the market.

The whole Solid team had an emergency meeting. Stakeholders were in panic and asked the development team if there was anything they could do to adapt to the new changes and save the company from going out of business.

The development team, on the other hand, was calm. In fact, it was so calm that that made stakeholders even more nervous and they demanded explanation of their calmness. Then developers explained to stakeholders that all they had to do was to go through all the specifications and change the parts that had to be changed, and then the development team would adapt the code to make specifications pass again. Stakeholders relaxed a bit and got down to work with developers and testers.

They needed a couple of weeks to make the necessary changes to the specifications, and then between a month and two to adapt the code to these changes. Then they released their product to their customers’ delight.

There were a couple of bugs here and there, but nothing critical. Testers reproduced them, developers added missing tests and fixed the bugs. Customers were more than happy.

And what about Whinners, you ask? What Whinners? Oh, Whinners, yeah. They went out of business and all their customers switched to Solid.