What is the best testing tool?

Someone posted to me a question through meetupcom, “Greg, what is the best testing tool?” I didn’t have room to reply. I posted my response inside the Nashville JUG Google Group we host, but I thought today, it would be better to capture it here.

Asking “what is the best tool” with no other context sounds like you are looking at the situation from the wrong point of view. Let me explain.

A few years ago, I inherited a java app with hardly any automated tests, had been worked on by 6 engineers before me for a couple of years, had been demoed in front of program management, and was non operational. It did stuff, but not much and was loaded with errors. I adopted an approach where I wouldn’t work on any bug/issue until I could write an automated test to reproduce the issue. This was painful; more than you can realize. The software was tightly coupled, lots of static singletons, i.e. global variables, and hard to isolate. Basically, my first test involved using JUnit to empty some database tables, programmatically ingesting a spreadsheet, and then inspecting the database for results. This test took 5 minutes to run. It probably ran through a big chunk of code.

The problem was the system, and no magic test tool was the answer. Changes in the beginning were slow, but as I gained momentum, I eventually reached over 60% test coverage. This may sound low, but within 3 months, I made a flawless demo before program management. They were impressed that it worked, let alone with no problems. The tool was starting to get used by the intended audience. A year later, reported bugs were fixed in one day, and never regressed. I reached a point where I worked for a whole day, and when the half the unit tests failed, I threw away all changes and went home for the day, depressed. I started fresh the next day and actually fixed the problem. Two days to fix it! With no automated test suite, that would probably have been released, and incurred a gob of new bugs to deal with.

I met weekly with the users and captured new feature requests, problems, and generally made this tool work really well. The customers were very happy. I was also empowered to rewrite whole sections that were slapped together hastily in the past, because I had the security of my automated test suite. I threw away code that wasn’t used and didn’t work. My boss showed me a chart where the total lines of code became less than when I first inherited, yet it did more than those other 6 people could squeeze out of it. That was a happy day!

What I’m saying is that I could have used JUnit, TestNG, ScalaTest (works on java too), or any other suite of tools like acceptance testing frameworks, load testing, etc. But what test tool was used wasn’t important. Adopting a strategy of making the whole thing subject to testing, staying test focused no matter how painful it is, paid off. At some times, the test suite took 1.5 hours to run. I spent three days speeding up the most expensive parts of the system, and cutting out certain tests, getting the same number of tests to run in 30 minutes. I came up with a comprehensive test suite and a smaller, smoke test that ran much faster. I also created a spreadsheet to track number of tests and total test time, along with a graph. As the test time grew, I would periodically halt development and polish up certain parts that made it too hard to run tests. I would run the test suite at least once a day to make sure things worked right.

This whole development period was some of the best coding I had done in a long time. Cranking out top quality code with a warm fuzzy green bar made me grin ear-to-ear. When I left that company, I cried a bit because I wouldn’t be working on that tool anymore.

In the 9th chapter of my book, Python Testing Cookbook, the first recipe captures a lot of what I wrote up above, with some more detail. To quote the recipe “Something is better than nothing,”

“Just don’t let anyone tell you that you are wasting your time building a long-running test case. An automated test suite that takes an hour to run and is exercised at least once a day probably instills more confidence than clicking through the screens manually. Something is better than nothing.” —Python Testing Cookbook, page 326

When I wrote chapter 9, I wanted to move beyond simple coded recipes, but instead general lessons I had learned in the realm of testing. These principles work, whether you are writing Python, Java, Scala, or anything else.