Testing Wins Should Come through Mastery, Not Luck

or for worse. We should, however, strive to test well enough that users must be crafty to cripple the software we stamp. Given the dearth of testers who could serve as mentors in the average organization, we can nonetheless, with conscientious effort, work to grow the herd for ourselves, our cra ft, and the future of testing.

Pages

User Comments

8 comments

Pete Dean

Excellent article!

"...actively talk about your software to people outside your immediate test team. ...Talk to developers, database administrators, security pros, performance experts, and the server team about your project. Each of these people plays a specific position on the team. They may not be able to tell you what the guy playing first should be doing, but they can tell you everything about what the short stop should be doing because that’s the position he plays."

Abolutely 100% spot on. I've been been doing this for years now. It's such a ridiculously simple suggestion, but has paid off for me so many times and saved my career more often than I care to admit (in public).

Bonnie – You bring up an important and interesting discussion to the table. Testing as a discipline offers endless learning opportunities to the tester – be it about the product, technology, testing techniques, end user needs, how to collaborate with the team in helping them understand the quality goals etc. Along with your suggestions that you have listed to help the tester grow, here’s one from a slightly different angle that helps them promote good team collaboration and push quality upstream that I have found beneficial – the tester helps the team understand the product’s quality goals and empowers them to enhance quality through their efforts in possible ways. For e.g. working with the development team in building a set of unit tests, with the build team in giving them a set of automated smoke tests that can be run to verify a new build etc. Through these steps a pro-active tester is able to build better team collaboration, improve product quality and more importantly create more time for him/her to work on core testing tasks that will help build his/her mastery as well as product quality.

When I saw the title of this article I thought the author was completely out of their mind. In all the years none of my "Testing Wins" were the result of luck. I was quite insulted for all Test Engineers/Quality Assurance Engineers in the business.

When I graduated from college (1984), there were no accepted practices for Software Development let alone Testing. We were making things up as we went along. Over time our processes became more standardized and our hard work was rewarded by very successful customer acceptance testing. Our customer was the Navy.

Carnegie Mellon had been tasked with developing a method for evaluating Contractors competing for DOD contracts. My group was asked to help develop some of the process requirements contractors would be evaluated. One of the major requirements was training programs for all aspects of a project. I was involved in evaluating a Unit Test training course (both the original and a revamped course based on our feedback). I was later asked to help develop a System Test training course and was asked co-teach the class. Eventually, my co-teachers dropped out due to schedule conflictswith their projects. I ended up running the class by myself for more that a year. My management allowed me to take the time off our project because it was good for my professional development. Eventually, the class was turned over to someone else.

Our process included design and code inspections. For the design inspections, we reviewed the english language version of the unit along with all documentation and test cases for the unit testing. For code inspections, we reviewed the code (either higher level language or machine language) and more detailed test procedures. Whenever a problem was found after the code had been implemented (usually in system test or after deployment), a problem report was initiated. Each problem was reviewed and a determination was made if there was a workaround or if it require patching the software. Patching was done in machine language and went throught the same inspection process as original design/code.

It was required that a test engineer be present for all inspections. The inspection could not be held under any circumstances if there was not a test engineer available.

The last major upgrade I worked on was when we transitioned from a company propriatary processor to an intel processor. This required us to retest all the functionality to insure that nothing was broken. I was the Software Test Manager for the project. In addition to managing a very small group of testers, I wrote test cases/procedures, reviewed test cases/procedures written by my team and ran tests at the Software test level (versus System Test which ran in the actual hardware - although I did that as well).

The code was written in FORTRAN (yes I am dating myself) and there was only vendor with a compiler for FORTRAN. I spent almost as much time debugging the compiler as I did testing our code.

I know that anyone reading this is thinking that the government can (and does) spent a lot of money to ensure that there are no bugs and it can afford to throw money away on silly things like designing test cases. Just before we started the conversion to an Intel processor, a new Software Manager started with my group. We already had a budget and schedule. We were very sure about our schedule and knew that the total budget was accurate. The only thing we weren't sure about was exactly how to divide up the budget.

We very firmly beleived that the money we spent on inspections was going to save us money in the testing phase. The new Software Manager kept having fits because he said we were spending to much money on the inspections and were going to be over budget. He insisted that we suspend the inspections. He was over ruled by just about everyone. He also decided that he was going improve the metrics by closing all the open problem reports. I told him that he did not have the authority to close them. He told me he was the software manager and he could close them. He very quickly found out that he could only recommend them for closure. Closing a problem report required a recommendation for closure at each step in the process. Since they hadn't been tested (because no solution was implemented) I did not recommend them for closure and they stayed open.

The end result was that we finished the project 30 days early and $400,00 under budget. The earlier in the process you find a problem the less expensive it is to fix. Because we inspected the test cases along with the design of each unit, design flaws were caught before the design was coded. Because unit testing was comprehensive, there was no "low hanging fruit for the Software Test group to find (unit testing was done by the developer). Problems found in Software Test were limited to the interface between units and there were very few because there were system documents which laid out the interfaces. When we got to System Test/Integration there were less than a handful of issues and they were problems that could only be found when running on the hardware.

Even though writing test cases may not have been the developers favorite task, they wrote better code because thinking about how to test their software identified design flaws or code issues. The developers also took great pride in the fact that the Software Test group was unable to find any problems with their product. The Software Test group took great pride in the fact that when they passed the software to the System Test Group very few problems were found. I remember one of the System Engineers telling me that we did such a good job of testing that he was bored because all his test cases worked.

Your product dictates how much can be spend on testing. You can't spend the same amount of money to test an app as you could on say tax software or on software going into the space shuttle.

But certainly in the example you gave of JP Morgan, that was a very expensive bug for something that was realatively easy to test. The probe that was sent to Mars (I think) and never heard from again as the result of one unit using metric and another using non-metric was a very costly mistake.

One year when I was doing our taxes, I found that TurboTax decided that I had paid too much in SocSec taxes. My husband and I worked at the same company and TurboTax added our SecSec taxes together and compared it to the max for a single person and told me I paid too much. If I had not caught it, it would have been very costly to me. It wouldn't have cost TurboTax anything, but it was a bug that should have been found before the software was sold.

Testing is everyone's responsibility.

Sorry to go on and on, but this hit a nerve. Boy do I miss testing. I find it a lot of fun.

About the author

Bonnie Bailey

Bonnie Bailey is a software test engineer for a health care information technology company. Bonnie is an avid reader of fiction and non-fiction, including software design, testing and development, disruptive and emerging technologies, business leadership, science, and medicine. She also enjoys writing. Find Bonnie online at bbwriting.com.