Writings From The Desk Of An Agile Sparkie

Building a Test Case Management solution

I’ve recently been looking at how to build a reasonable test case management solution (good!=word documents) for our company. I quickly learned this is not a very developed field. Mercury TestDirector seems to dominate the commercial field, with the other QA product companies (CompuWare, IBM-Rational) following suite, but not there yet.

Test Case Management as I understand it deals with the following objects:

* Test Plans

* Test Cases

* Test Labs

* Test Schedule

The main deliverables expected from a Test Case Management solution are:

* It should provide visibility into the testing process – both the plan and the actual execution.

* Is risk-oriented, focusing on the riskier aspects of the system first.

* Facilitates day to day management of the testing team.

* It should present a coherent picture connected to the Issue Tracking tool (what bugs are blocking tests, what tests need to be ready for a certain release, etc.)

* It should present a coherent picture connected to the Automated testing frameworks (If the smoke test / sanity failed during the night, the Test Case Managment should reflect that without requiring the QA engineer to manually copy+paste the information).

* Tracking pass/fail status for each test

* Tracking pass percentage per module per version per milestone requirements

* Dynamic priority management which affects to do list for testers

* Coverage of requirements by test cases (each change request should be linked to at least one test case ? Closing a change request requires successful test run ? )

* Management of test beds relevant for each test case

* Manage test cases that are blocked by other change requests (bugs/enhancements)

Based on the information I found I’ve been looking at TestMaster, TestLink and QATraq, but didn’t install any of them yet. The other ones really seem either dead or not ready yet.

I’m still trying to understand whether the correct approach is to get a test management tool, and try to connect it to your issue tracker, or to get a really customizable issue tracker (e.g. JIRA) and build what you need of a test management tool there. I’m still contemplating the pros and cons, and trying to understand how much of test case management is actually an issue tracking type activity, and what are the parts that are not. This is quite uncharted ground from what I’ve found so far, and I understand that part of having a “reasonable” solution is to skip some of the requirements and vote for simplicity.

A good friend which knows what he’s doing when it comes to managing QA efforts repeatedly tells me to aviod the bells and whistles and the complex reports metrics and processes, and go for simple worthwhile metrics, the reports and flows that are necessary to support them, and to focus on the substance. Thats a big part of what I consider to be a “reasonable” solution.