How to Evaluate a Software Testing Tool

Before starting with any software test tool evaluation it is essential to understand the problem you are trying to resolve and the requirements you have to address. Without a clear set of requirements you will never know if the software test tool you select meets your objectives and solves the problem.

In following this process we're trying to make sure that we are picking the right software test tool for the right job. Pick the wrong tool and you'll waste precious time and effort. Select the right software testing tool and you'll increase productivity, improve team motivation and enhance your product release quality.

1. Identify the problem and Define RequirementsBefore you even select a software testing tool you need to understand the problem you are trying to solve. This problem may be as simple as the need to compare large amounts of test result data or it may be as complicated as trying to manage the whole test process more effectively. Either way a short paragraph on the problem you need to solve by implementing a new software test tool will keep the whole evaluation process focused.

With the problem defined clearly you can move forward to defining requirements. Defining requirements for a software testing tool is no different to defining requirements for any other project. Requirements worth detailing would include non-functional as well as functional requirements.

Make sure you involve all the relevant people in the requirements capture process. It may be that the project manager has a completely different view on what the new software testing tool should achieve to the test manager. Being clear about the requirements of the different stakeholders up front will help avoid conflict and disappointment further along the evaluation process.

2. Assess the appetite for changeIf the people on the ground using the proposed new software test tool have no appetite to embrace changes to their working practices then you've got an uphill battle on your hands. If the people on the ground are the drivers for the new software testing tool then you'll probably have a fairly easy ride when you get to the trial and implementation stages. It is just important to make sure that you consider the appetite for change when planning the amount of time and effort required for implementation. It may even be, after careful consideration, that you decide to postpone the evaluation due to resource constraints within the team. Alternatively if you have someone on the ground that will act as a champion for the implementation then can ease your progress through the rest of the test tool evaluation process by encouraging him or her to talk about the benefits to the rest of the team.

3. Ascertain suitabilityThe next stage involves taking a number of candidate software testing tools and evaluating them for suitability against the original requirements you defined. One of the best approaches for this phase of the evaluation is to create an evaluation matrix. Basically assign each requirement an importance factor. Then for each software testing tool you evaluate them on a scale of 1 to 10 for suitability and then adjust the score based on the importance factor for the requirement. Final scores can then be totalled up for each software testing tool being considered and an overall leader identified. With a leader identified, out of a range of software tools, the trial phase can commence.

4. TrialIt is essential to trial a new software testing tool in a live project before committing to a full role out. Whilst a list of requirements being met, might look impressive on paper, it's not until you try something in a live environment that you really see how a product will perform.

During the trial it can be useful for those running the trial to have a check list of the requirements to consult. As the trial progresses the users can tick off the requirements as they see them successfully met. This approach keeps the trial objective and makes it less susceptible to people's emotions.

If the fit of the tool looks good during the trial then you may select the tool for roll out. If the fit isn't good and the problem you initially defined is not solved then you may want to go back the 'ascertain suitability' stage and choose another tool for a second trial.

5. ImplementationImplementation of a software testing tool is complex topic in itself. Aspects already considered like the appetite for change matter greatly here. Also worth consideration may be items like data conversion, training, deployment and support. At a minimum if a software testing tool is being rolled out across a whole team it is worth training one or two people within the team so that they can act as implementation champions. Having a local team member that everyone within the team can consult makes for a smooth roll out and will help ease any transition issues from legacy testing tools.

In short though the whole process of evaluating a software testing tool boils down to choosing the right tool for the job.