When selecting a car, you must make a series of choices. Do you want a sports car, sedan, truck, SUV, van, or something else? If I want to transport my family of six, a sports car probably isn't very well suited for this purpose. However, if I want to haul heavy loads, the truck is probably my best option. There is no one car that is perfectly multi-functional. A certain amount of compromise must be made to find a car that is well-suited to a particular purpose.

Similarly, no single quality assurance tool meets the needs of every project or type of software development. This poses a unique challenge for our (quality assurance) QA organization, because the easiest solution would be to settle on a single tool. This would enable everyone to become familiar with a single tool and leverage the knowledge of everyone else who also used the same tool. Unfortunately, this isn't possible for several reasons. First, there isn't a single tool that supports all the technologies we work with. Second, if we did this we would be giving up a lot of useful features to get a very generalized tool. Third, and most importantly, this would not give us the best software.

Since our overall goal as QA engineers is always to improve the quality of the software we are testing, we typically choose tools that are well-suited to the project being tested. There are several common technology sets that are used within the Church for development. Because of this, we can focus on these common technologies and have the majority of our projects covered.

Regardless of the technology used, there are a few key guidelines projects use when selecting which tool(s) to test with, including:

Strong community or commercial support – A successful tool requires strong community support. If a tool doesn’t have a strong community around it, then it is likely we will spend significant amounts of time fixing roadblocks. Likewise, if a commercial product doesn’t have strong support from the company, we will run into challenges down the road. It’s always easier and more rewarding to participate in a group of like-minded people trying to accomplish similar goals.

Ease of extensibility – Almost any tool will require some amount of modification or new features to do what we would like to do. This makes the ability to extend and customize a tool critical.

Integration with other tools and frameworks – No testing tool or framework lives in a world alone. Since the products we test are changing rapidly, most products rely on several testing tools or frameworks to fulfill their testing needs. This means test tools must work together and be easy to integrate with other reporting tools.

Easily updatable tests – One problem we have encountered is tests that weren’t easy to update as the product being tested continued to change. We simply don’t have enough time to continually rewrite tests so we must have tools that enable our tests to be easily updated along with the product being tested. We have also encountered tools that have no migration path for existing tests when a new version of the tool is released. We certainly want to avoid this.

Ability to leverage existing knowledge – A huge benefit to adopting one tool over the other is if it allows us to leverage existing knowledge from both developers and QA engineers. This is not a hard and fast rule, but is certainly a consideration. It is always easier to get people to use technologies and languages they are familiar with.

As no single tool meets all of these needs, we currently use a wide variety of testing tools throughout the Church and our list of tools will continue to change. Regardless of what tools we choose to use, it is important that we continue to use these criteria to closely evaluate their usefulness for their intended purpose based on these criteria.

Asiel Brumfield is a senior software quality assurance engineer for the Church.