The best way to approach software testing starts by planning a path for getting your software to production. A key step here is to identify a testing tool belt that aligns with the route you plan to take.

In the case of Gannett | USA Today Network, where I work, our testing tool belt has all the information on what we use and how we use it. It's the central location for documentation around the tools that make up our strategy for testing several products.

In this post, I will go through each category of tools in our tool set to give you a general overview of the testing solutions we use. It is important to understand that this is not a complete list of all the software testing tools available, of course—It’s just what we use. To get full buy-in from our developers, we have aligned the automation technology stack with the application stack.

Breaking Down Testing Tool Categories

There are hundreds of testing solutions available, including open source as well as licensed software testing tool vendors. Choosing the best tool isn't hard when you have a well-organized tool belt approach that outlines all of the testing tool categories needed for software testing. Here are the testing tool belt categories that we use at Gannett | USA Today Network.

CONFIGURATION MANAGEMENT

CONTAINERS

BACKEND

WEB

Mobile

Security

Performance

Exploratory

Cloud Testing

Test Management

Monitoring

Deployment

We use a wide range of application technology stacks at Gannett, which results in a large inventory of tools for testing.

Configuration Management

We are using Chef to download, install, and configure our cloud servers. Your infrastructure code deserves testing also. Test Kitchen is an integration tool for developing and testing infrastructure code locally or in the cloud on an isolated, targeted environment. Rubocop is a static code analyzer for Ruby code. It will enforce many standards and guidelines for your Chef Cookbook code. Food Critic is a linting tool for Chef Cookbooks; it comes with 76 built-in rules. Serverspec is a testing framework designed to check that your server has been configured correctly.

Backend

To test our backend applications from server-side, REST APIs, and databases, we use a variety of testing frameworks. Jasmine and Mocha are very similar. Both frameworks allow you to write your tests in the behavior-driven development (BDD) style for testing JavaScript code. Golang has internal testing libraries, and you can easily find additional extensions for testing. We built an in-house schema validator that allows us to create custom business rules to test hundreds of JSON files, and ensures our desktop and mobile application are configured correctly. If you are doing any API development and testing, Postman should be part of your tool belt.

Web Applications

We are using almost every flavor of Selenium webdriver. To have the developers buy into quality, and help contribute to testing, it is important to align your automation technology stack with the web application technology stack. For Javascript applications, we are using NightwatchJS, Polymer WCT, and Protractor. For Python applications, we are using Behave. Ruby is the only framework we are using that doesn’t align with our application stack, but there has been a great success story using Cucumber and RSpec.

Mobile Applications

It’s the same theme for core mobile native apps. We have aligned the technology stacks. The testing framework built around Google Espresso makes it easy for developers and test automation engineers to add test coverage to new functionality. The iOS project is an entirely different story. We have explored everything and continue to find that the iOS testing frameworks are not mature enough. Google EarlGrey has a lot of potential, but the framework needs a lot of love to mature and stabilize. It's a struggle to find cloud testing solutions for Google Espresso, Google EarlGrey, and XCUI Tests. We are only using XCUI Tests for visual testing using iOS Applitools SDK. We have a small footprint of Appium tests for mobile web testing.

Security

The security team is currently in discovery mode to find better solutions than our existing tools. They are exploring tools like Checkmarx, OWASP Zap, and infrastructure code scanning tools.

Performance

We have a solid foundation of performance testing for backend applications using open source Locust.io and Taurus tools. We are in discovery mode to find an open source tool that helps us analyze web application speed and performance. On the top of our list is an open source tool called sitespeed.io.

Exploratory

We use Applause for outsourcing our exploratory testing for the Web and mobile applications. It gives us confidence that our applications are ready to be shipped to production.

Cloud Testing Solutions

The cloud testing vendors use cloud infrastructure to spin virtual environments up and down on demand for software testing. We are performing all web and mobile web testing on Sauce Labs infrastructure. We are only testing Android apps in the cloud using AWS Device Farm, which allows us to test against real physical devices. iOS testing is handled by our on-prem Mac Mini Farm using XCode simulators. Applitools is our new hotness! Applitools is enabling teams to validate user interface (UI) components by creating automated tests with visual checkpoints through testing frameworks like Selenium, Appium, Espresso, and more.

Test Management

Enough is enough with Google spreadsheets to track manual tests. We have zero traceability to specifications, no way to schedule executions, no way to track our automated tests...and more. We are in discovery mode with two test management tools; Zephyr and TestRail. We are seeking a tool that allows us to trace our manual and automated tests to specifications. Stay tuned for a possible blog post on our decision between Zephyr and Test Rail.

Monitoring

We use New Relic to aggregate logs, events, or collect metrics from Chef, GitHub, Amazon, Jenkins, etc. on a rolling timeline, and this information can be handled in many different ways. As part of the monitoring and management of cloud servers, we use Splunk for log management. To communicate schedule maintenance windows or downtime, we use Status Page. VictorOps is more than just alerting, and it provides teams with a virtual environment where they can prepare for, react to and recover from downtimes.

Deployment

The key to deployment success starts by automating the entire cloud infrastructure to be disposable and continuously testing each stage of the continuous integration pipeline. When the code is committed or merged to GitHub, it synchronizes with Jenkins by using webhooks. We use Scalr to manage our servers in the cloud. Scalr sits on top of Amazon Web Services and Google Cloud Platform. Scalr allows us to easily bootstrap our Chef cookbooks as a self-service tool and help control our costs. Jenkins is the continuous integration tool of choice for 90% of the teams at Gannett. The core part of this CI architecture is repeatable and disposable for Jenkins Workers. The Jenkins Workers are configured from our Chef cookbooks, and that allows us to scale up as many Jenkins Workers as needed for building and testing 110 markets concurrently and tear them back down. We have one team using Drone and another team using TeamCity as their continuous integration tool of choice. To manage all of the different packages for mobile, desktop, and anything, we use a tool called Artifactory. Artifactory is designed to be a central hub for all of our packages built from Jenkins, Drone, or TeamCity. The 2017 roadmap includes Kubernetes and Docker containers.

We believe everyone on the team owns quality, and shifting left will enable faster feedback from testing and integrating code. Now the team can find problems sooner rather than later. Making this shift requires daily collaboration and planning to build quality into every stage of the pipeline and understanding all the testing tools that are needed to release an application with confidence. What’s in your testing tool belt? If you’re interested in comparing testing notes and tools, I’d be happy to do so.