Top 5 Features Your Test Analytics Tool Needs Right Now!

Enterprises are under constant pressure to improve software delivery cycles while maintaining – or even improving – quality. The only feasible way to iterate more quickly while ensuring quality is to implement Continuous Testing as part of the overall development strategy.

Continuous testing (CT) allows organizations to gather fast feedback per each code change (commit) as part of the continuous integration (CI) workflow. When implemented properly, CT can help organizations attain true DevOps status. The only real downside associated with CT is the vast amount of test results data that is generated with a high number of commits – and subsequent volume of test cases – throughout code cycles.CT Produces Significant Amount of Test Results DataWhile testing is a critical component in building a successful DevOps organization, the more tests executed across a large array of platforms (mobile devices, tablets, desktop browsers), the more test results data is created. Analyzing, understanding, and filtering the results data quickly is critical in order to prevent bottlenecking the DevOps process.

In a DevOps environment, the window to review and qualify test results has shrunk dramatically from days to hours (and even minutes in some cases). What’s no longer acceptable is a situation where a regression suite is running for 2-3 days and requires a similar amount of time to analyze results.
Our research at Perfecto tells us that organizations spend between 50-72 hours per regression cycle analyzing test results, filtering out noise and assessing failures which may impact their software releases. It is essential that teams have a better way to analyze data, triage issues, and act upon failures with the best possible insights. Unlike desktop browser testing, mobile devices create more complex issues as failures can be caused by network connectivity, device locks, improper object use, popups and more.

The Essential Toolbox for DevOps Analytics:When dealing with large test results datasets in a CI/DevOps environment, there are a few essential tools that teams need:

Executive dashboards that include quality heatmaps, continuous integration (CI) dashboards. In the screenshot above, users such as dev managers and QA managers can easily examine the pipeline, see CI trends related to time, build health and more. Upon finding an anomaly in the CI pipeline, managers and practitioners can easily drill down into the single test report and into the issue.

Single test report visibility with advanced reporting artifacts (Videos, screenshots, logs, access to Jira and Git, etc.)In the single test report, practitioners have access to the entire flow of test steps, video, logs, screenshots as well as access to Jira – even the ability to drill down into the source code to easily locate the issue.

Cross-platform reports show the UI/UX simultaneously across multiple screen sizes, resolutions, and platforms. Since the boundaries between the digital platforms need to be seamless, whether on desktop browsers or mobile devices, having the ability to see the UI/UX on multiple form factors (and layouts) at the same time is essential to assess overall quality.

Noise reduction tools powered by AI and analytics allow teams to filter out issues that are caused due to various issues that are NOT software defects – things like device connectivity issues, wrong scripting (bad objects, timeout issues, popup handling, etc.). Finding which error classifications are true application bugs is critical to efficiency. Having a tool with the ability to locate each failed test execution, the root cause, and the classification category is a huge productivity boost for both the test engineer and the developer. Things like popup handling, “element not found”, device connectivity etc., are among the top issues that slow down defect resolution activities.

Actionable insights with prescriptive ways to resolve issues. Once an issue is detected, classified and reported, having deep insight (see screenshot) brings practitioners halfway towards the resolution of the issue. In the example above, the error is due to a popup.

Bottom Line

To excel in software delivery for mobile, native, or responsive web apps, teams not only need better automation as part of their processes, test flows, and CI/CD workflows, but also a test results analytics platform to manage all of their test results data in a way such that the teams can evaluate the data, act upon it, and deliver iterations and features with confidence – and quickly!

Since DevOps involves a collection of team members from all parts of the SDLC process, the central platform needs to meet the needs of ALL team members.

Find this useful? Share it on...

Eran Kinsbruner is the Lead Technical Evangelist at Perfecto and Author of the Digital Quality Handbook and Continuous Testing for DevOps Professionals books. He is formerly the CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at ADT, Sun Microsystems, General Electric, and NeuStar. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and on his professional mobile testing blog at https://continuoustesting.blog/.

Our website uses cookies to give you the best online experience and provide functionality essential to our services. By clicking ‘Accept’ or by continuing to use our website, you are consenting to our use of cookies in accordance with our Privacy Policy

Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

You can adjust all of your cookie settings by navigating the tabs on the left hand side.

Strictly Necessary Cookies

Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.

disable

If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.