Visualization: Your secret weapon in testing

In God we trust; all the rest we test.

That phrase has become the holy grail of modern software development, including to me and my team at Wix.com. We test units, components, their integration, API(s), and even our own quality assurance (QA) processes.

We no longer can say that we trust our code or even our eyes, but we still get most of the information on the Internet (also during testing) visually. So why not integrate visualization into your testing tool? That's what we did.

Here are a few practical tips, based on our experience, on how to use visualization to make the day-to-day work of QA professionals, automation, and front-end developers easier.

Visualization as a tool of manual QA

The early involvement of QA in the development process produces a higher-quality product. At Wix, our QA people start working with us at the ideas/mock-up review stage, they write all the testing documentation, and eventually check the quality of finished features and report any bugs they found. Visualization is a great help in every step along the way. Here's how to do it.

1. Visually review future products—mock-ups, ideas, tech specs

It’s important for QA to have a good imagination—it helps to invent new ways of breaking things, to find new test cases and test scenarios, and to reproduce strange and impossible bugs. But this imagination shouldn't be used to envision the future product out of text, documentation, and tech specs.

Coming to the review, QA should already be armed with product sketches. This will free your mind to think about testing, rather than trying to imagine how the product will look.

When preparing for reviews:

Check the documentation with a textual description of the future product and draw your own primitive mocks of how you understood it from the docs. Ask your product/project manager about real mocks and compare. The drawings should be similar.

Check the mocks without reading any description of who will use this feature and how. Is the functionality clear to you from the mocks? Is it obvious how to use it without further explanation? How could you improve it in order to be more clear and native?

Make interactive mocks. Then you can concentrate on users scenarios, product behavior and tests you want to perform without thinking about switching images or slides. Our UX managers use our own Wix Code to produce interactive mockups.

Study your users. If you know that 80% of your users are on 15-inch screens and your mock looks overloaded with information on a 24-inch display, what do you think the experience will be like on a 15-inch laptop screen?

2. Visualize your testing documentation

Here are two tips:

Add mocks to your test plan and test cases. It will help new team members to understand your test docs, and remind you down the road why you wrote the set of tests in the first place.

Visualize test coverage. You can either highlight areas on mock that are already covered by tests or just count how many tests cases you already wrote out of your test plan.

3. Add a visual element to product testing

Here are a few things I like to visualize:

Checking links on the page. I use an add-on for Chrome called Check My Links. It highlights working links in green and broken ones in red. This helps me check that a page doesn't have broken links, and that the page is not overloaded with links.

UI testing. For this I use the Applitools Eyes Express Chrome add-on for comparing screenshots. You can create a screenshot of the baseline (how your product looks like right now, for example) and then each consecutive screenshot becomes a check of sanity/regression testing. It will highlight all the UI changes on the page compared to the baseline you have.

Alignment and sizes. Occasionally, there can be visual distortion on the page, which is usually caused by different font sizes, button sizes, shades, animations, and so on. To check that the layout is aligned properly, use a ruler such as the Page Ruler add-on for Chrome.

4. Reporting bugs

As a QA engineer, I hated it when developers were closing my bugs with "cannot reproduce" status. As a developer, I almost never read the text description of the bug. Title and screenshot or video were usually all the information I needed. That doesn't mean you should stop writing steps to reproduce and expected and actual results, but it does mean that you should:

Always include screenshots in your bug reports. Primary requirements for a screenshot tool are that it should have hotkeys to make screenshots really fast, it should automatically upload screenshots and provide a link that's easy to share, and it should have a small editor with at least crop and highlight functions. For this purpose I’m using Jing.

Consider adding videos to your bug report. It should be very concise and descriptive. I suggest using videos for cases where you cannot reproduce the issue every time, as well as for animations, shadows, and problems triggered by mouse-overs.

Show developers the issue on your computer when the bug can be reproduced only in a specific environment (OS/version of browser/screen resolution). Invite the developer to your office or use remote desktop tools such as TeamViewer or Chrome Remote Desktop.

You can use visualization in manual QA in every step of the quality assurance process, from planning to documenting, testing, and reporting bugs. And, as with any other tool, at some point you might want to automate the process.

Visualization in automation testing: It's complicated

Using visualization in manual QA is fairly straightforward, but it's more complicated with automation testing.

You usually run automation tests on remote computers in the cloud or elsewhere. This means you cannot easily visually inspect the outcome in the same way you would in a browser on your local machine without decreasing the performance of the tests. This makes it more difficult to understand what caused a test to fail and what should be done to remediate it.

Moreover, the sheer number of OS/browser/platform combinations make visual inspection even harder. So how can you automate testing of the UI if you need it to be cross-browser, cross-platform, both desktop and mobile, and probably with a multi-lingual support?

You should use visualization for two main purposes: Testing the product or feature and debugging the test. (If you want to check the tools I'm using, know more insights and tips and tricks, or see additional live code examples, see my talk at SeleniumConf Berlin. The presentation will be live-streamed, and available on YouTube following the conference as well.)

Visualization as a tool for developers

As a part of the TDD (test-driven development) process, I’m writing all kind of tests—unit, component, integration, end-to-end and, from time to time, making some manual smoke tests. Here's how you can add visualization in four key areas.

Unit testing

If you test units you don't usually need to visualize the unit you're testing. That's why the only thing I'm using visualization for is test coverage. There are a lot of plug-ins or built-in features in most of the popular IDEs. For example, I use Webstorm, which has a built-in code coverage test with a nice visualization of the result.

Component testing

If you're using component testing, you already need to render a component in order to test it. I use the open source TurnerJs library for rendering Angular components and directives. It has two main functions:

renderFromTemplate, which renders the template, creates new scope and applies the changes (running digest);

findByDataHook, a built-in selector you can use to query an element using the data-hook attribute. It allows you to see your rendered component, find parts of it by data-hook, and performs tests on them. If you use these tests, you won't need to write a lot of integration or end-to-end tests, saving you the time of running a build.

Integration and end-to-end tests

This is somewhat similar to the second part of this post, because we’re using the same tools for automation testing (in a production environment) and developers' integration/end-to-end tests (in local code).

The important point here is that the more tests and more functionality in your tests, the longer your build will take to run. And for development, the duration of a build can be very important. That's why I suggest adding visual tests as a part of your regular tests and running them in multiple threads.

Smoke manual tests

The most commonly used method to visualize your local changes is running a Web server that serves your static files on your local host. But there are cases when your product or feature is closely coupled to other products in production and that's either difficult to mock or you lack time to do so. Here are few suggestions to easily and quickly visualize your local changes over your production environment:

For small changes just use Chrome dev tools and override function or variable from the console.

For changing a whole file or few files, I prefer to proxy production requests to my local files. I do that using a HTTP proxy / HTTP monitor / reverse proxy tool called Charles from XK72. It can catch any request and point it to your local path.

In situations where the change is big and fundamental (like switching to new, major release of Angular), we’re using our own open-source experiment system. It allows us to push both old and new versions to production, but expose the changes only to developers. Then we run a full cycle of manual and automation testing, A/B test it for a period of time, and eventually switch to it fully.

Start small as you build up your visualization toolset

Visualization is an important tool in software development process that different teams can use at every stage in the product lifecycle. There are many opportunities to include visualization in your development process, starting from small things such as testing links on a page, all the way up to full automation coverage of that new UI you've been developing.

Think about visualization as a separate testing tool. Add it to test plans, plan time for it, and review use-cases to improve test efficiency, because, as with any other tool, it should be fast and easy to use. Remember, even the simplest tool can empower people to do great things.

Want to know more? Attend my presentation, "Screenshots in Automated Testing: When? How? Why?" at SeleniumConf Berlin. The presentation will be live-streamed, and will also be available on YouTube following the conference.