blog

Let's talk about testing

Test Automation – What is it Good for…

Much of the software development in website development is done using agile methods. Automation is usually necessary to keep up with the speed.

It’s not really a question of whether automation is useful. Only a few people would rather repeat the same tasks many times every day then to let a computer do them. The question is whether the benefits are bigger then the costs. There will always be costs; automation is an investment. Businesses expect a return for their investment. There are several advantages:

Requiring less manual work to reach the same test coverage

Being able to use testing resources for higher value work improving the user experience

Happier users -> more users

How to get started?

One of the critical parts in building test automation and automation suites is to have the right people to do it. There are other factors too, which I’ll briefly go through in this post. Tools, planning and working methods are other important factors.

Test automation suite?

What is it?

A test automation suite can be many things. It can and will evolve during its lifetime. The word test automation suite, at least for me, conveys a message of something massive, something that took a long time to build, covering the whole site functionality. That’s one definition, but there is no reason not to use a simpler, more agile approach.

A suite can be as small of automating a specific part of a site, for example, top navigation or a registration form. Anything, which needs a lot of steps to cover with every build, is a good candidate for automation. Can’t these things be tested manually then? Yes, but testers are humans too, you know. They will not be happy clicking the same 100 links or entering the 100 different email addresses required for good coverage. Does it matter that the tester is not happy about something. Well, that’s more of a HR question, but for me, it does matter. People need to be motivated to achieve the best results.

And the main reason for doing any testing is to achieve good quality. Saving the time from manually testing the forms and navigation will allow the tester to really do his/her job and concentrate on the more challenging test areas, for example, making sure the user experience of your site’s users is as good as possible.

And to be honest, no one will test all the links manually with every build. It’s only a matter of time before there is a serious problem when something important is missed in testing.

Okay, okay, we need automation. What next?

Requirements for successful automation project?

Here is a list of things that are important for a successful automation project.

Co-operation between development and QA teams in terms of communication about upcoming chances

Time. It will take some time to implement the automation. It will still be worth it, in many cases. Just remember to allocate some time for it. It’s important to keep working on automation iteratively, building more coverage as the site and the development work progresses. There will never be a two to four week period in the end of the project where you can just concentrate on building automation. Really. There isn’t.

Stability. If the site changes every day, it’s better to not start building an extensive suite. Again, start with the areas that are more stable. If not, you might face the chicken and the egg problem. You need a stable site to build automation but the site does not get stable when you don’t have it.

Iterative (agile) approach. Start from a single area in one browser. While doing this, you will see how well the automation works with your site. Once you are comfortable with the automation and how it works with your site, increase the coverage and expand to other browsers and platforms.

Tools. Use a tool set that’s easy to maintain. Use modern tools like Cucumber for writing the test cases and combine it with either Watir or Selenium. Whatever tool you use, make sure you can reuse your code. Cucumber is essentially a collection of navigation commands on your site combined with test cases that can be understood by other people, especially business analysts. They could even give you a hand in the test case creation by documenting the requirements in a format that can be used for test cases. Ideally, when using Cucumber, the requirements are the test cases.

Test automation is a tool to increase your test coverage, do not make it the software development project. You will not be allowed to keep working on the automation for months and months. While it is important to get the automation done properly, there is a risk that too much effort will be allocated for it. Like in any other form of testing, there is a trade-off between the time spent and the results achieved. 80-20 rule applies here too. Test automation is a tool to make your development project more successful in terms of quality and schedule. When you achieve that, you can be satisfied with the results.

The people

Can your average QA person do automation in a maintainable and reusable way? Most likely not. It requires a different skill set then manual testing. Or should I say, other skills in addition to the functional tester skills. A good solution is to have a team where some of the people are good in planning testing and test cases while others have the coding skills to maintain and build the test automation. If the team does not have the people with coding skills, most people start to automate testing with the help of a tool recording test cases. This will in most cases lead to major difficulties in maintaining the tests in the future. It can also be difficult to customize the tests for data-driven testing.

Required competencies

Automation requires development skills. You don’t need to be a high-level C++ programmer to create automation scripts. However, knowing how to implement code in a managed way will be necessary to get the automation done. And it is not only about completing the initial set of automation tests. The software/website you’re testing will typically receive updates every day/week. Some of those changes are going to break your test automation. Even if all the functionality on the site still works, some elements could have been moved or renamed causing the test automation not to find the element anymore. Elements on the page will need to be identified with the element attributes (for example, link text) or XPATH expressions, which are powerful but rigid. This is really the part where good understanding of HTML is required.

Software development practices

Maintaining any software, also test automation will benefit from good development process. One of the most important ones is using version control systems, for example, Subversion or git. It will enable maintaining the current version of the automation and tracking the changes done to it as well as branching your scripts for the current production system and the development version if there are major differences.

Development guidelines

Having a set of development guidelines to support the automation is needed to minimize unnecessary changes to the site. A big part of the technical implementation of test automation has to do with locating HTML elements on the page. It is good to establish naming conventions for the elements, and then stick to it. While it might be / is a good idea to fix naming of an element from something that’s obviously wrong, it can potentially cause test cases to fail. A typical case of this is could be something like renaming an element to something that’s clearer to other users:

Renaming the above unordered list would require the following change to an XPATH expression identifying the element:

//ul[@class='top-nav'] -> //ul[@class='top-navigation']

While this change does make sense, it can easily cause automation tests to fail. The fact that test cases sometimes fail is not really a problem; it’s a fact of life that needs to be managed by allocating time for maintenance work.

Allocating time for automation

The project manager creates a plan for his project. The plan includes time for QA after the development work is completed.

Project of 6 months:

2 month for requirements / user stories

3 months for development

1 month for QA

Now, what happens when development time is increased by 30%. That’s not at all unrealistic, is it? Even with this relatively low delay percentage, the whole time to QA is basically eaten up by the development work. And there is of course a commitment for a customer to deliver on the agreed date. An additional complication is that there are most likely areas that have not been tested before the QA cycle is supposed to end.

The team will find defects on the last week of the project, things that should have been discovered 3 months earlier. Maybe there are areas that cannot be implemented at all with the current architecture.

This can, especially for a new team, lead to a situation where test automation is never taken into use in a proper way. There is not a lot of time, people are not familiar with the tools and enough progress is not made. In such a situation, management can easily loose faith in the usefulness of automation in general.