Layered Architecture for Test Automation

Abstract

In test automation, code involved in testing is not only test logic, but also a bunch of other supporting code, like url concatenation, html/xml parsing, UI accessing, etc. Test logic can be buried in this unrelated code, which has nothing to do with test logic itself, making test code hard to read and maintain. In this article, the layered architecture of test automation is presented to solve this problem. In this layered architecture, the test automation code is divided into three layers: (1) test cases, focusing on the test logic of the application, (2) the domain layer, modeling the system under test in domain terms, encapsulating http requests, browser control, result parsing logic and providing an interface for the test cases layer, (3) the system under test, which layer 2 will operate directly on.

Problem

QA's job includes designing test cases, exploratory testing, performing regression tests, etc. While some tasks like exploratory testing require intuition and smarts, some others, such as regression tests, are repetitive and laborious. As more features are added to the system, time consumed by regression tests gets longer and longer.

Test automation solves this problem. With test automation, repetitive work like regression tests is done by computer, and tests cases are translated to computer program, so that QA can be freed from the burden of routine test repetition to focus on more creative work.

In test automation, code involved in testing is not only test logic, but also a bunch of other supporting code, like url concatenation, html/xml parsing, UI accessing, etc. For example, to test a web service which carries out operations like search by different keywords and return an xml containing certain information (like customer information), the test automation code must:

Assemble a URL based on the operation under test,

Send out a http request with some http libraries,

Interpret the response sent back from the web server and parse the xml,

Compare results returned to expected results.

In some test automation code, all this URL concatenation, html/xml parsing, XPath expression and test logic code gets written together, usually in one class or one method.

This form is easy to pick up and is intuitive initially, but it has its problems:

Test logic is hard to understand and modify. When test logic is embedded into a large amount of other unrelated code, it's difficult to see what is tested. To add new test cases, one often has to reread the supporting code and find out where is the best point to add new code. Test logic becomes hard to understand too.

Tests become fragile. Since test logic and supporting code like html parsing are mixed together, one small change in the 'contract' between the system under test and the test automation code can break the test automation. For example, if the UI changes, like moving an input element to a different div tag, or changing an ID of some UI element, all test code operating this part of the UI is affected.

Maintenance cost is high. There are generally several test cases for a particular part of a system, and a large section of each test case is similar. For example, they may all have to (1) assemble a URL based on the operation under test, (2) send out a http request with some http libraries, (3) interpret the response sent back from the web server and parse the xml, (4) compare results returned to expected results. Since this code is duplicated in all of the test cases, if anything changes, you will need to modify each test case.

Solution

The domain of software development has experienced the same thing, and developed a solution, that is 'Layered Architecture'. Basically, the value of layered architecture, to quote Domain-Driven Design, is:

"The value of layers is that each specializes in a particular aspect of a computer program. This specialization allows more cohesive designs of each aspect, and it makes these designs much easier to interpret. Of course, it is vital to choose layers that isolate the most important cohesive design aspects."

Though the focus is different in the domain of test automation, the fundamental problem is the same, so a similar solution can be applied:

Test Cases Layer

All (and only) test logic resides here. Test logic can be expressed concisely, with the help of the layer below. Test cases for different stories, scenarios, and corner cases rely on the same piece of code in the layer below, the only difference is in parameters or test data representing different cases.

Domain Layer

This layer will encapsulate operations to the system under test, like url concatenation, response xml/html parsing, rich-client GUI/browser control, etc. It will present the system under test in domain language, rather than in terms of xpath, sql, or html.

System Under Test Layer

Well, just the system being tested.

The test cases layer consists of multiple test cases. These test cases are based on the domain layer which encapsulating system under test in domain terms.
The Domain Layer accesses system under test directly.

Example

Say we are testing a restful web service. With this web service, you can search for some customer information, with telephone numbers as keyword.

To call this web service, the get http request in the following format should be sent out:

http://{endpoint}/subscribers?telephoneNumber={telephoneNumber}

The piped data returned contains the subscriber's name, phone number, address, and other information:

13120205504|ST|C|SQ|112|||FIRST|ST|W|Riverfront|BC|010|68930432|

Test cases for this service are (1) Search with a phone number which has an exact match, (2) Search with a phone number which has several exact matches, (3) Search with partial phone number.... The number of test cases is only limited only by the imagination of QA.

For each test case, the process is essentially the same: (1) assemble a URL containing the telephone number keyword, (2) send http get request with http library, (3) parse the piped data, (4) compare the data received with expected values. To avoid the problems mentioned before, we apply the layered architecture:

Test Cases Layer

The implementation of this layer is test framework related. In this example, we are using C# and NBehave (but not in a traditional way, please refer to the post 'Fix NBehave').

These test cases are written in C#, but but closer to plain English, so that they are business-readable (please refer to Martin Fowler's BusinessReadableDSL ). With this format, other roles, which have deeper understanding of business domain, can spot missing scenarios or cases.

It's even better if a framework which supports plain text test cases is picked up, like Cucumber in Ruby.

Variables with 'ACTION' suffix are lambda expressions. They give this code life.

SEARCH_WITH_ACTION is for sending the request to the web service and parsing the piped data returned. The code for CustomerService and Subscriber is in the domain layer, because this code is common supporting code for a variety of test cases.

Domain Layer

Class CustomerService is named after the real name of this web service. In the requirements document, daily interpersonal conversation, architecture map, and code, this web service is referred to by the same name. With this unified domain term, the ambiguity is eliminated. For a more complete introduction, please refer to the post Domain Based Testing

With this domain model, data verification can be carried out based on the object. For example, you can verify that the first name is 'Bei'

Assert.AreEqual("Bei", subscriber.FirstName);

Or the phone number starts with '010'

Assert.IsTrue(subscriber.TelephoneNumber.StartsWith("010"));

See the attached test automation source code for a complete working example illustrating the layered architecture. You can open it with Visual Studio 2008, or run it from the command line. You can execute ‘go.bat’ to run the example test, and test results are in the ‘artifacts’ directory. The solution source code consists of three projects. The project with ‘Client’ suffix contains the Domain Layer. The project with ‘Client.Spec’ suffix (spec here is short for specification) contains tests driving the development of this layer (with TDD). The project with ‘Stories’ contains the Test Cases Layer. This source code is tailored from a real project, so you can see some directories containing only one file which is overkill, but necessary if there are more files. Also some classes are returning hard-coded values, just to disconnect from the real system.

How does this solves the problem?

Problem: 'Test logic is hard to understand and modify'. Since we have a separated layer focusing only on test logic and making use of supporting code from the layer below, test cases can be expressed in a way that is similar to English, thus, difficulties in reading, reasoning, and modifying test code depend more on the coder's English skill than the code itself.

Problem: 'Tests become fragile'. Since we now have a domain layer isolating the test cases from the real system under test, any changes in the system, can be propagated solely to this newly added layer. If we change the code in this layer accordingly, tests cases depending on this layer can still run.

Problem: 'Maintenance cost is high'. Thanks to the encapsulation in the domain layer, duplicated code is removed from test cases and you only have to modify one piece of code. Also, since the services and domain models are modelling the system under test, the code is easier to understand and modify.

Frequently Asked Questions

Q: This solution seems complex, do I have to use this?

A: It depends on the size and complexity of the system under test. If the system is very small, and business logic is simple enough, this way is overkill. In this situation, even test automation is a waste of time. If it only takes a couple of minutes to manually test the system, why bother automating tests? For moderately complex systems, mixing test and supporting code can work. If the business logic is complex, I prefer layered architecture.

Q: This architecture requires an investment before real tests can be started, is that wasteful?

A: This is just another way of organizing code. Even if test code isn't organized in this way, code to perform url concatenation, xml/html response parsing, and result verification must be written anyway. With this architecture, you just have to break the code into different classes/methods. Plus, you don't actually have to fully implement these layers all at once. These layers are scenario-driven and test case-driven, and can be implemented as needed.

Q: Designing this requires substantial object-oriented experience, not all QA can do this.

A: I would say test automation is not only QA's responsibility. Other team members, including developers, should contribute.

About the Author

Li Bei (and Bei is his given name) is working in ThoughtWorks as a consultant. He is mostly interested in Domain Driven Design, Test Automation and Domain Specific Language. Bei would like to thank his coworkers. Working with them is always thought-provoking and enjoyable.

The idea of a helper layer for testing is great. Not sure if call it just domain layer, since it is not clear which domain it is referring to: app domain, or testing domain?

If testing domain, objects in that layer could be things like test, suite, etc. Clearly it is not (due to the example). In that case it is an App Domain, test helper layer.

Then, I have there another concept that itches. Why do we need a redundant domain layer to test app? Could it be the App is not exposed following the bizz domain, but a more obscure IT domain? Why can't I build that helper domain as an actually usable interface for app use?

Hi William, The 'domain' here is about concepts in testing, not application implementation. For example we test google search. We can have a domain object called GooglePage, with one method Search(keywords), sending out http request to google server. This encapsulation makes test purpose more clear. Think only in terms of black box testing, nothing to do with application implementation detail. They are independent.

I am seeing a cobbled together set of helper methods (and from your descriptions they seem extremely procedural).

I understand the benefits/need of these types of helper classes and I agree that they will help in the legibility of tests and help to reduce duplication. I cannot however justify this being called a "domain". At best I would call it a facade en.wikipedia.org/wiki/Facade_pattern, I believe its intent is much more in line with what you are doing.

Hi William, The 'domain' here is about concepts in testing, not application implementation. For example we test google search. We can have a domain object called GooglePage, with one method Search(keywords)

If it is about testing concepts, then we should go with entities like test, suite, assertion, etc. Those are from the testing domain.

If we go with GooglePage, and Search, we are talking about the application domain, since those are concepts from that domain. In this case, we can assume, the Google API does not provide an easy to implement Search interface, thus for testing we need to create a helper class. And that is perfectly fine. Still, I can call that an indirection layer, or facade, or why not, an adapter. If depends on what that layer does.

That is, the domain identification comes from the actual conceptual set, not from the use. You can have elements from one domain used to do business, and some others from the same domain used for testing.

Have a look at Concordion for writing test cases in plain english - www.concordion.org/It has proven an invaluable tool for the core banking system we are working with bridging the gap between business analysts and developers.We also use the pattern for writing business readable code - but more as a way for a developer setting up a test case in which required business objects need to be created prior to the execution of the Concordion test case.

Hi Greg, you are right, they can be called facade (or gateway martinfowler.com/eaaCatalog/gateway.html), from implementation perspective. There are different ways to implement this facade, for example: (1) HttpUtils.Get(url) or (2) CustomerService.SearchWithTelephoneNumber(phoneNumber). Both of them use http library to send out http request and read the response, and expose a simplified interface. The first one is built in http technical terms, while the latter one is in business domain terms. I prefer the second one since it focuses on what to do, rather than how to do. Thus the 'domain layer' is used here to emphasize using domain terms to build this layer. Hope this can dissipate the fog.

This is interesting. It's funny because I was blogging about the Layers pattern and unit tests at about the same time that this appears. I certainly agree that the Layers pattern can benefit unit tests. Here the author is talking about Layers in another dimension.

I've used this architecture for testing a rich Web client (written using ExtJS) from Java using Selenium RC, which lets you fire commands from the likes of Java at a remote agent that controls a browser. At the System-Under-Test layer, I have a Java proxy for the remote document.

A lot depends on the kinds of "help" the interposed layer provides. In my experience, intervening layers, while convenient, can limit the kinds of tests we can design. It's usually best for tests to operate the interface directly.