What are the different types of Test Automation Frameworks?

Test Automation Framework, can make testing a lot systematic and increase the chances of your next career switch. We have been discussing what is Test Automation Framework and why do we need it. The goal is the rapid release of efficient, effective and reliable software, and realizing the benefits of automation testing requires adaptable methods and innovative software tools that optimize data. Test automation frameworks guide this convergence of methodology and technology. The framework provides the user with various benefits that helps them to develop, execute and report the automation test scripts efficiently. Moving on in our pursuit to learn more, this article attempts to cover different types of Test Automation frameworks.

But why do we have different types at the first place? Framework should handle every aspect of automation, so all frameworks should be same, right? Nah! Some of the common scenarios include data-intensive tests where we have large set of Test data, some applications do have multiple common modules to be re-used in different scripts, Or what if you want scripts to be driven using business-application-keywords like click, search, login, etc.? As you might have guessed, automation frameworks vary depending upon the type of Business requirements, Test requirements, and how you want to organize different Testing aspects. Incorporating everything from coding standards and guidelines to test data and results reporting, a well-implemented automation framework offers an optimal combination of coverage, but with minimal maintenance and manual intervention. There is a divergent range of Automation Frameworks available now days. The 6 primary types of Test Automation frameworks are: Linear, Modular, Data-Driven, Keyword-Driven, Hybrid and BDD.

Linear Scripting

Linear as in direct. This is the first step to learning different types of Test Automation frameworks. I.e. scripting. Linear scripting, a first generation testing method, is nothing but directly scripting each & every functional flow within the application-under-test. You learn the alphabets, make words out of it and then start speaking & writing sentences. Simple, right? Yeah! It is very similar to the Record-and-Playback mechanism where every single interaction is captured and detailed out (along with checkpoints and validation steps) in the script/feature file.

Linear Scripting is a good way to quickly get started with some very basic automation. There is no programming or design expertise required to learn the record and playback feature of automation software 🙂 but it is far from scalable. Every action needs to be captured in the script which makes it very verbose. Additionally, the hard-coding restricts its re-usability. The tests lack reusable functions, and the data is a part of the test so tests cannot run with multiple data sets. With increasing coverage, it will result in a rapid increase in the number of test scripts/feature files. And with Agile, the functionality is sure to change on the way. What will you do then? Make changes in each & every script where you have used the changed-interaction? Not possible, right? Yet, it is crucial to understand linear scripting because it forms the basis for all other types of Test Automation frameworks.

Modular (or Structured) Framework

Modular as in sectional & linked. At the most basic level – we need to login to application every time to execute a test. Linear scripting will add code to login at the start of every test script. Now say after few days there is a change in the way user login. What will you do? Change the login code in every Test script? Nah! That’s quite lame design. As you might have guessed, the obvious logic says to write login code at one place and then reuse it at the start of all the test scripts. Now if the login functionality changes, what needs to be done? Yeah! Just change the login code at one place & whoa! All scripts refer to the modified code.

One of the popular OOPs concept called ‘Abstraction’, this particular approach forms the basis of Modular or Structured framework which organizes the code base into reusable routines or functions that can be called throughout the script to build efficiency in code re-usability and maintenance. As the script grows and becomes more complex, the need for creating reusable code becomes more apparent. The modular framework divides the entire AUT into number of logical and isolated modules. And what’s required for it? Yeah! A knowledge of functionalities that will be re-used in different scripts and programming know-how of functions. A simple approach can be to start linear scripting and then refactor the redundant code into re-usable functions as & when required. Re-usable functions result in clean, easy & short scripts with efficient maintenance. Why short? Because we have just replaced the 10 steps in all 20 test scripts (= 200 lines of code) with one function call (= 10 code lines + 20 function calls). That’s a reduction of 170 lines of code (200 – 30). This is just a beginning demonstrating the power of different types of Test Automation frameworks.

If we keep modularizing the application-under-test, what we finally get is a set of re-usable functions that can then be combined to form a Test script. Didn’t get it? Let me explain. Say we write different functions for login, search, add to cart, checkout and confirmation. Now these 5 functions (or modules) can be combined to form a Test script like ‘verifying order placement’. Or any other test script. The basic funda is to divide the application into re-usable modules (decomposition) and then conquer the Test scripts by just using (recombining) the required functions.

Some people call another twist to Modular framework as the Library Architecture Framework where the application is segregated into functions kept in code libraries which can be called within the test scripts whenever required. This mainly involves formulating reusable functions into library files (such as DLLs, APIs, library functions) and these library files can be called directly from any component.

The catch: Test data is still hard coded within the scripts.

Data-Driven Framework

Test Data is an important yet neglected testing concept. Test Data Management is crucial to Test team’s success. A simple text-box can have Test data ranging from numerals, alphabets, special characters, NULL, empty, etc. That way a simple text box script can be run multiple times to validate the input & output data. Every time you need to run a script with different test data, either write a new script or modify the existing script to incorporate the new Test data. How does it sound? Yeah! It sounds more manual than automation 😛 what’s the solution then? Let’s have a look…

Without doubt, the first step is to de-couple Test script & Test data into separate entities. Next obvious step is to somehow link these two. And how do you accomplish it? Yeah! Data-driven framework. De-couple the scripts & data by storing test data into separate files (it can be an excel file, CSV, XML or a database). Next – link these two by using yet another programming concept called ‘Parameterization’.

“A parameter is a special kind of variable, used in a subroutine (or function) to refer to one of the pieces of data provided as input to the subroutine. These pieces of data are called arguments.”

Instead of hard-coded test data, we replace it with a parameter (a variable) that can be loaded (actual data) from external sources. During execution, the test script will fetch this parameter’s value from pre-defined external source & use it for current script execution. The data is conventionally stored in “Key-Value” pairs. Thus, the key can be used to access and populate the data within the test scripts. Say ‘Username’ Key will hold values like ‘Deepanshu Agarwal’ & ‘Kanchan Kapoor’. And data extraction within test scripts is facilitated by a common library. This way we can run multiple instances of the same script using different set of test data. Problem solved! Additionally changes to test data doesn’t require corresponding changes to the test scripts. The data-driven framework offers more flexibility and ease of maintenance. As you might have guessed, data-driven framework requires testers to create (& maintain) test data sources & corresponding data reading mechanisms.

The catch: Once automation framework is in place, writing test scripts still require programming knowledge.

Keyword-Driven Test Framework

What if my Test script is like ‘Keyword Login – to Gmail – with Username zzz – Password xxxx’. Super-easy right? 😉 Yeah! This is not hypothetical. It’s possible. Just that ground work needs to be completed to reap the benefits later. The Keyword driven testing framework is an extension to Data driven Testing Framework in a sense that it not only segregates the test data from the scripts, it also keeps certain tasks (set of code OR Keywords) belonging to the test script into an external data file. The keywords and the test data are stored in a tabular structure. Keywords are self-guiding as to what actions needs to be performed on the application. This framework expands on the OOPs concept of Abstraction by hiding the implementation details of keywords from testers writing the scripts.

What’s the advantage? Once you are done with framework design, test script writing is just too easy. Just make an entry into the script table. What’s required for a script? Keyword (action) – Application Object (locator value) – Test Data. It’s like,

That’s it. My script for Gmail login is ready. Quite easy right? That’s the power of Keyword-driven framework. Just that there needs to be a central repository each for keywords code, object identifiers & the Test data files and how all this will be linked in my Test script.

Keyword: action to be performed on a GUI object. I.e. For a textbox keywords can be Enter Text, Verify Value, Verify Property, etc.

As you might have guessed, keywords should be derived depending upon the nature of the application. And then we can re-use the keywords in our scripts. The implementation and maintenance of such a framework does require programmers who can create the keyword execution mechanism. But this style enables less technically skilled staff to create test scripts. It also improves test script readability since for each test, the functionalities are documented in a table as step by step instructions. It would require some initial investment of time & cost to build a keyword-driven framework but once done, the benefits can be reaped in the long run.

Hybrid Framework

Everyone wants the best of each world. There are times when none of the types of automation frameworks mentioned above offer a best-fit solution. This is where the hybrid option comes into play. Hybrid as in fusion or amalgam, i.e. fusion of all different types of Test Automation frameworks to avail all the individual benefits. The most commonly implemented framework – pick and choose the capabilities of each framework, customized to current project needs and requirements. The goal is to pull on the strengths and mitigate the weaknesses of two or more frameworks. The best feature of a Hybrid framework is leveraging the benefits of all types of Test Automation frameworks.

As you might have guessed – there is no predefined structure to a Hybrid Framework. You can have modular approach and pull data from excel sheets. Or scripting can be keyword driven with certain common functions. Framework can leverage the Page Object model with environment details stored in property files. Or use a common library coupled with a data repository that combines both test I/O data plus action keywords. Although, the beauty of frameworks is that they allow for practically any combination. Depending upon the application & testing requirements – we can mix-and-match different types of Test Automation frameworks. Naturally, a hybrid approach may be more complex to set up initially. However, it could provide the greatest flexibility and highest level of scalability & maintainability if the trade-offs between the frameworks it incorporates are carefully evaluated and implemented.

Behavior Driven Development Framework

Behavior Driven Development (BDD) is a rising methodology to test your code. Behavior Driven Development allows automation of functional validations in easily readable and understandable format to Business Analysts, Developers, Testers, etc. Such frameworks do not necessarily require the user to be acquainted with programming language. In BDD, whatever you write must go into Given-When-Then steps.

Given: provides context for the test scenario about to be executed.

When: specifies the set of actions that triggers the test.

Then: specifies the expected result.

Example,

Given I navigate to the Facebook application

When I try to login with valid credentials

Then I should see that I logged in successfully

It’s like writing the documentation for application functionalities. And what do we need in order to implement BDD? Yeah! A Behavior Driven Development (BDD) Framework. It is centred on the use of a non-technical, semi-formal, natural language for test case specification. This process is facilitated by higher-level tools like Cucumber or JBehave in conjunction with testing tools such as Selenium. Its purpose is to facilitate test specification by business stakeholders as early in the development process as possible. A key drawback to this approach is that BDD scenarios are so abstracted that they are often at odds with technical reality. Furthermore, in practice, it is difficult to get participation from business stakeholders in creating or evaluating BDD scenarios.

Which Automation Framework Is Right for You

In an industry where time and resources are almost always in short supply, test automation stands out as a solution. When designed and implemented with the why, what, and how in mind, Automation frameworks lead to the benefits of test automation. And vice versa, i.e. a poorly designed, difficult-to-maintain automation solution limits the benefits of test automation.

There are a variety of test automation frameworks to choose from, all designed to make code more portable, reusable and easier to maintain. Choosing the appropriate Test Automation Framework depends on multiple factors. Based on your needs, you can evaluate each type and decide which framework(s) work best for your organization and testing. There is no single, cookie-cutter automation solution out there. This gives software teams the opportunity to assemble their own highly effective automation framework tools. The golden rule is to list down all the required components of a framework like Object repositories, Test data, Config files, Environment settings, generic files, program logic, reader utilities, error handling, reporting requirements, build tools and continuous integration. And then decide how you want all this to be organized to produce the desired results keeping in mind technical expertise, test requirements, application-under-test, test data management, re-usability, scalability & maintenance.

This article is not an exhaustive list of different types of Test Automation frameworks, but we covered the most popular styles used now-a-days. An organization could easily begin with a module-based approach and incrementally grow their framework to more complex styles over time depending on development and testing needs and the organization’s technical expertise.

Related Articles

Interview is the most important part of the employment process. It can make or break an opportunity. When it comes to Software Testing, almost all organizations are now looking for Automation engineers, SDETs, Selenium experts, Automation architects and what not. Since Manual testers are finding it tough to land a high-paying job switch, many have started learning the basics of Selenium automation (Yeah! Selenium is one of the most popular automation tool now-a-days). But interviewers demand practical experience. And interview questions reflect that view – starting from basic theoretical knowledge, slowly the interview will move towards – Explain Test Automation framework for your current project.

Selenium WebDriver makes direct calls to the browser using each browser’s native support for automation. It support multiple browsers, operating systems and programming languages. What’s more? It’s FREE, ‘Open Source’ as we call it! Why do you think it’s so popular? 😉 Though Selenium Webdriver is at the epicenter of automated software testing, it is supported by different tools to make Automation Testing easy, effective & efficient – The Automation Galaxy of Selenium Webdriver tools!

Test Automation framework plays a vital role in the success or failure of any software automation project. For testers – it forms the foundation to a successful career. Software Testing involves multiple components like application-under-test, test cases, test data, test environment, etc. and a good automation framework needs to take care of all of these. In our pursuit to explore the topic of automation frameworks, let’s get started with collating different aspects of software testing that are to be taken care when designing a Test automation framework.

About STS

Software Testing Studio is an attempt to share some incredible knowledge from industry leaders & experts, which should be helpful for anybody to start his/her career in ‘Software Testing’ or to progress it further. Apart from the technical nitty-gritties, one can also find some intellectual posts by industry experts sharing their Wisdom.