Hybrid Tests: Blurring the lines of the Automation Pyramid

Hybrid Tests: Blurring the lines of the Automation Pyramid

Mike Cohn’s Test Automation Pyramid is a guide that is frequently cited as a good practice to follow. It suggests that:

the bulk of your automated tests should be unit tests

the next largest group of tests should be developed to target the service layer which includes the web services and business layer of your application

there should be a very limited amount of UI tests, as they are more brittle and take the longest to execute

While it seems that just about everyone agrees with this in theory, I don’t see a lot of teams following this guide in practice. When there’s a new UI feature to test, we’re naturally inclined to automate those scenarios at the UI layer.

I’m with you! However, know that you don’t have to live in just one section of the Automation Pyramid. Just because you feel the need to test the feature at the UI layer doesn’t mean that every single step of the scenario has to be done there as well.

Let’s look at a few features and I’ll illustrate how to blur the lines within the pyramid. I’ll be using the ToolsQA site to demonstrate.

Feature 1: Search CatalogAs a user, I want to search the catalog so that I can find specific products

For this feature, the scenarios might include lots of different ways to search for a product. While this could certainly be done at the service layer, I’d feel a lot more comfortable automating this at the UI layer to ensure that the display of the search results (or error messages) is accurate. This is especially true for more advanced search result grids that allow for filtering, sorting, column organization, pagination, etc.

However, if you’re doing a ton of different search tests, it might make sense to do a few different variations on the UI, but write unit tests or utilize a Search service call for tests that are essentially exercising the same functionality but with different data.

Feature 2: Add Product to CartAs a user, I want to view the details page of a product and add the product to my cart.

For this feature, the main test would be to verify that the user can add a product to the cart. The steps involved in the scenario would be:

Search for the product

Locate the product within the search results

Click on the product

Click the Add to Cart button

Click Cart button and verify that the product is in the cart

In this scenario, you definitely want to make sure that the Add to Cart button works. Because of this, most people would concede to the fact that this scenario has to be automated on the UI. But if you evaluate the steps carefully, you’ll realize that step 4 is really where the testing begins. Everything else can be done outside of the UI.

From the work you’ve done on Feature 1, you already have automated scenarios for searching for a product, so it’s really tempting to reuse that framework for this test. Don’t do it! For one, it’ll be slower to execute this on the UI when in actuality, you don’t even have to. And more importantly, it’s not relevant to the test.

Let’s say there’s a regression with the Search feature and searching is now broken. If this Add to Cart scenario included Search, this test would now fail at the very first step, which means you’re losing the ability to use this test to verify what it’s actually intended to test: adding something to the cart.

Now, for this particular scenario, I’d still use the UI but I would utilize shortcuts. If I click on any product, I notice in the browser’s address bar that there’s a unique URL for each product. In this particular demo application, the unique identifier is the category + name. Your application might list the product’s ID. Knowing this, you can use your browser automation tool to go directly to your URL as opposed to navigating your way there through several steps.

We’ve now eliminated steps 1-3 and replaced it with a simple “Go to” and have eliminated the dependency on the Search feature. An additional benefit is that the test will run much faster and be less brittle.

Go to the product’s URL

Click the Add to Cart button

Click Cart button and verify that the product is in the cart

Feature 3: Remove Product from CartAs a user, I want to remove an item from my cart.

For this feature, the action happens in the cart. Yet, there are several steps that must occur before this can be tested.

Search for the product

Locate the product within the search results

Click on the product

Click the Add to Cart button

Click Cart button

Click the remove button next to the product and verify that the product is removed

The last step is the only thing that needs to be verified via the UI. We’ve already seen above how to eliminate steps 1-3. For step 4, you have already verified adding products to carts via the UI, so no need to do that yet again in this test. Instead, use a web service to place the item in the cart. The easiest approach would be to ask your developers if such a service exists. However, you can also find this on your own by inspecting the Add to Cart button in your browser’s dev tools.

The button will be an input element within a form. That form has everything you need to craft your service call. The action and method attributes on the form, along with the input name/value pairs are what to look for.

This will add the item to the cart, which eliminates the need for steps 1-4. We can also replace step 5 if we’d like to. Instead of looking for the cart button and clicking it, we can simply go to the cart’s URL. Again, faster and less brittle!

Add product to cart via web service call

Go to Cart url

Click the remove button next to the product and verify that the product is removed

If you’re worried about your UI not getting enough coverage, don’t. Notice that in all of the shortcuts in this demo, this same functionality was already exercised by some other automated scenario. So there was really no benefit to doing it again in another scenario. In your own projects, if you don’t have UI coverage for something that you think is important to cover on the UI, then go ahead and add it in, but once is good enough. For all other scenarios around that same area, utilize your application’s shortcuts. Your execution times and flaky test police will thank you for it.

Angie Jones

7 Comments

This is an insightful post. I’m currently not doing this, and my UI tests are slow, because for every test I have to start a new WebDriver, login, and navigate to the page under test. I want to keep separate WebDriver instances, but that “shortcuts” idea really rings a bell.
I thought I should avoid shortcuts and just use the UI, like a real user, but I get your point, that was already tested on some other check, so it’s just adding more time overhead. Cheers!

Xavier

Hi,
Nice article 🙂 However, when you arrive at this last scenario “Add product to cart via web service call, Go to Cart url…”, I think that the scenario is becoming too technical for the Business People to understand. How do you manage to keep them on board ? Do you have any insight on this ?
Thanks !
Xavier

Nice article, and a great primer to start thinking logically about testing to minimise UI interaction. After reading this and a recent tweet by the Friendly Tester I am attempting to find the ‘seams’ in our systems to improve how we automate our own testing.