Running SpecFlow Acceptance Tests in parallel on BrowserStack

Automated acceptance tests play a vital role in continuous delivery. Contrary to unit tests though, they’re quite hard to get right. This is not only because end-to-end testing is harder than testing single units, but also because of the way they need to be executed. You need the a fully working version of the application under test and a client that represents a real-world scenario. When you apply this to a browser based app, things can get complicated.If we want to test our website fully, we need to be able to test it on a variety of browsers, devices and screens. Building all this infrastructure is very costly and time consuming. But new companies such as BrowserStack and SauceLabs have a solution for that: they provide VM’s for all kinds of browsers, configurations and emulators. We can make use of their infrastructure by running our acceptance tests on their resources.

Another problem acceptance tests usually pose is that they are slow (in comparison to unit tests). By carefully maintaining your tests, you can get them as fast as they can possibly be, but they are slow by nature so there’s only so much you can do.

In this post I’ll work through an example on how to run a SpecFlow test remotely on multiple BrowserStack VM’s at the same time. These are the goals:

No need to build/buy infrastructure

Run tests simultaneously, so that adding configurations doesn’t slow down the process

Make it easy to add configurations

Make tests executable from the command-line, by anyone, at any time

1. The tests

As an example I’m going to use a simple test that executes a google search and then verifies whether results were returned:

Feature: Searching google Given I am on the google page When I search the web Then I get search results

Note that here I’m accessing the UI immediately in my step definitions. This is done for simplicity’s sake. In a real-word scenario, you probably want to use some indirection through the use of Page Objects. For more information on how to create maintainable specflow tests, you can refer to my article about Automated acceptance tests with Cucumber

2. The driver

To drive the browser I will be using Selenium (as seen in the example above). Selenium can drive the browser through an instance called a WebDriver. There are a few different WebDrivers available for IE, Chrome, FireFox, … but there’s also a driver called RemoteWebDriver. This is a driver that can drive a browser on a different machine. This is the one we will be using.

Nevertheless, we don’t want to tie any of our tests to a particular type of WebDriver. Luckily they all implement the IWebDriver interface.

2.1 Instantiating the driver

If we could use constructor injection, we could inject an instance of an IWebDriver into our step definitions. Unfortunately, SpecFlow doesn’t allow this. The next best thing we can do is use a Service Locator pattern. SpecFlow provides a Dictionary-like object called a ScenarioContext, which contains objects available to the current scenario. In the example above you can see that upon instantiation of the GoogleSteps-class, we get the driver from this dictionary. The next bit of code shows how we set up the driver at the beginning of a scenario:

Through the use of some SpecFlow hooks, we create a driver before each scenario and tear it down after the scenario finishes. In this example we just created a local FireFox driver. At this point we can execute our test and it will run successfully in a local FireFox instance.

2.2 Configuring the driver to use browserstack

The next step is executing the same test, but on a remote machine managed by BrowserStack. To do this, we need to create a RemoteWebDriver, and configure it accordingly. When you create a RemoteWebDriver, you need to provide two arguments:

The Url of where the RemoteDriver will accept commands (this is provided by BrowserStack)

The capabilities: this is a loosely typed dictionary of parameters that will be sent to the RemoteDriver. In this case, BrowserStack dictates which parameters you need.

So first of all, let’s create an instance of the DesiredCapabilities-class and set the properties accordingly:

The driver is still of type IWebDriver, so we don’t need to change anything to our steps. Provided we have entered the correct username and key, we can now run our tests remotely on a VM managed by BrowserStack.

2.3 Allowing BrowserStack to access local resources

In the above example, I’m only accessing a public website. Chances are that in a real world scenario you will be testing a QA environment or even a dev environment on your PC. These are usually not publicly accessible. To allow BrowserStack access to these resources, they allow you to set up a tunnel through your PC and then access the resources from your PC. To set this up, you need to:

3. Running tests from the command-line

The tests can now be ran from Visual Studio. To run these tests from the command-line, we can use the unit test runner. In this case I have used NUnit as the underlying test framework, so I’ll be using the NUnit runners. To run the tests from the command line you run the following command:

4. Running tests in parallel

Now that we have the tests running in BrowserStack from the command-line, it’s time to start running them simultaneously on various configurations. In order to run them on various configurations we need a few things:

There’s one problem with this approach: when we run NUnit through the console there’s no way to pass parameters to the tests. The general approach given on various forums is to set environment variables. Since we are running the tests simultaneously, this doesn’t help us. There’s one thing however that lets us parameterize how we run the tests: NUnit allows us to specify the configuration we want to run the tests with. So I have come up with the following approach:

Use one solution configuration for every configuration we want to test

Extract the os, os_version, browsername and version to the configuration

Use config transforms to vary these parameters based on the solution configuration

Use Powershell to run various instances of the NUnit-console process with different configurations

4.1 Creating different solution configurations

For each configuration we want to test, we will add a different solution configuration. Usually I start by deleting the standard release configuration and renaming the debug-configuration to something more sensible (eg: Win8Firefox33). You can add as many solution configurations as you want. It’s best to copy them from the original Debug-config as this will have all the settings necessary to be able to debug your code.

4.2 Extract the parameters to the configuration

We also need to change the capabilities to fetch these values from the config:

4.3 Add config transforms

When you have created the different solution configurations, you now need to add a config transform for each configuration. This is an example for my Win8Firefox33 configuration (note that the name is not important, it’s merely a convention)

This snippet retrieves the configurations and it start a job for each one. When all jobs are started it waits for all of them to complete and then receives and writes the output of all of them to the console one by one.

5. Reporting

Once we have run all our tests, we want to report the results. First of all, you can look at your tests performing live via the BrowserStack website. This is great and allows you visually verify any errors if the occur. It also shows you a complete log of everything that has happened in the scenario. The image below shows what you can see while the tests are running:

You can see in real-time which tests are running, how far they are and even follow on-screen what they are doing. That’s pretty awesome for live debugging and monitoring.

Apart from live debugging, we also need to create reports when tests have finished so we can check them afterwards or even archive them for later review. To do this, you can run the specflow tool:

7. Demo project

In the code above, I have omitted a few things for brevity’s sake. In order to be able to run msbuild, the nunit console runner and specflow, you need to ensure that all the paths are set up correctly (ie: added to your environment variables). Therefore, I have created a full working implementation, which you can find on GitHub.

To run it you need to do the following:

Clone the repository to your PC

Create a trial account on BrowserStack

Go to Account => Automate and copy the UserName and Value into the app.config