Software tester, tea drinker.

Tag: cucumber

Recently I have been diving into optimising performance of selenium cucumber tests, as there’s no single “make the tests faster” method I thought it’d be interesting to document all the different approaches I took (partly for my own future reference, and partly for anybody else who may find it useful.

Some of these are common sense, some of them don’t relate to code at all, some of them may be controversial, but at the end of the day YMMV, these are what worked for this particular project:

1) Reduce the number of tests

Ultimately the biggest performance improvement came from something this simple. An initial pass over the entire test suite revealed there were a number of tests that were either duplicates in some subtle way, redundant as they weren’t doing any more than a unit test, or otherwise unneeded. Pruning out these tests not only had the benefit of reducing total time taken for a test run, but also made reading the test suite as a whole much easier as they were now cleaner and only contained what was required.

2) Reduce redundant asserts in test steps

Again not really related to code, just common sense which often gets lost as a test suite builds up when there hasn’t been the time to go back over and rationalise everything as it grows. In the tests on this project there were many instances where we would assert something in one step and then assert the same thing again in a later step. Of course some of these were valid, but many were not, they were simply left in because the author didn’t realise that particular assert had already been done. There was also an example in this particular project of certain asserts being performed every single time a call was made to a page object; making it so these asserts were only performed the first time the object is called or whenever we were expecting the page to have changed had quite a big impact on performance.

3) Don’t use Thread.sleep()

This shouldn’t really need to be said, but don’t use Thread.sleep() anywhere, this is precisely what FluentWait or WebDriverWait is for.

4) FluentWait/WebDriverWait poll interval

Now we’re getting into the juicy details of WebDriver. We had quite a few instances in our tests of waiting for certain elements to appear on the page or contain certain text before continuing with the tests, nothing complicated:

Here we are specifying a timeout of 5 seconds in the wait, however we aren’t specifying what the polling interval is (i.e. how often it checks to see whether the condition is true). As it turns out, the default polling interval is 500ms, rather a long time to wait between checking. This 500ms rapidly builds into seconds and minutes if you’re extensively using waits throughout your test suite. By adding a third parameter we can specify our own polling interval:

A lot has been written on this subject elsewhere so I won’t go into too much detail, but not all element locators perform in the same way. Browsers are optimised to locate elements by ID, so try and use ID wherever possible. Badger your developers to add unique ID’s to all elements, it makes everything so much easier, I can’t stress this enough.

At the other end of the spectrum XPath is often slow and unreadable (though not always of course!), if you absolutely must use this form of locator then we’ve found CSS selectors are at least a little better and perform more reliably across browsers.

6) @BeforeAll instead of @Before hook

There were a few instances in our project where we had a before hook running something before each test that only really needed to be run once before all the tests (for example starting an email server, really didn’t need to stop/start it before every single test). Sadly for whatever reason cucumber-jvm has not implemented a global @BeforeAll hook, however as that post points out, writing your own is relatively simple.

7) Controversial – use JavaScript for form input rather than sendKeys()

sendKeys() is slow. However this is for good reason, Selenium tries to emulate user interaction, pressing each key as it types into a field. Though if you wish you can bypass this with JavaScript (setting the value of the element and triggering onchange for example):

Whether you would want to however is an entirely different matter. Going down this route does give a quite visible performance improvement, but at the cost of risking not triggering certain things if your website is expecting a user to type keys in a certain way, what if not everything is triggered onchange? This is a controversial one, and the purists amongst you will certainly not advocate this approach.

8) Use a different webdriver

In many cases this isn’t possible, you’re using selenium grid to conduct your testing across multiple browsers, but where you have the choice of webdriver to use, remember that Firefox isn’t the only one available!

On our own project we noticed for example that ChromeDriver performed our tests about 10% faster than the FirefoxDriver. Going beyond that you could even look into a headless browser driver such as GhostDriver which uses PhantomJS. Unfortunately for whatever reason some of our tests just refused to run with GhostDriver and there wasn’t much time to investigate. However as an aside we are using PhantomJS in conjunction with Jasmine for some of our unit tests, and just over 500 tests run around 5 seconds there.

For those that really want to stick with the FirefoxDriver then there is always the following snippet of code that may speed up things:

While this does speed things up by having the driver not wait for pages to finish loading before proceeding, it can introduce instability and unpredictable behaviour in tests. Use with care!

9) Use selenium grid

One final simple way to speed up things is to parallelise tests via Selenium Grid. There is a vast array of documentation on this elsewhere, but essentially you can have your tests running across multiple machines simultaneously.

In order to make this work of course you need to be following one of the most fundamental principles of writing automated tests, that your tests are independent. If there is any coupling between scenarios then clearly splitting up your scenarios to run across multiple machines isn’t going to work at all.

After finishing going through the code and implementing the above we had ended up reducing time spent running the entire test suite by 60%. The best thing about it all, none of this is complex! While there’s no catch-all “make the tests run faster”, a few simple steps can make all the difference.

Set in the trendy tech area around Old Street (exposed brick and pipe work indoors abound!), the day started off with spotting a minor bug while walking towards the venue (it was an hour out). Arrived shortly after, checked into the conference, picked up the obligatory name badge and collection of stickers and grabbed a croissant and cup of tea to start the day (I must commend them on the selection available, I much prefer Twinings Assam to the standard English Breakfast).

The first talk was from the creator of Cucumber, Aslak Hellesøy, on where the tool is heading. I guess the big information was around the commercial side: Cucumber Ltd being formed to allow for more full time development, and Cucumber Pro (at this moment still in a sort of closed beta state). The Cucumber Pro tool looks quite interesting and I had come across it while looking for alternatives to Relish for making feature files easily accessible. Their goal with it sounds to be more engagement with the BA side of the business by abstracting away from the dev world of source control and jenkins reports. All of that will still exist under the covers of course, but this tool will hopefully put a pretty sheen over the top, make it easy to publish feature files as documentation, make it easy to see which features are passing and failing. Everything relating to the features of the system all in one easily accessible and easily usable place. It’s certainly something I’d be interested in taking a look at when it’s ready, but I think the big test will be working with such a tool in practice, how well will it integrate into existing processes, how well will this layer of abstraction above dev tools improve collaboration. All that is yet to be seen, but the future is promising.

The second talk of the day I attended was Liz Keogh giving a rapid fire run through of the history of BDD and the mistakes made throughout. Useful to see how far we’ve come, what what we’ve learnt and hopefully take all that forward and make less mistakes in future.

The next few talks I listened to were mostly around collaboration and the value of user stories and documentation/readability. I particularly liked Seb Rose’s talk on the relationship between user stories and features and why there shouldn’t necessarily be a one to one mapping. I do strongly believe that ultimately features are the documentation of the system, not individual stories, stories get thrown away after they’re done! Multiple stories add more acceptance criteria to a “feature” of the system. If we stick to this then we can keep using the feature files as documentation and keep the number of scenarios to the minimum required (instead of a proliferation of near duplicate yet subtle tweaked ones where stories map one to one with feature files). Seb also announced “The Cucumber-JVM Book” will be released, which as far as I understood will be “The Cucumber Book” but specifically for Cucumber-JVM instead of Ruby.

The final talk of the day from Matt Wynne had his thoughts on the 6 stages a dev team goes through when implementing BDD. From burnt toast, to auto-burnt toast, to the three amigos, into eventual disillusionment, and some thoughts on what the final stage of transcendence might look like. I guess much like the Buddhist Nirvana, we all have to find our own way there. We can take common tips, but what works for one team may not work for another, and we may not all get there, but that is the ultimate goal.

Overall I thought it was well worth attending, there was at least something to take away from each talk, and definitely a few things that give food for thought to being implemented as soon as I’m back in the office.