Want to try Screenster on the cloud?

Selenium alternatives for testing automation

If you’re looking for an alternative to Selenium, there are plenty of options out there. Here are the options that our teams at AgileEngine and AjaxSwing used to automate UI testing:

UFT (based on QTP)

LeanFT

TestComplete

PhantomJS

Jasmine

Mocha

Protractor

Screenster

But how do these tools compare? And why search for an alternative to Selenium in the first place? Selenium holds the status of a go-to solution for UI testing automation, and it’s still a must-learn for testing automation engineers in 2018. This said, companies like Google, Kaspersky, and Samsung are already on the lookout for more efficient solutions. So what is the motivation for wanting to move beyond Selenium?

Love and Hate with Selenium

Spend enough time automating web applications, and you will most likely develop a love/hate relationship with Selenium. The truth is that it’s pretty much the only game in town to drive the browser through the API. That’s where the “love” comes from — you can start writing cross-browser code in your favorite programming language in a matter of hours. Sweet!

But spend enough time doing that for a real world application, and you will discover the ugly side of this coin that at times make you pull your hair out. In my experience, the following 6 pain points are near deal-breakers for automation with Selenium.

1. Selenium tests are unstable.
WebDriver libraries version trail the auto-updating browser, and there’s always something small that doesn’t quite work. Often when you get the new version of the Selenium libraries that is supposed to fix the issue, you discover that now something that used to work doesn’t work the same way anymore. Add to that multiple versions of browsers, and you are constantly chasing a moving target.

2. Good tests require good coders.
Yeah, I’m sure there are plenty of good developers automating web testing with Selenium. But for everyone’s benefit it’s optimal to have your best developers write product features and leave automation testing to lesser-skilled people. For better or worse, many companies leave UI test development to junior developers or non-CS graduates, which results in at best mediocre code quality that is a liability over time. Writing a good Selenium test requires understanding of the browser, CSS, AJAX, and application page structure.

3. Test maintenance is a drag.
This of course is true for any automated tests. The more of them you build, the more time you have to spend updating them for new features and dealing with false positives. The reason I hold Selenium culprit here is that it is a very low-level API that doesn’t offer much help to the developer in test maintenance like dealing with element locators, page screenshots and error capturing.

4. Selenium test suites take too long to run.
Write enough Selenium tests and your suite will take days to finish. No framework out there can make a browser run faster and Selenium has pretty cool support for grid that can run parallel tests on multiple computers. But very few people have actually set it up and again, it requires quite a bit of skill to keep that beast running.

5. Selenium IDE lacks crucial functionality
The concept behind the lightweight Selenium IDE is neat, but very few companies can actually use it for production purposes. The IDE doesn’t offer much in the way of functionality or browser support, and the tests it produces are, essentially, poorly implemented WebDriver code.

6. Low ROI
Selenium is de-facto the only browser automation API that at a low level gives you everything you need. But it requires a very capable technical test automation engineer, and even then the productivity is low.

What Selenium Alternatives are there?

I know what you are thinking. Sure, it’s easy to complain about the imperfect world of test automation. But unless there’s something better, this is just whining. Well, I’m here to share some of the promising alternatives to Selenium that people have used to get to a better state.

IMHO the main problem with Selenium is that it’s just too low-level. It’s definitely good enough for what it does, but if you want to increase the ROI, then I think you should consider picking up a higher-level framework or testing tool. In an ideal world, this would be a Selenium-based or Selenium-like tool that offers the simplicity of Selenium IDE combined with the advanced features offered by the WebDriver API.

Old School: desktop record/playback IDEs

Back in the 90s, desktop-based record/playback tools were all the craze. Tools like UFT, Ranorex, LeanFT, TestComplete, Telerik, and Sahi still fall into this category. Just like Selenium IDE, they allow a less-technical tester automate UI testing, which increases the ROI. While they can certainly do this job better than Selenium IDE, very few people consider them to be a real alternative to Selenium.

Even though solutions like UFT and TestComplete have an established user base, it mostly comes down with large enterprises. There are multiple reasons for this. Most record/playback tools tend to be giant beasts that every tester has to install, and they take weeks to learn. What’s more, the record/playback approach produces brittle tests written in some language that tester has to continue to maintain manually. Over the long term these kinds of tests tend to cost more than well-coded tests written by hand on top of something like Selenium.

Most of the teams I talked with have abandoned old-school IDEs in favor of hand-coded tests with a Selenium-like tool or framework. Given this, old-school record/playback solutions are an alternative that I wouldn’t recommend.

Browser alternatives

First, let’s make sure we compare apples to apples. Selenium is an API for browser automation. It is primarily used to drive a real browser like Firefox and Chrome giving exactly the same environment as the end user. But do you really need to drive the real browser to efficiently test the UI?

A once popular alternative to a real browser was testing in a “headless” mode, which implies running the browser without any UI. Chrome introduced headless mode in version 59, but originally, this functionality was available in dedicated headless browsers like PhantomJS.

A headless browser is not what end users will use with a website, but it’s pretty darn close. The downsides are that a headless browser like PhantomJS is almost as heavy as a real browser like Chrome, but passing tests in PhantomJS do not guarantee that they will pass in Chrome.

Another browser alternative is HttpUnit, a super-thin framework with a simple built based on a JavaScript engine. It’s light and fast, but because it’s very different from a real browser, it can leak bugs into production. Besides, it can result in tests failing due to HttpUnit itself.

IMHO, the alternatives to the real browser are not worth the risks of leaked bugs. The problems they are trying to solve are high resource use and slow execution speed. I would argue that one is better off coding simple JavaScript unit tests with frameworks like Karma and Jasmine and sticking with a real browser for slow-by-reliable end-to-end tests with frameworks like Protractor and Jasmine. More on this below.

Evolutionary: coding to a higher-level framework

So we want to stick with a real browser but get something less quirky and more productive than plain old WebDriver. In 2017 we have a lot of good choices for hand-coded tests, so let’s quickly look at the most popular ones.

Higher-level frameworks like Mocha, Jasmine, and Protractor rely on WebDriver to control the browser, but they offer more out of the box than direct Selenium API. They are cleaner and more fun to code with, they have a lot of syntactic sugar for BDD, TDD and advanced JavaScript programming. I would certainly look at using a higher-level framework as a modern alternative to Selenium, because of the following advantages:

Pre-integrated with other frameworks

BDD syntax for more readable tests

Easier for MVC frameworks like Angular, React and Ember

Improved asynchronous and timeout support

However, most of the fundamental issues that you’d run in Selenium are still the same. A lot of setup, testers have to be capable programmers, every action emulation and verification has to be coded manually, fragile locators break whenever the page changes, and there is no CSS or visual verification of any sort. All this results in brittle tests that take a long time to write and a long time to run.

Revolutionary: cloud based integrated platforms

Wouldn’t it be nice if there was a system that could just capture how humans normally do testing? This system would produce tests that could be used to regression test the application and detect changes.

The Holy Grail of automation testing has always been record/playback but the tools just didn’t quite get there. The main issue is that they never got good enough were no coding was needed, and the code they produced was so painful that a good developer would prefer to just write it from scratch.

Another major problem is that the majority of automated tests do very little to assert that the UI is not broken. None of them can fail a test if certain text is cropped or overlapped or colors are off. Missing images, inadvertent text changes and broken layouts just go undetected.

During the last couple of years, there’s been an emergence of a new generation of platforms that promise dramatic increase in productivity and the ease of use due to a fundamentally different model. Rather than coding to a framework, you get a cloud-based automation testing platform that works out of the box. This is what Screenster, our very own codeless platform for UI test automation is all about.

Screenster: codeless visual tests on the cloud

Introduced in 2016, Screenster is a web-based automation testing platform that is unlike anything else. Built for less technical testers and business users, it’s promises 10x productivity improvement compared to Selenium and other frameworks. How does Screenster address the typical problems of record/playback IDEs?

100% web-based
There is nothing for testers to install locally. All testers and developers use a shared server running on premise or on the cloud. And there aren’t any manuals to read either.

Visual baseline
As the tester interacts with the application, Screenster creates a visual baseline of the application and stores it on the server along with the user input. It detects the dynamic portions of the UI and offers to ignore them from future comparison. Regression test runs detect the changes and show them to the tester for approval to the baseline or bug reporting.

Codeless tests
Tests are stored in the database along with captured DOM and screenshots. Testers can browse page elements offline and update tests through innovative mouse-friendly UX. For skeptics: would you deal with code if you didn’t have to?

Smart Maintenance
Differences can be approved to the baseline with one click. Smart selectors are used to ensure that tests don’t break when elements are moved on the page or page structure changes.

The prerequisite for using Screenster is that UI state is repeatable, meaning that repeat executions of the same tests produce the same result. The approach with codeless visual tests may not cover all use cases but it should produce a significantly better ROI for 95% of the commonly covered scenarios. At the end of the day, it’s the focus on higher ROI that makes for a good Selenium alternative.