Selenium IDE is an add-on for Firefox that allows anyone to create macros through commands and then translate them into reproducible steps with the Selenium Firefox Driver. Commands include clicking on an element, typing into a field, and dragging an element from one location to another. The add-on can be downloaded from the Mozilla app store with tutorials on how to get started.

Selenium IDE

To understand why we were interested in incorporating Selenium IDE into our development cycle, you first need to understand where our testing infrastructure stands as a company. At Lucid Software, we have 13 full-time testers on two teams: a team of nine manual testers and a team of four testers dedicated to maintaining and improving our UI Automation framework. Lower-level tests, such as integration and unit tests, are the responsibility of each developer. Additionally, all developers write some UI tests for new features they implement. As a relatively young company at only five years old, we currently only have about half of our regression tests automated. Automating more of the regression tests is one of our top priorities company-wide.

As part of this effort, we began experimenting with Selenium IDE. From our investigation and experimentation, we found out a great deal about Selenium IDE. Hence, we present The Good, The Bad, and The Ugly.

The Good

Fast Setup

As a browser add-on, Selenium IDE can be set up very quickly. We have had four or five different developers and QA members try the tool, and all of them were able to get going in under 10 minutes.

Simple Interface

Selenium IDE doesn’t require a CS degree or knowledge of programming languages to set up most tests. In fact, it’s mainly just point-and-click. There is a whole list of possible assets to look for when it comes to debugging, and it highlights red where the test fails. Overall, the tool is built for any web user with even the simplest understanding of testing.

Quick Turnaround Time

Compared to writing Selenium tests with our framework, Selenium IDE was incredibly fast. We timed writing a simple test to make sure our Lucidchart editor loads. Even with most of the methods written and available in our homegrown Selenium framework, it took about ten minutes from the time the test was started until we were able to run it. With Selenium IDE, it took two.

The Bad

Only Runs in Firefox

About 25% of traffic to our sites is on Firefox, while almost 70% comes through Chrome. While it’s good for us to have tests in both Firefox and Chrome, we want our tests to mimic the workflows of the majority of our users.

Working With Canvas Elements Is Hard

And I mean really hard. In both of our products, the canvas element is about 80% of the editor’s real estate. With our homegrown Selenium solution, we make use of the Actions object to move things around on the canvas, click at locations, and drag and drop blocks. Without access to the Actions object in Selenium IDE, this became just a matter of trying to click at the correct offset from the canvas object to make selections and move them.

Lack of Conformity to a Standard

Selenium IDE is an open-source project and it shows. The list of commands and syntaxes varies widely. Some just use the command field and a target. Others require the target to be listed in the value field. Still, others require you to add specific fields or take some away. While the versatility is good, the lack of standards means that any tester needs to spend a lot of time learning the different commands and what they expect in order to do anything complicated. The simplicity and ease of the product are lost through poor adherence to standards. This problem will likely become worse as more commands are added.

The Ugly

The Waiting Game

This is the number one complaint around the web with Selenium in any fashion. With Selenium IDE, it is even worse. Waiting on a site to execute Javascript requires the browser to wait for a particular action. In Selenium IDE, it tries to execute as quickly as possible. For example, as soon as a button gets added to the DOM, it tries to click the button, even if the browser is still calculating the position and the button won’t work yet. You need to wait until the button is fully set up, but Selenium IDE moves on as soon as the button is added to the DOM. In order to work around this problem, the options in Selenium IDE are using a loop and building out some retry code or simply putting in a wait command to tell it to wait a little longer.

Access to the API

One of the advantages of our home grown Selenium framework is direct access to backend APIs. We use these to create users, add users to teams, create documents, and clean up at the end of the test. With Selenium IDE, we don’t have this access, so every single test creates a new user in our test environment that never gets cleaned up.

Reliability

To be completely frank, simple tests flake out around five percent of the time. Once we ramped up a little and tried to do some complex testing, we found we were getting false positives about 20-25% of the time. On these tests, we spent hours trying to fine-tune waiting periods, change the speed of the test, and click on more clear components. Even with hours of effort, the best we could get was about a 10% fail rate.

The Verdict

Selenium IDE is a user-friendly, easy-to-use tool. You can set it up very quickly, and anyone can use it. At Lucid Software, we have decided to add Selenium IDE to our toolset for our Quality Assurance Specialists, but only in a limited capacity at the discretion of each team member. It was not reliable enough or full-featured enough for us to warrant porting out old UI tests forward or even making it a primary tool for automation; the cost to maintain was just too high.

Our Quality Assurance Specialists have found a use for it in repeating the common tasks that they do every day. They use it to set up team situations, set up a new account with specific A/B tests, or even just clean up old testing documents by going through all the documents and deleting them. These are all simple tasks that can fail without hurting anything, as the QA Specialist can see what failed and simply finish the process manually. Selenium IDE accomplishes the task faster and allows the specialist to focus on other things while the process is being run.