Running E2E tests with Selenium *AND* Cypress

E2E test

An E2E(End to End) test is a common practice to ensure your software’s quality. Web developers want to make sure that the APIs are working through the UI, and check whether or not the UI is working as expected after API/UI changes. It is also helpful to ensure your service/website is supporting multiple browsers, especially when you expect users might be using really old browsers such as IE7, 8, or 9 — even though Microsoft officially announced they will stop supporting them (I can hear you crying.)

Developers can’t really blame users for browsing through those ancient browsers. My grandfather, for instance, had used a very old PC for long time. I can imagine he barely used his computer, had no idea how to upgrade the OS, and never understood necessity of such a change. I know many people like him, and I believe technology shouldn’t weaken them. Rather, it should assist people in this situation: various browsers should be supported for them.

Selenium

In order to support multiple browsers, we’ve been using Selenium, the de facto standard E2E framework which has been supporting web developers’ lives for a long time (via a third party cloud service). Through their APIs, we can run tests in different types of browsers powered by Selenium. This is helpful.

On the other hand, from my experience, maintaining Selenium tests takes too much time. When you change one part of the UI, you have to run the Selenium test, change the test code if it has failed, then re-run if necessary. Furthermore, running a Selenium test isn’t fun. In reality, it’s really slow, the error log is not always easy to read (at least, not for me), and it doesn’t describe what causes the error — not really debuggable. This experience makes people too lazy to write E2E tests when, in fact, we don’t have as many of these tests as we should.

Lately we had some problems with Selenium tests. The tests we have been running every time suddenly started failing. We didn’t make any code changes related to the failures, and error logs didn’t indicate what exactly happened. We asked the third party service provider’s support center (which is supposed to be a group of Selenium test professionals), however they couldn’t find anything from the test logs. We needed to find out an alternative way… then we discovered Cypress.

Cypress

Cypress is a brand new open source E2E testing framework. It only supports Chrome browsers, and only allows you to write tests in JavaScript. While it sounds like it’s not as good as Selenium, we found many benefits, including:

Fast to iterate. Once the browser is launched the test goes super fast.

Easy to write. The syntax is the same as other JavaScript test framework. Also it has useful and easy-to-use APIs.

Easy to debug. It has a ‘Time Travel’ feature, so you can go back to any point, see the state of the element, and JS error stack trace. Chrome’s developer tools are also available to use.

Quick Setup. It executes tests, generates videos and screenshots with almost no configurations.

It is fast, but sometimes it’s too much. It even reminds us how slow our app is. For example, it clicks an element before the event handler gets attached. As a patch, we added an attribute to the element when an event is attached. Something like this:

Then we made the test to wait until that attribute appears like:

This is not something we would’ve discovered with Selenium. Possibly some users would have had an issue, such as clicking right after the button was rendered and nothing happening. This fix is just a patch, so we can make it faster eventually.

Selenium *AND* Cypress

We ported some of our Selenium tests to Cypress tests, but before we did that we discussed what tests should be under which framework. These are the ideas on how we are going to port our E2E tests in the near future:

We want to minimize the number of Selenium tests to avoid using too much inspection time for unknown failures

We want to keep the tests maintaining their functionalities in old browsers

It is great to have two E2E testing frameworks, as one covers the other’s weakness. Adding Cypress didn’t take much time as we would have needed to spend on maintaining Selenium tests. I’m excited that our E2E tests will be more robust.