14 Sep 2009

I mentioned in my last post that I'd started to think we were going about Selenium testing all wrong. For years I've been writing Selenium tests using the Selenium IDE, so the tests are in HTML files. When doing pre-commit testing and on CI the tests are run with the HtmlTestRunner. This technique is fine and well - developing tests with the IDE is fast and they can be easily worked on an re-tested in the IDE until they are correct. However, some things have always bothered me and Adam Goucher's thoughts on the Selenium Value Chain crystallised some of those suspicions.

So what are the big advantages of using Selenium RC over just the IDE and HtmlTestRunner?

Test design

Selenium tests written in HTML tend to either become an amorphous blob, not testing one simple thing or proliferate to the point of having hundreds of small test files. With Selenium RC you can write Selenium test classes with multiple test case methods just as you would unit tests. Each test case can be neat, self-contained and share set up with other test cases in the same class. Also by abstracting out common command sequences into utility methods you can start to develop a higher level DSL for testing your application. In HTML tests this can be done by writing extension functions in Javascript but it's not as straightforward.

Data fixtures

Data set up is the biggest headache when writing Selenium tests in HTML. You can't drop out of selenese and set up a couple of domain objects so you end up either trying to run all your tests against a monolithic data fixture (a Sisyphean task if any of your tests make modifications to the data) or write some sort of URL endpoints that can set up and tear down data. We've done the latter and it's resulted in the single most bloated, horrible, misused bit of code in our app. Fixtures get re-used by other tests because they set up approximately appropriate data - until some change requires a change in the fixture and suddenly a bunch of unrelated tests fail. Selenium RC tests don't have this problem; you're in Groovy-land so you can access domain classes directly and use the invaluable build-test-data plugin to set data up.

Data verification

I mentioned previously the anti-pattern of false moniker testing and HTML based Selenium tests are terrible for this. If you have a test that creates some data, how do you verify it has done it correctly? The only way is to scrape the data on another screen - e.g. a CRUD list or show view. Ugh! With Selenium RC you can read the data from the database with standard Grails domain classes and verify your expectations directly.

I started experimenting over the weekend with getting Selenium RC tests written in Groovy to run using grails test-app -functional. It turns out to be really fairly straightforward and I've put some alpha code up on GitHub.

I've been trying to persuade our team to do this for ages! In addition to your points, some other positives from writing them in Java/Groovy and using JUnit:

1) Easy to organise and find your tests - no list of 300+ files in a single selenium suite, you can put them in packages and run a single test, single test class or a package or multiple packages very easily with a right click.

2) Easy tear down - if a "normal" selenium test fails half way through and leaves a fixture in a deliberately broken state it's going to break all the other tests that run after it. In JUnit you can do a tearDown to fix it despite the fail.

3) More natural to use other technologies to do functional testing - for instance our reliance on selenese for all functional testing has resulted in us writing gruesome tests and fictures to drive a RESTful JSON service via Selenium and the browser; I'm sure if we were writing our functional tests in JUnit we'd immediately have spotted the mismatch and used a REST client in groovy instead.

But the single biggest point is that as you say, selenese makes it very, very difficult to write well factored code. The other day we changed the id of a submit button used in perhaps 60% of our tests. Complete nightmare to fix all the selenium tests, because we'd never bothered to abstract away submitting the form because writing selenium extensions in javascript is too painful

Oh, and another positive - no having to remember to update allProjectTests.xml if you add or rename a test. It's all too easy to have orphaned tests knocking around. Of course we could improve the selenium plugin to generate the suite auto-magically - but why bother when every tool under the sun auto-generates JUnit suites from naming conventions in known directories?

I've been emailing back & forth with Peter Ledbrook today who was about to embark on working on amending the existing selenium plugin to use RC. It would be great to have Selenium as a "first class" functional testing option for Grails along with Webtest and G-Func. Stay tuned!

The one objection people have raised that I can't answer is that the Selenium IDE is a very convenient way to run single selenium commands (or manually run a few in a row). Not sure that this is really avoidable - break points Java/Groovy can do, but just executing one line of code in isolation is a bit more difficult!

Using Selenium RC opens the door to using the Page Object pattern, described here:

http://code.google.com/p/webdriver/wiki/PageObjects

Searching for it today I also found this presentation:http://www.slideshare.net/dantebriones/using-the-page-object-pattern

This gives you a systematic way of testing against a logical view of pages. Minor changes in pages only impact the page object for the affected pages - the tests using the page object are unaffected. This counters the common argument that "it's not worth investing in automated testing until the design is stable".

I've added some examples of how to use the page object pattern to the sample projects in the selenium-rc plugin. It's a really powerful and simple technique and it's once you start using that you really see the productivity gain of RC over record-and-playback selenese tests.

Hi Rob, I am new in Automation world.I am trying to automate my grail based application by using grails-plugin selenium rc.I faced 2 challenging points are1)how to setup Testdata.I wanted to use excel for that.But couldn't find any luck in groovy.there is many options in Java.So could you suggest me on this.For time being i m just using Property file(JAVA) to parametrized selenium testcases.

2)How to verify displayed data from database.Today I tried it by GORM in selenium testcases.It works.But on net i found it will raise issue when Domain become complex.is there any other way to do so.Please suggest.