Taking Screenshots with Selenium

I had written about using Selenium, TestNG, Cargo and Maven to write an automated smoke test suite. It would be useful if we could get a screenshot of the application under test at every stage of the test – the screenshots would be useful to diagnose test failures. In this post I describe a crude screenshotting mechanism that I have developed to take screenshots for tests running in Selenium.

Note: The code used for this post can be downloaded here. Due to WordPress restrictions, I had to rename it with extension .jpg. Kindly rename it to .tar.gz. Also, the code is a continuation from my previous post.

By default, Selenium shows the application under test in a browser frame. This frame, of course, cannot be maximized to occupy the entire screen. If Selenium Remote Control server is launched in multiWindow mode, the application under test is launched in a seperate browser window. To launch Selenium Remote Control server in multiWindow mode:

Now, if we use ScreenshottingSelenium instead of DefaultSelenium, we would get screenshots of the application in the target directory. But we would like to go one step further – we want to generate a nice HTML report of the screenshots. For this we create a Reporter class:

public class Reporter {
void startTest() {
// Called when a test is started; creates the directory
// for saving screenshot images of this test
}
void endTest() {
// Called when a test ends; creates an HTML summary
// of all screenshot images of this test
}
void recordStep(String description) {
// Creates a screenshot and saves the image to a file
}
public void close() {
// Called after all tests are run; creates an HTML summary
// of all tests run
}
}

The implementation of startTest(), endTest(), and close() are quite simple that I do not describe them here. recordStep is essentially the same as takeAScreenShotOfTheApp() earlier. The HTML reports are generated using couple of FreeMarker templates.

The ScreenshottingSelenium would be configured with a Reporter object; the ScreenshottingSelenium calls methods on the Reporter object at appropriate times. The startTest() method is called from the start() method. The start() method opens a browser window and this we assume signals the start of a test. The endTest() method is called from the stop() method. The stop() method closes the browser window, which we assume signals the end of a test. Method like click(), type(), open() etc call recordStep() method of the reporter object with a text description of the action just performed.

The Reporter object needs to be created before the testsuite starts execution. Also, the close() method on the reporter object should be called after all tests in the suite are executed. We make use to @BeforeSuite and @AfterSuite TestNG annotations for this purpose:

To see these ideas in action:
1. Download this code bundle. Rename the .jpg file to selenium-screenshots.tar.gz.
2. The bundle contains a small web application and its smoke testsuite. Uncompress the archive to some place on your disk.
3. Install the web application on your local Maven repository.

> cd selenium-screenshots
selenium-screenshots> mvn install

4. Run the integration testsuite:

selenium-screenshots> cd functest
functest> mvn integration-test

This will download Tomcat from the web, install Tomcat server, deploy our web application onto the server, open Firefox and IE browsers and test drives the web application on the browsers. The screenshots from these would be generated in the functest/target/screenshots directory.

Acknowledgements: This technique was inspired by a forum post by Dan Fabulich.

Update: I fixed a small bug in the instructions given to run the code bundle – thanks to davidm for pointing it out in the comments. I have also updated the code bundle with the URL pointing to a newer version of Tomcat. Some versions of Selenium have trouble launching some versions of Firefox. Hence, I have commented out the FireFox tests; instead the tests run on IE.

However, I’m on a Mac and as on Linux (and other OS) Firefox is the best browser to run. I note the configuration is buried deep in the package. Perhaps providing easy instructions to set up the browser cross platform would be the solution, I note that is already done for the location of IE but it is not clear what “or explicitly specify a path to IE like this,” which would maybe work for other executables, means. I am pretty sure the Java desktop screen capture will work cross other platforms.

Thanks again for this terrific work, I hope it can be used by many people (including myself!) without too much trouble.

One idea that I have seen mentioned is this – the testsuite does not use any assertion, but simply captures a series of images. Once the tests are run, the program compares the image captured at each step with some previously known “gold” images. This is a simple pixel-by-pixel comparison. If there is any mismatch, it is considered to be a test failure. A report of failures – what was expected and what was actually got – is send out to the developer. The developer goes through this report, and recognizes bugs. Sometimes the UI might have changed because of a new feature added or improved; in such cases the developer can set the new image as the “gold”.

1) The Tomcat URL is specified in functest/pom.xml. Search for element in the XML. Tomcat developers frequently purge old release bundles from their web servers. So the URL to the release bundle I specify in the POM becomes unreachable in a few months.

2) Selenium supports a wide range of browsers. You can choose an appropriate browser by editing the XML file in functest\src\it\testng.xml. In this file, you are essentially telling TestNG to pass a parameter named browser into the tests. The value of this parameter decides which browser to run the tests on. browser=firefox runs the test on FireFox. On Mac OS X, Selenium is known to be working well with Firefox 1.5.0.4 and 2.0. But I have noticed that it does not work well with FireFox versions later than 2.0. Since many people configure FireFox to update itself periodically, they might be running versions later than 2.0 – with which Selenium does not run well. You can explicitly download FireFox 2.0 and put it in /Applications and try running this – that should work fine.

1) The Tomcat URL is specified in functest/pom.xml. Search for element in the XML. Tomcat developers frequently purge old release bundles from their web servers. So the URL to the release bundle I specify in the POM becomes unreachable in a few months.

2) Selenium supports a wide range of browsers. You can choose an appropriate browser by editing the XML file in functest\src\it\testng.xml. In this file, you are essentially telling TestNG to pass a parameter named browser into the tests. The value of this parameter decides which browser to run the tests on. browser=firefox runs the test on FireFox. On Mac OS X, Selenium is known to be working well with Firefox 1.5.0.4 and 2.0. But I have noticed that it does not work well with FireFox versions later than 2.0. Since many people configure FireFox to update itself periodically, they might be running versions later than 2.0 – with which Selenium does not run well. You can explicitly download FireFox 2.0 and put it in /Applications and try running the tests.

Thanks,
Binil

davidm

June 28, 2007 at 5:54 pm

Thanks for these additional pointers. I’ve used Ant much more than Maven, and Selenium looks like its own beast, glad it is possible to configure.

I gave up trying to get it to work on Mac, am trying on Linux, but it just sits there at “Preparing Firefox profile..” This is Firefox 2.0.4. Maybe its the later version problem. What a mess.

Sorry that you could not get it working yet. My recommendation is to switch to FF 1.5 product line. I remember Selenium working very well with FF 1.5, but the switch to FF 2.0 didn’t go too well.

Thanks,
Binil

Tarjei Huse

July 16, 2007 at 11:55 am

Hi, I’m doing a masters thesis on using image verification like this on dynamic websites – i.e. websites where content changes and you want to check other things than the content (for example, is the font used correct?).

One of the things I’ve been considering is using Selenium to do this, so this post comes at a most opportune moment🙂

A few questions though: Is it possible to use this against a remote selenium server? If you want to use selenium for more than just one browser, this quickly becomes an issue.

Browsershots.org has a service where you can get screenshots of the whole page – not just the top half. The package is open source and written in Python. Well worth a look.

Well, I should stop writing and start downloading🙂

Regards,
Tarjei

Sunitha

August 30, 2007 at 12:09 pm

Hi,
It will be great if I can get the code which has these ideas in action. Unfortunately I am seeing a 404 when trying to access the code. IS the file moved? Where can I get the code from?

Hi,
My Selenium tests will be running under xvfb server.
For example, if i start my xvfb server on :1 , how could I record the screenshot.
The current example assumes the display to be localhost.Any idea how to make capture the screenshots from the remote display?