Automated Smoke Tests With Selenium, Cargo, TestNG and Maven

In the project I am working on right now, we have decent amount of unit tests. Some of the unit tests launch an in-memory HSQL DB and run the tests against them – this helps us catch bugs in schema, Hibernate mapping etc. We use a Continuous Integration server which checks out code from the repository every hour and runs the tests. This setup helps us catch some kind of bugs early, but some other errors still can happen – for example, errors in web.xml, javascript errors etc.

Also, we frequently deploy newer releases to QA. Before making a release, we do a “smoke test” where one of us runs through the major functional areas of the application. Sometimes we encounter bugs at this stage. The development web server is Jetty, but all deployments from QA upwards happen on Tomcat – minor incompatibilities between Jetty and Tomcat are identified very late, usually when the smoke test is run right after a deployment.

I devised a mechanism to run an automated smoke test, and I will describe the process here.

All the above tools are available free and open source. This is not an exhaustive tutorial on any of these tools. All I intend to do is show how these can be used together to write a smoke test process. I have broken down the integration of these tools into eight steps. This tgz bundle contains the files needed for each step – kindly rename the file to steps.tar.gz (without the .pdf extension) and uncompress it to your local disk.Note:I do not discuss creating the database schema for the test.

Step 0: Application under test
The web application we are going to test is a very simple one. On the first page there is a link, and when a user clicks on it she is taken to a form. The form is a simple Celsius-Fahrenheit converter; on submitting the form the user is shown the result of the conversion, and she is shown the form again so that she can try another conversion. There is a link at the bottom of the form which takes the user back to the index page. The application is developed using WebWork.
To run this app, get the file, uncompress it to your local disk, run:

Note: If you have not worked on Maven before, setting it up is real easy. Download the package from the project website, unzip it somewhere on your local disk, add a system variable MAVEN_HOME pointing to where you have installed Maven, and add $MAVEN_HOME/bin to the PATH. You can verify the installation by running mvn -v, which is expected to print the Maven version on the console.

Step 1: Configure an integration test suite
We create a new module, functest, to hold our functional tests. We configure for the following in the functest/pom.xml:

To run this test, we need to first start the Selenium RC server. The Selenium RC client driver communicates with the server; the server performs the actual task of launching of the browser and controlling it. Download Selenium RC server 0.9.0 version from OpenQA repository and launch the jar:

This will launch the Selenium RC server on port 4444.Note: Make sure that firefox executable is on the PATH of the command window from where Selenium RC server is launched.
Now, in another command window, run:

steps/step3/functest> mvn integration-test

If everything is set up fine, you would be seeing a Firefox browser open and close.

Step 4: Test for the title of the index page
So far we got the browser to launch, but we have not tested if the application is working fine. Now let us write a test which launches a browser, opens the index page of the application, and verifies the title of the index page.

Notice the use of Selenium RC client driver API to direct the browser to a particular URL, wait for the page to load, and geting the browser title string.
To run this test we need to start the application. On a new command window, run:

steps/step4/mywebapp> mvn jetty:run

In another command window, run:

steps/step4/functest> mvn integration-test

If everything is fine, you should see a browser open, load the index page, and close. You should also notice that the test passes. To test that this assertion really works, change the assertion, run the test again, and verify that the test fails.Note: Make sure that the Selenium RC server from step 3 is running.

Step 5: Adding one more test
Now we add another test to click on the link on the index page of the app, and verify that the page with the form comes up. We also refactored the opening and closing of the browser into seperate methods. We annotate these methods – openBrowser & closeBrowser – such that they are run before and after the tests respectively.

Note: Make sure you have the Selenium RC server and Jetty server running still.

Step 6: Starting and stopping Selenium RC server from the test
So far we have had Selenium RC server running in a command window. First step towards true automation start and stop the server from within the test. Firstly, stop the Selenium RC server running. We need to start the Selenium RC server before any tests are run, and we need to shutdown the Selenium RC server after all tests are run. TestNG’s @BeforeSuite and @AfterSuite annotations enable us to do so.

and verify that this works.Note: Make sure that you have shutdown the Selenium RC server started earlier.

Step 7: Controlling a Tomcat server from the test
Our tests still need a Tomcat server running. It would be really useful if we could start the Tomcat server, deploy our application onto the server, run the tests and eventually shutdown server. We use Cargo to let us do this – Cargo has a maven plugin which can be configured in functest pom to achieve this. The plugin download Tomcat binary from the web, installs it, deploys our web application onto it. After the tests are run, the plugin stops the Tomcat instance too.

Notice that we added our web application as a dependency to functest project.
Close the Tomcat server that was started earlier, and run the tests once again:

steps/step7/functest> mvn integration-test

If we have done things alright, tests should pass sucessfully.

Step 8: Running tests on multiple browsers
Right now our tests only run in Firefox. If we want to run it another browser, we need to change the third parameter to constructor of DefaultSelenium. TestNG allows parameters to be passed to @BeforeTest method, so we make use of that.

Notice that we abstracted the form submision into a private method – this enables us to use the code that performs the actual form submit in many tests.

Thats it!🙂 We can use this setup in a Continuous Integration server. It will download Tomcat from the web (only once!), install it, deploy our web application onto it, start Selenium RC server, open a browser window, test drive the application on the browser, close the browser window, stops Selenium RC server and eventually stops Tomcat server.

“Use Xvfb (X Windows Virtual Frame Buffer): If you want to run Selenium on a Unix server- without an X Windows display- or if you just don’t want to see the web browser windows popping up, use xvfb. This is an X server that just runs in memory, without a display.”

What I was wondering: Do you have any experience in setting the database in an initial state?

Also, I have some problems with existing tomcat configurations. To be more precise. I’d like to deploy my container and then copy some configuration (property files) to the tomcat/conf folder and put some jars (like log4j) in my tomcat/common/lib folder. How can I do this with Cargo?

> Do you have any experience in setting the database
> in an initial state?
We use a simple mechanism right now. We use Hibernate for ORM, and we use Hibernate utilities to recreate the schema based on our entities. We have our seed data defined in a bunch of excel files – one file per functional area. Each of these files have many worksheets. The name of the worksheet needs to be the name of the DB table; the first row of the sheet is the name of the columns in that table; all other rows contain data. We use DBUnit to populate the database with these Excel files. We invoke the Hibernate and DBUnit utilities with the appropriate DataSource – and once they are executed our (test) database is ready to be used.

A migration framework would have been better, but we haven’t yet reached the need for it, and hence havent integrated one into the application. Some approaches that can be used are discussed at http://www.infoq.com/news/ruby-migrations-java.

Please let me know your thoughts on this.

> I’d like to deploy my container and then copy some
> configuration (property files) to the tomcat/conf folder
> and put some jars (like log4j) in my tomcat/common/lib
> folder. How can I do this with Cargo?

the maven eclipse plugin does not generate eclipse project files for pom modules (in this case the functest module). My workaround is to set the packaging to jar, then generate the project files, then reset packaging to pom. Did you use eclipse? Any better way to do this?

Yes, I used Eclipse to create those projects and I also ran into the same issue.🙂 My workaround was the same as yours – comment out the pom element, run mvn eclipse:eclipse, and then uncomment the element.

1- Your integration-test phase should exclude any tests that might run in the test phase or you will end up running your unit tests again in the integration-test phase.
2- Same goes for your test-phase, it should exclude any integration tests.
3- You don’t have to use cargo m2 plugin to start and stop your tomcat server, you can achieve the same result using Cargo Java API and put the code in your @BeforeSuite method.
4- I implemented something similar to what you did using JUnit as a testing framework. I had to use JUnit as it was a requirement, but I agree with you that using TestNG is much cleaner (and easier to implement).

Thanks Bashar!
1 & 2: I have assumed that the functest module contains only integration tests. This module is a sibling of the webapp module. All unit tests would be put in the webapp module.
3: Agreed. If I can start the Selenium RC server from code, there really is no good reason to insist on starting Tomcat with the Cargo plugin.🙂 But I would have preferred to start both SRC and TC using a Maven plugin.

can you explain me how your tests are structured. In my case, I try to do a class for each scenario. I will have multiple scenarios for testing all my application. Do you declare a Selenium object by scenario so you have to connect and deconnect for each one ?

Here’s one way you can start and stop the Selenium Server automatically when using TestNG.
It uses one test class: IReporterSeleniumTest.class and includes a test listener: ISeleniumTestListener. If you do it this way, you don’t even need a testng.xml file.

I have a question on the step7 where we try to integrate tomcat in maven build. I tried to do that and I am getting this error during build. For me web-3.1-SNAPSHOT.war is built after this build is finished. So, I think I am having a chicken and egg situation. How can I avoid that?

From what you wrote I gather that you need to deploy the WAR onto Tomcat *before* the WAR is even built! A moments reflection will convince you that this is next to impossible. Is there reason why you want it this way? Are you doing this to ensure that the war you build is always a “good one” that passes some smoke test? If so, you could order the dependency to build the war first, then deploy it. The build succeeds if and only if the war is built properly *and* it is deployable onto TC.

Yes.. I am doing this to ensure that the war you build is always a “good one” that passes some smoke test. I haven’t used/done POM with dependency to build first and then deploy. I will look into it. But meanwhile, if you have any samples like that, can you send that to me?

BTW, have you ever seen the automated SeleniumServer cause a system to slow down? My test runs great with a SeleniumServer command line, but speed degrades quite quickly if I start the server per your method above.

Andre, Selenium in general is sluggish, but I have not noticed the method above being slower than the command line option. One thing you might have to investigate is the heap size of the Maven JVM – check if the VM is thrashing memory (set MAVEN_OPTS=-Xmx.. to control the heap size).

We are doing almost all of this – not smoke, but using Selenium, testng, tomcat, and maven to do integration tests (we are slowly expanding to do the entire build and release process)

Looking at adding Hudson to the process – but running into problems and I’m not really up on a lot of this – we need to stop tomcat and clear the database before tests are run, but the app we are testing (not currently deployed by Maven, but will be) locks onto the db and does not allow manipulation while running. Our solution is to uses the exec plug in to stop Tomcat, drop and recreate an empty database (or import a backup file with the db we want to test with) then restart tomcat.

Because Hudson runs as a tomcat app – which captures the output log files BTW to answer Ramu’s question in comment 34 – I’m looking at cargo as a way to just stop our app, and not tomcat. Currently our set up works ok as command line or from eclipse, but the point where tomcat is stopped kills Hudson which kills Maven.

One solution is to have a second instance of Tomcat running on different ports to handle Hudson, but I was wondering what was required in Maven to stop and restart one Tomcat application.

Help such as pointing to a link with this info would be very appreciated.

@rdekleijn Yes you can run the tests in multiple browsers using Selenium RC. . Do you mean multiple browser types for example (IE6, IE7, FF3) or multiple instances of the same type for example (3x IE6).