Hudson CI for Testing

We are heavily using Hudson for testing a product we are working on. Other than maintenance and some instability issues, it’s been a great experience working with Hudson. We started out with just a single machine running Hudson server and then soon moved to a master-slave configuration where we have setup one master connected to four slaves. We are currently running about 10-15 jobs on this setup at one point or another during the day and use it to run pre-checkins (well in a way *post*-checkins), BATs and regression runs.

Here are a couple of screenshots that I took when we were just getting started and were stabilizing Hudson and the product.

We have the following kinds of jobs running on this system:

Pre-checkins: We provide development team with a set of tests that they are supposed to run before they commit any code. To ensure that the development team was doing this, we have a job that runs pre-checkins tests on every commit. This helps us ensure that things remain green though out the day and we come to know of failures asap.

BATs: Basic Acceptance Tests as another category of tests that help us ascertain the daily quality of builds. Before the QA picks up a build for testing, we run the BATs against the daily official build and make sure that all tests pass.

Regression runs: We also use hudson to run regression tests. At time of taking this screenshots, we had only about a 1000 tests and hence had a single job which would run all the tests and would take about 6-7 hours to complete. Now, we have split up that job into multiple jobs by using multiple build stages that Hudson supports. I will talk more about this in a later post.

Helper jobs: We have parameterized jobs setup that help any one (dev or QA) to start any kind of test run against an official or sandbox build.

CI jobs: This is how Hudson was originally intended to be used. For each branch that we work on, we poll the SCM every 5 minutes to check for checkins and build the dev and QA code. This helps us ensure that the code any checkin hasn’t broken the QA build and that users have access to the latest artifacts/distributions at all times.

For the project, we had invested time into providing infrastructure to pick up official/sandbox builds and then using these builds to setup environment, configure tests and then execute the tests. We just had to put in some more time to change or add features to make it all work easier with Hudson. Now, instead of letting everyone fend for themselves when it comes to setting up environments for testing and then reporting the results, we use Hudson to automate that process to a very large extent and make these tasks more manageable and the product quality more visible.