This article is about Bromine, an open source functional web test automation tool for Selenium. We’ll go through the background for Bromine and explain
what Bromine is and why you should (or shouldn’t) use it. Finally we’ll introduce two real world scenarios in which Bromine is used.

If you have worked with test automation of web sites you have most likely heard of Selenium. For those of you who have not, Selenium is an
open source test automation tool that has a lot to offer, in particular capture
/ replay functionality and means to run your test in a distributed environment.
This is the foundation that Bromine builds on. In short Bromine can be described
as a management tool for Selenium. It offers is a way to organize your test scripts and easily run them in different browsers and operating systems.

Once installed you can access Bromine through your browser. From here you can create new projects that can contain an array of requirements
and test cases. For each test case you can upload a Selenium test written either
in Java or PHP. Once you have uploaded the script head over to the Test Lab
where you will be able to run either a single test case or a complete suite of
tests which you have organized under a requirement. If you have the necessary
capacity, the tests will be run in parallel saving you time compared to running them sequentially via the IDE or on a single machine through Selenium RC.

So, how did Bromine come to be? If you have ever worked as a software tester you know how daunting it can be testing that the same menu link
still works, build after build. This is where Selenium came into the picture. It
is open source, so there was no need to invest in a proprietary automation tool
which could easily have cost several thousand dollars per user in acquisition
and then come the service agreements on top of that. However, we soon realized
it was complicated to organize, run, and share tests between testers as Selenium
did not have anything to offer on that part. We then began the development of
what was then called TRM (Test Result Manager). This was back in 2007. From here
on we kept adding features to TRM and then in early 2008 we decided to rework
the whole thing and renaming the project Bromine. We decided on Bromine because
it is the 35th atomic number in the periodical table next to Selenium. This
phase of the development continued to about mid 2009 where we decided to
completely rework Bromine again, this time around using a PHP framework
(CakePHP) hoping that it would attract more people to contribute as it would be
easier to adopt the programming style. This led to Bromine 3 Beta being released
autumn 2009, which entailed a signification amount of improvements over the last
installment of Bromine. Today you can download Bromine 3 RC2 from http://seleniumhq.org
or http://www.brominefoundation.org/
and in the coming months we are planning to release Bromine 3 Final.

What is Bromine and why should I use it?

Bromine is not perfect. Actually, it’s quite far from what we we would consider perfect. It will fit some organizations just great and
others less so. We’ll try to explain what the pros and cons of Bromine are in
this section. But don’t take our word for it; give it a try. It’s free after all.

First off Selenium does functional web testing and Bromine makes using Selenium easier. Selenium is probably the most used functional web
testing tool out there. It works by starting a browser, injecting it with a lot
of JavaScript functionality and then executing JavaScript through the browser
which simulates user actions. A command could be ‘click("submit
button")’ or ‘verifyElementPresent("some div")’. The words
in quotation marks are what are called locators, i.e. some string that
identifies a part of a web site. This is mostly element ids or xpaths. Selenium
comes in a couple of shapes. The most used are Selenium IDE, a Firefox plugin
that does record and replay, and Selenium Remote Control (RC), which is a Java
server that listens for commands and upon receiving them opens and controls a
browser and sends the results back for each command. Lots of more info on
selenium can be found at http://seleniumhq.org

What Bromine does, and does well, is executing selenium RC tests, storing the results and presenting them in an easily understandable way,
all as an easily accessible PHP/MySQL web application. This is the core
functionality and this is what we’ve spent most of our time coding. The
rationale behind this is that when we first started using Selenium RC, these
were the most frustrating issues we faced. What good is a test if it’s hard to
run and the results are difficult to save and interpret? From this core
functionality we have begun to develop the business side of Bromine; requirement
and test case handling. This side is fairly bare bones at the moment, but we’re
planning on to focus our efforts in this direction over the next couple of releases.

A standard workflow in Bromine goes something along the lines of this: Management creates requirements and details which operating
systems and browser combinations these should be tested in. Test manager
creates test cases to cover these requirements. The tester records a
script in selenium IDE doing what the test case specified. Tester exports the
test script into Java or PHP and uploads it to Bromine. The tester selects the
test case (or requirement, or even an entire project) and presses the run icon,
which will launch the tests in the required OS/browser combinations (given that
you have test machines setup having the required OS/browser combinations of course).

The tests are scheduled, much like a printer job queue and executes when the test machines are available, i.e. not running any other tests.
The test script drives the browser and Bromine captures every command sent
between the test script and the RC server as well as the status of the commands.
This is then saved as a test run on the given test case and OS/browser
combination. The test case will be flagged as passed or failed based on the
results of the latest test run, and in turn any requirements the test case
belong to will be flagged as passed or failed as well. The poorest result takes
priority so that a single failed test case in a requirement with lots of test cases will mark the requirement as failed.

Bromine is unique in that it acts as a GUI for Selenium, closing the gap between technically minded tester and less technically minded
management. In addition, it is free and open source. We don’t feel that there are any other tools out there that fit this description.

Included in the Bromine RC2 release was Hudson and Sauce Labs integration, which are some features we’re pretty excited about.

Figure 2 - Bromine with the scheduling pane open for a single test case

A real life scenario at Test Lab I/S

Test Lab I/S is owned by Visti Kløft and Jeppe Poss, two of the Bromine founders, and specialize in automated testing. The following is an
example of why and how Test Lab uses Bromine, based on a couple of large
projects. Unfortunately it’s impossible to be more detailed due to NDA’s with the customers.

How do Test Lab I/S use Bromine in the real world? Bromine helps us solve issues in big test setups. We have tested large web applications
which included large number of sites, many localized versions, browsers, and large number of test cases.

In Bromine you organize your test cases and with one click you can start hundreds of test runs in different browsers. Imagine a project
with a hundred test cases that needs to be tested in four browsers (FF3, IE7,
IE8 and Safari) on three environments (test, QA and production). This gives 400
test runs each time we need to run our regression tests and that creates a few issues:

Capacity issue (load balancing)

Maintenance of the test scripts

Overview of the current state of the application (does all the features work?)

Capacity issue

Running 400 test cases takes time and to solve this issue load balancing is needed. Bromine does just that. Bromine looks at the available
Selenium RC (called nodes in Bromine) and will try to run as many tests as possible at the same time.

Maintenance of the test scripts

Having a hundred test scripts requires a lot of maintenance and instead of doing our own revision control you can just use SVN (or similar
tools). All the test scripts in Bromine are in a certain folder and if you put
that folder under revision control you can easily maintain your script without having to learn another new tool.

Overview of the current state of the application

Bromine gives us an excellent overview of the current state of the application. While it’s not out there doing exploratory testing for us,
we have it setup to test for known bugs as well as the core functionality of the
application under test. Are all the pages online? Does each page links to the
right pages? Does the rate, buy and signup functionality work? Having certainty
on these issues frees up time for tasks humans are good at; finding tricky bugs and making developers miserable.

A real life scenario at Roskilde Festival

The following case study is a blog post by Anders Nickelsen (twitter.com/anickelsen),
the leader of the QA-team in the IT-development group at Roskilde Festival, reproduced here with permission.

The organization

The Roskilde Festival located in Roskilde, Denmark, is the second largest music event in
Europe and entertains more than 100.000 guests over 4 days (+4 day warm-up)
annually. The festival is made possible by approximately 25.000 volunteers that
work either before, during or after the festival. Amongst these are 40
volunteers working throughout the year to develop and operate 10+ specifically
designed web-based information management systems used by the festival.

The challenges of volunteer developers

As with many open-source projects, the group of volunteer developers is very dynamic, since people can leave or join rather spontaneously.
Therefore, a key element is to provide new developers a simple entry into the
development process, and ensure that systems stay stable even if developers drop out.

Contrary to many open-source projects, developers are not end-users, so there is no obvious itch to scratch [1] until new developers have
experimented with the systems and technologies. Also, once new developers have
made contributions to the systems, they need to see it used by end-users in production soon after to avoid losing interest in the project.

Quality assurance by continuous integration

To provide sandboxes for experimenting and short ‘time-to-production’,
we have built a versatile quality assurance framework to allow new developers to
dive right into changing codes and try out new things, without the risk of
breaking any of the running systems. Through a series of verification steps,
both manual and automated, all system contributions are continuously integrated into the production mainline in a lean and reliable way.

The development process

Our setup builds around the philosophy that "developers need flexibility, operators need stability". To achieve this we use a chain
of environments. As contributions flow through these environments - from the
developers hand into production mainline - the developers flexibility becomes
limited as focus is turned to stability of a release. The chain consists of the
following 4 environments that are focused around per-system subversion repositories.

Development environments: VirtualBox images with replicates of the production environment that can be used locally by developers
to try out new things. The developers commit contributions from the images into subversion projects.

Preview environment (alpha-test): Central servers with exports of the subversion projects that allow developers and change managers to
check out subversion revisions to preview individual changes and run unit tests

QA environment (beta-test): Central servers with exports of the subversion projects that allow beta-testers to verify releases through
integration tests and acceptance tests, and release managers to test release
procedures. QA environments are separated from preview environments because they
are also used for end-user education, which requires them to be more stable and less frequently updated.

Production environment: Central servers that host production versions of the systems, which are used by end-users.

The test framework

The test part of the continuous integration framework consists of the following tools:

Hudson

Selenium

Bromine 3

Sauce OnDemand by Sauce Labs

Hudson automatically triggers basic testing in the preview environment on every commit to Subversion. Commits are considered atomic and
only include one change each, which is validated manually - also in the preview
environment. Atomicity enables easy rollback of changes and preserves traceability.

Integration tests and end-user acceptance tests are recorded using Selenium IDE and collected in Bromine. Before the release of a new
version, Hudson triggers Bromine to execute the entire test-suite of Selenium
tests from the previous release as a regression test on the new system deployed
in a QA environment. The tests are executed either on dedicated internal
Selenium clients, or in the Sauce Labs cloud via Bromine’s integration with
Sauce OnDemand. Bromine produces result reports which provide an overview of the
success rate of the tests. The result reports are automatically integrated into
Hudson, which makes the test result overview fast and clean. Futhermore,
end-user testers make exploratory testing in the QA environment to validate the released bug-fixes and features.

When the new release is approved, the Bromine test suite is updated to include new Selenium tests for the new bug fixes and features.

After release, the wheel turns once again with planning of a new release, where everyone - new as old - can participate, with small or big
changes, thanks to the integrated quality assurance framework.

Conclusion

Bromine is an open source functional web testing tool. It is based on Selenium which is also open source and probably the most used web
automation tool out there. Bromine’s main strength is its ability to make the
life of the Selenium powered tester easier. Bromine provides an easy way to
organize and run your test scripts. It also gives you the opportunity to tie in
the results of the test scripts with requirements and test cases. Bromine is a
web application, which is easy to access and use for a large number of users. A
headless Linux server with 256mb of RAM will probably host Bromine
satisfactorily for an average small business (up to 5 users online simultaneous,
more will require additional memory). Bromine integrates with some of the cool players in the industry like Hudson.

Bromine is not a magical tool which will make all your problems vanish in the blink of an eye. It’s just a tool and tools are nothing
without methods. You’ll need to remember your ISTQB syllabus, and recall that
automation is hard work and requires more abstraction than record and playback.
It definitely pays off in the end though, as we hope our case studies have shown.