Design

Project of the Month: Thucydides

By John Ferguson Smart, December 12, 2011

Results from large, automated runs of acceptance tests can be hard to understand quickly, especially for projects in active development. Thucydides synthesizes acceptance test results and generates useful charts and analysis that pinpoint errors and present a visual overview of the project development status. Better yet, it works with JUnit, Selenium, and easyb.

Acceptance Test Driven Development, or ATDD, is an advanced form of Test Driven Development (TDD) in which automated acceptance criteria  defined in collaboration with users  drive and focus the development process. This helps ensure that everyone understands what features are under development.

It is often said that Test Driven Development is not about testing, but design. In the same way, Acceptance Test Driven Development is as much about communication as it is about acceptance testing. Indeed, ATDD has a strong focus on team communication and reporting, aiming to ensure that the requirements being implemented are well understood by all parties. The great value of ATDD is its ability to express high-level requirements in the form of automated acceptance criteria that can help validate and clarify these requirements in business terms that both users and developers can understand.

Thucydides is a Java-based open source library designed to help teams implement Acceptance Test Driven Development more effectively. Thucydides was a Greek historian known for his astute analysis skills who rigorously recorded events that he witnessed and participated in himself. In the same way, the Thucydides framework observes and analyzes your acceptance tests, and records a detailed account of their execution.

Automated acceptance tests for Java applications can be written in plain old JUnit, or using more specialized Behavior Driver Development (BDD) tools such as Cucumber and easyb. These tools can help make your tests clearer and more expressive by describing and reporting on expectations in terms that the business can understand, typically structuring requirements in a "given-when-then" format. However real-world acceptance tests are (and should be) expressed in very high-level terms, and standard BDD reporting does not reach this high level.

Ideally, however, automated acceptance tests should be able to document your application's behavior and features in great detail. Thucydides takes this communication-focused reporting and documentation farther by providing a blow-by-blow account of the steps executed during each acceptance test, even including screenshots for each step in the case of Web tests. It also makes it easier to group the acceptance tests executed by story or by feature, giving a higher-level picture of project progress in terms of the number and relative size of passing and pending acceptance tests.

In a nutshell, Thucydides lets you define your acceptance criteria in terms of automated tests and high-level steps, and then reports on the execution of these steps when the tests are executed. The tests themselves can be implemented directly as JUnit tests, or using a BDD testing framework.

If you are implementing a Web application, it makes excellent sense to automate at least some of your acceptance criteria in the form of automated Web tests. As I demonstrate later, Thucydides caters to this need by providing very strong support for Selenium 2 testing. And if you also need to test back-end services (such as Web services, EJBs, and so forth), Thucydides plays nicely with JBoss's Arquillian testing tool for container-based testing.

The rest of this article takes Thucydides for a spin. To follow along, you will need to have JDK, Maven, and Firefox installed on your machine.

Obtaining the Sample Project

To save you a little time, we will be working with an existing Thucydides project. This sample project is one of the Thucydides showcase projects, which contain a several projects that illustrate how Thucydides works for different situations. To get started, you can clone or fork the Github Thucydides project, or just download the Thucydides source code. Once you have obtained the source code, simply open up the "thucydides-showcase-simple-webtests" project in your favorite IDE.

This site lets you search for Maven artifacts in the Maven Central Repository, and then view details about those artifacts.

To see the sample project in action, go to the project directory and type "mvn verify." This will run the Web tests in Firefox and generate some reports detailing the executed tests. Before we go any further, open up target/site/thucydides/index.html and take a look around. Here, you will see a summary of the Thucydides test outcomes in terms of the features we are trying to test (Figure 2).

Figure 2: A Thucydides aggregate report.

If you drill down into the features, you will also see detailed narratives describing each test, including screenshots for each stage of the test (Figure 3).

Figure 3: A Thucydides test report.

The rest of the article explains how Thucydides generates these reports in more detail.

Specifying Acceptance Criteria with Thucydides

The first step in any project is to consider the requirements. The aim of the sample project is to test two aspects of the Maven Central website: searching the repository for artifacts, and viewing the details of a given artifact. We could sum up these requirements in a couple of simple user stories:

A user searches for an artifact by group name and artifact ID.

A user displays the details for a particular artifact.

Thucydides helps you map automated acceptance criteria back to the original requirements they are supposed to be validating. Remember that, in agile approaches in general and in ATDD in particular, a feature cannot be considered "done" before all acceptance criteria are automated and passing. As I will demonstrate, the Thucydides reports give you a succinct vision of what requirements have been specified, what has been completed, and what is still waiting to be implemented (or "pending implementation," in BDD parlance). It can also keep track of progress in terms of the number of completed features over time.

For this to work, however, you need to tell Thucydides a little about your application requirements. Requirements are defined in the form of an annotated Java class, which, in our sample project, looks something like this:

As you can see, requirements are grouped into "features," annotated with the @Feature annotation. Each feature contains nested classes that correspond to the user stories (or use cases) that relate to this feature.

The next step is to define and automate acceptance criteria for these stories. In this project, I am using easyb, the Groovy-based BDD tool mentioned earlier. easyb is well suited to expressing high-level acceptance criteria in business-friendly terms. Before you start work on a requirement, it is good practice to outline your acceptance criteria in high-level terms, without necessarily going into the implementation details.

Tools such as easyb make it easy to write tests in this form. These will be executed during the automated tests and flagged as "pending." A simple "pending acceptance" criterion for the sample project is shown here:

using "thucydides"
thucydides.tests_story SearchForArtifactsByName
scenario "A developer looks for an existing Maven artifact by name and group id", {
given "the developer is on the Maven Central search page"
when "the user searches for artifacts with the name 'thucydides'"
then "the thucyidides core artifact should appear in the results list"
}

Notice how we use the thucydides.tests_story instruction to map this acceptance test back to the story it is testing. This is how Thucydides knows where to place the test results for this particular test in the overall test reports.

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task.
However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Video

This month's Dr. Dobb's Journal

This month,
Dr. Dobb's Journal is devoted to mobile programming. We introduce you to Apple's new Swift programming language, discuss the perils of being the third-most-popular mobile platform, revisit SQLite on Android
, and much more!