Using Serenity with Cucumber, Part 2

In the first post of this series, I got us set up with a Serenity repository using Maven as our build tool. I also started us on the path of applying Cucumber-JVM by putting a feature file in place. In this post we’ll start tying Cucumber into Serenity’s runtime operations.

Let’s jump right in with getting a test runner set up.

Test Runner

In Serenity, the Cucumber tests are run via a JUnit runner. This, however, requires a dedicated test runner that will actually know how to “run” the feature files which are, keep in mind, just text files. What Cucumber does is provide a mechanism that essentially consumes a Gherkin-style feature file and then uses its internal API to break that feature file down into parts, effectively crafting an abstract syntax tree.

Let’s start out by creating a package to hold our runner. We’re going to be operating in the src/test/java root package for now. Create a package in a way that you normally would: a domain package and then a more specific contents package. For purposes of this article, I’ll go with the following: com.testerstories.tutorial.todos. You can see I’ve used my own domain and then tagged on the addition of the application I’ll be testing, which is the TodoMVC application. Within that package, create another package called features.

Here’s a visual of what things should look like for you:

In this package, create a class called RecordTodos. This is going to be our runner class. So what goes in here? When you run your Cucumber-style scenarios with Serenity, you’re going to use a CucumberWithSerenity test runner. So let’s start off with the following code:

Java

1

2

3

4

5

6

7

packagecom.testerstories.tutorial.todos.features;

importnet.serenitybdd.cucumber.CucumberWithSerenity;

importorg.junit.runner.RunWith;

@RunWith(CucumberWithSerenity.class)

publicclassRecordTodos{}

Depending on how familiar you are with Cucumber, you might wonder how this runner is going to do anything with the feature file that we created in the last post. To answer that, one thing I’ll call to your attention to right now is that the package structure I’m using under src/test/resources is not the same as that under src/test/java. Does this matter? Strictly speaking, no.

If the feature files are not in the same package as the test runner class — as is the case with my example so far — then you also need to use a @CucumberOptions class to provide the root directory where the feature files can be found. Let’s add this to our code:

Okay, wait a minute. I just said provide the “root directory” where the feature files can be found. Yet here I’m including the entire directory as well as a specific feature file. That means this particular runner, when executed as part of the JUnit runner, will only execute that particular feature file. In Cucumber-JVM you will often have multiple runners. This is something I’ve found is a bit of cognitive friction for those coming to Cucumber from a language like Ruby. These multiple runners can be used to point at specific feature files, specific directories of feature files, or the entire root directory for all feature files.

Technically, this configuration using @CucumberOptions is not necessary. But to avoid using it, this means that your feature files (in src/test/resources) must be stored under the same package as the runner class (in src/test/java). In other words, you could match the runner package in your resources as such:

A convention of Cucumber-JVM is to automatically look for feature files in the resources directory that match the package from which the runner is executing. If you decide to go with your own convention, you simply have to add a bit of configuration, as I’ve shown, with the @CucumberOptions.

I should note here that the current Serenity documentation specifically states that if you go the package-matching route then you have to supply a “serenity.requirements.dir” property in your serenity.properties file. This setting is used to override the location of the requirements. So you would have to do this:

That being said, I haven’t found this to be true. Things work perfectly fine even if I do not include that property.

For purposes of this post, I’m going to stick with the approach I started with. I have never been a big fan of making the features repository (in src/test/resources) match the Java package structure. My current impression is that this conflates the two domains a bit and constrains how flexible I can be. That being said, it also depends on the configuration you do or do not want in place. If you abhor the idea of using @CucumberOptions, then a package matching structure would likely be good for you.

Cucumber Reporting

While you will likely be using the Serenity aggregate reports, you can certainly still generate Cucumber-specific reports as you need to. Keep in mind that the runner you are using is no different than that which you would use with Cucumber-JVM itself. So, for example, if you wanted to generate the simple Cucumber HTML output, you could add the following to your runner:

You can also hook into existing Cucumber reporting libraries, including them just as you would in a project that wasn’t using Serenity. That’s an important point to understand. You aren’t “giving up” anything by using Cucumber integrated with Serenity.

Crafting the Step Definitions

In Cucumber, each line of the Gherkin scenario maps to a method in a Java class. The conceptual name for these classes are called “step definitions” in the Cucumber context. These classes use annotations that match the Gherkin keywords, like @Given, @When and @Then. Those annotations are used to instrument methods that will be called when a given line is recognized from the feature file, via token extraction that is done from the regular expressions contained with the annotations.

Let’s first create a place to put these step classes. Under your todos/features package, create a steps package. In that package, create a class called RecordTodoSteps. So you’ll have this structure:

Let’s start by handling our Given step. Modify the source of the RecordTodoSteps as such:

Java

1

2

3

4

5

6

7

8

9

packagecom.testerstories.tutorial.todos.features.steps;

importcucumber.api.java.en.Given;

publicclassRecordTodoSteps{

@Given("^the todo application$")

publicvoidstart_the_todo_application(){

}

}

So here we’re matching a specific step from the feature file. In fact, if you are using a Cucumber-aware IDE — like IntelliJ or Eclipse — you should find that the feature file itself will now indicate that this particular step has been found in the code, whereas the other two have not. For example, in IntelliJ you’ll see something like this:

Executing the Scenario

Cucumber has the notion that any step succeeds unless an assertion or expectation makes it demonstrably fail. So right now we have a scenario that is ultimately pending, but with one passed step. Let’s check out what this looks like in the report, particularly given how I described the reports in the first post. Run this:

mvn clean verify

You’ll see a lot of output that I recommend you go through at some point to familiarize yourself with what is happening. This output is a combination of Serenity and Cucumber-JVM reporting. Once the execution completes, open the report at target/site/serenity/index.html.

On the “Overall Test Results” tab, you’ll see a “Tests” table at the bottom of the page. This contains the “Add a new todo” title line, indicating that this test has three steps. If you click on the title, you’ll be taken to the steps for the scenario. You should see something like this in your report:

The Given: Adding a Page

Here I’ll focus on adding a page object in the mostly traditional sense of how these things operate. Eventually I’m going to want to refactor away from “full” page objects and go more towards a “minimalist” style page object, putting focus instead on step libraries within Serenity.

First let’s add an action to our @Given method:

Java

1

2

3

4

5

6

7

8

9

10

11

12

packagecom.testerstories.tutorial.todos.features.steps;

importcucumber.api.java.en.Given;

publicclassRecordTodoSteps{

TodoPage todoPage;

@Given("^the todo application$")

publicvoidstart_the_todo_application(){

todoPage.open();

}

}

I’ve added a call to the open() method on a page object (todoPage) that doesn’t exist yet. So I need to create an instance of this TodoPage class. I’m actually going to create this class in my src/main/java package. To follow along, create a package — again, within src/main/java — of com.testerstories.tutorial.todos. If you used a different structure in src/test/java, make sure to use whatever you did there. With that package in place, create another package within it called pages. In that package create the TodoPage class. Just to be entirely clear, here’s what my current structure looks like, assuming you’ve followed along in both posts up to this point:

Make sure that this class extends the PageObject class as follows:

Java

1

2

3

4

5

6

packagecom.testerstories.tutorial.todos.pages;

importnet.serenitybdd.core.pages.PageObject;

publicclassTodoPageextendsPageObject{

}

If you go back to your RecordTodoSteps class, you’ll see that the line todoPage.open() does not have an error, even though you have not written any open() method. (Do make sure to add the import of the pages.TodoPage.) The reason you don’t have to provide an open() method is because Serenity is doing this for you. And that’s happening because you are are extending the TodoPage class with PageObject, and it is the PageObject which provides the open() method.

However, you do have to somehow indicate what page to actually open. There are various ways you can do this. For now, I’ll use the @DefaultUrl annotation on the page class:

Java

1

2

3

4

5

6

7

8

packagecom.testerstories.tutorial.todos.pages;

importnet.serenitybdd.core.pages.PageObject;

importnet.thucydides.core.annotations.DefaultUrl;

@DefaultUrl("http://todomvc.com/examples/angularjs")

publicclassTodoPageextendsPageObject{

}

You can certainly run your scenarios now (mvn clean verify) and you’ll find that the step still passes but now at least you see the browser appearing and navigating to the TodoMVC application for AngularJS.

The When: Adding the Test Action

Now let’s create a matching step definition for our ‘When’ step in the feature file. In the RecordTodoSteps class, add the following:

Java

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

packagecom.testerstories.tutorial.todos.features.steps;

importcom.testerstories.tutorial.todos.pages.TodoPage;

importcucumber.api.java.en.Given;

importcucumber.api.java.en.When;

publicclassRecordTodoSteps{

TodoPage todoPage;

@Given("^the todo application$")

publicvoidstart_the_todo_application(){

todoPage.open();

}

@When("^the todo action '(.*)' is added$")

publicvoidadd_a_todo_action(StringactionName){

}

}

You should find that your ‘When’ step from the feature file is now showing as matched if you are using an IDE. Since I’m going with a generic page object approach here, the next step is to add a method to the page object that will handle adding the provided bit of text (stored in actionName) to the element on the TodoMVC form that corresponds to the input text field. First let’s add the action:

Java

1

2

3

4

@When("^the todo action '(.*)' is added$")

publicvoidadd_a_todo_action(StringactionName){

todoPage.addActionCalled(actionName);

}

Now create that method in your TodoPage class and add the following code to it:

Feel free to run this. You’ll find that it works. The TodoMVC application shows up and the provided text is entered into the input field.

This should look very familiar to you if you have ever worked with Selenium before. You could also shorten this considerably using a Serenity convenience method that matches what you would see if you’ve ever done work with jQuery. Specifically, you could change the above method call as such:

Java

1

2

3

4

5

6

7

8

9

10

11

12

13

packagecom.testerstories.tutorial.todos.pages;

importnet.serenitybdd.core.pages.PageObject;

importnet.thucydides.core.annotations.DefaultUrl;

importorg.openqa.selenium.Keys;

@DefaultUrl("http://todomvc.com/examples/angularjs")

publicclassTodoPageextendsPageObject{

publicvoidaddActionCalled(StringactionName){

$("#new-todo").sendKeys(actionName);

$("#new-todo").sendKeys(Keys.ENTER);

}

}

This makes it a little simpler and removes the need for a few imports. However, Serenity gives an interesting API that lets you actually do this in a more readable fashion, with a bit more code but chained together:

Java

1

2

3

4

5

6

7

8

9

10

11

12

packagecom.testerstories.tutorial.todos.pages;

importnet.serenitybdd.core.pages.PageObject;

importnet.thucydides.core.annotations.DefaultUrl;

importorg.openqa.selenium.Keys;

@DefaultUrl("http://todomvc.com/examples/angularjs")

publicclassTodoPageextendsPageObject{

publicvoidaddActionCalled(StringactionName){

$("#new-todo").type(actionName).then().sendKeys(Keys.ENTER);

}

}

The main thing to understand here is that the $() method is defined on the PageObject class. This method can take an argument that is either a type of WebElement or a String that is an XPath or CSS Selector. The method is designed to return what is known as a WebElementFacade. I’ll come back to the WebElementFacade idea later. For now, just know that type() and then() are methods on a WebElementFacade instance and they return an instance of WebElementFacade. This is what lets you chain the methods together and still call Selenium actions, such as sendKeys().

The Then: Adding the Observable

Let’s finish off this post by adding the implementation of the ‘Then’ step. Add the following to your RecordTodoSteps class:

Java

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

packagecom.testerstories.tutorial.todos.features.steps;

importcom.testerstories.tutorial.todos.pages.TodoPage;

importcucumber.api.java.en.Given;

importcucumber.api.java.en.Then;

importcucumber.api.java.en.When;

importstaticorg.assertj.core.api.Assertions.assertThat;

publicclassRecordTodoSteps{

TodoPage todoPage;

@Given("^the todo application$")

publicvoidstart_the_todo_application(){

todoPage.open();

}

@When("^the todo action '(.*)' is added$")

publicvoidadd_a_todo_action(StringactionName){

todoPage.addActionCalled(actionName);

}

@Then("^'(.*)' should appear in the todo list$")

publicvoidaction_should_appear_in_my_todo_list(Stringaction){

assertThat(todoPage.getActions()).contains(action);

}

}

As before, you should see that the ‘Then’ step in your feature file indicates that it has been matched. All we have to do is implement that getActions() method in our page class. Do that by adding the following to the TodoPage class:

Java

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

packagecom.testerstories.tutorial.todos.pages;

importnet.serenitybdd.core.pages.PageObject;

importnet.serenitybdd.core.pages.WebElementFacade;

importnet.thucydides.core.annotations.DefaultUrl;

importorg.openqa.selenium.Keys;

importjava.util.List;

importjava.util.stream.Collectors;

@DefaultUrl("http://todomvc.com/examples/angularjs")

publicclassTodoPageextendsPageObject{

publicvoidaddActionCalled(StringactionName){

$("#new-todo").type(actionName).then().sendKeys(Keys.ENTER);

}

publicList<String>getActions(){

returnfindAll(".view").stream()

.map(WebElementFacade::getText)

.collect(Collectors.toList());

}

}

Here findAll(), defined on a PageObject, takes in a By instance which should be a selector. The rest of this is basically a functional representation of a map-reduce operation. I don’t want to get side-tracked into too many details here but essentially I’m finding all elements on the page that match the selector “.view”. I’m then streaming that collection and mapping to the getText() method of the WebElementFacade. Behind the scenes this handles using a Selenium WebElement and calling a method on getting the text. The collect() method is then used on the stream to get each item collected and create a list of string from that collection.

At this point you have a fully working test. Try it!

Make Sure It Fails

As with any test, you should be able to prove that it is capable of being wrong. So make an error in the text of the string being checked for in the ‘Then’ step. Then run the test again.

When you view the report, you’ll see the Then step is colored red and indicates an assertion error. You can click the “View Stack Trace” to see more details, which will show something like this:

Expecting: to contain:
but could not find:

Note that this stack trace is actually in a form that Serenity calls “simplified.” What this means is that any calls to instrumented code or test libraries are abstracted out of the final report. However, if you do want a non-simplified stack trace, you can provide a property in your serenity.properties file, stored in the root of your project, like this:

1

simplified.stack.traces=false

Screenshots

One thing to notice in the report is that the screenshot for the second step does not show the text actually entered in. The property serenity.take.screenshots can be set to configure how often the screenshots are taken. This is yet another setting that can be stored in serenity.properties. To see it in action , add this:

1

serenity.take.screenshots=AFTER_EACH_STEP

You could also use the setting DISABLED, which means no screenshots at all will be taken.

It Comes Down to Abstraction Levels

What I’ve shown here is that you can use Cucumber-JVM just as you would outside the context of Serenity. You can even generate the same reports you were used to with Cucumber. You saw that I generate the native Cucumber report as part of the @CucumberOptions. The point of this, however, is that the abstraction layer of Cucumber, at the English level, is now fully in place.

As I mentioned at the start of this post, I’m becoming more and more convinced that the abstraction layer provided by Cucumber is not necessarily effective or efficient.

What I’ve also done here, however, is further reinforce the traditional concept of the page object abstraction. And that’s important to note because managing and refining the appropriate levels of abstraction is important in automation. It’s easy to make mistakes it becomes even easier, in my opinion, when you have natural language feature files as part of your abstraction layer. This is just something I think testers need to be aware of.

In the next post, I want to refine what we did here and move towards the idea of better abstraction, away from the page objects. I believe that when you start to take this approach, the overall abstraction of feature files can become a little easier to manage.

About Jeff Nyman

Anything I put here is an approximation of the truth. You're getting a particular view of myself ... and it's the view I'm choosing to present to you. If you've never met me before in person, please realize I'm not the same in person as I am in writing. That's because I can only put part of myself down into words.
If you have met me before in person then I'd ask you to consider that the view you've formed that way and the view you come to by reading what I say here may, in fact, both be true. I'd advise that you not automatically discard either viewpoint when they conflict or accept either as truth when they agree.

You can change the Maven directory structure if you want. Keep in mind that Maven is convention-based. But you can use configuration to override that. To be honest, I don’t do that very much with Maven but, for example, you can change the source directory with a line in your POM like this: