No more DSL for acceptance/integration testing

Everybody seems to be using Fitnesse or Cucumber for integration testing these days. The main idea is that users (testers/business analists and programmers) can write what your program should do in plain text. The programmer then creates a small layer of code which gets interpreted and executes your actual program to see if the specified requirements are met.

Fitnesse example

For example a (hypothetical) piece of code in Fitnesse as it could have been used in the Port of Rotterdam:

This is something the testers and product owners can read and understand. Behind this layer of text Fitnesse will try to execute pieces of code. This is left as an exercise for the programmers to complete. They’ll need to write something like:

This is where the problems start, there is a lot of room for typos. Which will send you all over the codebase constantly translating Fitnesse text like “and it reports anchor down” to “andItReportsAnchorDown” during your searches.

Also people always say, because it is ‘plain text’ the testers, analists and product owners can instantly start writing tests. But this just isn’t true… It isn’t just writing down tests/requirements, it is programming with a DSL. Normal human text isn’t good enough, it still needs some DSL structure. They’ll have to learn to program in the Fitnesse DSL language. In almost all project I’ve worked on, it is the programmers writing and maintaining tests, people from the business stay far away from Fitnesse.

If you want to refactor the scenarios/fixtures in Fitnesse you’re in for a treat. Everything is bound by name only, and renaming code for example won’t rename the DSL! Using Fitnesse ‘sounds’ very agile, but it doesn’t really make the code agile at all. Most test code I’ve worked on is fragile, very ridged and not easy to refactor. Why not take advantage of that IDE and statically typed language if the programmers are writing and maintaining the tests?

Using code instead

Today we had a “ShipIt”-day (nautical-pun intended) at work, where we’re free to work on any project for 24 hours straight and we present our findings afterwards. This boosts the creative ideas and in the long run efficiency and innovation.

After using Fitnesse now for three years we’ve become used to it, and don’t ever question its usage. Although we feel the pain of the rigid/fragile test code on a daily basis. I decided to start rethinking the way we do acceptance tests.

My goals was to make our tests:

Simpler to write

Easier to refactor

Maintain the readability

Easier to reuse

Faster! (no DSL parsing)

This is what the initial design now looks like. The example used above would become the following:

@Jitness // Annotation that indicates this is a test class
@SuiteSetUp({ // This is called once for all the methods in the class below
@Execute(
type = ShipExecutor.class,
data = { "1001", "Maersk Bali", "TANKER", "D6DZ8", "7420211" })
})
})
@SetUp({
// @Execute annotations here will be called before each of the methods below
// The @SetUp annotation can also be placed on a single method in the test class
})
public class AnchorageTests {
public void givingCommandAnchorDownCausesShipToAppearInList() {
final AnchorageScenarios scenarios = new AnchorageScenarios();
scenarios.whenAShipReachedThePortCalled("Maersk Bali");
scenarios.andItReportsAnchorDown();
Assert.assertTrue(scenarios.theShipIsShownInTheListOfAnchoredVessels());
Assert.assertEquals(1, scenarios.theListOfAnchoredVesselsNowHasSize());
}
public void anotherTestHere() {
... etc ...
}
}

Using Jitness (working title) the scenarios will be implemented in exactly the same way we’ve already been doing. This makes it easier to transition from Fitnesse to this code.
The only bit of ‘magic’ is the execute annotation. You can write executors like this:

The main advantages of using this are obviously leveraging the full potential of your IDE, including renaming/refactoring. Changing methods and classes in your production code will automatically show up as changes in Jitness. And everything is easy to refactor because it is just code! This will eventually lead to much more agile code, simpler and easier to refactor. Obviously there is one main drawback, it is harder for testers to read and write. But if they pair up with a programmer this should be no problem I think. I’d rather have the testers learn a bit of coding than the programmers learn a text DSL which leads to fragile tests.

Another big advantage is execution time, using compiled code is always faster than parsing plain text or HTML at runtime. Also this code is compiled during the build, so there is no need for additional classpath (Maven) plugins. I’m really fond of the idea to use code to describe the tests, and I’m very curious how my colleagues are going to rate this tomorrow when we demo the “ShipIt” ideas. With a couple of more days work this can be fully tested and made robust enough to start using it and maybe even open source it.

What are your thoughts? Any ideas about improving the API? Should we just keep using Fitnesse/Cucumber or any other alternative DSL instead?

I haven’t been too big of a fan of external DSLs for any use case, not just for testing. Granted, Eclipse’s Xtext seems to be going in the right direction by providing tooling directly with your custom external DSL.

But most popular testing frameworks built to work with JUnit have chosen to provide APIs in the form of an internal DSL. For example: Mockito or JMock. The power of internal DSLs is precisely the fact that

– you can leverage the Java compiler to verify your code correctness
– you can use IDE autocompletion to write tests faster
– you can refactor your tests

I have similar thoughts. I feel that in the project I’m working on, cucumber is used as a poor man’s scripting language. A language without any type checking, parameter names or anything else.
From what I read, the original intention behind cucumber was to allow people to write behavioral tests which should be higher level than integration tests. I don’t understand exactly what this means and most of the cucumber tests I’ve seen were just integration tests automated using cucumber.
The only benefit I’ve seen is that QA are more inclined to write cucumber tests which is a great benefit. I don’t see QA writing tests using JUnit.
But the reason I write this comment is to ask, how is your framework different from JUnit or TestNG? It looks like it is trying to solve the same problem (automating tests) using the same methods. Did you take a look at these tools? Maybe you can use them without inventing the wheel all over again?