1. Introduction

In this article we will try to examine current state of the automated testing concept in the Java world.
As a reference, we will go through basic and advanced spock-framework features and compare them with what JUnit4/JUnit5/TestNG/Hamcrest/AssertJ/Mockito/whatever can offer instead.
We will try to understand Spock philosophy and find out both pragmatic and emotional answer to the questions:
should I use spock-framework in the year of 2016?
how to convince my manager that "yes, we should"?
how to convince my teammates that "no, we shouldn’t"?

2. What’s About Modern Unit Testing?

To my mind, automated testing is one of the most powerful software development concept suggested so far.
After receiving great attention as part of XP manifesto, it made possible to increase size of projects and complexity of code we are able to maintain by orders of magnitude.

JUnit, started by Kent Beck and Erich Gamma, played key role in the development of the automated testing idea as it was one of the first ever and surely the most popular and influential automated testing library.

TestNG, started by Cédric Beust, was inspired by JUnit and initiated to offer wider and more powerful range of functionalities than JUnit originally does, keeping the same general concept. Later, most of the TestNG initiatives were implemented by JUnit.

As recent study shows, JUnit is used by about amazingly 60% of Java projects and TestNG shows quite strong result of about 6%.

Both JUnit and TestNG shares the same framework architecture often called xUnit (surprisingly after SUnit and JUnit itself).
It’s described best by two concepts: test structure and assertions.
As test structure obviously is set of features to describe tests organization, and assertion is a function that verifies the behavior (or the state) of the unit under test.
If we look at the most simple JUnit and TestNG tests we will found them quite similarly implementing both concepts:

Actually, except for import statements, these code snippets are the same.
My feel is that, not arguing with all benefits of JUnit and TestNG, they both were frozen long time ago and no longer evolving.
JUnit is ready to overcome this problem with 5.0.0 release, scheduled for the end of 2016.
Spock recently has epochal 1.0 release and 1.1 is on the way (1.1-rc2 version is already out).
TestNG has only several 6.9.x releases with no significant changes in them.

So, there are several possibilities for passionate developers to continue evolving their test automation instruments. Let’s examine them shortly.

For now we can already examine second milestone release, and we can see that after a great work of JUnit team we will get a fresh new look at how JUnit should be implemented.
The most significant change is introduction of solid testing backend model, but it’s not a main topic of this article.
Let’s better check how the simplest JUnit 5 test looks like.
How many changes you can see?

Actually I hope in the year 2016, no one is writing tests that looks like ones I’ve already shown.
As a base rule, they should be at least two steps forward: test should have clear inner structure and assertion should clearly explain its intent:

2.3. Spock

Finally, we can make really big step forward and try completely new testing approach - Spock - test framework developed by Peter Niederwleser and Luke Daley.
Spock is greatly inspired by Groovy language philosophy and used Groovy dynamic nature a lot.
But the same as with Gradle, where you do not need to know Groovy to develop build scripts, here you need to know almost nothing about Groovy to use Spock.
If you treat Spock like separate test-specific language and learn its main concepts, you can easily use them in conjunction with plain old Java code.

As big steps are much more effective when they are done immediately, here is a very simple Spock test.
If you write tests that looks like the last example, it will be very simple for you to get what is going on there:

3. Why Try Spock?

Why someone should try to use Spock on their projects?
In the following sections I will demonstrate all benefits that can be gained from Spock usage.
Additionally, I will show drawbacks of such decision.
But despite both pros and cons there are two reasons why everyone should try Spock at least for their pet projects and later evaluate if it will be useful for the real ones:

Spock is the next generation test framework. What does it mean? Spock takes all the years of automated testing experience and pain and re-imagine how it should be implemented if we look at it from a different angle.
What if we do not need to use the same language for testing as we use for production code?
What if it can be language designed for testing specifically?
What if test context can be moved from comments to code?
What if there are different testing approaches and we should offer convenient features for all of them?
Spock asks plenty of What if? questions and based on them re-implement well-known testing approach in another (arguably better) way.

Spock is enterprise ready. Spock is more than 6 years old. Spock is already released 1.0 version and has stable API.
It’s completely mature framework that can be safely used without any fears.

4. How To Start With Spock?

As you can already see, it’s more than easy to start using Spock.
Begin with adding these three dependencies to your project:

If you already have test code you want to migrate to Spock either from JUnit of TestNG take there five easy steps:

Rename your test files from *.java to *.groovy and move them to src/test/groovy directory.

Extend your test class form spock.lang.Specification class.

Remove all @Test annotations and change return type for all test methods to def.

Enforce inner test structure by using following labels: setup for test object setting up, when for test actions and then for assertions.

Enjoy becoming a Spock developer in less than 15 minutes.

However, if you are starting from scratch, you will probably want to write more idiomatic Spock code.
The first step towards this goal is start using Spock assertions.

5. Spock Assertions

In the first place, each test framework offers two simple assertion constructions: assertEquals checks for equality of two objects and assertTrue checks if specified expression is true. However, using such constructions has two critical drawbacks:

when test fails, it’s hard to understand why it has failed;

test code is not maintainable, as it’s not obvious what was the intent behind each assertion.

Just try to look at following test failure logs and guess what is tested and why tests have failed:

Original Spock assertions show a little intent behind their code, but they are not less than perfect in demonstrating test context that lead to the failure. Moreover, all you need to start using them is to place boolean expression inside then block:

Spock assertions were so successful that they were ported to plain Groovy, which means that if you use JUnit or TestNG you can write you code in Groovy, immediatelly getting Spock assertions in your old-fashioned tests:

Here is one important thing about Spock philosophy as project.
If you do like traditional approach you can continue using it. Spock will just add some fancy Groovy magic to make it better.
In this concrete case, you can use Groovy traits to extract repeatable setup/cleanup behavior and reuse it easily across your test classes:

Instead of using static keyword for shared resources use @Shared annotation

2

@AutoCleanup annotation invokes close method of the shared resource after each test case

3

Tests can have individual setup blocks

4

Tests can have individual cleanup blocks

Let’s look at @Shared annotation closely.
It will demonstrate why it’s so important to use test-specific language for your tests instead of general-purpose language like Java.
When you develop your test (but not only test) code, each time you try to find language feature that suites best your need.
For example, if you need to share some resource within test class and you use Java, natural choice will be static.
But, what if you want to do the same in several test classes? You can extract this behavior to superclass like following:

If you look carefully, you will find that some of these tests will fail.
Single resource will be shared between all three test cases and you have no control over it.

Spock, as test-specific language, can keep in mind needs of test code developers and offer features that gives you more control on test code behavior.
In this example @Shared annotation clearly will share resource only within single specification. Our example is fixed:

7. Inner Test Structure

To my mind, the most important Spock feature comparing to other test frameworks (that do not have anything similar) is enforcing inner test structure.
While it’s generally a good practice to set up inner test structure using comments, Spock provides language features that make you do it.
It’s really great design decision, because comments, as they are not natural part of the code, becomes outdated very quickly.
When something is part of the code, it is always up to date.
Let’s look at the example:

You are already familiar with setup block, but pay attention on string (not comment!) that follows each test block describing what is going on inside it

6

expect block contains set of conditions that all should evaluate to true

7

when block contains set of actions on object under test

8

and block helps to split sets under when or then blocks

9

then block contains set of conditions that all should evaluate to true after previous when block

Great follow-up from the fact that test structure becomes part of the test code is that this information becomes available at runtime.
Projects like Spock reports leverage this information and, for example, generate very detailed reports based on it.
Just look at test report that is ready to publish for your team non-development folks:

8. Data Driven Tests

To my mind, while the previous feature is greatest and the most unappreciated Spock feature, this one is the most famed one.
Surely, it’s very powerful, but it’s needed only for some specific cases, while inner test structure is generally applicable.
I’m talking about data driven tests. Let’s look at quick example:

where is new test block used for defining variables that work as multiple test parameters for same test scenario

2

Test parameter can be defined as collection of values

3

Or just as assignment rule

4

Defined test parameters can be used as regular variables inside other test blocks

This example shows complex test case, when the same scenario is run against multiple test parameters within same equivalence class.
Test is considered as passed if and only if test scenario passes all test parameters.
Here is how test report looks for this case:

Another use case supported by Spock is reusing same test code for test cases from different equivalence class.
Here is example that demonstrate this and some other concepts:

This example shows dividual test cases, when test representation is separated from test code.
Each test parameter generate individual test case, that shares the same code, but not test result, with others.
Here is how test report looks for this case:

Traditional test frameworks mix up both concepts, and moreover provide very poor language support for data-driven test:

Pay attention that using Object[][] is very unsafe and unclear for test parameters set up, and even more, that TestNG does not supports test naming easily.

What is interesting, JUnit 5 instead of having concept of data-driven tests, has more general and very powerful concept of dynamic tests.
It means that test cases can be generated at runtime, based on provided data and any other source:

What Spock do is, again, come up with very convenient way to declare that test intent is that exception should not be thrown.
Pay attention that it’s done not through comment, but rather with language construction:

What is very disappointing is that TestNG has a very limited, closed and ugly solution for the same problem.
If you want to check exception error message, you should supply expected regexp to @Test annotation:

10. Mocking

As you probably already notice from the study shown at the beginning, industry standard de facto for all mocking activities is Mockito library.
As usual, if you like it (and probably you do), you can continue using it with Spock.
And again Spock offers you its own solution which uses fancy Groovy magic to make mocking more pleasant for developers.

By the way, providing all needed functionality for any kind of testing approach is a great part of Spock philosophy, which is clearly shown by this example.

Let’s now exam and compare Mockito and Spock solutions for various mocking tasks one by one:

10.1. Stubs

Stubs are used when you need to replace some complex behavior of units you depends on by simple one, like returning constant value or throwing exception.

then: '''first random generator is invoked once to generate password length,
then at least 8 times more and it is never invoked for anything else'''
InOrder inOrder = inOrder(random)
mockito inOrder.verify(random, times(1)).nextInt(10)
mockito inOrder.verify(random, atLeast(8)).nextInt(26)
mockito inOrder.verifyNoMoreInteractions()

One small detail is that for using Mockito invocations inside then block you need to make them always return true, because as failure is signaled by throwing exceptions, Spock can misinterpret value returned by Mockito.
This issue is simply fixed by adding following method to your specifications:

Just use mockito {mockitoMethod} for all Mockito interactions

staticdefmockito(def verify) {
true
}

10.3. Spies

Spies are used when you need to monitor and verify interactions with unit you are testing.
The difference is that mocks are used to fake unit behavior and monitor its usage, while spies monitors without changing original behavior.

For this case, Spock has introduced small but very useful feature. You can wrap any variable or method invocation by old() inside then block and it will be replaced by the corresponding value before execution of previous when block:

11.3. Testing async code

While two previous features were so small and minor, the next one is really important.
How can we test async code? Let’s forget for now that is should be turned to synchronous before testing.
Sometime it’s impossible, sometimes it’s not reasonable.
The most simple and the most strange approach is just ignoring that code is async:

While it will work sometimes, such approach has two problems: sometimes async call will finish long time before delay does and this time will be wasted, and sometimes async call will last a little longer than delay does and the test will fail.
The more intelligent approach is to add some logic to check if async call is finished:

The problem of this approach is that, actually, it’s a very clear test code anti-pattern. For many reasons, keeping non-linear logic in test code reduces tests reliability a lot.

Spock suggests two native features for testing async code.

First one is called 'polling conditions' and it evaluates assertions regularly until they become true or timeout is reached.
Assertion is considered as failed if it does not become true within specified time limit:

Second feature is called 'async conditions' and it is used to pass assertion expressions as callback for async code.
Assertion will be evaluated ones as usual when async call finishes or it will be considered as failed if it will be not reached within time limit:

What is more interesting is conditional disabling.
In JUnit it is implemented with Assumptions API.
Assumptions are very similar to assertions, but when they fail, it causes test to be skipped, not failed:

In Spock, the same feature is implemented as part of Ignore API.
Moreover, it has more various features like disabling and enabling conditions and predefined conditions (environment operating system, installed java version, system properties and environment variables):

Besides that, Spock has one more small and very useful feature, especially for functional testing.
If you need to re-run just one single test from the full suite, you no longer need to mark all the rest with @Ignore.
Please welcome self-describing @IgnoreRest annotation:

Moreover, Spock 1.1 introduces one more interesting way to disable test case called Pending Feature.
If you mark test case with a @PendingFeature annotation, it will have following behavior:
if it fails, it will be reported as skipped (ignored) test case, as test case for unimplemented feature is not important for overall test result;
if it passed, it will be reported as failed (sic!) test case, as it’s strange that test case for unimplemented features succeeded, so it should be explicitly reported:

If you need your test to change system properties there is always a challenge to restore them back after test execution.
Spock introduces annotation @RestoreSystemProperties, which does exactly the thing.
For the sake of example, another Spock annotation is used: @Stepwise, which makes Spock to execute tests in the order they are specified in the code.
Being damned in the unit testing worlds, it is more than useful for integration and functional tests:

14. Geb

And the last thing, that worth special mention is Geb.
Geb is a browser automation solution, developed by one of Spock main contributor Luke Daley.
It brings together the power of WebDriver, the elegance of jQuery content selection, the robustness of Page Object modelling and the expressiveness of the Groovy language.
It can be used for scripting, scraping and general automation — or equally as a functional/web/acceptance testing solution.
And preferable Geb testing approach is greatly inspired by Spock framework.
Just look at simple example:

Even if you like JUnit or TestNG so much that you don’t want to migrate them, you can introduce Spock into your project via Geb.
Geb is obviously the best in his role and if you need web functionally testing it is more than worth trying.

15. Why Use Spock?

Concise syntax
If you are tired with verbosity and ceremonies of Java, which are multiplied several times for test code, Spock is a great choice.
It uses Groovy magic and own features to make your test code short but still understandable.

Language for testing instead of testing struggling language
Naturally, you use language features to express your intent in test (and not only test) code.
If you use language designed specially for testing, you will obviously always can find constructions that were designed specially for your case.
And when you use general-purpose language like Java for testing you just try to make it fit your needs, which is not working well in all cases.

Clear test structure
Spock enforce inner test structure.
Being a generally good practice to do it via comments or method separation in traditional test frameworks, Spock makes it true in 100% test cases.

Code and reports readable for everyone
While Spock code is by design and by Spock philosophy can be read even by non-development folks like QA engineers or managers, it’s also possible to immediately share Spock reports with anyone interested without any additional post-processing.

Powerful built-in features
Spock has plenty of small and huge built-in features ready to cover any of your need immediately.
Even for your 1% chance test requirement there probably is a Spock feature that makes it easier and more pleasant to develop.

Extensions
Even if Spock does not have a good feature for your case, it has simple and powerful extension mechanism which you can use to make it better fit your needs.

One framework for any testing approach
Spock philosophy states that you should be able to use single Spock library for any possible testing approach.
So, Spock is useful for unit, integration and functional testing.
Spock has powerful native support of assertions and mocking.
Spock is good for TDD, BDD and chaotic testing.
And anyway, Spock can be greatly integrated with helper testing libraries like Hamcrest, Mockito or others, if you want to.

Behaves as one more JUnit runner
Inside, besides tons of pure magic, Spock is just one more JUnit runner.
It means that is integrates easily with build tools, CI/CD tools, reporting libraries and actually everything that accepts JUnit.

Works for both Groovy and Java
Being great part of Groovy world, just like many other Groovy projects, there is no great need to dig into Groovy to understand Spock.
After getting its main concepts, you can test both Groovy and Java code, using either Groovy or pure Java expressions inside test blocks.

16. Why Do Not Use Spock?

The first thing you should keep in mind about Spock, is that usually you can find corresponding features in JUnit or TestNG.
If no, Hamcrest/AssertJ/Mockito/someone can probably help you.
If still no, you can try to imitate some of Spock features even within JUnit/TestNG limitations.
Being not so beautiful, it will work in 90% of the cases, while Spock is just a fresh look on the traditional approaches or even when it introduces new features that can be reproduced.

Another thing is that Spock is now in the process of finding new development model.
Luke Daley has left the project long time ago.
Peter Niederwleser has stopped Spock contribution more than a year ago after joining Apple.
Now Spock is public Github project, and after some standstill after 1.0 release seems that now it is on the way to stable wide community development model, as 1.1 release is almost ready for GA.

17. When Use Spock?

The main precondition before start using Spock, is your team being ready to accept it.
There is no other limitation or significant expected problems for it.
Spock integrates well with all your other solutions, can be easily migrated from existing JUnit/TestNG code, does not require great additional knowledge or efforts to maintain.

Obviously, the simplest case is when you are about to start new project: just use Spock from the very beginning and enjoy it, nothing more.
If you have legacy code without unit testing coverage, the case is pretty similar: it’s not a problem to start using Spock for it immediately.

If you already have JUnit tests in your code, the good strategy for Spock migration is start developing new test cases via Spock.
As Spock is in some way just a JUnit runner both test suites are integrated into one view without any efforts.
During the further process, you can start continuous migration from JUnit to Spock one test class by one, which is very easy as renaming files and changing five lines of code for each test case.
For the TestNG, things are little bit harder, so good strategy will be keeping two separate test suites and quick migration from TestNG to Spock.

And the last great entrance point for Spock into your project is Geb.
If you are about to introduce functional testing into your web project, Geb will be the best choise, that besides all other benefits, will introduce Spock to your team.

18. Looking Further

Thanks for your attention!
If you do like or don’t like this article, please share your opinion.
I will appreciate your feedback (including surely negative one) in any form available for my attention (like social media, direct e-mail, or etc.).

If you are interested in this topic I can recommend you following resources: