This is the second instalment in the series Comparing Java Mock Frameworks. This time, I will take a look at how the frameworks instantiate test stubs and mock objects as well as their overriding capabilities. The whole series consists of the following posts:

At last, I managed to finalise this second instalment. I didn’t think that I would need so so long to make this first comparison. This is quite a lot of reading and trying out. At the end of the day, I hope you’ll find it useful. If there is any mistake in this post, I’m eager to know and to fix it.

For this post, I experimented with the frameworks to get a feel for how they addressed the following issues:

Are mocked objects strict by default (explicit or implicit verification)?

What are the main differences between the frameworks?

Remark: I didn’t dive into the details of each framework. It is not my intent to dive into the details of what each framework provides. I hope I didn’t overlook anything worth mentioning. If so, please tell me.

Different Approaches to Mocking

The tested framework fall into two different categories, depending on the way they create the mock objects:

Using proxy: Create a dynamic implementation of a class using the Java reflection API;

Allows redefining the public interface of a class;

Remapping classes: Remap the class in the class loader using the Java Instrumentation API;

Allows mocking objects that are created using the new operator;

Allows completely redefining a class;

The limitations of what the different frameworks can mock come from the approach their take.

It is also possible to annotate a class variable that shall be mocked. However, the annotation requires that some specific method be called to allow for the injection. The following code snippet extracted from the official documentation illustrates how to use the annotation.

jMock instantiates both stubs and mock objects in the same way. It can naturally mock interfaces using an instance of the Mockery class. The test case class must be annotated be annotated with @RunWith(JMock.class) for JUnit4.

By default mocked objects are strict. It is however possible to change this behaviour by declaring expectations and using either ignoring or allowing. The two are equals and one can choose the method that best expresses the intention.

context.checking(new Expectations() {
{
ignoring(aMock); // The invocation is allowed any number of times but does not have to happen.
allowing(anotherMock); // The invocation is allowed any number of times but does not have to happen.
}
});

jMock:

Mocks interfaces, abstract classes and concrete classes;

Cannot mock final classes. It will throw an exception when trying to mock such a class;

Does not mock final and static method. Instead, it delegates the call to the real implementation;

PowerMock is a framework that builds on either Mockito or EasyMock. It provides additional functionality to the underlying framework.

To use PowerMock with JUnit, you have to use the PowerMock test runner.

The @PrepareForTest annotation is used to tell PowerMock to prepare certain classes for testing (typically final classes, classes with final, private, static or native methods that should be mocked and also classes that should be return a mock object upon instantiation). The code snippet below illustrates how to annotate a test case to use PowerMock.

Remark: Because of some class loader issues in the EasyMock extension, I did not manage to make it work as documented. As I didn’t want to spend much time investigating the issue, I hope it will be fixed soon.

Mockito Extension

PowerMock provides a class called PowerMockito that is used for creating mock objects and initiating verification, and expectations. For example:

To make your test case Unitils-enabled, you must either extend the class public class UnitilsJUnit4 , or use the JUnit test runner UnitilsJUnit4TestClassRunner. The following code snippet illustrates both ways.

JMockit includes two mocking APIs. The first one is for “behaviour-based testing” and is named the “JMockit Expectations & Verifications” API. The second API is for “state-based testing” and is named “JMockit Annotations” API.

Mock objects are essentially defined using annotations. By annotating a variable with @Mocked, JMockit replaces any instantiation of the class in the test fixture with a mocked instance. JMockit also provides the @NonStrict annotation that leads to non strict mock objects, meaning that they will not fail when non pre-recorded methods are called on the object.

Any object annotated with @Mocked is non-strict as long as no expectation “strict” is defined on it, otherwise, it becomes strict.

@Mocked
MyInterface interfaceTestDouble; // This is a non-strict mock object but it will become strict if no strict expectations are defined on it
@Mocked
MyClass classTestDouble; // This is a non-strict mock object but it will become strict if no strict expectations are defined on it
@NonStrict
MyFinalClass finalClassTestDouble; // This is a non-strict mock object
@Test
public void someTest() {
// All objects are not null as they were injected by JMockit
interfaceTestDouble.someMethod();
classTestDouble.someMethod();
finalClassTestDouble.someMethod();
}

Because JMockit replaces the actual implementation of the mocked class, any instance of the class will be a mock. In the following code, both objects will be mock objects even though one was instantiated from the regular class.

@Mocked
MyClass classTestDouble; // This is a mock object
@Test
public void foo() {
MyClass myClass = new MyClass(); // This is a mock object as well!!!
}
...
public class AnotherClass{
private MyClass myClass;
public AnotherClass() {
myClass = new MyClass(); // This is a mock object when the instance is created in the scope of the test.
}
}

It is also possible to obtain a mock object for a given class by declare it as a parameter to the test method. A parameter of a test method is considered, by default, to be a “mock parameter”; it can optionally be annotated with @Mocked and/or @NonStrict.

Allows mocking without dependency injection (replaces instances even when created using new in the code under test);

Returns default values for non-void methods as 0 for primitive integers, empty collections and arrays, and so on;

Mock objects defined with @Mocked are non-strict by default. Defining expectations on them makes them strict;

JMockit Annotations API (for stated-based verifications)

Mock objects are used in behaviour-based verifications. Nevertheless, JMockit provides an API that allows for state-based verifications. I present this API for illustrative purposes, because JMockit is the only framework that provides a dedicated API to perform this kind of validations.

The @UsingMocksAndStubs annotation specifies the classes that must be mocked or stubbed out in the test fixture. Using this annotation, classes passed on as parameters to the annotation will be stubs. This annotation allows using a provided implementation of the stub or mock object to use. The list of classes provided to the annotation contains either production classes or mock classes that are annotated with @MockClass.

The following code snippet is excerpted from the official documentation.

In the above example, the static initializer of the Database class is also stubbed out, by specifying the special method name “<clinit>”.

There is one more way to mock classes with JMockit. It consists in defining what is called a mock class, which is a class with methods annotated with @Mock. The mock methods of the mock class with substitute for the actual implementations in the target class to mock . Here is an example from the official documentation.
This code snippet defines a mock class, with methods that will substitute for real methods of the target class.

So far, I can sum up the main characteristics of the frameworks in the following categories:

Type: Proxy-based or class remap, which determines the extent to which behaviour can be changed in the created objects;

Stubbing: Determines whether the framework allows stubbing, with or without extra configuration steps;

Default return values: Not all frameworks behave the same with default values for non-void methods;

Instances are strict/non-strict by default: Some frameworks make objects strict by default while others don’t and some others allow specifying the strictness. That can have some importance depending on your testing approach;

Declarative creation: The framework provides annotations or wrapper types to identify the objects to mock or stub out;

Injects dependencies: The framework injects the dependencies of the class under test that are identified as either mock objects or stubs;

Replaces instances created using new: The framework does not need that dependencies be injected into the class under test. It can replace instances created using the new keyword;

All the evaluated frameworks are capable of creating both mock objects and stubs. The differences lies in whether or not the frameworks objects are strict or not by default.

The table below summarises the collected information:

Framework

Type

Stubbing

Default return values

Instances are strict/non-strict by default

Programmatic creation

Declarative creation

Injects dependencies

Replaces instances created using new

Mockito

Proxy-based

Yes

0, null, false

Non-strict

Yes

Yes

No

No

EasyMock

Proxy-based

Yes

Strict instances throw exceptions. Non-strict return 0, null, false

Depends on the ways the object was created. Type can be changed at runtime

Yes

No

No

No

Mockachino

Proxy-based

Yes

0, null, false

Non-strict

Yes

No

No

No

jMock

Proxy-based

Yes

0, empty String, false, empty array, null

Strict

Yes

No

No

No

PowerMock

Class remap

Yes

0, null, false

Strict with EasyMock and non-strict with Mockito

Yes

No

No

Yes

Unitils

Proxy-based

Yes

0, null, false

Non-strict

No

Yes

Yes

No

JMockit

Class remap

Yes

0, empty collections, empty arrays,

Non-strict by default

No

Yes

Yes, replaces instances of @Mocked classes

Yes

The two different approaches to mocking allow for different kinds of usage. On the one hand, proxy-based framework do not allow mocking certain elements that are present in legacy code. They are of less help if the code under test does not allow for dependency injection for example. On the other hand, frameworks that are based on class remapping allow testing any kind of code. That comes in particularly handy for legacy code.

In what I have seen so far, I like the possibility to annotate the objects that will be mocked and have them injected. It can help write clean test code.

Conclusions

Since I wrote the ﻿first installment, I had time to think again about what I want to achieve in this series. Comments on the post also made me realise that I do not want to choose any framework. I just want to put together information that will help me understand each of them. This information will help me, and any one who might find it helpful, choose the most appropriate tool for the job at hand. I believe it is a question of taste. If it weren’t, all frameworks would look the same :-).

On the other hand, I believe that some other points such as the quality of the documentation as well as the easiness to get started with build tools such as Maven for example might be valid criteria. I shall consider those to help choose a framework.

To conclude, I can’ t really draw any conclusion just yet. We will see in the next posts how to actually use the framework to test some code.

There is a mistake in the section about jMock, though: mocks created with this API *are* strict by default. To record non-strict expectations on the objects created with the “Mockery#mock(Class)” method, one needs to use the “allowing(mock)” of “ignoring(mock)” methods. So, by default all unexpected invocations to jMock mocks will cause the test to fail, unless it was explicitly specified otherwise in the test. I suspect you missed the fact that jMock requires the use of @RunWith(JMock.class) on the test class, or that it extends a jMock-specific base class.

I found another mistake related to the strict/non-strict default, this time in the Unitils section. I think you mixed the new “Unitils Mock” API with the old “EasyMock support” API. The “@Mock” annotation only applies to the latter. In Unitils Mock, all mocks created with “Mock” fields are always non-strict; semantically, this API is just like Mockito.

I also spotted a few issues in the JMockit section – about which I know more than a bit ;):
1) First of all, the JMockit toolkit includes *two* mocking APIs, one for “behavior-based testing”, another for “state-based testing”. The comparison could cover both, but I would recommend only the first one (the “JMockit Expectations & Verifications” API), since it’s the one most similar to the mocking APIs of competing tools. So, the @Mock, @MockClass, and @UsingMocksAndStubs annotations would be disregarded, as they are part of the “JMockit Annotations” API.
2) The “@Mocked” annotation (which can be applied to an instance field or a test method parameter) declares a mocked type in the Expectations & Verifications API, but it alone does not make a mocked instance strict. Only if and when an expectation is recorded inside a “new Expectations() { … }” block is that a mocked instance becomes “strict”; otherwise, it becomes non-strict.
3) Tests written with the Expectations & Verifications API tend to be more succinct than equivalent tests written with other APIs. For a concrete example comparing JMockit and Mockito tests, see http://code.google.com/p/jmockit/source/browse/trunk/samples/mockito/test/org/mockitousage/JavadocExamples_JMockit_Test.java.

Maybe using “Strict by default” as a main characteristic for evaluating mocking APIs isn’t the best idea. Personally, I believe the important question is whether the API supports both strict and non-strict expectations/mocks, rather than which behavior is the default. In EasyMock there really isn’t a default, since the user has to choose the mock creation method each time; in JMockit the “default strictness” depends on whether and how expectations are recorded on the mocked instance/class, and on whether the @NonStrict annotation was used or not.

Thanks very much for the thorough review! Playing with all the frameworks got me confused at some point obviously ;-)

Regarding your remarks:

1/ I must say that JMockit s very rich and versatile, which makes it indeed hard to compare with the other when considering all its APIs. Therefore, and based on your comments on JMockit, I updated the post. I divided the section into two sub-sections, one for each API. I keep Annotations API for illustrative purposes. I hope everything is correct now :-)

2/ Fixed as well

3/ I wonder whether I could use your example with other frameworks, for the sake of comparison. You compared JMockit with Mockito, that’s one thing less to do ;-)

Regarding the “strict by default” criterion, I proposed it because it is good practice to usually have one mock object and one or more stubs in a test. Stubs are non-strict mock objects when created with mock frameworks. Given the fact that stubs must “outnumber” mock objects in tests, I found it a pretty good argument :-) Maybe I should clarify that point. On the other hand, what matters really is the ability of the frameworks to provide both behaviours so that objects can be used as mocks or stubs. As they can all do it, there is indeed nothing more to say.

Using JMockit, isn’t it a good practice to define stubs, which we know are stubs with @NonStrict and “the” mock with @Mocked? This would lead to more test fixtures but make the code very readable. What are the best practices with JMockit?

Two small corrections, though:
1) A mock field/parameter annotated with @Mocked will be non-strict as long as there are no *strict* expectations recorded on it. Expectations are recorded inside “expectation blocks” (zero or more such blocks in any test), which are defined as subclasses (typically anonymous) of either mockit.Expectations or mockit.NonStrictExpectations. The first class is similar to the one with the same name in jMock, and implies strict expectations unless specified otherwise. With the second class, all expectations recorded will be non-strict.
2) A parameter of a test method is considered, by default, to be a “mock parameter”; it can optionally be annotated with @Mocked and/or @NonStrict. This is part of the Expectations API, not the Annotations API.

I really appreciate the prompt edits to the technical sections of this second part, but I must say the “Conclusions” section is quite perplexing. First of all, it does not focus on the topic at hand: “Creating mock objects”. Secondly, it makes several clearly subjective claims, for which I cannot find support in facts nor in sound logical reasoning. But this is a topic for a future conversation, I believe. I only hope this series of posts is meant to be a fair and objective comparison between existing mock tools for Java.

I concur regarding the conclusion. Actually, when I wrote the post, I could not come to any real conclusion. I even realised that I was kind of biased in a way because I am used to a certain framework. The conclusion therefore sounded pretty subjective. This is definitely not the intent of the post.

Therefore, I updated the post. What do you think about the new sections observations and conclusions?

With regards to the summary table in the “Current observations” section, I have some thoughts:
1) Since all mocking APIs support stubbing, it would probably be better to remove this column, or alternatively add some detail which indicates the (small) differences between the various APIs; for example, whether the API has specific methods/annotations for stubbing versus mocking.
2) Concerning “mock object creation”, the big difference, I think, is in the support for two very different ways to obtain mock/mocked instances: the *programmatic* way (which relies on method calls) and the *declarative* way (which relies on field/parameter declarations, annotated or not). It would be interesting to classify each API in categories such as: “programmatic only” (EasyMock, jMock, Mockachino), “mostly programmatic” (Mockito, PowerMock), or “mostly declarative” (JMockit, Unitils Mock). Another dimension would be whether the mocking API provides an annotation for the declaration/configuration of mock fields, as do JMockit (@Mocked, @NonStrict), Mockito (@Mock), and PowerMock (@Mock).

hi I am confused with one thing that which is better among jMockit and Powermock with mockito framework ? Acc to me syntax of powermock is easy to understand so it will be easy. And Powermock with mockito also have all features similar to jMockit.
can you please elaborate in detail ?