The Problem with Mocks

The first post about mocks covered the basics of how to use
Python’s mock library.
Using mocks has many advantages (which we’ll discuss in
When and When Not to Use Mocks)
but they also have a downside: mocks can get out of sync with the real code
that they’re mocking, which can mean that
the tests might pass, even if the code is wrong. In this post we’ll explain
how this problem with mocks works, and some advanced features of the mock
library that you can use to minimize the problem.

Consider this class Bar, a class Foo that uses it, and a test for Foo:

>>>importmock>>>>>>classBar(object):...defsome_method(self,some_arg):...pass...>>>classFoo(object):...def__init__(self,bar):...# Foo calls a method that does not exist on Bar....bar.a_method(an_arg=23)...>>>deftest_foo_does_not_crash():...Foo(mock.MagicMock())...>>># The test passes, even though the code is wrong:>>>test_foo_does_not_crash()>>>>>># In production, when a real Bar object is used, Foo would crash:>>>Foo(Bar())Traceback(mostrecentcalllast):...AttributeError:'Bar'objecthasnoattribute'a_method'

The Foo class is wrong - it calls a Bar method that doesn’t exist, passing
in an argument that doesn’t exist. In production Foo will crash. But the test
for Foo passes because it uses a MagicMock in place of a Bar object. You
can call any method with any arguments on a MagicMock, so Foo doesn’t
crash.

This was a simple example, but there are all sorts of ways that mock objects
can be out of sync with the real objects they replace, causing tests to pass
even though the code is wrong:

The mock may have attributes, methods or arguments that the real
object doesn’t

The mock’s return values may differ from the real object’s, for example
it may return a different type of object that has different attributes

The mock’s side effects and behavior may differ from the real object’s,
for example maybe the mock fails to raise an exception when the real object
would do

These problems are most likely to occur when dependency code is modified and
you forget to update the code that uses it - the user code goes out of sync
with the dependency code but the tests still pass because they use mocks.

Autospeccing - a partial solution to the problem with mocks

To avoid the problem with mocks we want an automatic way to create mock objects
that have only the attributes, methods and arguments that the real objects
have, that have the same return values, side effects and behaviors as the real
objects, and that are updated automatically when the real classes are changed.

Unfortunately there’s no perfect solution to this, but Python’s mock library
does have a feature called
autospeccing
that gets us some of the way there.

Mock’s create_autospec()
function creates an object that has only the attributes, methods and arguments
that the real objects would have, and that crashes just like the real objects
would if you try to access something that doesn’t exist:

>>># Create a mock version of the Bar class by passing the real Bar class to>>># create_autospec():>>>MockBar=mock.create_autospec(Bar,spec_set=True)>>>>>># Now use the mock bar class to make a mock bar object:>>>mock_bar=MockBar()>>>>>># You can call methods that real Bar objects would have, passing arguments>>># that real Bar objects have:>>>mock_bar.some_method(some_arg=23)<MagicMockname='mock().some_method()'id='139778195937360'>>>>>>># Passing an argument that doesn't exist crashes:>>>mock_bar.some_method(some_arg=23,another_arg=True)Traceback(mostrecentcalllast):...TypeError:toomanykeywordarguments{'another_arg':True}>>>>>># Passing too many positional arguments also crashes:>>>mock_bar.some_method(23,True)Traceback(mostrecentcalllast):...TypeError:toomanypositionalarguments>>>>>># Passing too few arguments or missing a required argument also crashes:>>>mock_bar.some_method()Traceback(mostrecentcalllast):...TypeError:'some_arg'parameterlackingdefaultvalue>>>>>># Calling a method that doesn't exist crashes:>>>mock_bar.a_method()Traceback(mostrecentcalllast):...AttributeError:Mockobjecthasnoattribute'a_method'

As well as calling methods, accessing attributes that don’t exist will also
crash.

Trying to call the mock object, as in mock_bar(), will work if a real Bar
object would be callable (and will accept only the arguments the real objects
do). If real Bar objects aren’t callable then calling a mock bar object will
also crash.

Notice that we never specified the method name some_method or its argument
some_arg. create_autospec() figured these out automatically from the
real Bar class. Tests that use create_autospec() aren’t coupled to the
dependencies that they mock - if the Bar class changes then the next time you
run the tests create_autospec() will create mocks that match the new Bar
code, without any changes to the test code.

The spec_set=True in the MockBar = mock.create_autospec(Bar, spec_set=True)
call prevents you from writing the value of an attribute
that doesn’t exist on the real class, mock_bar.does_not_exist = True will
crash with AttributeError. Without spec_set=Truereading attributes
that don’t exist will crash but writing them will succeed.

In this case we used create_autospec() to create a MockBarclass, and
then created our own mock_bar object from the mock class: mock_bar = MockBar().
You can also create a mock object directly by passing instance=True to
create_autospec():

mock_bar=mock.create_autospec(Bar,spec_set=True,instance=True)

Autospeccing is recursive: if the real class has attributes then those
attributes will also be autospecced. For example here the Foo class has a
string attribute named some_attribute, a mock Foo’s some_attribute has
only those methods that a real Foo’s some_attribute would have:

>>>classFoo(object):...some_attribute="a_string">>>>>>mock_foo=mock.create_autospec(Foo,spec_set=True,instance=True)>>>>>># Calling a method that exists works:>>>mock_foo.some_attribute.decode()<MagicMockname='mock.some_attribute.decode()'id='139778195402064'>>>>>>># But calling a method that doesn't exist crashes:>>>mock_foo.some_attribute.does_not_exist()Traceback(mostrecentcalllast):...AttributeError:Mockobjecthasnoattribute'does_not_exist'>>>>>># Accessing an attribute that doesn't exist also crashes:>>>mock_foo.some_attribute.does_not_existTraceback(mostrecentcalllast):...File"<stdin>",line1,in<module>AttributeError:Mockobjecthasnoattribute'does_not_exist'

Trying to call an attribute will also crash if the real class’s attribute
isn’t callable.

create_autospec() actually isn’t used much in the Hypothesis tests currently,
but it probably should be. If you’re going to make a mock object, you should
probably make it using create_autospec() if you can.

spec_set

The Hypothesis tests often use a mock feature called spec_set that is similar
to create_autospec() but not as good. Instead of calling create_autospec()
you just instantiate a mock.MagicMock() (or more often a mock.Mock() in the
Hypothesis tests) and pass the class being mocked as a spec_set argument to
the mock constructor. As with autospec, mocks created in this way only have the
attributes and methods that the real object would have. Unlike with autospec,
you can still pass any arguments that don’t exist without crashing:

>>>classFoo(object):...defsome_method(some_arg):...pass...>>>mock_foo=mock.MagicMock(spec_set=Foo)>>>mock_foo.some_method(some_arg=23)<MagicMockname='mock.some_method()'id='140189369916688'>>>>>>># Calling a method that doesn't exist crashes:>>>mock_foo.method_that_does_not_exist()Traceback(mostrecentcalllast):...AttributeError:Mockobjecthasnoattribute'method_that_does_not_exist'>>>>>># But passing arguments that don't exist still passes:>>>mock_foo.some_method(arg_that_does_not_exist=23)<MagicMockname='mock.some_method()'id='140189369916688'>

Class variables aren’t recursively autospec’d either.

There’s also a spec argument (mock.MagicMock(spec=Foo)), it works the
same as spec_set but allows attributes that don’t exist to be written.

Passing a list of strings to spec_set

Instead of a class you can pass a list of strings as the value to the spec_set
or spec argument. Only those names given in the list will be accessible on
the mock object. Each name is accessible either as an attribute or callable as
a method, and returns an unconstrained MagicMock:

Passing a list of strings to spec_set or spec is an even weaker form of
specification, and it introduces duplication between the test code that creates
the mock and the real code that’s being mocked. If the real code is changed
then the mocks may need to be updated, otherwise the tests that use the mocks
may still pass even if the code they’re testing is wrong.

Limitations of autospeccing

Unfortunately create_autospec() isn’t a perfect solution to the problem with mocks.
It has a few limitations:

Instance variables created in __init__() methods don’t work.

For example here a real Foo object would have a bar attribute, but a mock
Foo object from create_autospec()doesn’t have bar:

Note that we had to drop the spec_set=True. This works but has the downside
that the tests are now coupled to Foo - if the Foo class changes, for
example to remove, rename or change the type of Foo.bar, then the above
mock will need to be updated (otherwise tests for the code that uses Foo
could still be passing even if the code is wrong because it still uses
Foo.bar which no longer exists on the real Foo).

This makes the Foo code a little more verbose than it otherwise needed to be,
and note that mock_foo.bar now returns a MagicMock (not the string BAR),
so if the code under test were to call a non-string method on bar it would
not crash in the tests (but would crash with the real Foo in production).

Finally, one more way to workaround this issue is to create a real Foo
object and pass that to create_autospec() instead of the Foo class:

But now some duplication between the tests and Foo has been introduced.
Again, if the real Foo.some_method() is changed then the above mock code
would also need to be updated otherwise the tests for the code that uses
Foo could still be passing even though the code is now wrong.

Autospec doesn’t automatically mock the behavior of the real class.

Your tests still need to use return_value and side_effect to simulate
raising exceptions and other behaviors of dependency objects. Again, this
introduces duplication between the tests and the dependencies being mocked -
if the dependency code is changed then the mocks may need to be updated, or
the tests could still be passing while the code is wrong.

Conclusion

As you can see, autospeccing in Python is a battle to write your code and
create your mocks in a way that minimizes the possibility of false-positive
test passes creeping in.

Most of the autospec limitations above can be avoided most of the time, but it
can’t always be done perfectly: there’s no way to automate simulating the
return_values, side_effects and behaviors of real classes without
introducing some amount of duplication between the real classes and the tests
that mock them. Duplication between real code and test mocks introduces the
possibility that over time, as the real code changes, the mocks will go out of
sync with the real code and false-positive test passes may creep in.