Monday, January 28, 2008

Testing: One expectation per test

In a previous entry I discussed why I prefer One Assertion Per Test. In that entry I give a few state based examples, but didn't address the topic of mocks. This entry is going to focus on how you can use mocks while maintaining focus on maintainability.

The SuiteRunner class is fairly straightforward, it's simply delegating the block on to the instance_eval method on the suite attribute, which is an instance of the Suite class. Even though the class isn't doing anything very interesting the test isn't completely simple. Due to the chain of method calls you need to set up 2 mocks and a total of 3 expectations. While the test isn't unmaintainable, if the description (the method name) isn't kept up to date you could easily lose the intent of the test.

A step in the right direction toward making this test more readable is to introduce a few stubs. However, before we decide what to stub we need to decide what this test is trying to verify. Given the description it seems that eval'ing in the context of the suite is what is being tested, thus the suite mock should probably remain. Since the suite mock is the focus of the test, the rest of the mocks are probably better expressed as stubs. The test can be written more concisely now that we've chosen to use stubs.

With one mock and one expectation the test expresses it's purpose. Code should not only express how it works, but also why it's been written in a particular way. The new test not only executes, but it conveys the intent of the test (even if the description becomes stale). Converting that test was easy, but tests that have behavior expectations and state based assertions can be a bit more complicated to clean up.

There's a lot going on in this test. Test::Unit even reports that 7 assertions have been met. Unfortunately, with that much going on in a test, it's hard to tell what the original intent of the test was. Even worse, it would be hard to change anything in the book_for method without breaking this test. The test is completely tied to the implementation and if that implementation changes you'll break one or more of the assertions within the test. What really bothers me about this test is the ability to break "one or more" assertions. If you change the reserve_for method to also take the hotel as an argument the test will immediately stop executing with the following error.

Knowing that an assertion failed is good, knowing that other assertions might also fail when you fix the first problem is not so good. The problem with this test is that it's verifying 7 different things. This can most likely be resolved by writing several different tests that all focus on one thing. (There are several tests that can be written, here are 5 examples)

The above tests take more lines of code than the original, but they are far more maintainable. You can change the implementation of the book_for method in various ways and only break the tests that are relevant to the change. The tests have become more robust. They've also become more readable because they express what their focus is. The tests that have a mock with an expectation are written to test a behavior that is expected. The tests that have an assertion are written to statefully verify that the object has been set as expected. When the implementation does inevitably change the tests can communicate on their own what the original author intended.

Writing tests in this way is actually quite easy if you follow a few simple suggestions.

If your test has an assertion, do not add any mock expectations (instead use stubs for any methods that need to be changed for the test).

If you add a mock expectation you probably don't need it to return a meaningful value (since you won't be verifying that value).

If you already have a mock expectation you should use the stubs method for any other method where you don't want to use the implementation.

Do not use .with when using a stub unless you need to. If you are stubbing, you don't need to verify the arguments.

When returning stubs, prefer stub_everything so that additional calls to the stub will not cause unnecessary exceptions to be raised.

The above suggestions focus on creating tests that assert (or expect) one thing at a time and specify as little other implementation as possible. Every expectation, .with, and .returns statement is specifying implementation. The less implementation you express in a test, the more that test becomes.

I consider each call to stub (or stub_everything), stubs, mock, expects, with, and returns to be 1 implementation specification point. Any test that has an implementation specification score of 5 or more has the high implementation specification smell. As with all smells, everything might be okay, but I'm going to take the extra time to see if anything can be broken up by moving responsibilities or creating smaller methods.

By limiting your tests to one assertion or one expectation you create tests that express their intent. By creating tests that specify as little implementation as possible you reduce the amount of noise taking away from the intent of the test. An additional consequence of creating tests that focus on intent is that they specify less and are therefore more robust.

12 comments:

Hi Jay ... hey my one issue with the one expectation per test approach is that I often end up performing the same setup/context repeatedly. Now in most cases this is ok and not much of an overhead, but in some circumstances it means having a slower test ... where it's faster to do the Arrange and Act (thinking of Bill Wake's 3 A's here ... Arrange, Act, Assert) once and then do the Asserts.

One assertion (or expectation) per test is a good rule for unit testing and a good guideline for functional testing. If your test requires a fair amount of setup that makes it run slowly, I assume you are writing functional tests, in which case you'll need to balance speed with purity.

I would first take a good long look at the code and make sure there's no other way to do what you want to accomplish. There might be a way to extract the slow code to a method, mock that method and turn this into a unit test.

If you can't figure out a way to tease things apart, then it's one of those situations where you'll need to go against the suggestion and do multiple asserts. There's no silver bullet. =)

I went through a very similar exercise yesterday with a coworker. I'm a big fan now of the style you describe as its definitely made the specs cleaner and clearer. I'd really suggest trying this out to anyone who's skeptical.

@jay: At the moment I feel comfortable with the 'how' but sometimes get stuck at the 'what', especially with controllers. At times I've written a spec for almost every line of a controller and that just feels wrong; at the same time, I don't want to be too broad and not catch situations where people are changing instance variable names and screwing everything up. Do you have any guidelines or 'mantras' if you will about what types of calls you test and what you avoid? Have you had any new ideas or thoughts since http://blog.jayfields.com/2007/09/rails-testing-controllers.html

The other week on the rspec list David Chelimsky posted this http://pastie.caboo.se/142403 which I think is really cool for using mocks and doing the one assertion per test. Not sure if this helps with the controller issues you guys have been having?

One objection I've seen from people is that if you were to test-drive it with one expectation per test (rather than starting with the big ol lump and then refactoring into multiple tests), you end up having to edit more than one test for each bit of production code you add.

Here's where the problem arises...Test #2 is passing right now, but you have to go back to Test #1 and stub MaidService.notify.

I don't really have a problem with that. I think it's perfectly simple. However I've encountered some resistance from people who claim that in making this test pass, I've made a previously passing test turn red. Is there merit to that argument, or do you think it might just be an excuse to avoid mock-based testing?

@Pat, I see breaking a previous test as a good thing. Ideally, changing the implementation would not break a previously written test. When adding new implementation does break a test it signals to me that either the test or the implementation can be written in a better way.

@Ana, I believe you could write the test as: it {@x.should be_nil}

Of course, I can't help but mention that I'd also be looking to Inline Setup

Pat,In this case, I don't think there is a way I'd improve the test or implementation. Unfortunately, sometimes you cannot avoid having to add things such as MaidService.stubs(:notify) in several tests. It's not ideal, but it sucks the least in my opinion.

Of course, the domain could be modeled in another way which might help, but that's outside the scope of the example.

Very nice post, Jay. How would you suggest writing tests for a method that throws an exception? Given certain input conditions, this method is expected to throw an exception, and I want to test for that. But there are other expectations to test for as well (such as confirming that the method closes a connection it has opened), and in order to separate these out, it seems like one would have to catch and discard the exception in all the other tests.

This seems like it would introduce a lot of clutter and introduce an irrelevant implementation detail to the other tests. (They don't care whether an exception is thrown; they should ideally be able to testing their piece in relative isolation.)