As I understand in theory under "unit" people mean method (in OOP). But in practice tests which verify some method in isolation are very fragile behaviour tests (verifying not the result but the fact that some dependency method was called). So I see a lot of people who by unit understand a small set of closely related classes. In this case only outer dependencies are mocked/stubbed and for dependencies which are inside unit real implementations are used. In this case there are more state, meaningful (according to specification) and not so fragile tests. So the question is how do you feel about these approaches and is it valid to call the second approach unit testing or may be it is some kind of low-level integration testing?

If you see some specific considerations about applying TDD by one of these ways of testing, I would be thankful for your thoughts.

3 Answers
3

Originally TDD came from the agile movement, where testing was written up front as a way to ensure what you coded remained correct given the specification that was now well-defined in the test code. It also appeared as a very important aspect of refactoring, in that when you modified your code, you could rely on the tests to prove that you hadn't changed the code's behaviour.

Then the tools people came along and thought they knew information about your code and could then generate test stubs to assist you in writing your unit tests, and I think this is where it all went wrong.

The test stubs are generated by a computer that has no clue what it is you're doing, it just mindlessly produces a stub for every method because that's what it's told to do. This means you have a test case for each method regardless of the complexity of that method or whether its suitable for testing in isolation.

This is coming at testing from the wrong end of the TDD methodology. In TDD you're supposed to figure out what the code is to do, and then produce code that achieves this. This is self-fulfilling in that you end up writing tests that prove the code does what the code does, not what its supposed to do. Combined with the auto-generation of method-based test stubs, you pretty much waste your time proving each tiny aspect of your code that can so easily prove to be wrong when all the little pieces are put together.

When Fowler described testing in his book, he referred to testing each class with its own main method. He improved this, but the concept is still the same - you test the entire class so it works as a whole, all your tests are bundled together to prove the interaction of all those methods so the class can be reused with defined expectations.

I think the test toolkits have done us a disservice, led us down the path of thinking that the toolkit is the only way to do things when really, you need to think more for yourself to get the best result from your code. Blindly putting test code in test stubs for tiny pieces just means you have to repeat your work in an integration test anyway (and if you're going to do that, why not skip the now-redundant unit test stage completely). It also means people waste a lot of time trying to get 100% test coverage, and a lot of time creating large amounts of mocking code and data that would have been better spent making the code easier to integration test (ie if you have that much data dependencies, unit test might not be the best option)

Lastly, the fragility of method-based unit tests just shows up the problem. Refactoring is designed to be used with unit tests, if your tests break all the time because you're refactoring then something has gone seriously wrong with the whole approach. Refactoring likes to create and delete methods, so obviously the blind per-method based test approach is not what was originally intended.

I have no doubts that many methods will get tests written for them, all of the public methods of a class should be tested, but you cannot get away from the concept of testing them together as part of a single test case. For example, if I have a set and a get method, I can write tests that put the data in and check the internal members are set ok, or I can use each to put some data and then get it out again to see if it's still the same and not garbled. This is testing the class, not each method in isolation. If the setter relies on a helper private method, then that's fine - you don't need to mock the private method to ensure the setter is working, not if you test the entire class.

I think the religion is getting into this topic, hence you see the schism into what's now known as 'behaviour-driven' and 'test-driven' development - the original concept of unit testing was for behaviour driven development.

A unit is most commonly defined as "the smallest testable part of an application". More often than not, yes, this means a method. And yes, this means that you shouldn't test the result of a dependant method, but just that the method is called (and then only once, if possible, not in every test for that method).

You call this fragile. I think that is incorrect. Fragile tests are those that break under the slightest change to unrelated code. That is, those which rely on code that is irrelevant to the functionality being tested.

However, what I think you really mean to say is that testing a method with none of its dependencies is not thorough. On that point I would agree. You also need integration tests to make sure that the units of code are correctly hooked together to make an application.

I really wanted to say behaviour tests are fragile. They often become false negative during codebase change. It happens less with state tests (but state tests are very seldom for unit testing)
–
IdsaAug 14 '11 at 22:15

@Idsa: I'm a little lost by your definitions. Behaviour tests are integration tests, testing a piece of behaviour as specified. Reading your original question, it seems that when you say state tests, you mean the same thing.
–
pdrAug 14 '11 at 22:26

by state I mean test which verifies the state, result of some function; by behavour I mean test which verifies not the result, but the fact that some function was called
–
IdsaAug 15 '11 at 7:26

@Idsa: In that case, I completely disagree. What you are calling state test, I call integration. What you are calling behaviour, I call unit. Integration tests by their very definition are more fragile. Google "integration test unit fragile" and you'll see that I'm not alone.
–
pdrAug 15 '11 at 10:09

there are a log of articles about testing but which of them share your opinion?
–
IdsaAug 17 '11 at 19:18

As the name suggests you are testing an atomic subject in each test. Such a subject is usually a single method. Multiple tests can test the same method, in order to cover the happy path, possible errors, etc. You are testing behaviour, not internal mechanics. So unit testing is really about testing a class' public interface, i.e. a specific method.

In unit testing, a method needs to be tested in isolation, that is, by stubbing/mocking/faking any dependencies. Otherwise, testing a unit with 'real' dependencies makes it an integration test. There is a time and place for both types of tests. Unit tests ensure a single subject works as expected, standalone. Integration tests ensure that 'real' subjects work together correctly.

not quite, a unit is any isolated subject, just because the automated tooling prefers to treat a method as this doesn't make it so, nor make it the best. 'Isolated' is the key here. Even if you test methods, you should also test the private ones too.
–
gbjbaanbMar 7 '13 at 14:56