Finding the Balance Between Unit & Functional Tests

In my previous blog post I wanted to set the scene for this post on finding the balance between Unit and Functional tests. In that post we discussed the evolution of the Testing Pyramid and concluded that because it is only a model, we don’t have to strictly accept that we should always have more Unit tests than Functional tests. It’s about much more than just looking at the quantity of tests (i.e. we must look at the RISK). In this post, I want to explore how we might try to find the balance between Unit / Functional tests by working through a few abstract case studies.

It is important to remember that for every project the balance between Unit and Functional tests will vary and there is no hard and fast way to get the correct balance. Instead there should be ongoing discussion and collaboration between all team members, as well as the stakeholders & end users, to determine that the balance of testing mitigates the most risk possible.

I thought that I would start on a lighter note with some toilet humor! For this case study I want us to imagine that we can only use eitherUnit Tests or Functional tests, but not both! Imagine that a new men’s bathroom is being constructed in some public building (how exciting….). It consists of all the normal components you would expect including; urinals, wash basins, paper towel dispenser, mirrors etc.

Suppose that you could only perform unit testing on this new bathroom. So you can test individually each component in the bathroom, individual unit tests for the urinals, wash basins etc. All of the unit tests come back as passed, so everything in the bathroom is working as intended in isolation – is this enough to confidently say that everything in the bathroom is working? Well, imagine that you walk into the bathroom now and see this:

Oh dear, that could be awkward! Whilst the urinals worked fine individually at a unit level, when they were both put into the system (the bathroom) we uncovered a flaw in the design of the bathroom. Hopefully you can see what I am getting at here, we really needed a functional test that covered something like using all of the urinals concurrently.

Now let’s look at the other extreme – we can only use full end to end functional tests for our new men’s bathroom. The full end to end test covers everything that a visitor would do in one trip to the bathroom, i.e:

Use the urinal

Wash their hands in the basin

Check themselves in the mirror

Dry their hands

We will also assume that the bathroom has x5 urinals, x5 basins, x5 towel dispensers and x5 mirrors. As this is a functional end to end test (NO unit testing), we have to check the full “path” through the system (i.e. using each urinal, basin, towel dispenser and mirror).

At a minimum we will want to make sure that we use each component once in our testing, so we will need at least 5 tests.

Remember, if the test fails at any point during our “test” then the whole test will fail, and we won’t be able to complete the test case – our toilet user will leave the bathroom feeling disappointed (and possibly embarrassed, depending on what went wrong!)

So we look at the end to end test results and we can see that on test 3 of our “visit to the bathroom” test case there was a failure. Remember this is an end to end test so we aren’t sure where exactly the failure is right away, we have to look closely at the detail being returned by the test to see where the problem was…. Eventually we uncover a problem with one of the urinals being blocked, so in this case the poor chap is unable to use the urinal and the test ends there when he leaves the bathroom!

So we go ahead and fix that urinal and run test 3 again. The test fails again! This time one of the sinks isn’t working, but that issue wasn’t uncovered in the previous test because we didn’t even test sink 3 as we “failed” the test as we didn’t successfully complete the prerequisite (using Urinal 3)…

So we fix the sink and run the end to end test again and this time the test fails because the mirror is broken!

That’s now three end to end tests that have been run (and remember end to end tests are the most “expensive” in terms of time and development cost) and each one has revealed different problems that could all have been quickly identified by unit tests.

In fact, we still haven’t had a complete test execution pass successfully yet for test 3!

In addition to the above, to avoid the problem that we uncovered in the 2 toilet picture, we will want tests where two or more components are being used concurrently.

We want to do the minimum in our example, so let’s have end to end tests where all the urinals are being used, then all the basins etc. That would be another 4 tests as in the image.

You might also want to test that certain urinals can be used at the same time as certain basins etc. so again more tests for those, but we won’t document them here…

This is a very silly example obviously, but I hope that it gives some insight in to the problems that can potentially arise from being solely reliant on functional or unit tests. Whilst having functional testing of every possible journey through the system might be possible in theory, in reality it is unlikely to be realistic and even if it is, it is probably going to lead to significantly increased time in identifying and resolving issues (especially as the system grows and becomes more complex)

Case Study 2: The New House – Why we need Unit AND Functional

Let’s next look at another abstract case to see why bothUnit and Functional tests are normally required. In this case, a brand new house has just been built, and before the new owner takes possession of it, everything in the house needs to be “tested”. This is a popular analogy and whilst it is not quite correct, we can use it to better understand that differences between unit and functional tests.

So let’s start with unit tests, which in this case would be represented by the building inspector visiting the house and running various tests on each of the internal components. He will want to individually test everything inside the house such as:

The Plumbing

The Electrical Wiring

The doors

The windows

And so on…

He will test that each part of the house works correctly and meets the building code.

The functional tests in this scenario are analogous to the new homeowner visiting the same house and testing it from his point of view (i.e. the end user). Whilst he obviously is concerned that the individual components of the house all work correctly, he assumes that they have all been tested at the unit level by the building inspector above.

The homeowner is more interested in testing what the house is going to be like to live in, so he might want to test scenarios such as:

Will there be enough space to install a play room?

Will he be able to catch enough sunlight through the windows whilst he has breakfast in the morning?

Does the design of the house allow for effective movement around the property?

Would he be able to have guests over for dinner parties?

And so on….

All of these are functional scenarios / tests performed from the users perspective, they couldn’t realistically be covered by the tests done by the building inspector, i.e. the unit tests. The point is that, to thoroughly test this “system” (the new house) we need tests from different types of testers, that come from different perspectives.

To summarise our new house example, we could say that:

Unit tests tell the House Developer that the inner components of the house are working

Functional tests tell the House Developer that the house meets the homeowners requirements

So to then relate this to the testing of a piece of software, we could simply say:

Unit tests tell a Software Developer that the code is doing things right

Functional tests tell a Software Developer that the code is doing the right things

I want to switch gears now and look at a more technical example with some actual dummy code where we will test a method in one of our classes. Suppose that we have a system to handle processing of employee expenses. In our code we have a class “Expense” and we want to test the method within that class that formats the Expense reportdepending on the country that the report applies to. The code might look something like this:

public class Expense {
// ... Other methods and properties ommited for clarity
public void formatExpenseReport(String countryCode)
{
switch(countryCode) {
case("UK"):
float vatRateForCountry = callDbForVatRate(countryCode);
String fullCountryName = callDbForCountryName(countryCode);
// .. other calls to the DB for this particular country
// .. other code to create the report for this country
break;
case("FR"):
// .. other code and calls to the DB
break;
// other cases...
}
}
}

So the method “formatExpenseReport” takes a string of a countryCode when called and the switch statement is executed. For each case that is executed there are calls to the database to populate the required attributes to format the report for that country, for example with UK above for the vatRateForCountry and fullCountryName.

The tests for the “formatExpenseReport” method above can all be unit tests that do not hit the database. This can be achieved easily by stubbing the calls to the database for the attributes. With those attributes stubbed we can thoroughly unit test our method and be confident that it is working as intended by creating unit tests that cover each logical path through the method.

But before we shipped the code to production, we would also want to test the the data being returned from the database was accurate and formatted correctly in the way that our method expects. So we could add a small number of functional tests (in addition to our series of unit tests) that tested the same method but used an actual instance of the Expense class that actually called the attributes from the database for a subset of country codes.

Using this combination of many unit tests that quickly test the method, and a small number of functional tests that include the call to the database, we can have a high degree of confidence that the method is working as intended. This is achieved whilst conducting our testing in a much more efficient manner than if we had to call the database to populate the attributes for the method each time for every single test.

To summarise everything this post:

Unit and Functional tests cover testing of the system from different perspectives

It is unlikely to be optimal to rely exclusively on one type or the other

Used coherently, they can make testing much more efficient

Determining the balance of tests is an iterative and ongoing process that depends on the requirements, risks, and nature of the system itself. It can be best determined by analysis and collaborations between team members and stakeholders.

I hope that this post was of use, please feel free to leave comments below.

Tags:

Great post, Jim! I really like the analogies with house construction, they make a lot of sense! In particular, I like the analogy between functional tests and dinner parties, it really made me think that integration tests are supposed to be about “features” instead of “slices”.