Thursday, January 27, 2005

Python unit testing part 1: the unittest module

Python developers who are serious about testing their code are fortunate to have a choice between at least three unit test frameworks: unittest, doctest and py.test. I'll discuss these frameworks and I'll focus on features such as availability, ease of use, API complexity, test execution customization, test fixture management, test reuse and organization, assertion syntax, dealing with exceptions. This post is the first in a series of three. It discusses the unittest module.

The SUT (software under test) I'll use in this discussion is a simple Blog management application, based on the Universal Feed Parser Python module written by Mark Pilgrim of Dive Into Python fame. I discussed this application in a previous PyFIT-related post. I implemented the blog management functionality in a module called Blogger (all the source code used in this discussion can be found here.)

The unittest module (called PyUnit by Steve Purcell, its author) has been part of the Python standard library since version 2.1.

Ease of use

Since unittest is based on jUnit, people familiar with the xUnit framework will have no difficulty picking up the unittest API. Due to the jUnit heritage, some Python pundits consider that unittest is too much "java-esque" and not enough "pythonic". I think the opinions are split though. I tried to initiate a discussion on this topic at comp.lang.python, but I didn't have much success.

API complexity

The canonical way of writing unittest tests is to derive a test class from unittest.TestCase. The test class exists in its own module, separate from the module containing the SUT. Here is a short example of a test class for the Blogger module. I saved the following in a file called unittest_blogger.py:

All in all, not a huge number of APIs to remember, but it's enough to draw some people away from using unittest. For example, in his PyCon 2004 presentation, Jim Fulton complains that unittest has too much support for abstraction, which makes the test code intent's less clear, while making it look too different from the SUT code.

Test execution customization

The canonical way of running tests in unittest is to include this code at the end of the module containing the test class:

if __name__ == '__main__':
unittest.main()

By default, unittest.main() builds a TestSuite object containing all the tests whose method names start with "test", then it invokes a TextTestRunner which executes each test method and prints the results to stderr.

In conclusion, it's fair to say that unittest offers a lot of flexibility in test case execution.

Test fixture management

I already mentioned that unittest.TestCase provides the setUp and tearDown methods that can be used in derived test classes in order to create/destroy "test fixtures", i.e. environments were data is set up so that each test method can act on it in isolation from all other test methods. In general, the setUp/tearDown methods are used for creating/destroying database connections, opening/closing files and other operations that need to maintain state during the test run.

In my unittest_blogger example, I'm using the setUp method for creating a Blogger object that can then be referenced by all test methods in my test class. Note that setUp and tearDown are called by the unittest framework before and after each test method is called. This ensures test independence, so that data created by a test does not interfere with data used by another test.

Test organization and reuse

The unittest framework makes it easy to aggregate individual tests into test suites. There are several ways to create test suites. The easiest way is similar to this:

Here I created a TestSuite object, then I used the makeSuite helper function to build a test suite out of all tests whose names start with "test". I added the resulting suite to the initial TestSuite object via the addTest method.

A suite can also be created from individual tests by calling addTest with the name of the test method (which in this case does not have to start with test). Here is a fragment from unittest_blogger.py:

The first line runs a TextTestRunner with the default terse output and using the suiteFew suite, which contains only 2 tests.

The second line increases the verbosity of the output, then runs the suite returned by the suite() method, which contains all tests starting with "test" (and all of them do in my example).

The suite mechanism also allows for test reuse across modules. Say for example that I have another test class, which tests some properties of the sort() method. I saved the following in a file called unitest_sort.py:

As I said previously, unittest provides its own custom assertions. Here are some of the reasons for this choice:

if tests are run with the optimization option turned on, the standard Python assert statements will be skipped; for this reason, unittest provides the assert_ statement, equivalent with the standard assert, but that will not be optimized away

the output of the standard Python assert statements does not show the expected and actual values that are compared

For example, let's make the testDeleteAllEntries test fail by comparing the value of get_num_entries() with 1 instead of 0:

There's no indication of what went wrong when the actual and the expected values were compared.

Dealing with exceptions

The unittest_sort module listed above has 2 examples of testing for exceptions. In the test_sort_exception method, I first test that calling sort with an undefined function as a sort function results in a NameError exception. The test will pass only when NameError is raised and will fail otherwise:

10 comments:

Anonymous
said...

Great! Especially looking forward to the part on "py.test", what I have seen so far about "py.test" looks very good.

Currently I am using unittest and doctest together (verifying my docstrings becomes one of my unittest tests). Each test gets put into the place where it feels more natural, usually doctest, but if a testing framework is required (loops, setup, teardown, etc.), it goes into unittest, instead of fussing with doctest.

forces unittest executing tests in defined and easy modificable order. I'm used to do so.

Just my 5 cents

Facility to run test in defined order, but break the run after first failed test lacks to me in unittest. I know how to do it (runnig each test namelly and testing its result) but I don't like it and it's hard to extend.

It is 2013. This is the first example of unittest in Python that I have actually understood. I now actually understand where everything is supposed to be placed when creating test suites. Thank you, a million times over, thank you!

I tried to use this code on tearDown() but it shows me exception on invalid syntax in I guess it is due to time function I have import the library even though why this syntax error is showing please suggest and help.