I'm working on a plugin for my test runner, nose (​http://somethingaboutorange.com/mrl/projects/nose/) to ease testing of django apps. It works along the lines of the django tests, setting up a test database (or schema) based on the selected settings file and installing the INSTALLED_APPS there, and tearing all of that down at the end of the test run.

It's currently in early alpha -- if you want to check it out, please don't use it with a settings file that points to production data. While I don't believe there are any data-destroying bugs... well, you get the idea.

Ok; here's a walkthrough of v1 of the Django testing framework. Note that this is not the end of development; it is just an attempt to set up a generic test framework that is the equal of the existing test/runtests.py. The next step is to implement facilities for fixtures, testing templates, contexts, views, etc.

core

Added a 'test' target to django-admin

Test target gets the settings.TEST_RUNNER method in setings.TEST_MODULE, and invokes it

Test method is provided with a list of modules to test, and a verbosity for error reporting.

Added a '--verbosity/-v' option, which is passed to a few of the targets

Modified syncdb to allow customized verbosity of output messages.

Modified syncdb to operate in a 'non-interactive' mode, so you can call syncdb and disable the call to create a superuser

Modified the parameters of the post_syncdb signal to pass on verbosity and interactivity options.

conf

Added two new default settings: TEST_MODULE and TEST_RUNNER. These strings identify the module name, and the module symbol that will be used by the django-admin 'test' target to start the test set. Points to the 'simple' test runner by default.

contrib

Modified the post_syncdb handlers for the contenttypes, auth, and sites applications to cater for the new verbosity and interactivity information.

test

Moved the copy of doctest.py into the test package.

simple.py

An implementation of a 'simple' testing strategy - i.e., the same strategy currently used by tests/runtests.py

Searches for any doctest in the models.py or tests.py module

Searches for any unittest in the models.py or tests.py module

Composes a unitest.TestSuite of all these tests

Sets up the test db before running the entire suite, and tears it down after running the suite.

The simple test runner can also be provided with a list of 'extra' testCases that will be added to the test suite. This is used by the django tests to insert model validation tests.

testcases.py

Some utilites that customize the default behaviour of doctest and unittest.

Original intent was to add some TestCase extensions to this module that implement fixture/db setup type procedures.

Here's the next piece of the testing puzzle - a method for testing contexts and templates produced by views.

It acts a little bit like an automatable web browser. The interface allows users to make GET and POST requests; what is returned is a 'test decorated Response' - a normal HttpResponse object, but with extra attributes (specifically, 'context' and 'template') that describe the contexts and templates that were rendered in the process of serving the request.

If a single template was rendered, response.template and response.context will point to the template and context dictionary respectively. If multiple templates were rendered, response.template and response.context will be lists, set such that response.context[n] was used on response.template[n]. If no templates were rendered, response.template and response.context will be None.

Cookies are also handled in a limited fashion, and are preserved for the lifespan of the Browser instance. A simple helper interface exists to assist with submitting to login pages so that you can test @login_required views.

This is all acheived by directly hooking into the WSGI interface, so no server instance is required for the Browser to be used. Template and context details are obtained by listening to a new signal that is emitted whenever the render() method is invoked on a template.

As noted in the patch - this is not intended to reproduce or replace tools such as Twill or Selenium. While these tools are good tests of client behaviour, they cannot check the contents of Context or Template rendering details (except at the level of 'is the rendered output correct').

As noted with the previous patches, the approach here is not to enforce a particular testing framework, but to provide the tools that allow anyone to test any aspect of their Django application.

Added some revised patches, incorporating some comments that have been received. For ease of applying, they are combined; to allow them to be attached to TRAC, there are two parts. Both the revised-* patches should be applied using patch -p0 from the django trunk directory.

(In [3658]) Refs #2333 - Added test framework. This includes doctest and unittest finders, Django-specific doctest and unittest wrappers, and a pseudo-client that can be used to stimulate and test views.

(In [3660]) Refs #2333 - Added 'test' target to django-admin script. Includes addition of --verbosity and --noinput options to django-admin, and a new TEST_RUNNER setting to control the tool used to execute tests.

(In [3661]) Refs #2333 - Modified runtests script to use new testing framework. Migrated existing tests to use Django testing framework. All the 'othertests' have been migrated into 'regressiontests', and converted into doctests/unittests, as appropriate.

(In [3706]) Refs #2333 - Added a TEST_DATABASE_NAME setting that can be used to override the 'test_' + DATABASE_NAME naming policy. This setting is then used in runtests.py to restore the use of 'django_test_db' as the Django model/regression test database. Thanks to Michael Radziej for the feedback.

(In [3707]) Refs #2333 - Re-added the template rendering signal for testing purposes; however, the signal is not available during normal operation. It is only added as part of an instrumentation step that occurs during test framework setup. Previous attempt (r3659) was reverted (r3666) due to performance concerns.

I've been using this framework, and I find it's generally more convenient to actually allow errors during page production to propagate up to the test harness, rather than getting eaten and turned into 500 pages. This can be easily accomplished by changing the big "try" in django.core.handlers.get_response to re-raise exceptions in the final "except" clause.

I'd submit a patch, but this really ought to be a setting of some kind (so you can still test 500 error page generation) and I don't know the best way to do that. Right now I use "PYTHONPATH=~/django_src/ python manage.py test [myapp]" to swap in a Django svn checkout that has this modification hard-coded in whenever I want to do that.

Phase 3 - Fixtures

Phase 3 of the Django testing framework is Fixtures - the ability to set up the test database to contain a specific data so that the test harness doesn't have to manually create and delete data. As an added bonus, the approach taken here allows for fixtures to be used as a crude backup mechanism, or as a crude schema evolution mechanism.

To make fixtures as simple as possible, I have used the serialization framework, and added a mechanism to load files of serialized data into the database. The patch (fixtures.diff) does the following:

Look in a 'fixtures' directory under each app directory (similar to the way that initial_sql looks in the directory named sql).

Look for a file foo.EXT, where EXT is the file extension appropriate for that serializer (e.g., foo.xml, foo.json). The file extension is the name of the serializer; json fixtures are the default, but any other serializable format is possible using the --format option.

Looks through each of the directories named in settings.FIXTURE_DIRS for fixtures named foo.EXT, and installs all of those.

All fixture data is added in a single transaction (i.e., single call to installfixture = single transaction). This is to accomodate forward dependencies in fixture data.

Renames the 'initial_data' target in manage.py to 'custom_sql'. This is to preserve the ability to provide table-modifying statements, but to discourage the use of the target for data insertion.

Adds a 'flush' target to manage.py. This removes all data from all tables, then executes post_syncdb on all models (so that any autogenerated content is recreated).

Adds a 'dumpdb' target to manage.py. This allows the developer to dump the contents of the entire DB, or a subset of apps, in a nominated fixture format.

Adds a django.test.testcases.TestCase. This is an extension of unittest.TestCase that flushes the database at the start of each test, then does an automated fixture installation. Sample usage:

class MyTest(django.test.TestCase):
fixtures = ['foo','bar']
def test_feature1(self):
# proceed as if 'foo' and 'bar' fixtures has been loaded
def test_feature2(self):
# proceed as if 'foo' and 'bar' have been loaded;
# effects of test_feature1 removed from DB.

Adds code to perform a database flush at the start of each doctest. This slows down the testing process s a bit; however, if you have two doctests (or a doctest and a unit test) in the same application, you get inconsistent results depending on the order of test execution. The alternative is to require/expect users to manually call flush at the start of each doctest for those doctests that are likely to experience order-related issues.

Adds import shortcuts to allow:

from django.test import TestCase
from django.test import Client

Adds a unit test for unit testing with fixtures :-)

Design decisions worth discussing

I've nominated JSON as the default fixture format; mostly because its the most mature serializer that isn't XML :-)

There is a get_sql_flush implementation on each database backend; this method does the heavy lifting of the manage.py flush target.

I've migrated test_client to use the fixtures framework rather than the management.py dispatcher hack for setting up initialization data.

Open issues:

Should syncdb install a fixture with a known name (e.g., post_syncdb)? This would allow for the automated installation of initial data on sync.

Database flushing is a very database specific thing. As a result, the get_sql_flush implementation is mostly in the database backends:

Postgres requires that all TRUNCATEd tables with constraint dependencies appear in a single TRUNCATE request. However, this is only available in Postgres 8.1 or later. Postgres also requires that sequences be reset. A solution for earlier versions of Postgres is required.

I can't comment on Oracle or MSSQL, as I don't have a test bed. This requires someone with access to these databases to get the 'flush' command working on these platforms.

There is one additional change that I would recommend, but is not part of the patch - removal of the 'reset' and 'install' targets in manage.py. These two targets take the output of 'get_sql_' calls, and apply them to the database - except that this isn't what is required anymore. With all the syncing, fixturing, and custom SQL, the get_sql_ calls are useful for debugging, but not really useful as actual reset/install targets.

The shape of this patch has been shaped by the discussion I have had with Jacob and others on the django-dev mailing list.

Noteworthy changes since the last version:

manage.py has loaddata, dumpdata and flush targets to load fixtures, dump fixtures, and flush the contents of the database, respectively.

manage.py loaddata foo.json will look for JSON fixtures named foo in the application fixture directories, the directories named in FIXTURE_DIRS, and in the absolute path (i.e., loaddata /bar/whiz/foo.json will load the specifically named file).

manage.py loaddata foo will look for any fixture type named foo; if foo.xml and foo.json are found in the same fixture directory, an error is raised, and the fixture installation is rolled back.

manage.py install has been removed in favour of syncdb; the 'pass sqlall into the database' approach misses all the signals and indexing that syncdb performs, retrofitting install to have these features would be non-trivial, and ultimately would yield nothing more than a subset of the functionality of syncdb

manage.py sqlinitialdata has been deprecated, with a message directing to the new name sqlcustom. This is a rename to indicate that initial data should not be in SQL format, but in the new fixtures format.

The only open issue at this point is database backend compatibility. SQLite, MySQL, and Postgres 8.1+ are covered. Oracle _should_ work AFAIK, but is untested. ADO is completely untested. Postgres 7.x and 8.0 will not work. The only effect of this patch on unsupported databases is that fixtures wont work, and the fixture-based unit tests fail.

Documentation will be forthcoming once this final design has been approved.

Postgres 7.x doesn't have a TRUNCATE statement; Postgres 8.0 added TRUNCATE, but it has a problem. If Table1 contains references to Table2, the TRUNCATE approach fails because the table contraint rules kick in. I've tried putting the TRUNCATE calls into a transaction, but it didn't seem to help - the constraint rules seemed to kick in during the transaction.

Postgres 8.1 fixed this problem by allowing you to specify multiple tables in a single TRUNCATE statement:

TRUNCATE table1, table2;

which doesn't activate the constraint checker.

Now; I'm not denying that the problem with 8.0 may exist (at least partially) between my keyboard and my chair; I might have just messed up in my use of transactions when I was testing with 8.0. However, this still leaves the Postgres 7.x issue. 7.x will need to regress to either deletion of rows (like the SQLite implementation), a table destroy/resync, or a complete database destroy/recreate. Supporting 7.x also introduces the need to identify what database version is running so that the correct solution can be applied.

This is all made more difficult by the fact that I have very limited access to a Postgres 8.0 install, and no access to a Postgres 7.x install, so my ability to test these earlier versions is somewhat restricted. Any assistance on this front would be greatfully accepted.

Strange, postgresql 7.4 and 7.3 have a TRUNCATE statement, ​see docs. I'm not sure about earlier releases, but I'd be astonished if it wouldn't be in at least 7.2, too. But it still won't work with foreign key constraints (and this is a strict no, as described in the docs.)

I think that dropping/recreating all tables is not so bad, at least that's something you can easily do for any database. I wouldn't bother but do this if the TRUNCATE fails, and then you don't need a version check. If you don't like it, you could also disable the foreign key constraint temporarily with

I stand corrected on TRUNCATE in 7.x. I had a quick look and couldn't find it, but I'll admit that I didn't look that hard once I knew that TRUNCATE in 8.0 wasn't going to work. Thanks for the pointer.

The reason I went with TRUNCATE over DELETE is speed. Fixtures need to be added and removed for every test case in the system, so the fixture installation process needs to be as fast as possible. I initially started with a 'drop constraints and DELETE' approach, but then I remembered the speed advantage in TRUNCATE.

The removal of constraints is a little messy because an implied constraint is created whenever a REFERENCES to a previous table is added, and its a little nasty to find the name of these implied constraints. I'm currently playing with another approach based around putting DEFERRED/DEFERRABLE in the table definitions; I'll let you know how I go.

DEFERRED with DELETE work, but TRUNCATE automatically gets its own transaction, so it doesn't work.

Inserts are *much* faster without foreign key constraints. But most of the time you don't insert a lot of data, or do you?

I'd simply try to use TRUNCATE, and fall back to completely drop all tables and resync if TRUNCATE fails. With a bit of luck, people will improve this for their favourite database and provide patches ;-)

Isn't that a problem with mysql, too? #2720 describes a bug that means that mysql foreign key constraints are created in the wrong way ...

(In [3706]) Refs #2333 - Added a TEST_DATABASE_NAME setting that can be used to override the 'test_' + DATABASE_NAME naming policy. This setting is then used in runtests.py to restore the use of 'django_test_db' as the Django model/regression test database. Thanks to Michael Radziej for the feedback.

What about TEST_DATABASE_USER and TEST_DATABASE_PASSWORD, etc.? (I'm running on a shared host whose only support for databases is one-per-user.)