Description

Hi there,

Here is another issue with multi database setting and tests about fixture loading.
I have a setup with one legacy database and one dedicated to django.

In order to pass my tests, I load 2 fixture files. One is some administrative data for every test case and relating only to the django dedicated DB. The second fixture file is about test data which are loaded in the two files.

The fixtures load correctly on the django database but will fail to load in the legacy database.
The fixture loader notices that the administrative data fixture file has NO data and will rollback at this point.
While this might be fine in a single database setup, it shouldn't rollback here since data were already loaded in the other database.

# If the fixture we loaded contains 0 objects, assume that an
# error was encountered during fixture loading.
if objects_in_fixture == 0:
sys.stderr.write(
self.style.ERROR("No fixture data found for '%s'. (File format may be invalid.)" %
(fixture_name)))
transaction.rollback(using=using)
transaction.leave_transaction_management(using=using)
return

According to me:

it would make sense to issue a warning, but one should probably add the database name in order to discard any confusing error message.

one should rollback if such error occur (the user has been notified about it)

Maybe a better solution would be to let the test framework load a fixture in both database in a single call but that would require a major rework of the loaddata command and could possibly have side effects if one uses same class but different objects on different databases.

Empty fixtures, regardless of whether they are universal or targetted at a single database, are considered an error, since there's no reason to define an empty fixture; this error message is printed as part of the test process.

Marking invalid; if I've missed something important, please reopen, preferably with a clear test case that demonstrates why this is a problem.

In some cases, you can have a non empty fixture to be declared as empty.

Basically, the multi database dev have been made/tested with a master/slave configuration in mind. A django/legacy point of view didn't seem to be considered.

So let's say I have a django database in which I store django applications plus my own and another database which has legacy data.
Let's also consider my applications work with some custom models and a lot with legacy data.
At some point in my tests, I will probably create a fixture that will load data in only one of the two databases, let's say I load a set of complementary data in the legacy database.
And here comes the drama.
The fixture will load well for the legacy DB, but will be considered as empty for the django database.
As once the fixture is marked as empty, the fixture loaded will rollback the data. However, the test will not stop here and will keep on going.

So in the end, I am extremely confused with what happened.

I had en empty fixture message but no empty fixtures

I have half of my fixtures loaded in my test anyway (fixtures after the rollback are still loaded).

There is nowhere in the multi database page that states anything on the fixtures.

There is nowhere in the documentation that states that a fixture should have data for ALL the databases

I agree this isn't much an issue if you use the loaddata directly as you know pretty well what you did.
However, when this happens within a test case, it's just a nightmare to understand what happens.

I might write a project but it will takes some time to get it written.

Now I understand the error case. The rollback isn't the actual problem. The error here is that there is now a difference between the number of objects in a fixture and the number of objects that are loaded into a given database.

If an error is found, it is rolled back because individual fixtures can be interdependent - objects in one fixture can reference objects in another. This means that if one fixture file has an error, the entire fixture setup must be called into question.

If a fixture contains no objects, it is an error. This is well established Django behavior that exists because certain serializers don't distinguish between an unparseable fixture file and an empty fixture file. In order to catch the error case, we need to raise an error if a fixture exists, but is "empty". This doesn't cause an error in the test, but it does cause many error messages to be raised to stderr.

The problem introduced by multi-db is that if your router has an allow_syncdb definition, it's possible for a fixture to have valid data, but none of those objects will be loaded into the database. However, the object count is only incremented when an object is actually loaded, not when it is parsed, so a fixture is reported as "empty" when no objects are loaded, rather than when no objects actually exist.

The fix for this is fairly easy; I should be able to commit a patch shortly.

As a side note; this problem can also be avoided using the per-database fixtures I described when I closed the ticket earlier. If a fixture nominates the fixture 'foo', and you provide 'foo.default.json' and 'foo.legacy.json', then only the relevant fixture will be loaded onto a given database.