Description

I noticed that the ManyToManyField ignored my "use_for_related_fields = False" setting, and investigated by looking into the source code. It seems that the ManyToManyField always uses _default_manager, and the ForeignKey-fields use _base_manager. If this is by design, I have failed to find the documentation explaining why the two kinds of lookups should have different behaviour (and what to do if you want custom behaviour for the objects-manager, but not for ManyToMany lookups), and this should then be considered a plea for a documentation update.

Change History (31)

I can't think of an obvious reason why use_for_related_fields isn't honored on m2m fields. However, introducing this change has the potential to introduce some fairly major backwards incompatibilities.

ManyToManyRelatedField uses different Manager than ForeignKey-field →
use_for_related_fields=False is not honored by reverse FK or M2M related managers

Type:

New feature →
Bug

This isn't just M2M fields, it's reverse FKs as well. ForeignRelatedObjectsDescriptor, ManyRelatedObjectsDescriptor, and ReverseManyRelatedObjectsDescriptor all unconditionally use _default_manager rather than _base_manager as the superclass for the dynamic manager they create (the only exception is for internal deletion purposes), and ForeignRelatedObjectsDescriptor.create_manager and create_many_related_manager both create a dynamic manager class whose get_query_set method gets its initial queryset from super(). In other words, regardless of the value of use_for_related_fields you always get a related-manager that is a subclass of and behaves the same as your default manager.

The only place "use_for_related_fields = False" (the default, in theory!) is actually honored currently is for OneToOneField (SingleRelatedObjectDescriptor and ReverseSingleRelatedObjectDescriptor both use _base_manager), and internal deletion traversal.

This definitely does not match the documented behavior... and fixing it is also likely to break people's code. The fix is simple, if we actually want to make it - just always use _base_manager instead of _default_manager in all of those descriptors. (ensure_default_manager already sets _base_manager to be the same as _default_manager if use_for_related_fields = True).

I'm changing the categorization here to "bug." When we've clearly documented how a feature is supposed to work, and it doesn't work, fixing it is not a "new feature," regardless of how backwards-incompatible the fix is. Our usual policy is that we fix bugs regardless -- but I'm sure exceptions to that have been made in the past when there was a likelihood of breaking lots of code, and this might be a case for such an exception.

The first commit there has failing tests demonstrating that we currently don't do what our documentation says.

The second commit has the fix (essentially s/_default_manager/_base_manager/g in related.py.)

And the third commit fixes the two places in Django's own test suite where we were assuming the semantics of use_for_related_fields=True without specifying it. This is a pretty good indicator that fixing this will break other peoples' code as well.

The result of the branch is that all tests pass and the behavior of related managers matches what our documentation says it is.

I must admit I haven't done enough research yet, so the above suggestion might be missing something crucial...

It seems just fixing this issue to match our docs would cause problems at least for some users. So, even if we want to do that, then we would likely want to invent a new flag behaving as the currently documented use_for_related_fields and deprecate the old one so that users have time to adapt to the change in behaviour.

This is an open source project. If you truly feel that this is the single greatest bug facing Django, and can't imagine how we've let it linger for so long, YOU CAN FIX IT YOURSELF. You can fork the repo, make the necessary change, and use that fork in your own work. Using an anonymous account to howl at the moon about how hard done by you are because nobody else will volunteer to fix your problem for free is neither helpful, nor endearing.

I just implemented it to the ForeignKey relations, and it seems to works fine.
It is also backwards compatible, and I think it may be possible to deprecate progressively the « use_for_related_fields » option.

I don't think putting a big warning in the docs "this does not work" is sufficient, especially when there are definitely avenues for fixing the underlying problem. It's the lazy way out. Doing nothing is also bad. But until someone steps up to actually do the work, then it's still going to be an issue.

Now, does my proposal above have any significant drawbacks? It's a combination of previous talk and my discussions with Loic on IRC.

​PR6175 deprecates use_for_related_fields. "to-one" relations go through _base_manager which can be specified with the base_manager_name model option and "to-many" relations go through _default_manager which can be specified with the default_manager_name model option.

We tried to add a blanket "use for every relations" manager in the form of related_manager_name but ​eventually decided against. For finer control over relation managers, I think comment:27 is the way to go but this is a new feature which deserves a ticket of its own.