Change History (25)

Some things aren't pickleable. It's a known limitation at the Python level, and is unlikely to change. Is Django itself trying to do this somewhere? Is there an extremely good reason this should be possible? I'm closing this as wontfix. Please feel free to reply if you've got good answers to these questions.

Now request.user has been changed to a SimpleLazyObject rather than changing the request class to actually lazily return the true user object, this will choke.
Maybe this is an acceptable regression, but it's something I tripped up on in a project recently.

Is this regression related to the change to using pickle.HIGHEST_PROTOCOL throughout Django? The memcached backend used that, but locmem didn't. Does that mean this isn't a regression after all, and depended on the non-canonical behavior of locmem using the lowest pickle protocol?

FYI, the current code that makes request.user a SimpleLazyObject is from changeset [16305], which was a re-working of [16297], to fix #15929.

A comment cleaned up along the way said:

# If we access request.user, request.session is accessed, which results in
# 'Vary: Cookie' being sent in every request that uses this context
# processor, which can easily be every request on a site if
# TEMPLATE_CONTEXT_PROCESSORS has this context processor added. This kills
# the ability to cache. So, we carefully ensure these attributes are lazy.

which seems like a pretty good reason for making request.user lazy.

Would it be possible to fix this instead by fixing the chain of events somewhere else? e.g. should any access of request.session result in setting the Vary: Cookie header?

As a work around, can't you just pickle a real User model object instead of request.user?

That would still require a code change when upgrading to 1.4, right?

Yes, I meant it as a complement to your original argument. As you said, being able to pickle request.user directly was never documented, so whether or not it's a release blocking regression is debatable.

Would it be possible to fix this instead by fixing the chain of events somewhere else?

No, I don't think so.

e.g. should any access of request.session result in setting the Vary: Cookie header?

Yes, it should. Any access of the session means the response you are generating is almost certainly dependent in some way on values in the session, which means serving that same response as a cached response to other users would be at best wrong, and at worst a security issue. This applies even more strongly, if anything, to accessing request.user in particular. So it's quite important that request.user remain lazy, and that accessing it trigger Vary: Cookie on the response.

@kmike: That doesn't quite work, but it is close. My attached patch works in some cases, if someone can verify that it fixes the issue with a real User instance in environment showing the bug, I will commit it.

I added the code to SimpleLazyObject not LazyObject, to minimize impact with other subclasses of LazyObject. SimpleLazyObject is not designed to be subclassed really - you are supposed to use it just by passing a callable to the constructor.

There is one potential backwards compatibility concern here: if you had used a SimpleLazyObject with a callable that was itself pickleable, it would previously have been possible to pickle that SimpleLazyObject without the patch. Potentially, the object could have been pickled in the 'lazy' state, without the callable having been called.

However, I've tested this, and it seems that - for whatever reason - calling pickle.dumps causes the the callable to be called. Therefore, the object is not serialised in a 'lazy' state - it is always 'evaluated'. And this is just the same as it currently is, except we have made it explicit. The only difference now is that SimpleLazyObject._setupfunc is never available now after unpickling, but that shouldn't be a problem since it is never used.

The previous patch only works for objects that have no state. The basic problem is that due to proxying the __class__ attribute, the pickling machinery is confused (I think), and so it stores a reference to the class of the wrapped object, rather than SimpleLazyObject.

So, we need to define the __reduce__ method for full customization, rather than the simpler __getstate__ method. However, for some reason, the presence of the __class__ attribute proxying causes this to fail - if you define __reduce__ on SimpleLazyObject, it is never called when pickling unless you remove the __class__ hacking.

This takes us back to using __getstate__ (which, for some reason, doesn't suffer from the same problem as __reduce__). But now we have to cooperate with the fact that pickling will skip SimpleLazyObject and store a reference to the wrapped class, and define __getstate__ so that it returns the state from the wrapped object. This appears to work:

def__getstate__(self):ifself._wrapped is empty:self._setup()returnself._wrapped.__dict__

The result is that on unpickling, the SimpleLazyObject wrapper disappears entirely, which should be OK.

I still get failure if SimpleLazyObject is wrapping a builtin, and nothing but removing the __class__ proxying appears to be able to fix that. But that bug has always been there, and this change doesn't affect it.

I've confirmed that with this patch you can pickle request.user, and on unpickling get an object that compares equal to the original, so I'm therefore committing my improved patch.