Source code for google.appengine.ext.ndb.key

## Copyright 2008 The ndb Authors. All Rights Reserved.## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License."""The Key class, and associated utilities.A Key encapsulates the following pieces of information, which togetheruniquely designate a (possible) entity in the App Engine datastore:- an application id (a string)- a namespace (a string)- a list of one or more (kind, id) pairs where kind is a string and id is either a string or an integer.The application id must always be part of the key, but since mostapplications can only access their own entities, it defaults to thecurrent application id and you rarely need to worry about it. It mustnot be empty.The namespace designates a top-level partition of the key space for aparticular application. If you've never heard of namespaces, you cansafely ignore this feature.Most of the action is in the (kind, id) pairs. A key must have atleast one (kind, id) pair. The last (kind, id) pair gives the kindand the id of the entity that the key refers to, the others merelyspecify a 'parent key'.The kind is a string giving the name of the model class used torepresent the entity. (In more traditional databases this would bethe table name.) A model class is a Python class derived fromndb.Model; see the documentation for ndb/model.py. Only the classname itself is used as the kind. This means all your model classesmust be uniquely named within one application. You can override thison a per-class basis.The id is either a string or an integer. When the id is a string, theapplication is in control of how it assigns ids: For example, if youcould use an email address as the id for Account entities.To use integer ids, you must let the datastore choose a unique id foran entity when it is first inserted into the datastore. You can setthe id to None to represent the key for an entity that hasn't yet beeninserted into the datastore. The final key (including the assignedid) will be returned after the entity is successfully inserted intothe datastore.A key for which the id of the last (kind, id) pair is set to None iscalled an incomplete key. Such keys can only be used to insertentities into the datastore.A key with exactly one (kind, id) pair is called a top level key or aroot key. Top level keys are also used as entity groups, which play arole in transaction management.If there is more than one (kind, id) pair, all but the last pairrepresent the 'ancestor path', also known as the key of the 'parententity'.Other constraints:- Kinds and string ids must not be empty and must be at most 500 bytes long (after UTF-8 encoding, if given as Python unicode objects). NOTE: This is defined as a module level constant _MAX_KEYPART_BYTES.- Integer ids must be at least 1 and less than 2**63.For more info about namespaces, seehttp://code.google.com/appengine/docs/python/multitenancy/overview.html.The namespace defaults to the 'default namespace' selected by thenamespace manager. To explicitly select the empty namespace passnamespace=''."""__author__='guido@google.com (Guido van Rossum)'importbase64importosfrom.google_importsimportdatastore_errorsfrom.google_importsimportdatastore_typesfrom.google_importsimportnamespace_managerfrom.google_importsimportentity_pbfrom.importutils__all__=['Key']_MAX_LONG=2L**63# Use 2L, see issue 65. http://goo.gl/ELczz_MAX_KEYPART_BYTES=500

[docs]classKey(object):"""An immutable datastore key. For flexibility and convenience, multiple constructor signatures are supported. The primary way to construct a key is using positional arguments: - Key(kind1, id1, kind2, id2, ...). This is shorthand for either of the following two longer forms: - Key(pairs=[(kind1, id1), (kind2, id2), ...]) - Key(flat=[kind1, id1, kind2, id2, ...]) Either of the above constructor forms can additionally pass in another key using parent=<key>. The (kind, id) pairs of the parent key are inserted before the (kind, id) pairs passed explicitly. You can also construct a Key from a 'url-safe' encoded string: - Key(urlsafe=<string>) For esoteric purposes the following constructors exist: - Key(reference=<reference>) -- passing in a low-level Reference object - Key(serialized=<string>) -- passing in a serialized low-level Reference - Key(<dict>) -- for unpickling, the same as Key(**<dict>) The 'url-safe' string is really a websafe-base64-encoded serialized Reference, but it's best to think of it as just an opaque unique string. Additional constructor keyword arguments: - app=<string> -- specify the application id - namespace=<string> -- specify the namespace If a Reference is passed (using one of reference, serialized or urlsafe), the args and namespace keywords must match what is already present in the Reference (after decoding if necessary). The parent keyword cannot be combined with a Reference in any form. Keys are immutable, which means that a Key object cannot be modified once it has been created. This is enforced by the implementation as well as Python allows. For access to the contents of a key, the following methods and operations are supported: - repr(key), str(key) -- return a string representation resembling the shortest constructor form, omitting the app and namespace unless they differ from the default value. - key1 == key2, key1 != key2 -- comparison for equality between Keys. - hash(key) -- a hash value sufficient for storing Keys in a dict. - key.pairs() -- a tuple of (kind, id) pairs. - key.flat() -- a tuple of flattened kind and id values, i.e. (kind1, id1, kind2, id2, ...). - key.app() -- the application id. - key.id() -- the string or integer id in the last (kind, id) pair, or None if the key is incomplete. - key.string_id() -- the string id in the last (kind, id) pair, or None if the key has an integer id or is incomplete. - key.integer_id() -- the integer id in the last (kind, id) pair, or None if the key has a string id or is incomplete. - key.namespace() -- the namespace. - key.kind() -- a shortcut for key.pairs()[-1][0]. - key.parent() -- a Key constructed from all but the last (kind, id) pairs. - key.urlsafe() -- a websafe-base64-encoded serialized Reference. - key.serialized() -- a serialized Reference. - key.reference() -- a Reference object. The caller promises not to mutate it. Keys also support interaction with the datastore; these methods are the only ones that engage in any kind of I/O activity. For Future objects, see the document for ndb/tasklets.py. - key.get() -- return the entity for the Key. - key.get_async() -- return a Future whose eventual result is the entity for the Key. - key.delete() -- delete the entity for the Key. - key.delete_async() -- asynchronously delete the entity for the Key. Keys may be pickled. Subclassing Key is best avoided; it would be hard to get right. """__slots__=['__reference','__pairs','__app','__namespace']def__new__(cls,*_args,**kwargs):"""Constructor. See the class docstring for arguments."""if_args:iflen(_args)==1andisinstance(_args[0],dict):ifkwargs:raiseTypeError('Key() takes no keyword arguments when a dict is the ''the first and only non-keyword argument (for ''unpickling).')kwargs=_args[0]else:if'flat'inkwargs:raiseTypeError('Key() with positional arguments ''cannot accept flat as a keyword argument.')kwargs['flat']=_argsself=super(Key,cls).__new__(cls)# Either __reference or (__pairs, __app, __namespace) must be set.# Either one fully specifies a key; if both are set they must be# consistent with each other.if'reference'inkwargsor'serialized'inkwargsor'urlsafe'inkwargs:(self.__reference,self.__pairs,self.__app,self.__namespace)=self._parse_from_ref(cls,**kwargs)elif'pairs'inkwargsor'flat'inkwargs:self.__reference=None(self.__pairs,self.__app,self.__namespace)=self._parse_from_args(**kwargs)else:raiseTypeError('Key() cannot create a Key instance without arguments.')returnself@staticmethoddef_parse_from_args(pairs=None,flat=None,app=None,namespace=None,parent=None):ifflat:ifpairsisnotNone:raiseTypeError('Key() cannot accept both flat and pairs arguments.')iflen(flat)%2:raiseValueError('Key() must have an even number of positional ''arguments.')pairs=[(flat[i],flat[i+1])foriinxrange(0,len(flat),2)]else:pairs=list(pairs)ifnotpairs:raiseTypeError('Key must consist of at least one pair.')fori,(kind,id)inenumerate(pairs):ifisinstance(id,unicode):id=id.encode('utf8')elifidisNone:ifi+1<len(pairs):raisedatastore_errors.BadArgumentError('Incomplete Key entry must be last')else:ifnotisinstance(id,(int,long,str)):raiseTypeError('Key id must be a string or a number; received %r'%id)ifisinstance(kind,type):kind=kind._get_kind()ifisinstance(kind,unicode):kind=kind.encode('utf8')ifnotisinstance(kind,str):raiseTypeError('Key kind must be a string or Model class; ''received %r'%kind)ifnotid:id=Nonepairs[i]=(kind,id)ifparentisnotNone:ifnotisinstance(parent,Key):raisedatastore_errors.BadValueError('Expected Key instance, got %r'%parent)ifnotparent.id():raisedatastore_errors.BadArgumentError('Parent cannot have incomplete key')pairs[:0]=parent.pairs()ifapp:ifapp!=parent.app():raiseValueError('Cannot specify a different app %r ''than the parent app %r'%(app,parent.app()))else:app=parent.app()ifnamespaceisnotNone:ifnamespace!=parent.namespace():raiseValueError('Cannot specify a different namespace %r ''than the parent namespace %r'%(namespace,parent.namespace()))else:namespace=parent.namespace()ifnotapp:app=_DefaultAppId()ifnamespaceisNone:namespace=_DefaultNamespace()returntuple(pairs),app,namespace@staticmethoddef_parse_from_ref(cls,pairs=None,flat=None,reference=None,serialized=None,urlsafe=None,app=None,namespace=None,parent=None):"""Construct a Reference; the signature is the same as for Key."""ifclsisnotKey:raiseTypeError('Cannot construct Key reference on non-Key class; ''received %r'%cls)if(bool(pairs)+bool(flat)+bool(reference)+bool(serialized)+bool(urlsafe)+bool(parent))!=1:raiseTypeError('Cannot construct Key reference from incompatible ''keyword arguments.')ifurlsafe:serialized=_DecodeUrlSafe(urlsafe)ifserialized:reference=_ReferenceFromSerialized(serialized)ifreference:reference=_ReferenceFromReference(reference)pairs=[]elem=Nonepath=reference.path()foreleminpath.element_list():kind=elem.type()ifelem.has_id():id_or_name=elem.id()else:id_or_name=elem.name()ifnotid_or_name:id_or_name=Nonetup=(kind,id_or_name)pairs.append(tup)ifelemisNone:raiseRuntimeError('Key reference has no path or elements (%r, %r, %r).'%(urlsafe,serialized,str(reference)))# TODO: ensure that each element has a type and either an id or a name# You needn't specify app= or namespace= together with reference=,# serialized= or urlsafe=, but if you do, their values must match# what is already in the reference.ref_app=reference.app()ifappisnotNone:ifapp!=ref_app:raiseRuntimeError('Key reference constructed uses a different app %r ''than the one specified %r'%(ref_app,app))ref_namespace=reference.name_space()ifnamespaceisnotNone:ifnamespace!=ref_namespace:raiseRuntimeError('Key reference constructed uses a different ''namespace %r than the one specified %r'%(ref_namespace,namespace))return(reference,tuple(pairs),ref_app,ref_namespace)def__repr__(self):"""String representation, used by str() and repr(). We produce a short string that conveys all relevant information, suppressing app and namespace when they are equal to the default. """# TODO: Instead of "Key('Foo', 1)" perhaps return "Key(Foo, 1)" ?args=[]foriteminself.flat():ifnotitem:args.append('None')elifisinstance(item,basestring):ifnotisinstance(item,str):raiseTypeError('Key item is not an 8-bit string %r'%item)args.append(repr(item))else:args.append(str(item))ifself.app()!=_DefaultAppId():args.append('app=%r'%self.app())ifself.namespace()!=_DefaultNamespace():args.append('namespace=%r'%self.namespace())return'Key(%s)'%', '.join(args)__str__=__repr__def__hash__(self):"""Hash value, for use in dict lookups."""# This ignores app and namespace, which is fine since hash()# doesn't need to return a unique value -- it only needs to ensure# that the hashes of equal keys are equal, not the other way# around.returnhash(tuple(self.pairs()))def__eq__(self,other):"""Equality comparison operation."""# This does not use __tuple() because it is usually enough to# compare pairs(), and we're performance-conscious here.ifnotisinstance(other,Key):returnNotImplementedreturn(self.__pairs==other.__pairsandself.__app==other.__appandself.__namespace==other.__namespace)def__ne__(self,other):"""The opposite of __eq__."""ifnotisinstance(other,Key):returnNotImplementedreturnnotself.__eq__(other)def__tuple(self):"""Helper to return an orderable tuple."""return(self.__app,self.__namespace,self.__pairs)def__lt__(self,other):"""Less than ordering."""ifnotisinstance(other,Key):returnNotImplementedreturnself.__tuple()<other.__tuple()def__le__(self,other):"""Less than or equal ordering."""ifnotisinstance(other,Key):returnNotImplementedreturnself.__tuple()<=other.__tuple()def__gt__(self,other):"""Greater than ordering."""ifnotisinstance(other,Key):returnNotImplementedreturnself.__tuple()>other.__tuple()def__ge__(self,other):"""Greater than or equal ordering."""ifnotisinstance(other,Key):returnNotImplementedreturnself.__tuple()>=other.__tuple()def__getstate__(self):"""Private API used for pickling."""# If any changes are made to this function pickle compatibility tests should# be updated.return({'pairs':self.__pairs,'app':self.__app,'namespace':self.__namespace},)def__setstate__(self,state):"""Private API used for pickling."""iflen(state)!=1:raiseTypeError('Invalid state length, expected 1; received %i'%len(state))kwargs=state[0]ifnotisinstance(kwargs,dict):raiseTypeError('Key accepts a dict of keyword arguments as state; ''received %r'%kwargs)self.__reference=Noneself.__pairs=tuple(kwargs['pairs'])self.__app=kwargs['app']self.__namespace=kwargs['namespace']def__getnewargs__(self):"""Private API used for pickling."""return({'pairs':self.__pairs,'app':self.__app,'namespace':self.__namespace},)

[docs]defparent(self):"""Return a Key constructed from all but the last (kind, id) pairs. If there is only one (kind, id) pair, return None. """pairs=self.__pairsiflen(pairs)<=1:returnNonereturnKey(pairs=pairs[:-1],app=self.__app,namespace=self.__namespace)

[docs]defroot(self):"""Return the root key. This is either self or the highest parent."""pairs=self.__pairsiflen(pairs)<=1:returnselfreturnKey(pairs=pairs[:1],app=self.__app,namespace=self.__namespace)

[docs]defnamespace(self):"""Return the namespace."""returnself.__namespace

[docs]defid(self):"""Return the string or integer id in the last (kind, id) pair, if any. Returns: A string or integer id, or None if the key is incomplete. """returnself.__pairs[-1][1]

[docs]defstring_id(self):"""Return the string id in the last (kind, id) pair, if any. Returns: A string id, or None if the key has an integer id or is incomplete. """id=self.id()ifnotisinstance(id,basestring):id=Nonereturnid

[docs]definteger_id(self):"""Return the integer id in the last (kind, id) pair, if any. Returns: An integer id, or None if the key has a string id or is incomplete. """id=self.id()ifnotisinstance(id,(int,long)):id=Nonereturnid

[docs]defflat(self):"""Return a tuple of alternating kind and id values."""flat=[]forkind,idinself.__pairs:flat.append(kind)flat.append(id)returntuple(flat)

[docs]defkind(self):"""Return the kind of the entity referenced. This is the kind from the last (kind, id) pair. """returnself.__pairs[-1][0]

[docs]defreference(self):"""Return the Reference object for this Key. This is a entity_pb.Reference instance -- a protocol buffer class used by the lower-level API to the datastore. NOTE: The caller should not mutate the return value. """ifself.__referenceisNone:self.__reference=_ConstructReference(self.__class__,pairs=self.__pairs,app=self.__app,namespace=self.__namespace)returnself.__reference

[docs]defserialized(self):"""Return a serialized Reference object for this Key."""returnself.reference().Encode()

[docs]defurlsafe(self):"""Return a url-safe string encoding this Key's Reference. This string is compatible with other APIs and languages and with the strings used to represent Keys in GQL and in the App Engine Admin Console. """# This is 3-4x faster than urlsafe_b64decode()urlsafe=base64.b64encode(self.reference().Encode())returnurlsafe.rstrip('=').replace('+','-').replace('/','_')

# Datastore API using the default context.# These use local import since otherwise they'd be recursive imports.

[docs]defget(self,**ctx_options):"""Synchronously get the entity for this Key. Return None if there is no such entity. """returnself.get_async(**ctx_options).get_result()

[docs]defget_async(self,**ctx_options):"""Return a Future whose result is the entity for this Key. If no such entity exists, a Future is still returned, and the Future's eventual return result be None. """from.importmodel,taskletsctx=tasklets.get_context()cls=model.Model._kind_map.get(self.kind())ifcls:cls._pre_get_hook(self)fut=ctx.get(self,**ctx_options)ifcls:post_hook=cls._post_get_hookifnotcls._is_default_hook(model.Model._default_post_get_hook,post_hook):fut.add_immediate_callback(post_hook,self,fut)returnfut

[docs]defdelete(self,**ctx_options):"""Synchronously delete the entity for this Key. This is a no-op if no such entity exists. """returnself.delete_async(**ctx_options).get_result()

[docs]defdelete_async(self,**ctx_options):"""Schedule deletion of the entity for this Key. This returns a Future, whose result becomes available once the deletion is complete. If no such entity exists, a Future is still returned. In all cases the Future's result is None (i.e. there is no way to tell whether the entity existed or not). """from.importtasklets,modelctx=tasklets.get_context()cls=model.Model._kind_map.get(self.kind())ifcls:cls._pre_delete_hook(self)fut=ctx.delete(self,**ctx_options)ifcls:post_hook=cls._post_delete_hookifnotcls._is_default_hook(model.Model._default_post_delete_hook,post_hook):fut.add_immediate_callback(post_hook,self,fut)returnfut

# The remaining functions in this module are private.# TODO: Conform to PEP 8 naming, e.g. _construct_reference() etc.@utils.positional(1)def_ConstructReference(cls,pairs=None,flat=None,reference=None,serialized=None,urlsafe=None,app=None,namespace=None,parent=None):"""Construct a Reference; the signature is the same as for Key."""ifclsisnotKey:raiseTypeError('Cannot construct Key reference on non-Key class; ''received %r'%cls)if(bool(pairs)+bool(flat)+bool(reference)+bool(serialized)+bool(urlsafe))!=1:raiseTypeError('Cannot construct Key reference from incompatible keyword ''arguments.')ifflatorpairs:ifflat:iflen(flat)%2:raiseTypeError('_ConstructReference() must have an even number of ''positional arguments.')pairs=[(flat[i],flat[i+1])foriinxrange(0,len(flat),2)]elifparentisnotNone:pairs=list(pairs)ifnotpairs:raiseTypeError('Key references must consist of at least one pair.')ifparentisnotNone:ifnotisinstance(parent,Key):raisedatastore_errors.BadValueError('Expected Key instance, got %r'%parent)pairs[:0]=parent.pairs()ifapp:ifapp!=parent.app():raiseValueError('Cannot specify a different app %r ''than the parent app %r'%(app,parent.app()))else:app=parent.app()ifnamespaceisnotNone:ifnamespace!=parent.namespace():raiseValueError('Cannot specify a different namespace %r ''than the parent namespace %r'%(namespace,parent.namespace()))else:namespace=parent.namespace()reference=_ReferenceFromPairs(pairs,app=app,namespace=namespace)else:ifparentisnotNone:raiseTypeError('Key reference cannot be constructed when the parent ''argument is combined with either reference, serialized ''or urlsafe arguments.')ifurlsafe:serialized=_DecodeUrlSafe(urlsafe)ifserialized:reference=_ReferenceFromSerialized(serialized)ifnotreference.path().element_size():raiseRuntimeError('Key reference has no path or elements (%r, %r, %r).'%(urlsafe,serialized,str(reference)))# TODO: ensure that each element has a type and either an id or a nameifnotserialized:reference=_ReferenceFromReference(reference)# You needn't specify app= or namespace= together with reference=,# serialized= or urlsafe=, but if you do, their values must match# what is already in the reference.ifappisnotNone:ref_app=reference.app()ifapp!=ref_app:raiseRuntimeError('Key reference constructed uses a different app %r ''than the one specified %r'%(ref_app,app))ifnamespaceisnotNone:ref_namespace=reference.name_space()ifnamespace!=ref_namespace:raiseRuntimeError('Key reference constructed uses a different ''namespace %r than the one specified %r'%(ref_namespace,namespace))returnreferencedef_ReferenceFromPairs(pairs,reference=None,app=None,namespace=None):"""Construct a Reference from a list of pairs. If a Reference is passed in as the second argument, it is modified in place. The app and namespace are set from the corresponding keyword arguments, with the customary defaults. """ifreferenceisNone:reference=entity_pb.Reference()path=reference.mutable_path()last=Falseforkind,idornameinpairs:iflast:raisedatastore_errors.BadArgumentError('Incomplete Key entry must be last')t=type(kind)iftisstr:passeliftisunicode:kind=kind.encode('utf8')else:ifissubclass(t,type):# Late import to avoid cycles.from.modelimportModelmodelclass=kindifnotissubclass(modelclass,Model):raiseTypeError('Key kind must be either a string or subclass of ''Model; received %r'%modelclass)kind=modelclass._get_kind()t=type(kind)iftisstr:passeliftisunicode:kind=kind.encode('utf8')elifissubclass(t,str):passelifissubclass(t,unicode):kind=kind.encode('utf8')else:raiseTypeError('Key kind must be either a string or subclass of Model;'' received %r'%kind)# pylint: disable=superfluous-parensifnot(1<=len(kind)<=_MAX_KEYPART_BYTES):raiseValueError('Key kind string must be a non-empty string up to %i''bytes; received %s'%(_MAX_KEYPART_BYTES,kind))elem=path.add_element()elem.set_type(kind)t=type(idorname)iftisintortislong:# pylint: disable=superfluous-parensifnot(1<=idorname<_MAX_LONG):raiseValueError('Key id number is too long; received %i'%idorname)elem.set_id(idorname)eliftisstr:# pylint: disable=superfluous-parensifnot(1<=len(idorname)<=_MAX_KEYPART_BYTES):raiseValueError('Key name strings must be non-empty strings up to %i ''bytes; received %s'%(_MAX_KEYPART_BYTES,idorname))elem.set_name(idorname)eliftisunicode:idorname=idorname.encode('utf8')# pylint: disable=superfluous-parensifnot(1<=len(idorname)<=_MAX_KEYPART_BYTES):raiseValueError('Key name unicode strings must be non-empty strings up'' to %i bytes; received %s'%(_MAX_KEYPART_BYTES,idorname))elem.set_name(idorname)elifidornameisNone:last=Trueelifissubclass(t,(int,long)):# pylint: disable=superfluous-parensifnot(1<=idorname<_MAX_LONG):raiseValueError('Key id number is too long; received %i'%idorname)elem.set_id(idorname)elifissubclass(t,basestring):ifissubclass(t,unicode):idorname=idorname.encode('utf8')# pylint: disable=superfluous-parensifnot(1<=len(idorname)<=_MAX_KEYPART_BYTES):raiseValueError('Key name strings must be non-empty strings up to %i ''bytes; received %s'%(_MAX_KEYPART_BYTES,idorname))elem.set_name(idorname)else:raiseTypeError('id must be either a numeric id or a string name; ''received %r'%idorname)# An empty app id means to use the default app id.ifnotapp:app=_DefaultAppId()# Always set the app id, since it is mandatory.reference.set_app(app)# An empty namespace overrides the default namespace.ifnamespaceisNone:namespace=_DefaultNamespace()# Only set the namespace if it is not empty.ifnamespace:reference.set_name_space(namespace)returnreferencedef_ReferenceFromReference(reference):"""Copy a Reference."""new_reference=entity_pb.Reference()new_reference.CopyFrom(reference)returnnew_referencedef_ReferenceFromSerialized(serialized):"""Construct a Reference from a serialized Reference."""ifnotisinstance(serialized,basestring):raiseTypeError('serialized must be a string; received %r'%serialized)elifisinstance(serialized,unicode):serialized=serialized.encode('utf8')returnentity_pb.Reference(serialized)def_DecodeUrlSafe(urlsafe):"""Decode a url-safe base64-encoded string. This returns the decoded string. """ifnotisinstance(urlsafe,basestring):raiseTypeError('urlsafe must be a string; received %r'%urlsafe)ifisinstance(urlsafe,unicode):urlsafe=urlsafe.encode('utf8')mod=len(urlsafe)%4ifmod:urlsafe+='='*(4-mod)# This is 3-4x faster than urlsafe_b64decode()returnbase64.b64decode(urlsafe.replace('-','+').replace('_','/'))def_DefaultAppId():"""Return the default application id. This is taken from the APPLICATION_ID environment variable. """returnos.getenv('APPLICATION_ID','_')def_DefaultNamespace():"""Return the default namespace. This is taken from the namespace manager. """returnnamespace_manager.get_namespace()