google.appengine.ext.ndb.model module

Summary

Model and Property classes and associated stuff.

A model class represents the structure of entities stored in the
datastore. Applications define model classes to indicate the
structure of their entities, then instantiate those model classes
to create entities.

All model classes must inherit (directly or indirectly) from Model.
Through the magic of metaclasses, straightforward assignments in the
model class definition can be used to declare the model’s structure:

class Person(Model):
name = StringProperty()
age = IntegerProperty()

We can now create a Person entity and write it to Cloud Datastore:

p = Person(name='Arthur Dent', age=42)
k = p.put()

The return value from put() is a Key (see the documentation for
ndb/key.py), which can be used to retrieve the same entity later:

p2 = k.get()
p2 == p # Returns True

To update an entity, simple change its attributes and write it back
(note that this doesn’t change the key):

p2.name = 'Arthur Philip Dent'
p2.put()

We can also delete an entity (by using the key):

k.delete()

The property definitions in the class body tell the system the names
and the types of the fields to be stored in Cloud Datastore, whether
they must be indexed, their default value, and more.

Many different Property types exist. Most are indexed by default, the
exceptions indicated in the list below:

StructuredProperty: a field that is itself structured like an
entity; see below for more details

LocalStructuredProperty: like StructuredProperty but the on-disk
representation is an opaque blob; unindexed

ComputedProperty: a property whose value is computed from other
properties by a user-defined function. The property value is
written to Cloud Datastore so that it can be used in queries, but the
value from Cloud Datastore is not used when the entity is read back

GenericProperty: a property whose type is not constrained; mostly
used by the Expando class (see below) but also usable explicitly

JsonProperty: a property whose value is any object that can be
serialized using JSON; the value written to Cloud Datastore is a JSON
representation of that object

PickleProperty: a property whose value is any object that can be
serialized using Python’s pickle protocol; the value written to the
Cloud Datastore is the pickled representation of that object, using the
highest available pickle protocol

Most Property classes have similar constructor signatures. They
accept several optional keyword arguments:

name=<string>: the name used to store the property value in the
datastore. Unlike the following options, this may also be given as
a positional argument

indexed=<bool>: indicates whether the property should be indexed
(allowing queries on this property’s value)

repeated=<bool>: indicates that this property can have multiple
values in the same entity.

write_empty_list<bool>: For repeated value properties, controls
whether properties with no elements (the empty list) is
written to Datastore. If true, written, if false, then nothing
is written to Datastore.

required=<bool>: indicates that this property must be given a value

default=<value>: a default value if no explicit value is given

choices=<list of values>: a list or tuple of allowable values

validator=<function>: a general-purpose validation function. It
will be called with two arguments (prop, value) and should either
return the validated value or raise an exception. It is also
allowed for the function to modify the value, but calling it again
on the modified value should not modify the value further. (For
example: a validator that returns value.strip() or value.lower() is
fine, but one that returns value + ‘$’ is not.)

verbose_name=<value>: A human readable name for this property. This
human readable name can be used for html form labels.

The repeated and required/default options are mutually exclusive: a
repeated property cannot be required nor can it specify a default
value (the default is always an empty list and an empty list is always
an allowed value), but a required property can have a default.

Some property types have additional arguments. Some property types
do not support all options.

Repeated properties are always represented as Python lists; if there
is only one value, the list has only one element. When a new list is
assigned to a repeated property, all elements of the list are
validated. Since it is also possible to mutate lists in place,
repeated properties are re-validated before they are written to the
datastore.

No validation happens when an entity is read from Cloud Datastore;
however property values read that have the wrong type (e.g. a string
value for an IntegerProperty) are ignored.

For non-repeated properties, None is always a possible value, and no
validation is called when the value is set to None. However for
required properties, writing the entity to Cloud Datastore requires
the value to be something other than None (and valid).

The StructuredProperty is different from most other properties; it
lets you define a sub-structure for your entities. The substructure
itself is defined using a model class, and the attribute value is an
instance of that model class. However it is not stored in the
datastore as a separate entity; instead, its attribute values are
included in the parent entity using a naming convention (the name of
the structured attribute followed by a dot followed by the name of the
subattribute). For example:

Structured property types can be nested arbitrarily deep, but in a
hierarchy of nested structured property types, only one level can have
the repeated flag set. It is fine to have multiple structured
properties referencing the same model class.

It is also fine to use the same model class both as a top-level entity
class and as for a structured property; however queries for the model
class will only return the top-level entities.

The LocalStructuredProperty works similar to StructuredProperty on the
Python side. For example:

However the data written to Cloud Datastore is different; it writes a
‘Person’ entity with a ‘name’ attribute as before and a single
‘address’ attribute whose value is a blob which encodes the Address
value (using the standard”protocol buffer” encoding).

Sometimes the set of properties is not known ahead of time. In such
cases you can use the Expando class. This is a Model subclass that
creates properties on the fly, both upon assignment and when loading
an entity from Cloud Datastore. For example:

Note: this property exists for plain Model instances too; it is just
not as interesting for those.

The Model class offers basic query support. You can create a Query
object by calling the query() class method. Iterating over a Query
object returns the entities matching the query one at a time.

Query objects are fully described in the docstring for query.py, but
there is one handy shortcut that is only available through
Model.query(): positional arguments are interpreted as filter
expressions which are combined through an AND operator. For example:

Keyword arguments passed to .query() are passed along to the Query()
constructor.

It is possible to query for field values of structured properties. For
example:

qry = Person.query(Person.address.city == 'London')

A number of top-level functions also live in this module:

transaction() runs a function inside a transaction

get_multi() reads multiple entities at once

put_multi() writes multiple entities at once

delete_multi() deletes multiple entities at once

All these have a corresponding *_async() variant as well.
The *_multi_async() functions return a list of Futures.

And finally these (without async variants):

in_transaction() tests whether you are currently running in a transaction

@transactional decorates functions that should be run in a transaction

There are many other interesting features. For example, Model
subclasses may define pre-call and post-call hooks for most operations
(get, put, delete, allocate_ids), and Property classes may be
subclassed to suit various needs. Documentation for writing a
Property subclass is in the docstring for the Property class.

Contents

For flexibility and convenience, multiple constructor signatures are
supported.

The primary way to construct a key is using positional arguments:
- Key(kind1, id1, kind2, id2, …).

This is shorthand for either of the following two longer forms:
- Key(pairs=[(kind1, id1), (kind2, id2), …])
- Key(flat=[kind1, id1, kind2, id2, …])

Either of the above constructor forms can additionally pass in another
key using parent=<key>. The (kind, id) pairs of the parent key are
inserted before the (kind, id) pairs passed explicitly.

You can also construct a Key from a ‘url-safe’ encoded string:
- Key(urlsafe=<string>)

For esoteric purposes the following constructors exist:
- Key(reference=<reference>) – passing in a low-level Reference object
- Key(serialized=<string>) – passing in a serialized low-level Reference
- Key(<dict>) – for unpickling, the same as Key(**<dict>)

The ‘url-safe’ string is really a websafe-base64-encoded serialized
Reference, but it’s best to think of it as just an opaque unique
string.

If a Reference is passed (using one of reference, serialized or
urlsafe), the args and namespace keywords must match what is already
present in the Reference (after decoding if necessary). The parent
keyword cannot be combined with a Reference in any form.

Keys are immutable, which means that a Key object cannot be modified
once it has been created. This is enforced by the implementation as
well as Python allows.

For access to the contents of a key, the following methods and
operations are supported:

This returns a Future, whose result becomes available once the
deletion is complete. If no such entity exists, a Future is still
returned. In all cases the Future’s result is None (i.e. there is
no way to tell whether the entity existed or not).

May be raised by transaction functions when they want to roll back
instead of committing. Note that any exception raised by a transaction
function will cause a rollback. This is purely for convenience. See
datastore.RunInTransaction for details.

Model instances are usually called entities. All model classes
inheriting from Model automatically have MetaModel as their
metaclass, so that the properties are fixed up properly after the
class once the class is defined.

Because of this, you cannot use the same Property object to describe
multiple properties – you must create separate Property objects for
each property. E.g. this does not work:

**kwds – Keyword arguments to pass to the constructor of the model class
if an instance for the specified key name does not already exist. If
an instance with the supplied key_name and parent already exists,
these arguments will be discarded.

Returns

Existing instance of Model class with the specified key name and parent
or a new one that has just been created.

Each keyword argument will be used to set a corresponding
property. Keywords must refer to valid property name. This is
similar to passing keyword arguments to the Model constructor,
except that no provisions for key, id or parent are made.

retries=N: Retry up to N times (i.e. try up to N+1 times)
propagation=<flag>: Determines how an existing transaction should be

propagated, where <flag> can be one of the following:
TransactionOptions.NESTED: Start a nested transaction (this is the

default; but actual nested transactions are not yet implemented,
so effectively you can only use this outside an existing transaction).

TransactionOptions.MANDATORY: A transaction must already be in progress.
TransactionOptions.ALLOWED: If a transaction is in progress, join it.
TransactionOptions.INDEPENDENT: Always start a new parallel transaction.

xg=True: On the High Replication Datastore, enable cross-group

transactions, i.e. allow writing to up to 5 entity groups.

WARNING: Using anything other than NESTED for the propagation flag
can have strange consequences. When using ALLOWED or MANDATORY, if
an exception is raised, the transaction is likely not safe to
commit. When using INDEPENDENT it is not generally safe to return
values read to the caller (as they were not read in the caller’s
transaction).

This looks like StructuredProperty on the Python side, but is
written like a BlobProperty in Cloud Datastore. It is not indexed
and you cannot query for subproperties. On the other hand, the
on-disk representation is more efficient and can be made even more
efficient by passing compressed=True, which compresses the blob
data using gzip.

Computed properties cannot be set directly, but are instead generated by a
function when required. They are useful to provide fields in Cloud Datastore
that can be used for filtering or sorting without having to manually set the
value in code - for example, sorting on the length of a BlobProperty, or
using an equality filter to check if another field is not empty.

ComputedProperty can be declared as a regular property, passing a function as
the first argument, or it can be used as a decorator for the function that
does the calculation.

A class describing a typed, persisted attribute of a Cloud Datastore entity.

Not to be confused with Python’s ‘property’ built-in.

This is just a base class; there are specific subclasses that
describe Properties of various types (and GenericProperty which
describes a dynamically typed Property).

All special Property attributes, even those considered ‘public’,
have names starting with an underscore, because StructuredProperty
uses the non-underscore attribute namespace to refer to nested
Property names; this is essential for specifying queries on
subproperties (see the module docstring).

The Property class and its predefined subclasses allow easy
subclassing using composable (or stackable) validation and
conversion APIs. These require some terminology definitions:

A ‘user value’ is a value such as would be set and accessed by the
application code using standard attributes on the entity.

A ‘base value’ is a value such as would be serialized to
and deserialized from Cloud Datastore.

The values stored in ent._values[name] and accessed by
_store_value() and _retrieve_value() can be either user values or
base values. To retrieve user values, use
_get_user_value(). To retrieve base values, use
_get_base_value(). In particular, _get_value() calls
_get_user_value(), and _serialize() effectively calls
_get_base_value().

To store a user value, just call _store_value(). To store a
base value, wrap the value in a _BaseValue() and then
call _store_value().

A Property subclass that wants to implement a specific
transformation between user values and serialiazble values should
implement two methods, _to_base_type() and _from_base_type().
These should NOT call their super() method; super calls are taken
care of by _call_to_base_type() and _call_from_base_type().
This is what is meant by composable (or stackable) APIs.

The API supports ‘stacking’ classes with ever more sophisticated
user<–>base conversions: the user–>base conversion
goes from more sophisticated to less sophisticated, while the
base–>user conversion goes from less sophisticated to more
sophisticated. For example, see the relationship between
BlobProperty, TextProperty and StringProperty.

In addition to _to_base_type() and _from_base_type(), the
_validate() method is also a composable API.

The validation API distinguishes between ‘lax’ and ‘strict’ user
values. The set of lax values is a superset of the set of strict
values. The _validate() method takes a lax value and if necessary
converts it to a strict value. This means that when setting the
property value, lax values are accepted, while when getting the
property value, only strict values will be returned. If no
conversion is needed, _validate() may return None. If the argument
is outside the set of accepted lax values, _validate() should raise
an exception, preferably TypeError or
datastore_errors.BadValueError.

Things that _validate(), _to_base_type() and _from_base_type()
do not need to handle:

None: They will not be called with None (and if they return None,
this means that the value does not need conversion).

Repeated values: The infrastructure (_get_user_value() and
_get_base_value()) takes care of calling
_from_base_type() or _to_base_type() for each list item in a
repeated value.

Wrapping values in _BaseValue(): The wrapping and unwrapping is
taken care of by the infrastructure that calls the composable APIs.

Comparisons: The comparison operations call _to_base_type() on
their operand.

Distinguishing between user and base values: the
infrastructure guarantees that _from_base_type() will be called
with an (unwrapped) base value, and that
_to_base_type() will be called with a user value.

Returning the original value: if any of these return None, the
original value is kept. (Returning a differen value not equal to
None will substitute the different value.)

Note: Unlike Django, auto_now_add can be overridden by setting the
value before writing the entity. And unlike classic db, auto_now
does not supply a default value. Also unlike classic db, when the
entity is written, the property values are updated to match what
was written. Finally, beware that this also updates the value in
the in-process cache, and that auto_now_add may interact weirdly
with transaction retries (a retry of a property with auto_now_add
set will reuse the value that was set on the first try).

This is mainly used for Expando and for orphans (values present in
Cloud Datastore but not represented in the Model subclass) but can
also be used explicitly for properties with dynamically-typed
values.

This supports compressed=True, which is only effective for str
values (not for unicode), and implies indexed=False.