Not Logged In

m01.mongo 0.11.1

This package provides a mongodb object mapper framework including zopetransaction support based on some core zope component libraries. This packagecan get used with or without zope.persistent and as a full replacement for theZODB. The package is not heavy based on zope itself and can get used in anypython project which requires a bridge from mongodb to python object.

======README======

IMPORTANT:If you run the tests with the --all option a real mongodb stub server willstart at port 45017!

This package provides non persistent MongoDB object implementations. They cansimply get mixed with persistent.Persistent and contained.Contained if you liketo use them in a mixed MongoDB/ZODB application setup. We currently use thisframework as ORM (object relation mapper) where we map MongoDB objectsto python/zope schema based objects including validation etc.

In our last project, we started with a mixed ZODB/MongoDB application where wemixed persistent.persistent into IMongoContainer objects. But later we whereso exited about the performance and stability that we removed the ZODBpersistence layer at all. Now we use a ZODB less setup in our applicationwhere we start with a non persistent item as our application root. All requiredtools where we use for such a ZODB less application setup are located in them01.publisher and p01.recipe.setup package.

NOTE: Some of this test use a fake mongodb located in m01/mongo/testing and someother tests will use our mongdb stub from the m01.stub package. You can runthe tests with the --all option if you like to run the full tests which willstart and stop the mongodb stub server.

NOTE:All mongo item interfaces will not provide ILocation or IContained but thebass mongo item implementations will implement Location which provides theILocation interface directly. This makes it simpler for permissiondeclaration in ZCML.

The MongoContainer can store non persistent IMongoContainerItem objects in aMongoDB. A MongoContainerItem must be able to dump it's data to valid mongovalues. This test will show how our MongoContainer works.

Condition---------

Befor we start testing, check if our thread local cache is empty or if we haveleft over some junk from previous tests:

We can't access another item from the same type from another parent container:

>>> pprint(LOCAL.__dict__) {}

>>> eu = root['europe']

>>> transaction.commit()

>>> pprint(LOCAL.__dict__) {}

>>> eu['cs'] Traceback (most recent call last): ... KeyError: 'cs'

>>> transaction.commit()

As you can see the KeyError left items back in our thread local cache. We canuse our thread local cache cleanup event handler which is by default registeredas an EndRequestEvent subscriber for cleanup our thread local cache:

The MongoStorage can store non persistent IMongoStorageItem objects in aMongoDB. A MongoStorageItem must be able to dump it's data to valid mongovalues. This test will show how our MongoStorage works and also shows thelimitations.

Note, this test uses a fake MongoDB server setup. But this fake server is faraway from beeing complete. We will add more feature to this fake server if weneed them in other projects. See testing.py for more information.

Condition---------

Befor we start testing, check if our thread local cache is empty or if we havelet over some junk from previous tests:

The ObjectId is also use as our __name__ value. See the MongoContainer andMongoContainerItem implementation if you need to choose your own names:

>>> obj.__name__ u'...'

>>> obj.__name__ == unicode(obj._id) True

A mongo item also provides created and modified date attributes. If we initialize an object without a given created date, a new utc datetime instanceget used:

>>> obj.created datetime.datetime(2...)

>>> obj.modified is None True

A mongo storage item knows if a state get changed. This means we can find out if we should write the item back to the MongoDB. The MongoItem stores the statein a _m_changed value like persistent objects do in _p_changed. As you can seethe initial state is ```None``:

>>> obj._m_changed is None True

The MongoItem also has a version number which we increment each time we changethe item. By default this version is set as _version attribute and set bydefault to 0 (zero):

>>> obj._version 0

If we change a value in a MongoItem, the state get changed:

>>> obj.title = u'New Title' >>> obj._m_changed True

but the version get not imcremented. We only imcrement the version if we savethe item in MongoDB:

Now we can add a sample MongoStorageItem to our storage. Note we can only use theadd method which will return the new generated __name__. Using own names is notsupported by this implementation. As you can see the name is an MongoDB24 hex character string objectId representation.

Now get the mongo item version. This should be set to 1 (one) since we only added the object and didn't change since we added them:

>>> item._version 1

If we now commit the transaction, the version get increased by one:

>>> transaction.commit() >>> item._version 2

If you now load the mongo item from the MongoDB aain, you can see that thetitle get changed:

>>> item = storage[__name__] >>> item.title u'New Title'

And that the version get updated to 2:

>>> item._version 2

>>> transaction.commit()

Check our thread local cache before we leave this test:

>>> pprint(LOCAL.__dict__) {}

=====================Shared MongoContainer=====================

The MongoContainer can store non persistent IMongoContainerItem objects in aMongoDB. A MongoContainerItem must be able to dump it's data to valid mongovalues. This test will show how our MongoContainer works.

Condition---------

Befor we start testing, check if our thread local cache is empty or if we havelet over some junk from previous tests:

As you can see the KeyError left items back in our thread local cache. We canuse our thread local cache cleanup event handler which is by default registeredas an EndRequestEvent subscriber for cleanup our thread local cache:

A MongoObject can get stored independent from anything else in a MongoDB. SuchMongoObject can get used together with a field property calledMongoOjectProperty. The field property is responsible for set and get suchMongoObject to and from MongoDB. A persistent item which provides such a MongoObject within a MongoObjectProperty only has to provide an oid attributewith a unique value. You can use the m01.oid package for such a unique oidor implement an own pattern.

The MongoObject uses the __parent__._moid and the attribute (field) name asit's unique MongoDB key.

Note, this test uses a fake MongoDB server setup. But this fake server is faraway from beeing complete. We will add more feature to this fake server if weneed them in other projects. See testing.py for more information.

Condition---------

Befor we start testing, check if our thread local cache is empty or if we havelet over some junk from previous tests:

But after adding the mongo object to our content which uses aMongoObjectProperty, the mongo object get located and becomes the attributename as _field value. If the object didn't provide a __name__, the same valuewill also get applied for __name__:

>>> content.obj = obj >>> obj.__parent__ <content 42="">

>>> obj.__name__ u'obj'

>>> obj.__name__ u'obj'

After adding our mongo object, there should be a reference in our thread localcache:

After we commited to the MongoDB, the mongo object and our transaction datamanger reference should be gone in the thread local cache:

>>> from m01.mongo import LOCAL >>> pprint(LOCAL.__dict__) {}

As you can see, our collection contains 1000 items:

>>> storage = root['storage'] >>> len(storage) 1000

batching--------

Note, this method does not return items, it only returns the MongoDB data. Thisis what you should use. If this doesn't fit because you need a list of the realMongoItem this would be complicated beause we could have removed marked itemsin our LOCAL cache which the MongoDB doesn't know about.

Let's get the batch information:

>>> storage.getBatchData() (<...Cursor object at ...>, 1, 40, 1000)

As you an see, we've got a curser with mongo data, the start index, the totalamount of items and the page counter. Note, the first page starts at 1 (one)and not zero. Let's show another ample with different values:

But we can force to count the result based on limit and skip arguments by useTrue as argument:

>>> cursor.count(True) 3

As you can see batching or any other object lookup will left items back in ourthread local cache. We can use our thread local cache cleanup event handlerwhich is normal registered as an EndRequestEvent subscriber:

An important part in batching is ordering. As you can see, we can limit the batch size and get a slice of data from a sequence. It is very important thatthe data get ordered at the MongoDB before we slice the data into a batch.Let's test if this works based on our ordable number value and a sort orderwhere lowest value comes first. Start with page=0:

The RENormalizer is able to normalize text and produce comparable output. Youcan setup the RENormalizer with a list of input, output expressions. This isusefull if you dump mongodb data which contains dates or other not so simple reproducable data. Such a dump result can get normalized before the unit testwill compare the output. Also see zope.testing.renormalizing for the samepattern which is useable as a doctest checker.

Since not every strategy is the best for every applications and we can'timplement all concepts in this package, but we will list here some imporvements.

values and items----------------

The MongoContainers and MongoStorage implementation will load all data withinthe values and items methods. Even if we already cached them in our threadlocal cache. Here is an optimized method which could get used if you need toload a larg set of data.

Note: I don't recommend to call keys, values or items for large collectionsat any time. Take a look at the batching concept we implemented. ThegetBatchData method is probably what you need to use with a large set of data.

AdvancedConverter-----------------

The class below shows an advanced implementation which is able to convert a nested data structure.

Normaly a converter can convert attribute values. If the attributevalue is a list of items which contains another list of items, then you need touse another converter which is able to convert this nested structure. Butnormaly this is the responsibility of the first level item to convert it'svalues. This is the reason why we didn't implement this concept by default.

- switch from Connection to MongoClient recommended since pymongo 2.4. Replaced safe with write concern options. By default pymongo will now use safe writes.

- use MongoClient as factory in MongoConnectionPool. We didn't rename the class MongoConnectionPool, we will keep them as is. We also don't rename the IMongoConnectionPool interface.

- replaced _m_safe_insert, _m_safe_update, _m_safe_remove with _m_insert_write_concern, _m_update_write_concern, _m_remove_write_concern. This new mapping base class options are an empty dict and can get replaced with the new write concern settings. The default empty dict will force to use the write concern defined in the connection.

0.9.0 (2012-12-10)------------------

- use m01.mongofake for fake mongodb, collection and friends

0.8.0 (2012-11-18)------------------

- bugfix: add missing security declaration for dump data

- switch to bson import

- reflect changes in test output based on pymongo 2.3

- remove p01.i18n package dependency

- improve, prevent mark items as changed for same values

- improve sort, support key or list as sortName and allow to skip sortOrder if sortName is given

- added MANIFEST.in file

0.7.0 (2012-05-22)------------------

- bugfix: FakeCollection.remove: use find to find documents

- preserve order by using SON for query filter and dump methods

- implemented m01.mongo.dictify which can recoursive replace all bson.son.SON with plain dict instances.

0.6.2 (2012-03-12)------------------

- bugfix: left out a method

0.6.1 (2012-03-12)------------------

- bugfix: return self in FakeMongoConnection __call__method. This let's an instance act similar then the original pymongo Connection class __init__ method.

- feature: Add `sort` parameter for FakeMongoConnection.find()

0.6.0 (2012-01-17)------------------

- bugfix: During a query, if a spec key is missing from the doc, the doc is always ignored.

- implemented _m_initialized as a marker for find out when we need to trace changed attributes

- implemented clear method in MongoListData and MongoItemsData which allows to remove sequence items at once wihout to pop each item from the sequence

- improve MongoObject implementation, implemented _field which stores the parent field name which the MongoObject is stored at. Also adjsut the MongoObjectProperty and support backward compatibility by apply the previous stored __name__ as _field if not given. This new _field and __name__ separation allos us to use explicit names e.g. the _id or custom names which we can use for traversing to a MongoObject via traverser or other container like implementations.

- Implemented __getattr__ in FakeCollection. This allows to get a sub collection like in pymongo which is a part of the gridfs concept.