Financial Charts with App Engine Tutorial Pt 3: Caching Data

In part 2 the chart app was successfully loaded with historical data using appcfg.py’s bulkloader tool and the remote_api module. However as queries over time series data quickly become expensive when we begin to deal with months and years, we should use some kind of caching mechanism to prevent too many datastore operations from slowing down our chart display.

App Engine’s powerful built-in object caching framework is memcache. With memcache we first check if the data we’re looking for is in memory and if so we will retrieve it from there, removing the need for any datastore querying. If not we will retrieve it using GQL as done in part 2 of the tutorial.

As our sample data, 3 Month Eonia, is updated once a day at a fixed time ideally we flush the cache just after the web scraper scheduled task is run and the updates are made to the datastore. Hence the datastore should in theory only be read once a day, making our chart app run virtually entirely in memory and speeding up the end user’s experience.

Finally a small tip: In GQL queries you cannot test properties for None explicitly, however for Integer and FloatProperty at least, you can test for <0 which will return the required None values.
Suggestions? Please share…
I’m a big fan of Google App Engine as a method of rapid application development. However, programming in the cloud brings new issues and concepts to think about over plain old client server stuff. Things like sharding and it’s consequences require design time thought in order to avoid problems later on in the development cycle. Shout if you see anything that should be discussed!