Database migrations are a popular pattern, particularly with Ruby on Rails. Since migrations specify how to mold old data to fit a new schema, they can be helpful when you have production data that must be converted quickly and reliably.

But migrating models in App Engine is difficult since processing all entities sequentially is difficult, and there is no offline operation to migrate everything effectively in one big transaction.

What are your techniques for modifying a db.Model "schema" and migrating the data to fit the new schema?

MigratingModel intercepts the conversion from the raw datastore entity to the full db.Model instance. If current_migration_version has fallen behind the class's latest migration_version, then it runs a series of migrate_N() methods which do the heavy lifting.

On a busy site, the migrate() method should retry if db.put() fails, and possibly log a critical error if the migration didn't work.

I haven't gotten there yet, but at some point I would probably mix-in my migrations from a separate file.

Final thoughts

It is hard to test on App Engine. It's hard to get access to your production data in a test environment, and at this time it is difficult-to-impossible to make a coherent snapshot backup. Therefore, for major changes, consider making a new version that uses a completely different model name which imports from the old model and migrates as it needs. (For example, User2 instead of User). That way, if you need to fall back to the previous version, you have an effective backup of the data.