I was recently in the situation of wanting to transition traffic gradually from an old deployment to a new deployment. It’s a large production system, so rather than just switching the DNS entries to point at the new deployment, I wanted to be able to shift the traffic over in a couple of controlled steps.

It turns out, Apache’s mod_proxy makes this relatively straight forward. You can choose which resource for which you want to move traffic, and easily update the proportion of traffic for that resource which should go through to the new env. Might be old news to some, but not having needed this before, I was quite impressed by Apache2’s configurability:

# Pass any requests for specific-url through to the balancer (defined below)
# to transition traffic from the old to new system.
ProxyPass /myapp/specific-url/ balancer://transition-traffic/myapp/specific-url/
ProxyPassReverse /myapp/specific-url/ balancer://transition-traffic/myapp/specific-url/
# Send all other requests straight to the backend for the old system.
ProxyPass /myapp/ http://old.backend.ip:1234/myapp/
ProxyPassReverse /myapp/ http://backend.ip:1234/myapp/
# Send 50% of the traffic to the old backend, and divide the rest between the
# two new frontends.
<Proxy balancer://transition-traffic>
BalancerMember http://old.backend.ip:1234 timeout=60 loadfactor=2
BalancerMember http://new.frontend1.ip:80 timeout=60 loadfactor=1
BalancerMember http://new.frontend2.ip:80 timeout=60 loadfactor=1
ProxySet lbmethod=byrequests
</Proxy>

Once the stats verify that the new env isn’t hitting any firewall or load issue, the loadfactor can be updated (only need to graceful apache) to ramp up traffic so that everything is hitting the new env. Of course, it adds one extra hop for serving requests, but it’s then much safer to switch the DNS entries when you *know* your new system is already handling the production traffic.

As a first project, I want to see if it’s possible to get Kodi (formally XBMC) packaged as a snap. I’m not yet sure if this is possible, but the plan is to install Ubuntu on a separate microSD and setup deb2snap and see whether I can create a snap (minus certain dependencies). Obviously then install the snap on my snappy microSD and profit! (The devel’s in the details of course.)

I’m not sure that deb2snap will work with Ubuntu Trusty (the default image for RPi2), as it was recently announced that XMir will no longer be supported for anything less than Wily. We’ll see how it goes.

I already found and reported my first issue while testing snappy on a VM last week, which got fixed for todays stable release. Let the fun begin…

After recently writing the re-usable wsgi-app role for juju charms, I wanted to see how easy it would be to apply this to an existing service. I chose the existing ubuntu-reviews service because it’s open-source and something that we’ll probably need to deploy with juju soon (albeit only the functionality for reviewing newer click applications).

I’ve tried to include the workflow below that I used to create the charm including the detailed steps and some mistakes, in case it’s useful for others. If you’re after more of an overview about using reusable ansible roles in your juju charms, checkout these slides.

Although when deploying this service we’d be deploying a built asset published somewhere else, for development it’s easier to work with a local file in the charm – and the reusable wsgi-app role allows you to switch between those two options. So the last step here is to add some Makefile targets to enable pulling in the application code and creating the tarball (you can see the details on the git commit for this step).

With that done, I can deploy the charm with `make deploy` – expecting it to fail because the wsgi object doesn’t yet exist.

2. Adding the wsgi, settings and urls

The current rnr-server code does have a django project setup, but it’s specific to the current setup (requiring a 3rd party library for settings, and serves the non-click reviews service too). I’d like to simplify that here, so I’m bundling a project configuration in the charm. First the standard (django-generated) manage.py, wsgi.py and near-default settings.py (which you can see in the actual commit).

An extra task is added which copies the project configuration into place during install/upgrade, and both the wsgi application location and the required PYTHONPATH are specified:

Rinse and repeat. You may want to occasionally run upgrade-charm in a separate terminal:

$ juju upgrade-charm --repository=../.. ubuntu-reviews

to verify that you’ve not broken things, running the hooks as they are triggered in your debug-hooks window.

Other times you might want to destroy the environment to redeploy from scratch.

Sometimes you’ll have dependencies which are not available in the distro. For our deployments,
we always ensure we have these included in our tarball (in some form). In the case of the reviews server, there was an existing script which pulls in a bunch of extra branches, so I’ve updated to use that in the Makefile target that builds the tarball. But there was another dependency, south, which wasn’t included by that task, so for simplicity here, I’m going to install that via pip (which you don’t want to do in reality – you’d update your build process instead).

I’ve separated the syncdb out to a manual step (rather than running syncdb on a hook like config-changed) so that I can run it after the postgresql service had been updated with the custom field, and also because juju doesn’t currently have a concept of a leader (yes, syncdb could be run on every unit, and might be safe, but I don’t like it conceptually).

To work around the second and third points above, I updated to remove the ‘south’ django application so that syncdb will setup the correct current tables without using migrations (while leaving it on the pythonpath as some code imports it currently), and added the old non-click reviews app to satisfy some of the imported dependencies.

By far the most complicated part of this change though, was the database settings:

Why are the database settings so complicated? I’m currently rendering all the settings on any config change as well as database relation change which means I need to use the global juju state for the relation data, and as far as juju knows, there could be multiple database relations for this stack. The current charm helpers gives you this as a dict (above it’s ‘relations’). The ‘db’ key of that dict contains all the database relations, so I’m grabbing the first database relation and taking the details from the postgres side of that relation (there are two keys for the relation, one for postgres, the other for the connecting service). Yuk (see below for a better solution).

Full details are in the commit. With those changes I can now use the service:

A. Cleanup: Simplify database settings

In retrospect, the complicated database settings are only needed because the settings are rendered on any config change, not just when the database relation changes, so I’ll clean this up in the next step by moving the database settings to a separate file which is only written on the db-relation-changed hook where we can use the more convenient current_relation dict (given that I know this service only has one database relationship).

Summary

Reusing the wsgi-app ansible role allowed me to focus just on the things that make this service different from other wsgi-app services:

Creating the tarball for deployment (not normally part of the charm, but useful for developing the charm)

Handling the application settings

Handling the database relation and custom setup

The wsgi-app role already provides the functionality to deploy and upgrade code tarballs via a config set (ie. `juju set ubuntu-reviews build_label=r156`) from a configured url. It provides the functionality for the complete directory layout, user and group setup, log rotation etc. All of that comes for free.

Things I’d do if I was doing this charm for a real deployment now:

Use the postgres db-admin relation for syncdb/migrations (thanks Simon) and ensure that the normal app uses the non-admin settings (and can’t change the schema etc.) This functionality could be factored out into another reusable role.

Remove the dependency on the older non-click reviews app so that the clickreviews service can be deployed in isolation.

Ensure all dependencies are either available in the distro, or included in the tarball

After recently writing the re-usable wsgi-app role for juju charms, I wanted to see how easy it would be to apply this to an existing service. I chose the existing ubuntu-reviews service because it’s open-source and something that we’ll probably need to deploy with juju soon (albeit only the functionality for reviewing newer click applications).

I’ve tried to include the workflow below that I used to create the charm including the detailed steps and some mistakes, in case it’s useful for others. If you’re after more of an overview about using reusable ansible roles in your juju charms, checkout these slides.

Although when deploying this service we’d be deploying a built asset published somewhere else, for development it’s easier to work with a local file in the charm – and the reusable wsgi-app role allows you to switch between those two options. So the last step here is to add some Makefile targets to enable pulling in the application code and creating the tarball (you can see the details on the git commit for this step).

With that done, I can deploy the charm with `make deploy` – expecting it to fail because the wsgi object doesn’t yet exist.

2. Adding the wsgi, settings and urls

The current rnr-server code does have a django project setup, but it’s specific to the current setup (requiring a 3rd party library for settings, and serves the non-click reviews service too). I’d like to simplify that here, so I’m bundling a project configuration in the charm. First the standard (django-generated) manage.py, wsgi.py and near-default settings.py (which you can see in the actual commit).

An extra task is added which copies the project configuration into place during install/upgrade, and both the wsgi application location and the required PYTHONPATH are specified:

Rinse and repeat. You may want to occasionally run upgrade-charm in a separate terminal:

$ juju upgrade-charm --repository=../.. ubuntu-reviews

to verify that you’ve not broken things, running the hooks as they are triggered in your debug-hooks window.

Other times you might want to destroy the environment to redeploy from scratch.

Sometimes you’ll have dependencies which are not available in the distro. For our deployments,
we always ensure we have these included in our tarball (in some form). In the case of the reviews server, there was an existing script which pulls in a bunch of extra branches, so I’ve updated to use that in the Makefile target that builds the tarball. But there was another dependency, south, which wasn’t included by that task, so for simplicity here, I’m going to install that via pip (which you don’t want to do in reality – you’d update your build process instead).

I’ve separated the syncdb out to a manual step (rather than running syncdb on a hook like config-changed) so that I can run it after the postgresql service had been updated with the custom field, and also because juju doesn’t currently have a concept of a leader (yes, syncdb could be run on every unit, and might be safe, but I don’t like it conceptually).

To work around the second and third points above, I updated to remove the ‘south’ django application so that syncdb will setup the correct current tables without using migrations (while leaving it on the pythonpath as some code imports it currently), and added the old non-click reviews app to satisfy some of the imported dependencies.

By far the most complicated part of this change though, was the database settings:

Why are the database settings so complicated? I’m currently rendering all the settings on any config change as well as database relation change which means I need to use the global juju state for the relation data, and as far as juju knows, there could be multiple database relations for this stack. The current charm helpers gives you this as a dict (above it’s ‘relations’). The ‘db’ key of that dict contains all the database relations, so I’m grabbing the first database relation and taking the details from the postgres side of that relation (there are two keys for the relation, one for postgres, the other for the connecting service). Yuk (see below for a better solution).

Full details are in the commit. With those changes I can now use the service:

A. Cleanup: Simplify database settings

In retrospect, the complicated database settings are only needed because the settings are rendered on any config change, not just when the database relation changes, so I’ll clean this up in the next step by moving the database settings to a separate file which is only written on the db-relation-changed hook where we can use the more convenient current_relation dict (given that I know this service only has one database relationship).

Summary

Reusing the wsgi-app ansible role allowed me to focus just on the things that make this service different from other wsgi-app services:

Creating the tarball for deployment (not normally part of the charm, but useful for developing the charm)

Handling the application settings

Handling the database relation and custom setup

The wsgi-app role already provides the functionality to deploy and upgrade code tarballs via a config set (ie. `juju set ubuntu-reviews build_label=r156`) from a configured url. It provides the functionality for the complete directory layout, user and group setup, log rotation etc. All of that comes for free.

Things I’d do if I was doing this charm for a real deployment now:

Use the postgres db-admin relation for syncdb/migrations (thanks Simon) and ensure that the normal app uses the non-admin settings (and can’t change the schema etc.) This functionality could be factored out into another reusable role.

Remove the dependency on the older non-click reviews app so that the clickreviews service can be deployed in isolation.

Ensure all dependencies are either available in the distro, or included in the tarball

I’ve been writing Juju charms to automate the deployment of a few different services at work, which all happen to be wsgi applications… some Django apps, others with other frameworks. I’ve been using the ansible support for writing charms which makes charm authoring simpler, but even then, essentially each wsgi service charm needs to do the same thing:

Setup specific users

Install the built code into a specific location

Install any package dependencies

Relate to a backend (could be postgresql, could be elasticsearch)

Render the settings

Setup a wsgi (gunicorn) service (via a subordinate charm)

Setup log rotation

Support updating to a new codebase without upgrading the charm

Support rolling updates a new codebase

Only three of the above really change slightly from service to service: which package dependencies are required, the rendering of the settings and the backend relation (which usually just causes the settings to be rerendered anyway).

After trying (and failing) to create a nice reusable wsgi-app charm, I’ve switched to utilise ansible’s built-in support for reusable roles and created a charm-bootstrap-wsgi repo on github, which demonstrates all of the above out of the box (the README has an example rolling upgrade). The charm’s playbook is very simple, just reusing the wsgi-app role:

Everything else is provided by the reusable wsgi-app role. For the moment I’ve got the source of the reusable roles in a separate github repo, but I’d like to get these into the charm-helpers project itself eventually. Of course there will be cases where the service charm may need to do quite a bit more custom functionality, but if we can encapsulate and reuse as much as possible, it’s a win for all of us.

If you’re interested in chatting about easier charms with ansible (or any issues you can see), we’ll be getting together for a hangout tomorrow (Jun 11, 2014 at 1400UTC).

I’ve been writing Juju charms to automate the deployment of a few different services at work, which all happen to be wsgi applications… some Django apps, others with other frameworks. I’ve been using the ansible support for writing charms which makes charm authoring simpler, but even then, essentially each wsgi service charm needs to do the same thing:

Setup specific users

Install the built code into a specific location

Install any package dependencies

Relate to a backend (could be postgresql, could be elasticsearch)

Render the settings

Setup a wsgi (gunicorn) service (via a subordinate charm)

Setup log rotation

Support updating to a new codebase without upgrading the charm

Support rolling updates a new codebase

Only three of the above really change slightly from service to service: which package dependencies are required, the rendering of the settings and the backend relation (which usually just causes the settings to be rerendered anyway).

After trying (and failing) to create a nice reusable wsgi-app charm, I’ve switched to utilise ansible’s built-in support for reusable roles and created a charm-bootstrap-wsgi repo on github, which demonstrates all of the above out of the box (the README has an example rolling upgrade). The charm’s playbook is very simple, just reusing the wsgi-app role:

Everything else is provided by the reusable wsgi-app role. For the moment I’ve got the source of the reusable roles in a separate github repo, but I’d like to get these into the charm-helpers project itself eventually. Of course there will be cases where the service charm may need to do quite a bit more custom functionality, but if we can encapsulate and reuse as much as possible, it’s a win for all of us.

If you’re interested in chatting about easier charms with ansible (or any issues you can see), we’ll be getting together for a hangout tomorrow (Jun 11, 2014 at 1400UTC).

Just documenting for later (and for a friend and colleague who needs it now) – my notes for setting up openstack swift using juju. I need to go back and check whether keystone is required – I initially had issue with the test auth so switched to keystone.

First, create the config file to use keystone, local block-devices on the swift storage units (ie. no need to mount storage), and using openstack havana:

Once everything is up and running, create a tenant and user with the user having admin rights for the tenant (using your keystone unit’s IP address for keystone-ip). Note, below I’m using the names of tenant, user and role – which works with keystone 0.3.2, but apparently earlier versions require you to use the uuids instead. Check with `keystone help user-role-add`).

Just documenting for later (and for a friend and colleague who needs it now) – my notes for setting up openstack swift using juju. I need to go back and check whether keystone is required – I initially had issue with the test auth so switched to keystone.

First, create the config file to use keystone, local block-devices on the swift storage units (ie. no need to mount storage), and using openstack havana:

Once everything is up and running, create a tenant and user with the user having admin rights for the tenant (using your keystone unit’s IP address for keystone-ip). Note, below I’m using the names of tenant, user and role – which works with keystone 0.3.2, but apparently earlier versions require you to use the uuids instead. Check with `keystone help user-role-add`).

I’ve been working on some more support for ansible in the juju charm-helpers package recently [1], which has effectively transformed my juju charm’s hooks.py to something like:

# Create the hooks helper, passing a list of hooks which will be
# handled by default by running all sections of the playbook
# tagged with the hook name.
hooks = charmhelpers.contrib.ansible.AnsibleHooks(
playbook_path='playbooks/site.yaml',
default_hooks=['start', 'stop', 'config-changed',
'solr-relation-changed'])
@hooks.hook()
def install():
charmhelpers.contrib.ansible.install_ansible_support(from_ppa=True)

And that’s it.

If I need something done outside of ansible, like in the install hook above, I can write a simple hook with the non-ansible setup (in this case, installing ansible), but the decorator will still ensure all the sections of the playbook tagged by the hook-name (in this case, ‘install’) are applied once the custom hook function finishes. All the other hooks (start, stop, config-changed and solr-relation-changed) are registered so that ansible will run the tagged sections automatically on those hooks.

Why am I excited about this? Because it means that practically everything related to ensuring the state of the machine is now handled by ansibles yaml declarations (and I trust those to do what I declare). Of coures those playbooks could themselves get quite large and hard to maintain, but ansible has plenty of ways to break up declarations into includes and roles.

It also means that I need to write and maintain fewer unit-tests – in the above example I need to ensure that when the install() hook is called that ansible is installed, but that’s about it. I no longer need to unit-test the code which creates directories and users, ensures permissions etc., or even calls out to relevant charm-helper functions, as it’s all instead declared as part of the machine state. That said, I’m still just as dependent on integration testing to ensure the started state of the machine is what I need.

I’m pretty sure that ansible + juju has even more possibilities for being able to create extensible charms with plugins (using roles), rather than forcing too much into the charms config.yaml, and other benefits… looking forward to trying it out!

[1] The merge proposal still needs to be reviewed, possibly updated and landed :)

I’ve been working on some more support for ansible in the juju charm-helpers package recently [1], which has effectively transformed my juju charm’s hooks.py to something like:

# Create the hooks helper, passing a list of hooks which will be
# handled by default by running all sections of the playbook
# tagged with the hook name.
hooks = charmhelpers.contrib.ansible.AnsibleHooks(
playbook_path='playbooks/site.yaml',
default_hooks=['start', 'stop', 'config-changed',
'solr-relation-changed'])
@hooks.hook()
def install():
charmhelpers.contrib.ansible.install_ansible_support(from_ppa=True)

And that’s it.

If I need something done outside of ansible, like in the install hook above, I can write a simple hook with the non-ansible setup (in this case, installing ansible), but the decorator will still ensure all the sections of the playbook tagged by the hook-name (in this case, ‘install’) are applied once the custom hook function finishes. All the other hooks (start, stop, config-changed and solr-relation-changed) are registered so that ansible will run the tagged sections automatically on those hooks.

Why am I excited about this? Because it means that practically everything related to ensuring the state of the machine is now handled by ansibles yaml declarations (and I trust those to do what I declare). Of coures those playbooks could themselves get quite large and hard to maintain, but ansible has plenty of ways to break up declarations into includes and roles.

It also means that I need to write and maintain fewer unit-tests – in the above example I need to ensure that when the install() hook is called that ansible is installed, but that’s about it. I no longer need to unit-test the code which creates directories and users, ensures permissions etc., or even calls out to relevant charm-helper functions, as it’s all instead declared as part of the machine state. That said, I’m still just as dependent on integration testing to ensure the started state of the machine is what I need.

I’m pretty sure that ansible + juju has even more possibilities for being able to create extensible charms with plugins (using roles), rather than forcing too much into the charms config.yaml, and other benefits… looking forward to trying it out!

[1] The merge proposal still needs to be reviewed, possibly updated and landed Read more

While writing charms for Juju a long time ago, one of the things that I struggled with was testing the hook code – specifically the install hook code where the machine state is set up (ie. packages installed, directories created with correct permissions, config files setup etc.) Often the test code would be fragile – at best you can patch some attributes of your module (like “code_location = ‘/srv/example.com/code’”) to a tmp dir and test the state correctly, but at worst you end up testing the behaviour of your code (ie. os.mkdir was called with the correct user/group etc.). Either way, it wasn’t fun to write and iterate those tests.

But support has improved over the past year with the charmhelpers library. And recently I landed a branch adding support for declaring saltstack states in yaml, like the above example. That means that the install hook of your hooks.py can be reduced to something like:

…letting you focus on testing and writing the actual hook functionality (like relation-set’s etc. I’d like to add some test helpers that will automatically check the syntax of the state yaml files and template rendering output, but haven’t yet).

Hopefully we can add similar support for puppet and Ansible soon too, so that the charmer can choose the tools they want to use to declare the local machine state.

A few other things that I found valuable while writing my charm:

Use a branch for charmhelpers – this way you can make improvements to the library as you develop and not be dependent on your changes landing straight away to deploy (thanks Sidnei – I think I just copied that idea from one of his charms). The easiest way that I found for that was to install the branch into mycharm/lib so that it’s included in both dev and when you deploy (with a small snippet in your hooks.py.

Make it easy to deploy your local charm from the branch… the easiest way I found was a link-test-juju-repo make target – I’m not sure what other people do here?

In terms of writing actual hook functionality (like relation-set events etc), I found the easiest way to develop the charm was to iterate within a debug-hook session. Something like:

write new test+code then juju upgrade-charm or add-relation

run the hook and if it fails…

fix and test right there within the debug-hook

put the code back into my actual charm branch and update the test

restore the system state in debug hook

then juju upgrade-charm again to ensure it works, if it fails, iterate from 3.

Use the built-in support of template rendering from saltstack for rendering any config files that you need.

I don’t think I’d really appreciated the beauty of what juju is doing until, after testing my charm locally together with a gunicorn charm and a solr backend, I then setup a config using juju-deployer to create a full stack with an Apache front-end, a cache load balancer for multiple squid caches, as well as a load balancer in front of potentially multiple instances of my charms wsgi app, then a back-end loadbalancer in between my app and the (multiple) solr backends… and it just works.

While writing charms for Juju a long time ago, one of the things that I struggled with was testing the hook code – specifically the install hook code where the machine state is set up (ie. packages installed, directories created with correct permissions, config files setup etc.) Often the test code would be fragile – at best you can patch some attributes of your module (like “code_location = ‘/srv/example.com/code'”) to a tmp dir and test the state correctly, but at worst you end up testing the behaviour of your code (ie. os.mkdir was called with the correct user/group etc.). Either way, it wasn’t fun to write and iterate those tests.

But support has improved over the past year with the charmhelpers library. And recently I landed a branch adding support for declaring saltstack states in yaml, like the above example. That means that the install hook of your hooks.py can be reduced to something like:

…letting you focus on testing and writing the actual hook functionality (like relation-set’s etc. I’d like to add some test helpers that will automatically check the syntax of the state yaml files and template rendering output, but haven’t yet).

Hopefully we can add similar support for puppet and Ansible soon too, so that the charmer can choose the tools they want to use to declare the local machine state.

A few other things that I found valuable while writing my charm:

Use a branch for charmhelpers – this way you can make improvements to the library as you develop and not be dependent on your changes landing straight away to deploy (thanks Sidnei – I think I just copied that idea from one of his charms). The easiest way that I found for that was to install the branch into mycharm/lib so that it’s included in both dev and when you deploy (with a small snippet in your hooks.py.

Make it easy to deploy your local charm from the branch… the easiest way I found was a link-test-juju-repo make target – I’m not sure what other people do here?

In terms of writing actual hook functionality (like relation-set events etc), I found the easiest way to develop the charm was to iterate within a debug-hook session. Something like:

write new test+code then juju upgrade-charm or add-relation

run the hook and if it fails…

fix and test right there within the debug-hook

put the code back into my actual charm branch and update the test

restore the system state in debug hook

then juju upgrade-charm again to ensure it works, if it fails, iterate from 3.

Use the built-in support of template rendering from saltstack for rendering any config files that you need.

I don’t think I’d really appreciated the beauty of what juju is doing until, after testing my charm locally together with a gunicorn charm and a solr backend, I then setup a config using juju-deployer to create a full stack with an Apache front-end, a cache load balancer for multiple squid caches, as well as a load balancer in front of potentially multiple instances of my charms wsgi app, then a back-end loadbalancer in between my app and the (multiple) solr backends… and it just works.

A number of times over the past few years I’ve needed to create some quite complex migrations (both schema and data) in a few of the Django apps that I help out with at Canonical. And like any TDD fanboy, I cry at the thought of deploying code that I’ve just tested by running it a few times with my own sample data (or writing code without first setting failing tests demoing the expected outcome).

class MigrationTestCase(TransactionTestCase):
"""A Test case for testing migrations."""
# These must be defined by subclasses.
start_migration = None
dest_migration = None
django_application = None
def setUp(self):
super(MigrationTestCase, self).setUp()
migrations = Migrations(self.django_application)
self.start_orm = migrations[self.start_migration].orm()
self.dest_orm = migrations[self.dest_migration].orm()
# Ensure the migration history is up-to-date with a fake migration.
# The other option would be to use the south setting for these tests
# so that the migrations are used to setup the test db.
call_command('migrate', self.django_application, fake=True,
verbosity=0)
# Then migrate back to the start migration.
call_command('migrate', self.django_application, self.start_migration,
verbosity=0)
def tearDown(self):
# Leave the db in the final state so that the test runner doesn't
# error when truncating the database.
call_command('migrate', self.django_application, verbosity=0)
def migrate_to_dest(self):
call_command('migrate', self.django_application, self.dest_migration,
verbosity=0)

It’s not perfect – schema tests in particular end up being quite complicated as you need to ensure you’re working with the correct orm model when creating your test data – and you can’t use the normal factories to create your test data. But it does enable you to write migration tests like:

which keeps me happy. When I wrote that I couldn’t find any other suggestions out there for testing migrations. A quick search now turns up one idea from André (data-migrations only), but nothing else substantial. Let me know if you’ve seen something similar or a way to improve testing of migrations.

A number of times over the past few years I’ve needed to create some quite complex migrations (both schema and data) in a few of the Django apps that I help out with at Canonical. And like any TDD fanboy, I cry at the thought of deploying code that I’ve just tested by running it a few times with my own sample data (or writing code without first setting failing tests demoing the expected outcome).

class MigrationTestCase(TransactionTestCase):
"""A Test case for testing migrations."""
# These must be defined by subclasses.
start_migration = None
dest_migration = None
django_application = None
def setUp(self):
super(MigrationTestCase, self).setUp()
migrations = Migrations(self.django_application)
self.start_orm = migrations[self.start_migration].orm()
self.dest_orm = migrations[self.dest_migration].orm()
# Ensure the migration history is up-to-date with a fake migration.
# The other option would be to use the south setting for these tests
# so that the migrations are used to setup the test db.
call_command('migrate', self.django_application, fake=True,
verbosity=0)
# Then migrate back to the start migration.
call_command('migrate', self.django_application, self.start_migration,
verbosity=0)
def tearDown(self):
# Leave the db in the final state so that the test runner doesn't
# error when truncating the database.
call_command('migrate', self.django_application, verbosity=0)
def migrate_to_dest(self):
call_command('migrate', self.django_application, self.dest_migration,
verbosity=0)

It’s not perfect – schema tests in particular end up being quite complicated as you need to ensure you’re working with the correct orm model when creating your test data – and you can’t use the normal factories to create your test data. But it does enable you to write migration tests like:

which keeps me happy. When I wrote that I couldn’t find any other suggestions out there for testing migrations. A quick search now turns up one idea from André (data-migrations only), but nothing else substantial. Let me know if you’ve seen something similar or a way to improve testing of migrations.

Android UI Fragments look like a great way to build re-usable elements within an app, but they don’t work exactly as expected out of the box (well, exactly is I expected – but that could be a lack of experience with android):

I’d expected this event to be routed directly to my fragment class without the containing Activity class needing to know about it – but instead, the addGoal() method is expected on the containing Activity instead.

To connect the fragment event directly to a click handler on the fragment class (so the view doesn’t need to be handling the fragment events, you can do the following instead (thanks Brill Pappin):

Android UI Fragments look like a great way to build re-usable elements within an app, but they don’t work exactly as expected out of the box (well, exactly is I expected – but that could be a lack of experience with android):

I’d expected this event to be routed directly to my fragment class without the containing Activity class needing to know about it – but instead, the addGoal() method is expected on the containing Activity instead.

To connect the fragment event directly to a click handler on the fragment class (so the view doesn’t need to be handling the fragment events, you can do the following instead (thanks Brill Pappin):

The last exercise in the Go Tour – parallelizing a web crawler – turned out to be quite a bit more interesting than I’d expected. If anyone has suggested improvements from which I can learn a bit more, or there own solutions posted, let me know – my exercise solution is on github. I’ve tried to stick to the tour content (ie. only using channels rather than the sync package for accessing shared data).

Spoiler Alert: If you are learning Golang and haven’t yet worked through the Go-Tour, go and do so now. If you get stuck, keep struggling, take a break, try again in a few days etc., before looking at other peoples’ solutions.

The solution I ended up with has a Crawl() function very similar to the original, just with two extra function parameters:

Access to the shared state of which urls have been crawled and when all Crawls() have finished etc., is managed via those channels in the processCrawls() go-routine, so that the main() can simply call the first Crawl() and then wait to quit. I want to check how cheap the temporary creation of a channel is (for the return value of the startCrawlFn above) – I think I saw this method on an earlier GoLang tutorial example, but otherwise I’m happy with the solution :-).