Creating a live, topic specific mirror of OpenStreetMap in PostGIS

Introduction

In this article, we will do a walk-through of creating a live mirror of OSM for a specific country or region and for a specific set of OSM features. For this example, we will fetch all building data for Angola into a PostGIS database, and update that database with new features as they arrive in OSM. Providing an OSM mirror in this way is a powerful tool to pair the power of OSM with the power of QGIS and PostGIS. Now you will be able to do offline analysis of the data in OSM. How does this all work? Here is a little diagram that illustrates the underlying architecture (click for a larger version):

Once the docker containers start running, the PostGIS database will be initialised, the country.pbf file imported using imposm and then osm-enrich will start running to update the username and timestamp for each feature. You can track its progress like this:

docker logs -f dockerosm_osmenrich

Which should show the update stream from OSMENRICH:

Update for 381148400Update for 381148402Update for 381148404Update for 381148406Update for 381148408Update for 381148410

After waiting a minute or two you should be able to establish a connection to the database from QGIS:

From where you can then drag and drop your OSM mirror layers into your project:

Depending on the size of your clip area / country area, the OSMENRICH process will take some time to run. Be kind to the OSM servers and don't use a larger clip area than you need as the OSMENRICH process will put some pressure on the OSM servers if you are running it from scratch too often. Once your mirror is initially updated from OSMENRICH, all of your records will have a user name and date associated. You can see the current status of OSMENRICH with this simple query in QGIS' dbmanager tool:

selectcount(*) as total,(select count(*) from osm_buildings where changeset_user is null) as to_updatefrom osm_buildings;

Which will tell you the total number of records in total and the number of records still due to be updated via OSMENRICH.

Making a time series visualisation

Now that you have the mirror running, it is possible to use QGIS to do desktop based analysis and visualisation on OSM data. In this example, we will create a view first that shows only records that have been updated by OSMENRICH and then use that view as the basis for our time manager layer. We do this because timemanager does not like it when we have a table with partially populated timestamps. Here is the SQL for the OSMENRICH populated view:

Test the query and then create a view from it, either by using a CREATE VIEW sql statement or by using the feature in QGIS' dbmanager that lets you quickly turn any query into a view as shown below.

Now refresh your browser panel using the (circled below) refresh icon and then drag the view into your project. Create a bold style for your view based layer so that the buildings will be easy to see when you animate the time series.

Next, use the timemanager plugin to add your view to an animation series as shown below. Make sure to check 'Accumulate features' option for your project so that the animation shows the accumulation of buildings over time.

I chose a time interval of 1 month but you may wish to adjust that depending on your needs. I should also mention that timemanager can be a bit fiddly sometimes - if things aren't working right (for example the 'play' button not showing), try restarting QGIS, it seems to help sometimes...

Now test your time series visualisation by scubbing the time slider left and right - you should see the progression of new buildings that have been added to the map.

I then output my series as an animated GIF (check this blog article for hints on creating videos), how to do it is beyond the scope of this article. If you know a bit of imagemagick, here are the options I used: