Category: Mapping

Introduction

This blog is going to look at the mathematical calculations involved in rendering 2D GIS vector data, based on a particular area that we want to see on a map. I.e. converting vertices from map space into view space, and back.

For each vertex in a feature geometry, we need to perform a set of operations to calculate the screen position, based on our chosen location, scale, and rotation. Also, most applications will use top left as the origin rather than bottom left for map projections.

Normally, we would use client software such as OpenLayers, or application software such as GeoServer to make these calculations (and render the points, lines, or polygons) for us, however, occasionally it is more convenient to make the calculations required ourselves. E.g. generation of an image file on a server when the data is not available as a WMS through GeoServer.

At thinkWhere, we have implemented these calculations in a number of programming languages over the years as the languages we have used have changed. We have gone from C++, through C# and Actionscript (Flash), to Python and Javascript.

The Maths

To make the calculations we use Affine Transformations. For 2D data, these take the form of 3×3 Matrices which can be applied to a point location, producing a modified point.

We can create a number of transformations, each of which represents an action on the data. E.g translate, scale, or rotate the point.

These can be combined through matrix multiplication into a single transformation.

We can also calculate the inverse of a matrix, making it just as easy to perform the reverse set of actions.

The beauty of this approach is that we can apply a number of actions to each vertex in a small number of arithmetic operations making them quick to apply. Even for rotation, the sin/cosine values are calculated once, and the calculated matrix values used in simple multiplications and additions from then on. Once the completed matrix has been generated, the calculations required for each vertex are:

x' = (a * x) + (b * y) + c
y' = (d * x) + (e * y) + f

The Good News

The good news is that once we have created a class to perform the matrix arithmetic, we don’t have to think about it again. We can then create a sequence of actions which will transform our map coordinates to view coordinates.

Translate the data based on our chosen centre point – centring the data at 0,0

Scale the data based on our chosen view scale

Rotate the data around the centre if required

Flip the horizontal axis to switch the origin from bottom left to top left

Translate again to the centre of our view (based on the view size)

The example below shows this in action.

Application

This sample application has been written using an HTML5 Canvas to draw a GIS dataset from a GeoJSON file. It uses the turf library to assist in reading the GeoJSON files.

It uses an implementation of affine transformations to calculate the necessary transform to convert map coordinates in the GeoJSON file to screen coordinates. A standard class called CanvasRenderingContext2D draws the features. We have added additional members to make the drawing easier. This object already implements affine transformations. You can apply individual actions using the rotate(), scale(), and translate() methods, which alter the current transformation to allow you to build up a set of actions, or you can supply the six relevant parameters of the matrix in a single call to transform().

We have used the approach of calculating the matrix ourselves, in order to also produce the inverse transformation. This is then used to make the reverse calculation I.e. for a point clicked on the map, find the map coordinate that this relates to in order to then search for the clicked feature within the source data.

The code below uses the view control values to create the required transformation, and pass this to the CanvasRenderingContext2D

When a data file is loaded, the maximum extent of the features is calculated, and the controls are set up accordingly. When the controls are modified the transformation is adjusted, and the features redrawn.

Loops and more

Since it it quite simple to set up a set of actions passing in the various parameters, we can make use of loops to run though a variety of settings and start having some fun.

We get to work on some great projects here at thinkWhere, but we’re particularly proud of the project we’ve just started with the Humanitarian OpenStreetMap Team (HOT) to develop the Tasking Manager version 3.0 (TM3). It’s great to work in an industry where the work we do with maps can have such a tangible impact on the humanitarian effort. thinkWhere support this wherever we can by supporting MapAction by both supporting my own personal volunteering (which you can read more about here) and through fundraising efforts such the running the Stirling Marathon, which is open for sponsorship here so please donate if you can! Therefore being given the opportunity to also get involved with the HOT Community and deliver TM3 is something we’re extremely proud of.

The current HOT Tasking Manager (TM2) coordinates volunteer mapper contribution to OpenStreetMap, meaning the global network of HOT volunteers can map affected areas efficiently providing disaster responders on the ground such as MapAction, MSF and the Red Cross access to detailed mapping during the response.

Following a significant increase in the capture of OSM data through initiatives such as Missing Maps, and subsequent loads on existing servers and software, the development of TM3 aims to better meet the needs of mappers, validators and project managers. This will be achieved by taking advantage of the very latest advances and innovations in web development frameworks and methodologies.

“We are very excited to be working with thinkWhere to develop the next generation of HOT’s Tasking Manager application. The team at thinkWhere brings a wealth of geospatial development experience, talent and insight to the project. Used by thousands of people around the world, our Tasking Manager software is the key technology component that enables our humanitarian mapping work and having thinkWhere as partners ensures the development project will be a success and deliver a great result for our community”.

In early February 2017, we travelled to HOT’s office in Washington DC for the project initiation meeting with various project stakeholders to discuss initial requirements and wireframe designs.

This engagement with some of the largest users of the Tasking Manager to discuss functionality and solicit feedback on how features might be implemented was a great way to start the project. A long, but very productive day, the discussion involved representatives from Missing Maps, YouthMappers, TeachOSM, GeoCenter, the US State Department’s Humanitarian Information Unit, Mapbox, HOT Project Managers, American Red Cross and Ethan Nelson, the lead community development volunteer.

thinkWhere’s Chief Executive Alan Moore said…

“We’re really delighted and privileged to be working with the team at HOT. Redevelopment of the Tasking Manager will be key to the future growth and sustainability of the humanitarian mapping effort across the world. The development work flows naturally from the innovative work we’ve been doing recently on theMapCloud, our new spatial data platform, and we’re keen to bring those advantages to the benefit of HOT and the global mapping community”.

Here at thinkWhere we’ve recently been working with the automation tools in QGIS. The processing toolbox is a much underrated feature in QGIS, lacking the jazzy graphics of the ArcGIS equivalent it is, however, just as useful and functional from the point of view of doing multi-stage GIS analysis.

We recently had occasion to create a tool for a client that counted the number of features within a polygon and provide statistics for subsets of the features. Let’s try this again, for the sake of demonstration and talk about trees.

Here we have a series of suburbs arranged as polygons and trees as points. As you can see there are 5 types of tree in the dataset. A Spatialite or PostGIS database would make quick work of this, but you can also use the QGIS processing toolbox to iterate through this and count each aspect of the data.

Effectively from the point dataset we need to extract each of the attributes.

Extract by Attribute from Trees “Type = Beech” etc.

This creates effectively a subset of the input points dataset which can then be counted via the polygons.

We can then use the count points in polygon to count the number of points that sit inside each of the neighbourhoods.

Count Points in Polygon From Beech Subset of Input Points by Input Polygon, Create field “Beech”

Using standard QGIS this will create a new dataset each time you run an algorithm, but the great strength of the processing modeller is that these intermediate steps can run in the background as temporary layers, rather than confusing your users with multiple layers.

Of course what needs to happen now is that the next iteration (Birch) will need to be counted using the polygon dataset you created for the counts in the Beech dataset. If you think about it, the geography of that dataset is the same as the Input Polygon (the neighbourhoods). It also carries forward all the attributes for the layer as well. So for each iteration you are not using the original input polygons, but adding the new attribute information on for each count.

In the processing toolbox this looks like this:

When you run the model, you get a polygon shapefile with a new set of attributes showing the number of each of the tree types in the area. You can then generate graphs or use the data for other purposes.

This is only one of the many ways that the automation tools in QGIS can help you. It is also a really useful system for analysis. QGIS is hugely flexible and has a load of options for this. To learn more and see details of our QGIS training offerings please see our training page.

Aside from the work I do here at thinkWhere I also volunteer for the humanitarian mapping charity, MapAction. This year MapAction are fortunate to be part of the BBC Radio 4 appeal and have our message presented by Alexander Armstrong (Presenter of Pointless, amongst other things) which will air on Christmas Day at 7:54am and 6.56pm and on 29th December at 3:27pm. As part of the drive to raise awareness of MapAction, thinkWhere has posed some questions to me to find out more about what it’s like being a MapAction volunteer…

Can you tell us a bit about what MapAction does and how it helps during humanitarian crises?

MapAction is a humanitarian mapping charity that works through skilled volunteers. Its aim is to save lives and minimise suffering by making the response to humanitarian emergencies as efficient and effective as possible through the use of maps and spatial data.

When a disaster strikes, people’s lives can be destroyed within a matter of seconds. In a landscape transformed by the disaster, the challenge for responders is to know where to start.

MapAction deploys skilled volunteers to the scene of a disaster within hours of an alert. In the acute phase of a response, our mapped analysis helps coordinate search-and-rescue efforts and the delivery of emergency aid. As the situation on the ground evolves, it helps responders understand the changing needs of survivors.

As a long-term measure we also help to prepare government authorities, responders and communities in vulnerable countries for the impact of disasters before they strike.

Consequently over the last 11 years I’d heard lots about what it meant to be a MapAction volunteer, how much difference MapAction is able to make during an emergency and how rewarding being a volunteer was. MapAction as an organisation is also very well-known and respected within the wider GIS community.

Given I chose to follow a similar GIS career as my sister I was therefore also driven to become a volunteer, so eventually applied during a recruitment drive and was very pleased to be successful and able to join MapAction myself in 2014.

The MapAction team at a recent disaster simulation exercise

What kind of training do you get to prepare you for any missions you might get sent on?

MapAction volunteers are expected to participate in at least seven training weekends a year, keeping us up to date with the latest developments in the humanitarian and technical community.

Every year MapAction also runs an intensive simulation exercise, focusing on developing the skills we need to deliver our mapping service in the potentially challenging context of an emergency response.

Our training helps us make judgement calls about what sort of map products we should create given the time available and the volume of rapid requests coming in.

Recognising that the world is dealing with more conflict-related emergencies, MapAction has also boosted its security training so we can deploy safely to environments where the situation can be highly unpredictable.

Training involves simulations that enable us to ‘learn by doing’ in a practical and safe environment. This has included practising first aid, emergency communication procedures and negotiating with combatants. We hope none of this will ever be needed, but MapAction make a point of always preparing the team for all eventualities.

How do you balance your time between volunteering and working at thinkWhere?

MapAction is a big commitment. Aside from deploying to the field for emergencies we also provide remote support to emergencies when required, as well as actively training together as a team throughout the year, so the time commitment is significant.

Luckily I have a very understanding girlfriend (thanks Maddie!) and a very flexible employer in thinkWhere. thinkWhere are great at accommodating my MapAction training and volunteering around my work commitments, such as recently allowing me the time to deploy to Northern Iraq. Without this flexibility my volunteering with MapAction wouldn’t be possible.

Team thinkWhere raising money for MapAction by completing Total Warrior in 2015

Where in the world does your volunteering take you?

MapAction deploy to emergencies all around the world, a list of which can be found on the website here: maps.mapaction.org. MapAction doesn’t currently have any teams engaged in active responses, however since September we’ve responded to a number of high profile emergencies including the conflict related crisis in Nigeria affecting as many as 14 million people and Hurricane Matthew in Haiti, Jamaica and the Bahamas.

In November I also went on my first deployment with a fellow volunteer Naomi to Northern Iraq to map aid needs and distribution as a result of the conflict related crisis affecting the region.

QGIS training in Erbil, Iraq

What (from your own personal experience) is the most rewarding thing about volunteering for MapAction?

The most rewarding thing is being able to directly apply the skills I have from my work life into a completely different context i.e. the humanitarian world, in order to make a difference and help those in need. Meeting and working with the rest of the MapAction team who also share the same drive and commitment is also very rewarding.

Have you ever met the Royal Patron of MapAction, Prince Harry?

I haven’t personally met Prince Harry, although he did recently attend ‘Triplex’ in Norway which is a major international humanitarian field exercise organised by the International Humanitarian Partnership (IHP), where MapAction volunteers were participating. Prince Harry has an active interest and great knowledge of our work having been MapAction’s Royal Patron for the last 10 years since 2007. Maybe I’ll meet him one day!

How can others get involved with MapAction?

MapAction will be recruiting a new intake of volunteers in the New Year, so anyone with the right technical skills and an interest in joining our close-knit team should keep an eye on MapAction’s website and social media channels for more information about the skills and commitment required.

It’s certainly not for everyone and requires a flexible employer, like thinkWhere, to be willing to release you at short notice. But the benefits you gain in terms of honing your professional skills, making a difference and meeting like-minded people make it incredibly rewarding.

How can people help MapAction without becoming a volunteer?

The single easiest thing people can do is donate. MapAction doesn’t receive funding from large public appeals, so we rely on the generosity of our supporters. Your donation means we can continue to be responsive, agile and ready for action at any time.

MapAction’s approach has a powerful multiplier effect. Every pound you donate will influence the allocation of many thousands, if not millions, of pounds of humanitarian aid. And that means many more lives potentially saved.

MapAction is often referred to as the “best-kept secret” of the humanitarian world. The video featuring Prince Harry on the MapAction website aims to raise MapAction’s profile, so more people will get behind this unique organisation and help get more aid to more people.

How can we find out more?

Visit mapaction.org and watch the Prince Harry video which brings to life the pivotal role that MapAction plays in coordinating humanitarian aid.

The MapAction BBC Radio 4 appeal being presented by TV personality Alexander Armstrong goes out on Christmas Day at 7:54am and 6.56pm.

Recently I had the occasion to do a bit of viewshed analysis in QGIS. I have done these before using tools in ArcGIS, but this was the first time I had the pleasure of doing this kind of analysis in QGIS. I was impressed by the simplicity and flexibility of the toolset.

The views from our office at Stirling are fairly legendary, with Stirling Castle, much of the Ochil Hills and the Wallace Monument all visible from our windows. But how to quantify this – what can we see from our office? This is what a viewshed does – using a height model and a position on the map it will tell you what you can (theoretically) see.

As a starting point, we can use the Ordnance Survey’s 50m Digital Elevation Model: Terrain 50.

So now we have the heights of all the hills and glens of central Scotland represented in a raster.

The function we are going to use is the Grass r.viewshed function which, rather conveniently, does most of the heavy mathematical lifting for you.

The R.Viewshed function is a GRASS function which integrates nicely into QGIS allowing you to use the algorithm without delving into the GRASS interfaces.

So here we are using the Elevation Model called Terrain 50, the coordinates identifying the viewing position (thinkWhere’s office in Stirling).

We are on the first floor and without measuring the height of the building and our average employee standing up, we have estimated the height of the viewing point at 5m.

The viewshed analysis allows you to set up a height to offset the target elevations as well. This would be very useful for an analysis of windfarm visibility or for a radio mast.

A good way to think about it is like this:

In this example, the viewer is looking to see what part of the landscape they can see. The height of the person is the viewing position and there is no height offset for heights on the landscape.

In this example, the viewing position is on the top of a hill and looking out at objects of a defined height.

So we can run this algorithm in QGIS to compute the theoretical view from our office.

The output looks a bit like this. Now it needs a little bit of processing to get it to something a bit more meaningful.

First of all, we’ll make this a vector for better symbology options. We can use the GDAL “Polygonize” function for this. The way that QGIS integrates these different applications into one place makes life so much easier for the GIS analyst.

Bringing the QGIS symbologies into play, we can show the viewshed like this:

However, we can have more fun than that.

Projecting a hillshade on to the Digital Elevation Model and bringing in some Ordnance Survey Vector data allows us to produce something far more meaningful.

Using the hillshade analysis (that is the way to create the really funky 3d effect on the landscape) and bringing in some of the Vector Map District from the Ordnance Survey you can bring this map alive. Using only a very small amount of the data and concentrating on the physical features a viewshed analysis can be easily understood.

Using the blend mode stops the viewshed from obscuring the underlying data in the hillshade raster, making the information easy to understand. A few labels from the VMD names dataset helps give a little bit of context.

Viewsheds are very heavily used in renewables, telecommunication, and planning. QGIS, along with Ordnance Survey Open Data allows this to be done by anyone with access to a computer.

We meet many different people trying to do different things here at thinkWhere and one of our attendees at our QGIS Skills of Analysis & Statistics was particularly interested in looking at place names. Place names are a really interesting way of telling what kind of history a place has had. The Island of Jura is a great example of this deriving from an old Norse word meaning Animal Island. Common Scottish names such as Tarbert and Ben Mor are Gaelic derived place names translating as a narrow strip of land or a big hill respectively.

While we often wonder about the originality of our forbears in how they choose to name things, these are the footprints of history in maps we use every day. They teach us about where we come from and how the peoples who come before us lived with their landscape. It’s very romantic and also a terrific thing for some GIS analysis.

QGIS has some nice ways of helping with this kind of thing, alongside OpenStreetMap which is a rich source of place name data. The Ordnance Survey Open Names dataset would also work well for this kind of analysis.

Duns & Dums

A common word found in Scottish place names is “Dun” meaning a hill fort or roundhouse in the Celtic tradition. This is sometimes rendered as “Dun” (as in Dundee or Dunfermline) or as “Dum” (As in Dumfries or Dumbarton). Less common in Scotland, although still prevalent in Ireland is the prefix “Don” (As in Donegal), however this gets confused in Scotland due to the River Don. Don in Scots Gaelic is more associated geographically with the Celtic river goddess.

In QGIS we can use the expression:

“name” ILIKE ‘%dun%’ OR “name” ILIKE ‘%dum%’

To select or filter for any words that contain those strings of letters.

The ILIKE function (a case-insensitive LIKE) is a great way of doing these kind of analyses and you can use the “%” wildcard to make selections in almost any context.

With a spot of good symbology and the use of the Print Composer to show multiple maps on the same page. You can show the distribution of a few different place name components. Having consistent maps next to each other on a page like this helps to show distribution.

A really effective way of showing this kind of data can be to use the heatmap. The last few versions of QGIS have put this into the standard symbology tool which allows you to generate them on the fly without the need to create a whole new raster. This makes it much easier. This is what the distribution of “dun”/”dum” components looks like.

The big problem with this kind of analysis is reliance on the data. The OpenStreetMap relies on users adding in data themselves and that makes an analysis of this nature a bit shaky with crowd sourced data.

The OS Open Names dataset would be a great way of doing that, but it is a huge dataset that would need a database set up for it. We do cover database creation in our advanced QGIS course, but it would be a good follow up.