Display Big Data

Problem: Displaying Big Data On A Google Map

There are two key features of web mapping. The first is that the interface is resources constrained: it is a browser. The second is that generally you are expected to be able to drive as much data and functionality to that browser as you would expect on a desktop product. In fact, the user doesn’t even want to consider how difficult it is to achieve a smooth, immersive map-based experience, they just want to experience what the map is showing them.

This tutorial will show you a way of displaying a large number of polygons on a Google Map. We will avoid the use of very large caches in lieu of node.js. We will build on what we learned about building a US Census Tract Database in an earlier blog post.

This is what we are shooting for, its a Google map with the 70,000 or so US Census Tract polygons overlaid on top:

Points To Note

There is no cache, we will not be “optimizing” this system at all. A great deal can be done with node and node-mapnik but this example uses both of these technologies in the most crude of manners. So that means you can take this much further if you want to take the wheels off and have a proper look. additionally the styling is also somewhat bleak, Mapnik provides a vast array of styling options. This example stays very simple.

Installs

You will have to have an array of software available on your development machine to get this tutorial running. The list is written out above. My recommendation, if you are starting from scratch is to first look at PostGIS and then Mapnik. For this example we are using the census tract data provided by the US Census Bureau and assembled in this earlier Sparkgeo blog post

So shoot for this order:

PostGIS > Mapnik > Node > Node-Mapnik

the other pieces to this are javascript libraries which need to be available to the client. So that means you point at Google Maps in the usual documented manner, and you need to download the Wax libraries and point to them as you would any other javascript file.

I will certainly not be able to provide better information on installation than those provided by each of the excellent authors. I will however suggest that you try this on a linux based machine. Ideally ubuntu is a good bet. I have successfully done this on a mac and on CentOS (thanks to the support team at Webfaction). I have not attempted this in a windows environment.

Checks

Ok, so you have managed to install the above softwares. lets do a couple of checks…

This is simple but there are some critical parts. Firstly, don’t forget to reference Wax & Google Maps, nothing will work without them. Secondly, Yup you are right there are inline styles and scripts. Yuk! But its for demo purposes, ok ;). You will also notice there is only one tag inside the body. The div “map” fills the screen and is populated entirely by Google Maps. The two inline styles ensure that the screen is filled.

The inline javascript script is where the magic happens. There are four things that happen to observe. Firstly we set the variables, the starting location (start) and the starting zoom level(dmZoom). Then we build the tile request. This request basically defines what the client is expecting to see from our node application. If you are building this on a local machine local machine, your tile url might be:

http://localhost:8000/{z}/{x}/{y}.png

instead. You can find out more about this in the Wax docs. The third piece is the definition of the Google Map, which you can find out lots from the Google Maps docs. Finally we add the wax layer to the map.

The Back End

Our Node app will start its own little webserver, so you need to put it in a sensible place. If you are running local host it can really go anywhere, but if you are putting this i a public place, then you should consider this location a little more carefully. It will also have to speak to PostGIS, of course. The node app which I called cencon.js looks like this:

The script above basically gathers the dependencies, defines the database connection string then scopes out how to deal with each request while it listens on a specific port (8000, in this case). Its up to the client to ask for tiles and this node application will deliver. So the relationship between what wax is doing and what node-mapnik is doing is absolutely critical. Wax will define what tiles are required and request them from the node application. The node application will take each individual request and throw them at Mapnik and by extension PostGIS. Mapnik’s resulting image files will be sent back to the client and wax will overlay it on the Google Map and cache them for a short time in the client browser session.

The application code above refers to two external js files: sphericalmercator and tile. Both of these files have very conveniently been provided by Springmeyer in the Node-Mapnik bindings. Look at: sphericalmercator.js and Tile.js. The third dependancy is the tile_symbols.xml which defines the styling associated with the node-mapnik overlay:

I’ll reiterate once more that the styling here is very simple, a single feature type being styled in a single manner. This could go a great deal further!

Start Serving

We have our architecture in place and all that is left is to start the service:

[will]$ node cencon.js

That’s it, if you want this service to persist beyond the terminal session you can do:

[will]$ nohup node cencon.js &

Summary

This tutorial has focused on how to access datasets from postgis on Google Maps via mapnik and node. The benefit of this process is to be able to disseminate large amounts of data very quickly without any need for a dense and difficult to manage web cache of map images. There is definitely a question as to how this approach would scale to many users, but the flexibility of the node framework suggests that this could be accounted for.