Tag Archives: geoJSON

Post navigation

So I spent most of yesterday trying to find what causes my work to crash Safari under iOS7: turns out it’s the gizmo that clusters the CC markers. Not going to cure that in a hurry!

Then I tried for several hours to add the info features from leaflet’s chloropleth/geoJSON example to my script. I got the LAs to highlight on mouseover, but not to de-highlight on mouseout. Nor could I get the data control to pick out data from the LA geoJSON file.

So I reasoned ‘if I can’t add their stuff to mine, can I add my stuff to theirs?’ That is, could I swap in my LA and CC data sets and use leaflet’s code to colour them and pick out data to be displayed in their external control? This seemed to go OK until I added in the geocoder (the bit for entering an address to zoom to that area of the map. This code failed, so everything that should have been processed after it wasn’t even reached.

I went back to trying to add leaflet’s functions to my otherwise functional code. Still no joy.

I refactored my code so it was in a more logical order:

preparatory functions,

drawing basic map

adding scale, geocoder, reset and help controls to map

adding LA data layer

adding CC data layer

adding layers on/off control

and made the reset control call a URL from a simple configuration file, so that when the client actually puts this work online, they only need to update the configuration file, not hack around in the reset script.

I still couldn’t get the mouseout bit to work. I knew this code was being called: if I replaced it with document.write(“rude word”); then rude words were written.

So this evening I revisited leaflet’s example, determined to get it to work. This example has all the functionality built into a script in the html file, not a separate ‘external’ script. Not really the way I want but I’m running out of time…

I realised that the geocoder was being called but just failing somewhere. I’m not sure how I worked it out but the fail point was that geocoder calls a function in my main external script to limit its searches to Scotland – the same bounds as are applied to my map. (Without this search-area limiting, searching for EH10 postcodes shows Walthamstow.) But this script is never invoked, so the function isn’t callable. So instead of calling that function, I’ve copied it into the geocoder. Now that works! And so do all the other bits. I can make the reset script call the configuration file, so long as they are in the right order in the html header (i.e. configuration before the reset function that depends on it – so perhaps the issue was that the geocoder was calling the main script before it was available.

There was another wrinkle adding in the CC marker code. Something doesn’t like a variable called location. Changing that to ccLocation worked.

Select each LA in turn, then do Vector > Geometry tools > Simplify geometries. The options I used were: but with a different file for each LA. For example, Shetland’s 160,883 vertices were reduced to 5416.

Select the newly-created object, then do Layer > Save selection as vector file, with options

So now I had 32 separate geoJSON files. To combine them, I used the process described here to make a single laBoundaryData.js file. That’s my geoJSON file on which the above styling magic works.

Finding LA extrema

In future versions of this map, I’ll want to zoom into individual LAs. The easy way to do this will be (I think) to find the furthest north, south, east, west points of each LA. So I adapted my code to find Scotland’s extrema to work on each LA:function drawMap() { //BMR 2014_02_25 //find extreme points var north = 0; var south = 90; var west = 90; var east = -90; for (var i = 0; i < coordinates.length; i++) { var laDatum = coordinates[i]; // the other way round from the getScotlandBounds because geoJSON latlongs are in the opposite order to leaflet latlogns var longitude = laDatum[0]; var latitude = laDatum[1];

It’s a complete hack but I copied, pasted and edited the results into my laData.js file to get line such as [“Angus”, “http://www.angus.gov.uk/commcouncil&#8221;, 56.986816427679120, 56.46164866362316, -3.407021822671358, -2.420365421269425],

If I’d been clever, I’d have made the script write the extrema data to the file. But this is a one-off and so it would have taken longer to write code to do it than to do it myself.

Awe and respect to the leaflet programming folk. I feared I might need to dig into their actual code to switch off these unwanted polygons. But it’s as simple as adding an option object to the otherwise unoptioned code. That is, from this

So a good conversation with Napier’s visualisation expert and I’m now more aware of some of the ways to make my code more bombproof – too be implemented this weekend, if paper-writing and having a life allow.

I’m not too bothered that this uses 6 colours when there should be a four-colour solution. I’d be more picky about the actual colours used. Changing the colours is easy – just changing up to 6 values in a function in my main script. Changing which LAs each colour is applied to involves opening the huge geoJSON file full of LA data, then finding, say, East Ayrshire, then changing the value of colour_code immediately below it.

Manually copying and pasting 32 sets of bits of file is fraught with difficulty, as any fule kno. So there had to be an easier way to assemble the individual LA geoJSON files into one javascript file. I did a lot of comparison of the brackets and guts of the working-so-far leaflet example and my geoJSON files. My file began with

I spent a while today trying to drop a marker at the user’s entered location. It’s easy to do on(zoomed) add marker but removing it at the next zoom is so far beyond me. Back to fighting with LA borders – the theatre of battle being styling them

I knew from leaflet’s example that leaflet does style geoJSON – so the issue must have been with my geoJSON files. When in doubt, copy: I copied leaflets’ code for colouring US state geoJSON data, substituted in the Scottish LA names and a piece to colour them by code rather than population density, and lo and behold it works.

Truth – it took a while to isolate the co-ordinate lines, so I’m only a wee way into substituting in the LA co-ordinates. But here’s the proof:

I was encouraged to try again to obtain up-to-date LA boundary data from the OS. This time I obtained a set of shape files. The relevant shape file was district_borough_unitary_region.shx. I opened that in QGIS, removed the non-Scottish areas and saved the result as Scotland.qgs. I then selected and saved each LA in turn as a geoJSON vector file (with CRS = WGS84/EPSG4326 as before). So this got me up-to-date accurate LA boundaries. But these are quite large files: averaging 2MB each. (Of course Highland was much bigger than Clackmannashire, for example.)

To obtain simplified files, I selected each LA in turn, then did Vector > Geoprocessing tools > Simplify (options: ‘use only selected features’, tolerance = 50, ‘save to new file’, ‘add to canvas’) so that I had a patchwork of separate LAs:

I then selected each simplified LA in turn and saved them as geoJSON files. The resulting files are about a twentieth of the size of the unsimplified files. Yet zooming right in shows an acceptable (to me) fit to the unsimplified boundaries and coastlines. Yeehah.

We have layers

And now the LA group is off by default, while the locater and layer controls are expanded by default:

So after ac couple of days of literature reviewing, and spending Saturday being social, I was back at the code-face today.

Task 1: understand – and maybe cure – the failing geocoder

Entering single-initial letter postcodes such as EH3 7PX works – the map zooms to the right place. Entering textual places such as Merchiston, Leith Links, Macmerry and Dalry also works. But EH10 5DT still appears to be in Walthamstow, instead of Merchiston. EH12 6AP appears to be slightly east of Walthamstow, instead of near Murrayfield.

I noticed that EH10 5DT and E10 5DT both give the same place (in east London), as do EH12 6AP and E12 6AP. So Nominatim is omitting the H in some cases. Interestingly, the geocoder works with Macmerry’s postcode (EH33 1PL).

To see whether it was my fault, I created a stripped-down version from scratch: the javascript contains just the map declaration, the code to fill it with tiles and the geocoder. Same result – so it’s the geocoder, not my code. I feel far less guilty but I’ll need to fix this.

Task 2: understand how leaflet.js handles geoJSON files

I worked my way through http://leafletjs.com/examples/geojson.html. I got most of it to work, except for styling the output. I then got a trial LA boundary to work – this trial had 4 co-ordinates. If it will work for four, it will work for many, many more – but will take longer.

Task 3: make geoJSON files of LA boundaries

With thanks to a Leaflet mailing list member for some sage advice, I’ve installed QGIS. This could open the 18-month old UK local authorities file I obtained from https://www.sharegeo.ac.uk/handle/10672/305. (It couldn’t handle the .sld files I obtained from the Ordnance Survey.) The way I worked – there may well be better ways – was to select an LA, then do Layer > Save selection as a vector file… with options

format = geoJSON

CRS = WGS84/EPSG4326

It took a while to work out the correct CRS option. I was surprised it wasn’t WGS84/PseudoMercator/EPSG3857 because that appears to be Leaflet’s default CRD. Then again, what do I know about converting points on am irregular 3D surface to points on a flat plane?

Anyway, that gave me 32 nice geoJSON files in a little less than 2 hours.

Task 4: display LA boundaries

Using my stripped-down version of the code, I added in 32 routines to display the LA boundaries. They were still in the default blue – remember I’d not cracked styling them. There may be an advantage to separate routines and data files. If an LA ever changes, I’d just need to make new geoJSON files for that LA and its neighbours, and each can be coloured (if this ever works!) separately. Anyway, by adding var = <LA_name> [ to the beginning of each file and ]; to the end of each, then appropriately commenting off all routines except for the LA I was testing, I made sure each displayed OK.

I then tried a few methods to programatically assemble the files into one huge array of the form var laBoundaryData = [ [content of first geojson file], [content of second geojson file], ….[content of last geojson file] ]; Another hour down the pan and no joy, so I did it the hard way, copying and pasting the contents of the geoJSON files intovar laBoundaryData = [],[], [
]
];

It almost worked straight off – all the LAs apart from Shetland and Scottish Borders displayed as they should. Shetland and Scottish Borders had some weird horizontal lines, but removing them from the huge file, then replacing their code from their individual geoJSON files got rid of these nasty artefacts. I guess I had pasted the code in slightly the wrong place first time round.

Then the code for displaying the LAs is just a simple traversal of the array of LA boundary data. And all this without going anywhere near the command line.

So with my map displaying all LA boundaries, it was time for…

Task 5: putting it all back together

I’d thought this would be easy – just replace the code lumps for calculating the map centre, its bounds and displaying the CCs. Not a bit of it! Failure points included not calling the data, scripts and even the css in the right order, the functions for calculating centres and bounds not being called properly – aarrgh. Anyway after a couple of hours, it’s all there. It’s slow (I guess need to simplify the LA shapes before writing them to geoJSON format), the geocoder is still a mystery but it works. Yeehah!

The initial result

zoomed into the West End of Edinburgh – see the markers for individual CCs

My reward for all that – 2 more hours doing stuff to a real CC’s website!

I couldn’t leave it alone this evening – 5 hours after I got home and I’ve done the 6 LAs on Scotland’s mid-east coast. The wee bit that was missing from Aberdeen has been restored. (I processed that dataset again and the missing points magically reappeared. So here it is in glorious technicolour:

The points are very close to the LA boundaries on OpenStreetMap tiles, which is quite encouraging. The dashed line is OpenStreetMap and the solid line is Leaflet’s rendering of the converted data.

Fife is a 4-part multipolygon, that is there are four polygons in the bits that represent Fife. I guess these are the Isle of May, Inchkeith and Inchcolm. Each part of a multi polygon requires separate processing via the hero of the hour(s), http://gridreferencefinder.com/batchConvert/batchConvert.php.

So I’m dreading doing Shetland, Orkney and the Western Isles. But perhaps they don’t need to be marked or coloured because they are clearly separate from the mainland.

Less heroic have been Mac TextEdit (no RegEx), TextMate (one successful RegEx, then fell over and won’t stop trying to process a long finished-with Angus file) and Word (it has done most of the heavy kiting but has fallen over at least 10 times). I’ll forgive Eclipse for chugging away with RegExes very slowly but I won’t forgive it for completing the RegEx then hanging so I can’t save the result.

Get rid of the dross at the start and end of the of the file, so that I’m left with just sets of [number,number],<new line>[number,number], …[number,number],<new line>[number,number].

Save.

Manually copy and paste the first set into the heroic web page, and tell it to convert. If it moans, there is more than one set in what I’ve copied, so go back and sort it out. At this point, Word will fall over – hence the multiple saves,

When a clean data set is in the hero, it will convert the data to lines containing some unwanted blurb and the data in LatLong format.

Copy that bit to a new TextEdit file called <LA-name>.js, then leave some spaces.

Convert the next data set, i.e the next polygon in the multipolygon.

Copy and paste that data into the TextEdit file.

Repeat steps 8-12 until all polygons are in the TextEdit file. Save and close it.

Open it in Eclipse. Do the RegEx to remove the unwanted blurb. (To play it safe, do a few hundred lines at a time, saving each time.

Save and close the file – Eclipse is about to fall over.

Open the file in Word again, then replace all multiple new lines with a suitable swearword.

Find first mention of Scotland – it’s about 7000 pages in. It’s to do with Angus.

Find second mention of Scotland. This is in the code marking the start of the data for Clackmannanshire.

Select all the data for Angus, then cut it out and paste it to a new document. Save that as Angus.txt.

Repeat steps 4 to 6 to get data files for all 32 Scottish LAs.

Such a shame they are in National Grid format, not lat/long.

proj4leaflet claims to handle other projections. Trying it but the huge size of the clackmannashire data file seems to be killing Eclipse. So have abandoned that, going back to what we have so far and trying to convert NG co-ords to latlong.