Richard's diary

Recent diary entries

Folks, I've made an exciting discovery. I've stumbled across a country on the map which has 46 million inhabitants but is barely mapped. There appears to have been an import several years ago of a poor-quality dataset, and since then it's languished untouched. There's no indigenous mapping community. Can we help this poor beleaguered country to get a decent map?

Ok, you may have figured where I'm talking about. It's the rural US: 72% of the landmass, 15% of the population. And it needs your help.

The problem

For the past couple of months I've been using idle moments to address a particular, and very widespread issue in OSM's coverage of the rural US - the highway=residential problem.

TIGER data, which forms the bedrock of OSM data in the US, classes roads by CFCC - 'Census Feature Class Code'. By far the most prevalent is A41 - "Local, neighborhood, and rural road, city street, unseparated" - and the TIGER import translated this to highway=residential. This was a good fit in urban areas but covers a multitude of sins in rural areas - everything from good, fast state highways to rutted forest tracks or worse.

The effect is that our map of the rural US shows pretty much everything, save the biggest roads, as a residential road. Tarmac road with sweeping curves and a painted centreline? highway=residential. Gravel road? highway=residential. Forest track? highway=residential. Vague two-foot clearing through the woods where someone perhaps rode 50 years ago? highway=residential. Ploughed field? Etc, etc.

This is a typical example from an agricultural area. A good-quality road with centreline, running east-west: a smaller access road, running north: and nothing at all running south. In OSM, this is mapped as a crossroads with highway=residential roads in all four directions. Having the edge of a ploughed field marked as highway=residential doesn't make for a great map, nor does it make for good routing results. "In 100m, turn left." "BUT THAT'S A FRICKING FIELD YOU ACCURSED MACHINE." Sigh.

But if you really want some fun, find a less cultivated area - this forest, for example. Look at all those lovely residential roads, tagged exactly the same as a paved city street. Except none of these are paved. A few might be gravel. Many don't really appear to exist at all.

Most of this remains unchanged. In some areas a dedicated user has cleared it up and there've been a few energetic nationwide editors, but it's a massive job. It's pretty much endemic - even just a few miles from San Francisco, a hotbed of OSM activity, you'll find examples.

Perhaps we shouldn't have imported TIGER in these rural areas, but just let the map grow at its own pace. That way, the important roads would have been surveyed, traced, or imported one-by-one, and the thickets of near-impenetrable tracks would probably have never made it in. But we are where we are, and though I'm generally sceptical of "armchairing" far-flung data, we have a big heap of flat-out wrong data and no other strategy to deal with it.

A framework for fixup

So I've been fixing up roughly along these lines, though obviously adjusting for local sensitivities and network considerations:

Most of this is fixable from imagery. There are also some good datasets: Arnold, Forest Service data, various state data, etc.

In forests 90% of highway=residential should really be tracks. In the plains, the majority is either track or unpaved road, often in grids, but with the occasional paved through route. In Missouri, Tennessee, Kentucky and eastwards you start to see more paved roads.

Personally, my main priority has been to identify and retag paved through routes. Often these can be identified by squinting at the map: a river bridge is a tell-tale indicator, or a road with wide curves, or one linking settlements. Sometimes you just need to look at the aerial imagery and pan around. Of course, it's not just the highway tagging that needs fixing - ref tags and geometries would benefit from attention, too - but you can't do everything, so my chosen challenge has been to get the tagging sane.

I just use plain vanilla Potlatch 2 for fixup, accelerated by assigning common tags to function keys. One day it'd be nice to build something MapRoulette-like to tackle the issue, a bit like HotOrNot (TrackOrCack? HighwayOrLieway? RoadOrFOAD?). But for now a normal editor does fine.

So if you're sitting in your armchair with an itchy OSM finger, resist the temptation to normalise some tags or trace some buildings or whatever else you might usually do. Come and fix up the rural US.

Task-based fixup workflows, such as MapRoulette, To Fix, and the HOT Tasking Manager, have become popular in OSM in recent years, presenting a series of map-improvement tasks for a user to complete.

MapRoulette is wondrous in many ways but has three issues. First, if you're using an online editor, the time taken to open each task is significant - around 3-5s to open a new instance of iD for each task. Second, that online editor has to be iD, so it's not suited for those of us who use P2. Third, the process for creating your own tasks is fairly cumbersome, especially if you're working on a small series of tasks to be completed by one or a few people.

Potlatch 2 now implements a fast, light way of moving through a series of tasks. It's intended for small-scale personal work rather than planet-sized challenges. There's no 'resolve' button - it's just a way of moving between locations efficiently.

It works like this:

Create (or find) a GPX or GeoJSON file containing the locations.

If it's a GPX file, it should contain waypoints. Each one can have a description in a <desc>, <cmt>, or <sym> element.

If it's a GeoJSON file, it should contain Point features. Each one can have a description in the 'name' property (or, failing that, the first property).

Open it in Potlatch 2 with the Tasks button.

You can either load a file from disk, or type a URL and click 'Fetch URL'. (For the latter, note that the usual nonsense about needing a crossdomain.xml file at the root of the server applies, sadly.)

Move from task to task with the palette.

The map will centre on the first location. When you want to go onto the next one, click the forward arrow. To go to the previous one, click the back arrow. And that's it.

I've created a set of tiles from US Forest Service road data for the 155 US National Forests.

This is to help with TIGER fixup in these rural areas, where tracks, trails and entirely non-existent paths are often tagged with a bare "highway=residential". The US Forest Service data is greatly superior to the original TIGER data and has metadata on surface type/quality, but is unsuitable for automatic import into OSM because it would overwrite mappers' existing work in these areas.

and they're included in the editor-imagery-index list used by P2, iD and Vespucci. The tiles are available up to z19. Use of Potlatch 2's new floating imagery window mode is recommended, so that you can work from both Bing imagery and these tiles at the same time.

I know. Two new features in the space of a week. Don't get too used to it.

P2 now has a 'Show floating window' checkbox in the Background menu. Select this, and you'll get a second set of imagery in a floating window:

The window follows your main cursor location. So you can have Bing in the main window and Ordnance Survey StreetView in the floating window, or a blank background in the main window and Bing in the floating window, or whatever you like.

There's a "lock zoom" checkbox to stop the floating window zooming in or out - for example, edit at z18 with Bing but see OSSV at z16; or edit at z14 with Bing but have the floating window at z19 for a close-up view. It also respects the "max_zoom" parameter of the editor imagery index.

I coded most of this at a hack weekend a year ago but had never got round to finishing it off. Hope it's useful.

At the same time, P2 now makes a passable fist of rendering multipolygons where the tags are on the relation rather than the outer way (yeuch). It's not perfect and it won't do any of that crazy "advanced multipolygon" stuff. But it'll do until we finally get an area datatype.

And so is Potlatch... just about. (Insert obvious Flash reference here.)

Potlatch 2's core purposes are (a) being an intermediate-level editor and (b) annoying Germans. It's never going to get major new functionality or important rewrites, but for those of us happily using it, I've always intended to add and fix little things over time.

In the summer it gained support for the editor-imagery-index project, which provides a central list of background imagery suitable for tracing. Today it has a couple of improvements to the long-standing feature where you shift-click away from a way to add a new point (roughly comparable to JOSM's "Improve Way Accuracy" mode, but we don't do modes round here) - which is a really useful technique for TIGER fixup.

I've also added the ability to memorise tag combinations and recall them by a single keypress. Very simple: press shift and a function key to memorise the tags from the current selection. From then on, pressing that function key will add the tags to the selected items. (Those with long memories may remember this is a Potlatch 1 feature, and one I've always meant to add to P2!)

I have a couple of larger features which are 90% coded but were never quite finished, both firmly "intermediate-level" tools. As and when I get the time, I'm hoping to finally finish them off and get them out there.

It's built with (patched) OSRM and a complex custom profile. It takes account of elevation, cycle routes, surface quality and more. All routes are fully draggable; you can export to GPX, TCX, and PDF, and save routes if you create an account. The cartography is specially designed for the site.

Here's a ride along the Rhine:

Or if you fancy cycling over Mont Ventoux:

Routing details

If it doesn't follow a route you'd expect it to take, this is usually because surface tags are missing.

For example, here in France, the canalside path is tagged as 'highway=path' with no surface tags. cycle.travel guesses that paths in rural areas have poor quality surfaces, so will try not to route along them. Adding a 'surface=gravel' tag to this path, which the aerial imagery suggests, will make the router like it. (Access tags are also good.)

Miscellaneous notes

The tileserver is a little slow - please be gentle!

There are occasional inconsistencies in the tiles - old styles that haven't refreshed yet.

You can't route between the UK and mainland Europe (there's a big lake in the way. Only Chris Froome is allowed to cycle through the tunnel and look how far it got him)

I'm planning on weekly updates but it'll be less often at first.

Known issue with highway=trunk, bicycle=yes getting undue prominence.

Known issue with fahrradstrassen/fietsstraten not being prioritised.

You can switch from miles to km if you create an account and set user preferences. It'll be a bit smarter about defaulting to km for Europe soon, but I need to do a bit of work to enable that. Edit: Now defaults to km for routes in Europe.

Still lots to improve but I hope you like it - and, as ever, thanks to all the mappers who have contributed all the lovely data.

You can post comments/bugs/suggestions here, of course, or on the site forum itself.

Like the moo of a cow, everyone knows what a highway=cycleway means. Yes, there are variations. Lots of them. But in the absence of any other supporting information (like a surface tag), you can make an assumption that the above pic probably won't be too far wrong.

But just like no-one has heard a fox, no-one knows what a highway=path means. It could be anything.

So if you are using highway=path because "it makes it easier for data consumers", it doesn't. It makes it a pain in the arse for data consumers. Right now I am consuming data for a cycling router and highway=path is the bane of my life. When I see "highway=cycleway" it's a safe bet I can route a bike down there, whereas when I see "highway=path; bicycle=yes" then maybe a bike might want to go there, or maybe it's a steep drop over a precipice with a rocky surface but, by some quirk of arcane legislation, bikes are allowed.

For the love of God, if you must use highway=path, please, please, please, please add a surface tag with a commonly-used value.

For those who don't follow the mailing lists or the OSMF blog, a heads-up: the licence redaction will begin this week. Read full details here. There will be no interruption to mapping but you should be aware that there might be changes in your area.

At the bottom, select "OSM", then "File: Open..." to load your .osm file. Do this for each one:

You'll see them appear as layers in the list. By default they're shown in 'Potlatch' style, which means they look the same as the rest of the map. You can choose any currently-loaded stylesheet to show them in, but I like to use 'GPS', which displays them as a clear cyan line:

Close the window and go back to the map. Look! Your .osm file is being displayed!

It's being shown as a background layer. In other words, it won't be uploaded next time you save. (Because unthinking imports are bad, right kids?) So to add it to the map, you need to pull it through from the background layer to the main map.

Couldn't be easier. Just alt-click. (Or, if you're using a Linux system that reserves alt for its own nefarious purposes, shift-control-click.)

And that really is all there is to it.

Even more awesome

This is just scratching the surface, but there's lots more you can do with vector background layers. Basically, Potlatch does the hard work of making data usable, so you don't have to faff around with a bunch of preprocessing scripts.

You can open from local disc (like this) or fetch from the Internet. You can open .osm, .gpx, .kml or - yes - even shapefiles. (If loading from local disc, put the .shp, .dbf and .shx files in a .zip).

Wrong projection? No problem: P2 can reproject from OSGB or NAD83 (and we're happy to add new projections if needs be).

Data too dense? Just enable the 'Simplify' button and Potlatch will run a Douglas-Peucker line simplification over it, at the strength you've specified in the Options dialogue.

And the cleverest thing of all: MapCSS-based tag transformations. You can supply a stylesheet-like text file that tells Potlatch how to map tags in the source data to OSM standards. Use MapCSS selectors as normal, and the special 'set' and 'delete' instructions to control the output. It's best expressed through an example, so here's one I used for OS VectorMap District data.

Lots of you will have seen the interest recently in switching from proprietary mapping providers to OpenStreetMap - blog postings by Nestoria and StreetEasy, Wired's article, and so on. We started a Twitter hashtag, #switch2osm, and it's rather taken off.

It's a one-stop shop for would-be switchers, showing them how to get started with using and generating OSM tiles; _why_ they might want to; and where to look for more information.

Huge thanks to Harry, Kai, and lots of other people who've helped with this (and especially Matt for the layout). We'll be continuing to refine the site in the forthcoming weeks, but we're happy that it's ready to launch now. So - OSM army, get to it and promote it :)

Paris mapper Pieren has written an excellent half angry, half regretful diary entry that sums up why unthinking imports are such a curse to OSM. For those that don't speak French, I'm posting a (slightly loose) translation here. Apologies for any errors of translation.

[start quote]

I sadly discovered today that the import of buildings in the 16th arrondissement [district of Paris] has been brutally completed by someone else, by means of the simple import of a file.

I'd been working on this area for several weeks and yes, that might seem too slow for some. But if I was progressing slowly, that’s because I was taking the time to check names and one-way streets on the ground, block by block; to clean up errors (disappeared names and POIs); to add extra information such as traffic lights, missing vélib [cycle hire] points, missing names, barriers, other POIs, addresses. Everything that a semi-automatic import doesn’t do (after all, I could have done the import myself in 15 minutes long ago).

I've already carried out this work for other arrondissements (smaller ones, true) over a long period. You only have to read my OSM diary to see the list. I was three-quarters of the way through working on this arrondissement, but here I'm stopping, disgusted. I shall leave the work of completing the missing information to others, so that the arrondissement may be consistent in a far future (if ever), as well as the other arrondissements of Paris.

I understand that many people work on each area, and it’s better like that. But a tiny bit of co-ordination and investigation, to check what others were doing, would seem to me to be a minimum before embarking on such an import.

I cycled two-thirds of the Way of the Roses, Sustrans' newest challenge cycle route, last week - from Morecambe, on the west coast, over the Dales to York. (The route continues to Bridlington, but I'm leaving the small bit of unmapped route there for our East Yorkshire correspondent to finish.)

A beautifully scenic route, well chosen, and generously signed - which is just as well, as I was cycling 'blind', the official route map not having turned up. There were some pretty strenuous hills; the worst of the lot was one on an off-piste detour I foolishly took out of Settle, having mapped the official route there already. Actually, the flattest bit (railway paths notwithstanding) was the most boring - the section from Ripon to York, which was fairly featureless countryside. The Ouse crossing on the rickety old Aldwarke Bridge was an experience, though!

OSM's National Cycle Network coverage is astounding and one of the reasons why everyone loves OpenCycleMap.

With the sun finally emerging once again (yay) we've got the chance to fill some of the gaps and make it really useful. Anna and I went out on Saturday to map a recently opened section of National Route 45 (south of Worcester), and it occurred to me that a few afternoons like that would complete coverage of several high-profile routes.

So I had a look at the map and have identified a few that could be ticked off in an afternoon by nearby mappers. Obviously, some are already in hand - Gregory W cycled most of NCN 1 last year, for example, and I've got a few planned for this year.

The new Way of the Roses*: a coast-to-coast route being launched this year, roughly Morecambe-Settle-Harrogate-York-Bridlington. East of York it's fully mapped. Morecambe to York is not yet fully signed. But it'd be great to have it mapped on OSM at launch.

Hadrian's Cycleway (NCN 72) and the Reivers Route (NCN 10) could be completed with a little effort.

A few gaps around Sheffield could also be completed fairly easily.

Any takers? Or any other gaps in big routes that people have spotted?

(I posted this to talk-gb too and there's been some discussion already.)

But much though I might like the kind words for Potlatch's drag'n'drop POIs - and much though others might giggle at NickB's new surname - this is the really important bit:

Gordon Brown, the UK prime minister, has just rediscovered cooperativism as a way of galvanising people to vote Labour. He would have been much more in tune with the times if he had widened it to include the open source movement in all its different aspects. It is one of the most interesting phenomena of our times, a kind of global mutual society.

Hoorah for small down-at-heel Midlands towns, for every street in Burton-on-Trent should now be mapped.

Bob Coates has done large chunks east of the Trent (Winshill etc.); Paul Sladen made a whirlwind visit by narrowboat and coloured in the town centre and thereabouts very attractively; and I've done a whole bunch of very exciting housing estates west of the Trent, plus a load of other stuff. There's probably the odd bit missing but nothing particularly significant that I can spot.

Doubtless there's lots of landuse one could add, service drives into industrial premises, POIs (well, "interest" might be stretching it - this is Burton after all) and even postboxes for those who like that sort of thing. I'm not particularly struck on making a huge effort to do that in the absence of aerial imagery, but will probably colour in a few bits now and then - principally residential and industrial areas, I guess - in a kind of slow, incremental way.

Google has a really enlightened guy called the Data Liberation Front. His role is to make it easy for people to get their data out of Google - rather than it being locked in.

Usually, people are locked in by the lack of an export feature, or an obscure file format. In mapping, people are locked in by licences.

In Google Maps' case, you can create your own work by tracing over aerial imagery. But you can't use this work elsewhere, because of the licences and terms of use. (The phrase "derived work" usually crops up around now.)

Google could fix this by saying that tracing from their imagery is ok - just like Yahoo have done. On my blog, I looked into the legalities of this and concluded there's nothing in law stopping them from doing so. It's entirely their decision.

So - please vote for the Data Liberation Front to fix this! Click here, sign in with a Google Account, and tick the box. And tell your friends.

I've been playing with mkgmap over the last week, with the aim of creating a Garmin map of the National Cycle Network from OSM data. It's effectively a new version of the one I did the other year, but with better styling (via a .TYP file), contours, and so on.

NCN mapping on OSM is coming on in leaps and bounds this year, not least due to Gregory Williams' Three Corners Cycle Ride. So I figured this was a great opportunity to make it available for GPS, and so tackle the familiar "now which way do I go here...?" problem. The screenshot above would have helped Gregory find his way out of Gloucester. :)

The installation procedure is the same as any such map: connect your Garmin via USB, go into Mass Storage Mode (Setup->Interface), open the Garmin folder (creating one at the top level if there's not one already) and copy gmapsupp.img in.

I'd very much like to hear comments on what could be improved, in particular with the styling, and the choice of what's rendered at each zoom level. I've not as yet paid a lot of attention to POIs and would particularly welcome suggestions and comments on that. (I think I might remove the bus-stops before NAPTAN drowns us all.)

Routing is enabled but does not consistently choose the quiet routes I'd like it to. If anyone knows how to make it suck less, let me know; t�he eTrex's routing algorithm is famously impenetrable. Caveat emptor. Source will be available when I get round to uploading it to svn.

Regional and Local routes are both rendered in the same blue. I'm anticipating the abolition of Regional Routes in favour of three-digit National Routes. ;) The National Byway is shown in brown (let joy be unconfined).

Many thanks to Andy Gates for his Garmin cycling map, which has been a great inspiration and especially helpful in figuring out what goes where; and also for having the generosity to release the source. Several of the icons and some colouring are taken from his work. Thanks to the Scottish Mountaineering Club for the contour files. Thanks to Andy Allan whose fault all this open cycle mapping is, and yes, you're right, taking a photo of an eTrex screen is a PITA. And thanks, of course, to Steve Radcliffe and the mkgmap community for a superb, ever-improving program - for example, shield support has just been added, which really lifts the appearance of the map - and for being consistently helpful.

Update: The source is now in svn here, and full instructions are on the wiki.

Cycling round yet another Burton housing estate last night, I was amused to find a cul-de-sac called Deferrers Croft. It's clearly meant to be named after De Ferrers, some local hero or other. Unfortunately the lack of a space on the street sign makes it look like a street dedicated to procrastinators.

This was topped, however, by the next cul-de-sac along: Telmah Close. Hamlet backwards. Ugh, just ugh.

Unfortunately xybot isn't sentient, because otherwise, in the words of General Dreedle, we could take him out and shoot him.

xybot has not announced his presence on talk-gb, yet 'he' has taken it on himself, without consultation, to change "denomination=Church of England" to "denomination=church_of_england" here (and doubtless elsewhere). This similarly. Nor has he announced himself on talk-ie, yet this.

And what, pray, is this all about? User xybot, but created_by=Potlatch 0.11b? The bot knows how to work Potlatch? Blimey.

It is so, so tempting to write a xyrevertbot. If you see an automated user called General Dreedle you'll know I've succumbed.