Open Street Map Binary Protocol

This page is about additions to permit a particular way of using the OSM Binary Format for mobile devices in a tiled way that permits caching, omission of duplicate features and intelligent diffs (todo).

The idea is to provide mobile applications with the minimum download possible to get enough data to draw maps and provide routing functions.

The API will take three queries: tile, ts and have (optional). For routing requests an additional tileto and detaillevel queries will be added.

eg: bmap.php?tile=tileid&ts=31&have=neighboursdownloaded

Where the tileid is 32bit int representing the bbox required (as calculated by lonLatToTileId function below), using a form of quadtileing (see QuadTiles). We default to using 31 of the available bits to avoid problems with negative numbers with broken things like PHP. This gives quite small tiles, so the ts parameter is used to specify a different tile size.

Providing a list of the surrounding tiles already downloaded means that ways that overlap with them can be omitted from the data for this tile (this is optional on the caller, a default value of 0 will indicate no surrounding tiles downloaded). A list of omitted ways will be included in the output.

The output will just include 'interesting nodes', and ways. Routing information is to be calculate from comparing all the ways within the tile and any with identical coordinates should be considered a junction node.

The protocol is being developed as a php caching proxy, it is also hoped to become an official OSM API direct from the database. Features listed on this page will each individually indicate whether they are implemented in each of these forms.

Usage and intended use

The OSM Mobile Binary Protocol is intended for the following types of clients:

Performance

Downloading a large tile over the west and middle of Edinburgh using both the XML API and the PHPProxy gives the following sizes:

Version

Tile

TS

Protocol

Size

Compressed By

Additional

0.5

1007353

21

OSM XML API

7260292

OSM XML gz

791627

89%

OSM XML 7z

584595

91%

Binary API

349571

95%

Binary API gz

227218

97%

35%

Binary API 7z

180941

98%

20%

This gives a 69% reduction in download size from 7ziped xml file to 7ziped binary protocol file. If client doesn't support 7zip then the reduction is 71% by using the binary protocol. Clients without compression experience a 98% reduction.

Downloading several adjacent (neighboring) tiles probably further increases performance, as ways passing several tiles are only downloaded once. A test for such case (eg a small city) would also be interesting, as well as a full planet dump (reducing compressed planet from 900MB to an estimated 180MB). --Stefanb 14:13, 2 October 2007 (BST)

Binary output format

The formats base structure is a series of binary blocks, which specify a type followed by encoded data. To avoid parsing issues with block separator values each block is preceded by its size.

Length - 32bit unsigned integer length (length does not include the 4 bytes used to specify the length)

To save space long/lat values are encoded as 32bit signed integers from the floating point values to six decimal places (eg lat/lon * 1000000). This gives ~11cm accuracy which is plenty for any mapping project. Time/Date fields are the number of seconds since the Unix Epoch (January 1 1970 00:00:00
GMT). Strings are truncated to 255 and are given as a length followed by characters utf8 encoded.

OSM nodes are not transmitted in their entirety, only interesting nodes are included, (nodes with some additional meta data - poi's).

Ways reduce the data needed to be transmitted by only sending the positions of nodes as long/lat pairs, the first long/lat pair using 32bits then subsequent ones are 16bit diffs from the previous node position, if the diff is too big then fake nodes filling the gap are given.

Ways that cross more than one tile would end up being downloaded more than once, to avoid this the 'have' parameter is used, ways that have been omitted are listed so that deleted ways can be detected.

Tile Definition (optional)

This can be used at the beginning of each tile definition if multiple tiles are packed into the same stream.

OSM Feature Tags

Translations

Client specific tagging

Specific client applications will be able to specify which tags they are interested in (no point sending tags to an app if it doesn't do anything with that tag).

Implementation Status: PHPProxy(Yes), OSMAPI(No)

Compression

Because processing power on a mobile device might not be such a big deal as bandwidth (satellite, GPRS roaming...), it is suggested (=optional, not mandatory) to use HTTP compression using mod_gzip or alternatives.

Client support of this is detected by use of the HTTP header: Content-Encoding: gzip

Implementation Status: PHPProxy(Yes), OSMAPI(Yes)

Diffs

A more advanced mechanism could be employed to indicate the date the oldest out of the requested and neighbouring tiles was last downloaded.

Giving tile requests of the form:

bmap.php?tile=257882462&ts=29&have=41&since=1190994262

Implementation Status: PHPProxy(No), OSMAPI(No)

Routing

Finding out which tiles one needs on his mobile device might be tricky without knowing the route and vice versa - it is impossible to compute a route (optimal, suboptimal, any) without having all the tiles locally.

Options:

Client could request all the tiles in the ellipse between the start and destination point (focus points of ellipse) and try finding a route in there, gradually increasing the ellipse area until a satisfactory route is found.

Resolution - Level of Detail

On a way from A to B the traveller might only be interested in certain kinds of roads (paved, motorways, trunks, not residential, not pathways, not service roads....). Travelling on a motorway trough a big, well mapped city between A and B might result in great deal of unnecessary transfer.

Levels of detail for downloading routing information:

Full detail (default, done :) )

Navigable Ways only (would need to specify if route is foot/cycle/car, no museums, buildings, restaurants...gas stations?)

Route plus x km left and right (for finding detours, snack, or finding a way back to tour if forced to make a wrong turn. full detail?)

Route plus connecting ways only (just beginnings, good for seeing street names as you pass them)