Jochen Topf wrote:
> On Thu, Jan 18, 2007 at 11:40:50AM +0100, Frederik Ramm wrote:
>> (a) Which tiles are outdated? One could of course just re-generate
>> all existing tiles but that would be very time-consuming (I guess
>> about 20 days of round-the-clock computing for a modern PC; less of
>> course if many take part in the distributed computing). I will try to
>> evaluate the last-change-timestamp in the planet file and use that to
>> determine which tiles are out of date.
>>>> (b) If I want to regularly find outdated tiles, the perfect way would
>> of course be to process a full change log instead of the limited RSS
>> feed. Ideally, I would want to ask the server "please send all nodes
>> changed or added since <timestamp>", then use this regularly to re-
>> generate all affected tiles. Does someone know if a feature like this
>> is planned or even likely? - The other approach is, again, using the
>> weekly planet files to find out which nodes have changed. But this
>> would lead to weekly updates only.
>>>> I am not saying that we should abandon the current tile regeneration
>> based on the RSS feed, but it seems to me that it needs to be
>> supplemented by something else if it alone cannot guarantee that our
>> tiles are halfway current.
>> How about this: For every update the API could increment a version counter
> for the tile this update is in. So you have an every-increasing counter
> for every lowest-level tile. Whenever you render a tile you store the
> current counter for this tile. It is now very easy to find out how many
> changes there were between the time you rendered the tile and the
> current counter. The details on how to handle higher-level tiles have to
> be worked out of course.
This sounds like a bad idea because you would have to perform a huge
number of queries, one for each tile (is tile[n].current version >
tile[n].last version). Why not just use last update timestamp or a
globally incrementing counter, then you need only ask "Send me all tiles
with timestamp > [last time I checked]" which is really simple for the
DB to answer (as long as you index the field).
> The API would need some code to calculate the tile number from the
> lat/lon coordinates. The biggest problem would probably be that for
> changes in segments and ways you have to get the participating nodes first
> and find their coordinates and then tile numbers. But with some clever
> optimization this could be fast enough. We don't have to check both ends
> of a segment for instance, because generally they are very close. So it
> wouldn't really matter in most cases.
>> The benefit of this approach is that it is totally generic. Every
> renderer can use the same counter infrastructure to find outdated tiles.
Only if the different renderers always use the same tiles. Currently,
they don't - e.g. there are renderers that use a mercator projection,
and others that use a simple rectangular latitude and longitude for use
in products like Google Earth or WorldWind.
> This is important because we'll get more and more renderers and special
> maps and layers. Also it doesn't use the main database at all, the
> counters can be kept somewhere separate, maybe even in memory. So it has
> no performance impact on our most critical asset. Compare this to any
> kind of "diff" approach where the database has to do all the work.
The database is there to do work - that is what it is for. Storing
things in memory only reduces the amount of memory the database has to
use for doing the work itself. If you want to avoid loading the
database, use a mirror of the database on another machine (using the
tools to load planet.osm into a new database). People here always seem
to write scripts that duplicate huge chunks of the database into memory
e.g.:
script to measure the length of all the segments:
http://svn.openstreetmap.org/utils/osm-length
vs. my sql query that does the same thing:
http://lists.openstreetmap.org/pipermail/dev/2007-January/002861.html
or the thing to turn planet.osm into a postGIS database for Mapnik.
I might have a go at rewriting that at some point.
Robert (Jamie) Munro
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 249 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20070118/1125a7a3/attachment.pgp>