Image extraction from TMS cache

Image extraction from TMS cache

Hi,

I assume the new TMS cache with the files TMS.data and TMS.key uses some
standard. What tools can I use to extract the images from that cache?
I do some post-processing for offline mapping with Mapserver.

Re: Image extraction from TMS cache

Hi,

This is +/- "propietary" standard of Apache Java Caching System
(https://commons.apache.org/proper/commons-jcs/). We are currently
using Indexed Disk Cache as store for tiles, so the easiest way
forward, is just to use JCS, to read the file, and then use keySet()
method, to get all the data that is stored in cache.

> Hi,
>
> I assume the new TMS cache with the files TMS.data and TMS.key uses some
> standard. What tools can I use to extract the images from that cache? I do
> some post-processing for offline mapping with Mapserver.
>
> Thanks,
> Holger
>
> _______________________________________________
> josm-dev mailing list
> [hidden email]> https://lists.openstreetmap.org/listinfo/josm-dev

Re: Image extraction from TMS cache

Hi,

Is this something I can do on the command line or do I need to write a
Java program? What would be a good starting point? I was assuming that
there is some kind of cache explorer, but I couldn't find one.

Thanks,
Holger

Am 28.05.2015 um 22:14 schrieb Wiktor Niesiobedzki:

> Hi,
>
> This is +/- "propietary" standard of Apache Java Caching System
> (https://commons.apache.org/proper/commons-jcs/). We are currently
> using Indexed Disk Cache as store for tiles, so the easiest way
> forward, is just to use JCS, to read the file, and then use keySet()
> method, to get all the data that is stored in cache.
>
> Cheers,
>
> Wiktor
>
> 2015-05-28 20:30 GMT+02:00 Holger Mappt <[hidden email]>:
>> Hi,
>>
>> I assume the new TMS cache with the files TMS.data and TMS.key uses some
>> standard. What tools can I use to extract the images from that cache? I do
>> some post-processing for offline mapping with Mapserver.
>>
>> Thanks,
>> Holger

Re: Image extraction from TMS cache

Hi,

There is no general general cache explorer for JCS, as it is
implementation dependant, what it stores in the cache. Generally -
what's in there, it's just plain Java objects.

So what you could do, is to create small java program, that would
extract all the files from JOSM cache, based on josm.jar. And further
functionality just depends on what you want to do with cache content.

> Hi,
>
> Is this something I can do on the command line or do I need to write a Java
> program? What would be a good starting point? I was assuming that there is
> some kind of cache explorer, but I couldn't find one.
>
> Thanks,
> Holger
>
>
>
> Am 28.05.2015 um 22:14 schrieb Wiktor Niesiobedzki:
>>
>> Hi,
>>
>> This is +/- "propietary" standard of Apache Java Caching System
>> (https://commons.apache.org/proper/commons-jcs/). We are currently
>> using Indexed Disk Cache as store for tiles, so the easiest way
>> forward, is just to use JCS, to read the file, and then use keySet()
>> method, to get all the data that is stored in cache.
>>
>> Cheers,
>>
>> Wiktor
>>
>> 2015-05-28 20:30 GMT+02:00 Holger Mappt <[hidden email]>:
>>>
>>> Hi,
>>>
>>> I assume the new TMS cache with the files TMS.data and TMS.key uses some
>>> standard. What tools can I use to extract the images from that cache? I
>>> do
>>> some post-processing for offline mapping with Mapserver.
>>>
>>> Thanks,
>>> Holger

Re: Image extraction from TMS cache

My idea was to write a small program that calls TMSLayer.getCache() to
get the TMS cache. But to do that I need to initialize JOSM
(preferences, TMS layer, and tile cache). How would I do that? Is there
something like a headless mode where I can hook into? I tried to call
some of the initialization methods, but there are too many dependencies
and I would need a running JOSM to make that work.

Does it work to write a plugin that can be called from the command line,
but without the GUI?

The other option is to re-implement the tile cache code without some of
the dependencies. That seems to be a lot of work. It needs modifications
every time the JOSM tile cache is modified.

> So what you could do, is to create small java program, that would
> extract all the files from JOSM cache, based on josm.jar. And further
> functionality just depends on what you want to do with cache content.
>
> 2015-06-01 20:45 GMT+02:00 Holger Mappt:
>> Is this something I can do on the command line or do I need to write a Java
>> program? What would be a good starting point?
>>
>>
>> Am 28.05.2015 um 22:14 schrieb Wiktor Niesiobedzki:
>>> This is +/- "propietary" standard of Apache Java Caching System
>>> (https://commons.apache.org/proper/commons-jcs/). We are currently
>>> using Indexed Disk Cache as store for tiles, so the easiest way
>>> forward, is just to use JCS, to read the file, and then use keySet()
>>> method, to get all the data that is stored in cache.
>>>
>>>
>>> 2015-05-28 20:30 GMT+02:00 Holger Mappt:
>>>> I assume the new TMS cache with the files TMS.data and TMS.key uses some
>>>> standard. What tools can I use to extract the images from that cache? I
>>>> do
>>>> some post-processing for offline mapping with Mapserver.

> My idea was to write a small program that calls TMSLayer.getCache() to get
> the TMS cache. But to do that I need to initialize JOSM (preferences, TMS
> layer, and tile cache). How would I do that? Is there something like a
> headless mode where I can hook into? I tried to call some of the
> initialization methods, but there are too many dependencies and I would need
> a running JOSM to make that work.
>
> Does it work to write a plugin that can be called from the command line, but
> without the GUI?
>
> The other option is to re-implement the tile cache code without some of the
> dependencies. That seems to be a lot of work. It needs modifications every
> time the JOSM tile cache is modified.
>
> BTW, was the tile cache code modified recently? My cache was reset (erased)
> yesterday.
>
> Holger
>

Re: Image extraction from TMS cache

Hi,

I re-package josm-latest.jar into my cachextractor.jar to have a self
contained archive. I missed to copy data/preferences.xsd, which killed
my JOSM preferences. Now I copy directory data in addition to **/*.java
and the cache read works.

What do I need to do to shut down JOSM if it was started with the
functions in JOSMFixture.init() (no GUI mode)? Looks like there is a
background process. I call System.exit(0), but I would like to stop all
running processes first.

The other thing that caused me some headache was the export disk file
format. The old cache that wrote tile files to disk wrote *.png files. I
tried different PNG writers, but my files were always about 8 times
bigger. It took me some time to figure out that even JPEG files were
written with the .png file name extension. Now I save the tiles as JPEG
and everything is fine. Does the current cache have information about
the original tile file (format, data)?

Regards,
Holger

Am 12.12.2015 um 20:09 schrieb Wiktor Niesiobedzki:

> As for your problem - this is kind of the problem we face also writing
> the unit tests. It is solved by calling:
> org.openstreetmap.josm.JOSMFixture.createUnitTestFixture().init();
>
> Though keep in mind, that this is part of the testing code, so it's
> located here:
> http://josm.openstreetmap.de/browser/josm/trunk/test/unit/org/openstreetmap/josm/JOSMFixture.java>
> This should give you a hint, how to initiate "headless" mode.
>
> Cheers,
> Wiktor
>
> 2015-12-12 18:52 GMT+01:00 Holger Mappt:
>> My idea was to write a small program that calls TMSLayer.getCache() to get
>> the TMS cache. But to do that I need to initialize JOSM (preferences, TMS
>> layer, and tile cache). How would I do that? Is there something like a
>> headless mode where I can hook into? I tried to call some of the
>> initialization methods, but there are too many dependencies and I would need
>> a running JOSM to make that work.

Re: Image extraction from TMS cache

Hi Holger,

If you use josm code, there is a directory locking code, that prevents
from accessing cache simultaneously from different processes, as they
do not share memory structures that store current disk file layout and
may lead to corrupted reads/writes. So you should have JOSM shut down
first, before extracting the PNG.

There is no information stored about original file format. But you can
try to extract CacheEntry objects, which getContent() should return
original file content. But you will need to do file format
identification by your self using "magic" (like file), as uses this
magic to identify file format.

> Hi,
>
> I re-package josm-latest.jar into my cachextractor.jar to have a self
> contained archive. I missed to copy data/preferences.xsd, which killed my
> JOSM preferences. Now I copy directory data in addition to **/*.java and the
> cache read works.
>
> What do I need to do to shut down JOSM if it was started with the functions
> in JOSMFixture.init() (no GUI mode)? Looks like there is a background
> process. I call System.exit(0), but I would like to stop all running
> processes first.
>
> The other thing that caused me some headache was the export disk file
> format. The old cache that wrote tile files to disk wrote *.png files. I
> tried different PNG writers, but my files were always about 8 times bigger.
> It took me some time to figure out that even JPEG files were written with
> the .png file name extension. Now I save the tiles as JPEG and everything is
> fine. Does the current cache have information about the original tile file
> (format, data)?
>
> Regards,
> Holger
>
>
> Am 12.12.2015 um 20:09 schrieb Wiktor Niesiobedzki:
>>
>> As for your problem - this is kind of the problem we face also writing
>> the unit tests. It is solved by calling:
>> org.openstreetmap.josm.JOSMFixture.createUnitTestFixture().init();
>>
>> Though keep in mind, that this is part of the testing code, so it's
>> located here:
>>
>> http://josm.openstreetmap.de/browser/josm/trunk/test/unit/org/openstreetmap/josm/JOSMFixture.java>>
>> This should give you a hint, how to initiate "headless" mode.
>>
>> Cheers,
>> Wiktor
>>
>> 2015-12-12 18:52 GMT+01:00 Holger Mappt:
>>>
>>> My idea was to write a small program that calls TMSLayer.getCache() to
>>> get
>>> the TMS cache. But to do that I need to initialize JOSM (preferences, TMS
>>> layer, and tile cache). How would I do that? Is there something like a
>>> headless mode where I can hook into? I tried to call some of the
>>> initialization methods, but there are too many dependencies and I would
>>> need
>>> a running JOSM to make that work.