Links

Tuesday, January 06, 2009

gpx and exif

Using a GPS

We got our first GPS last Christmas, in preparation for our trip to Ireland
in the spring (awesome trip, BTW). That was a Garmin nüvi 270, which is the basic hardware
device preloaded with maps of the US and Europe. Buying a device without the
Europe maps, and then adding them back would have been a little more expensive.
The device was quite useful on the trip.

As I'm a man, I've had more need for the device than my wife, and I had
been leaving it in my car. So for my birthday this year, my wife got me a
basic device, the Garmin nüvi 205. She wanted 'hers' back.
When I started hiking a bit more this
fall, I took it with me on the hikes, because it sucks to get lost. I could
also kinda figure out where I was based on the shape of the track the
device was generating,
compared to maps showing trails.

The big problem with taking a nüvi hiking is that the battery only
lasts 4-5 hours. It never ran out, but came close a few times. Another
problem people may have with older devices is that they don't seem to have the
nice tracking function that is really what you want in the device, to show
you visibly where you've been on the map. Our one year old nüvi 270
doesn't do the tracking thing, near as I can tell. Lastly, it's not terribly
convenient to slip into your pocket; it has a very sensitive touch screen and
a easily switched on/off switch at the top. I found an old Palm Pilot
leather case, with a hard 'front side' to prevent accidental
touches through the case, that ended up being a perfect fit
(saving $20 or on the Garmin case; I'm a pack-rat), but still a tight
fit for the pocket, and you have to slip it in and out the case just so.

For Christmas, my wife ended up getting me a Garmin eTrex Venture
HC, which is the basic GPS hiking model. The maps, compared to the
nüvi suck, but that's ok, even the default Garmin maps don't
include enough detail for hiking. This device handles track data
much better than the nüvi, in that you can pre-load a bunch
of tracks into the device and then display them while you're hiking.
I've got a bit of a long-winded procedure to generate tracks from
existing trail maps and Google Earth (see below), which then shows
me something close to the actual trails while I'm hiking.

Besides being used for live tracking, the other thing I've been wanting
to do is to correlate the pictures I've been taking while hiking with the
GPS, so that I can associate a fairly precise location with the pictures.
So that's how I spent a bit of my xmas break; writing that program.

What is GPX?

GPX is a XML file that your Garmin device will poop out giving you a
braindump of what it knows; "favorites" you've set up, track logs for where
you've been, etc. The file format is described pretty well on
this site.
The only thing I couldn't quickly figure out was the units for
the elevation; meters.

Actually, getting that GPX file can be a little tricky. You'll need to
connect your GPS device to your computer, and for Garmin use the software
they provide on a CD to pull the GPX file out, or for the Mac use
RoadTrip. For RoadTrip,
I always create a new folder for each GPX file I want to create, copy just
the stuff I want from the "most recent import" (or whatever), and then
export that folder, which exports it to a GPX file with the same name as
the folder. A bit non-intuitive, but you'll figure it out.

Once you have the GPX file, you can open it directly in Google Earth.
Google Maps doesn't appear to directly eat GPX files, but will eat
KML files, and you can easily convert a GPX file to a KML file using the
gpsbabel program.

What is EXIF?

EXIF is a standard for metadata embedded in image files. The site
http://www.exif.org/ explains all,
I guess. The spec is a bit dry. All sorts of metadata can get added
to images by your camera, including all the camera settings used
when the picture was takem, model information, and for this purpose,
GPS information.

Here
is an example of the sorts of information that gets stored as EXIF data
for a photo.

Two great tastes ...

So, now that we have a bunch of images, and a GPX file, it's a
SMOP
to get the time of the photo, calculate the GPS coordinates given
that time, and then stamp them back into the photo.

It looked to be a difficult slog to deal with the EXIF data myself, so some
reading quickly led me to the
exiftool
program which can do all manner of slicing and dicing of EXIF data for
your images.

The program I wrote reads in the GPX file, and then for every image
pulls out the time the photo was taken with exiftool, and calculates the GPS coordinates
for that photo, stamping that data back into the image with exiftool.

The program, gpx2exif, is housed here:
http://muellerware.org/hg/gpx2exif/,
is written in Python, may required version 2.5 or above, and also requires that you
have exiftool installed.

In addition to stamping the images with GPS data (actually, creating new copies of the
images with the GPS EXIF data), it also creates a KML file you can load into
Google Earth to 'test' the locations that got stamped. In case your camera's
clock is not synchronized to the GPS (hint, hint). If your times are off,
read the exiftool help, there's a way to adjust the times of your photos in
one swell foop.

Once you've got the GPS data stamped in your images, sites like Flickr
and Picasa will show you "map" versions of your sets, and do other stuff
with the geo data. The map view for my White Pines set at Flickr is
here
and the map view for the same set at Picasa is
here.

What's next

Turns out you can do all sorts of interesting analysis of the data
in the GPX file, like:

calculate distance travelled

calculate speed

figure out when you stopped for a break

plot data onto Google Maps or Google Earth

generate elevation maps

You should be able to do all this stuff in a web browser, in fact,
by writing the analysis code in JavaScript. Given that you can access
Flickr cross-site via their
'JSONP'-ish support,
associating photos with the GPS data is something you can also probably
do in the browser. We'll see. I'm a little worried that the number of
data points and expensive math required will be a bit much for normal
JavaScript processing; I may need to use a Google Gears worker to
offload some of that processing.

Notes

exiftool rocks; I was was happy to not have to deal with reading/writing
EXIF data myself.

My camera stores times in "local" format. Would have been nice
if it used UTC. Do any cameras do this? I made an assumption that
the camera, and computer you are running gpx2exif on, are running
at the same local time. Again, use exiftool to "fix"
this, if it's wrong.

I still can't figure out the secret to the findall() method
ElementTree. bugger.
Seems like a great API, I just can't use it. The XML processing wasn't that
complex, so
minidom,
which I'm very familiar with, was fine.

Neither Flickr nor Picasa will do anything with your EXIF GPS data unless
you specifically tell them to; presumably for privacy reasons. For
Flickr, the setting is
here;
for Picasa, the setting is
here.

The resulting map views from Flickr and Picasa aren't terribly pleasing
to me; in fact, the KML "test" file I produce from gpx2exif is
way more interesting. I think because you can see the actual trail,
but also the markers I used (default ones) work better than thumbnails
that Picasa uses, and the markers used by Flickr can't be disambiguated
when they're too close, like they can in Google Earth.

On the Garmin eTrex device, if you "save" a track that
you've made (hiked), it will strip the time values out. Make
sure you export the track off the device before saving; the
time values are (obviously) critical to determining the
locations for your photos.

To pre-load a set of trails for a park onto the device, I do
the following. Get a version of the trail maps (prolly from a PDF
from the park site) and convert to a JPG file. Bring up Google Earth,
find the park, and add an image overlay for the image file you created;
set the transparency down so you can see the trails and Google Earth
detail. Hopefully there's enough detail in the image, and Google Earth
so that you can move/resize the image overlay close enough. Then
create some new line segments in Google Earth, tracing over the trails.
I couldn't figure out how to export those line segments directly out
of Google Earth, but if you "mail" the folder they are in to yourself,
you will get a KMZ file, which is just a zip file containing a KML file.
Garmin tools like RoadMap don't eat KML, but you can convert the
KML to a GPX using gpsbabel, and then import that.
Voilà; trails to follow on my device.

When traveling long distances now, I've become completely dependent
on the GPS, and it's very nice to have when you're not on the
interstate. In fact, I've been actively avoiding interstates as much
as I can now; traveling back roads through small towns is much more fun.
You basically don't have to keep track of where you are,
what roads you're on, where to turn, etc. As long as you got
the destination plugged in right. And then I find myself racing against
the ETA the GPS displays prominently.
I called my wife at one point when I was coming home from hiking trip and
the conversation went something like this:

3 comments:

Thanks for documenting this. In the meantime, exiftool has been extended with a geotagging feature and can read GPX files. Assuming 4 hours difference between my camera time and UTC, I have had success with: exiftool -geotag mytracks.gpx "-geotime<${createdate}-04:00" -o processed/ .