Standard test images for Remote Sensing

October 23, 2011

When doing a course in Computer Vision last year I was introduced to the Lena image:

This was originally a scan from Playboy magazine in 1972, but has taken on a life of its own as a test image in the field computer vision. The (very interesting) history of it is described on the Wikipedia page and in the Complete Story of Lena. The image, freely available online, has been used as a test image for a huge amount of research in computer vision including image compression, face recognition, edge detection and more. Of course, Lena isn’t the only test image used in computer vision and image processing, there are many others (see, for example, these images). The use of a standard set of test images has a number of benefits:

It allows easy comparison between methods – Statistical measures of error will be comparable as they are based on the same underlying image data. Visual comparisons by humans will also be possible.

A range of test images can be chosen which range from easy to difficult to process – Lena is a good image to use for compression tests as it has a number of areas with high detail, but also larger flat areas. The subtly varying tones of the skin are also important, particularly when dealing with compression to lower colour depths.

Everyone working in the field can have access to the same images – 30 years ago, if you didn’t have a colour scanner (which were very expensive at the time), there was little you could do to get a colour test image – but the free distribution of the Lena image changed that. Similarly now, everyone from a Professor at a top research lab to a hobbyist working in their garage can have access to the same original data and perform tests which can easily be compared with the state of the art.

Research becomes more reproducible – Using test images means that other researchers can get hold of the data used in a study, and if they either get access to the code used to produce the results, or manage to re-implement the method themselves from a description in the paper (not always easy), they can then try and reproduce the results.

So, the question is:

Why don’t we have similar test images in remote sensing?

One of the things I leant on my computer vision course is that image processing for remote sensing is actually very similar to image processing for computer vision (whether it is face recognition, checking for cracks in contact lenses or anything else). In both fields, when you’re trying to do something (for example compress a photo or classify a satellite image) there are many methods to choose from. You want to choose the best for your situation, or decide whether you need to develop a new approach, and so you need to be able to compare the approaches easily.

A simple set of test remote sensing images would allow this. Of course the major questions are:

What sort of images do we want to pick?

How on earth are we going to deal with copyright, file format and distribution issues?

The second question is probably the more difficult (although less ‘scientifically important’) question – but there are a number of possible solutions, including getting imaging companies to release a few small portions of one of their images for free, or using freely-available data such as Landsat data. File format issues shouldn’t be too difficult – providing images as ENVI .bsq files (with a header file) and ERDAS .img files should allow easy use in the most common software, and make it fairly easy to read the data manually into another filetype. As for distribution, maybe a nice friendly university could be persuaded to host a website for us…

Now, the more interesting question: what sort of images do we want to pick? As with the test images discussed above, we want to have a range of image types and a range of image ‘difficulties’. As a first guess I’d suggest something like the following:

Small segment of urban area – Interesting because of the huge range of land covers, and notoriously difficult to classify. Three resolutions (high, medium, low) would be good, allowing comparisons between results at different resolutions. Ground truth data would be brilliant, but is likely to be expensive to collect.

MERIS (300m)

Landsat (30m)

IKONOS (4m)

Airborne (0.5m)

Extended area of vegetation – useful for testing vegetation data extraction algorithms. In this case, a phenological time-series would be useful, as well as different resolutions (as above).

Mixed landscape – a mixture of land-cover types including water, vegetation, urban and others. Preferably again at a number of resolutions.

These images don’t need to be huge – whole Landsat scenes would become unwieldy and take too long to process – but they should be large enough to contain enough pixels for statistically significant model development.

The main difference between the images available for testing computer vision algorithms and the images needed for remote sensing is that remotely sensed data must have good metadata. These images must be provided with full metdata including instrument type, bands used, resolution, date/time of collection, calibration data and map reference data. Ideally data should be provided fully atmospherically and geometrically corrected.

Of course, to do this would be a lot of work – far more work than most academics (including myself) have time for – but I think it would be an important and useful resource.