Landsat batch download from Google and Amazon

2015-06-28

Landsat is the work horse for a lot of remote sensing applications with it’s open data policy, global data vailability and long spanning acquisition time-series. The USGS Bulk Downloader however is clunky, depends on special ports being open on your network an can not be scripted to suit needs like automatic ingestion of new acquired Landsat-8 scenes. Fortunately Google and Amazon provide mirrors to a lot of the Landsat datasets which can be used for scripted bulk downloading.

Landsat download from Google

Google copies most parts of the Landsat archive to their servers to be accessed by its EarthEngine. This data can be processed in the EarthEngine but also downloaded through the Google Cloud Storage platform.

requirements

You will need Python and the Google Cloud Storage utilities gsutil to access the Cloud plattform. If you have Pip, included in Python >2.7.9, you can install it with:

download a folder

The nice thing about gsutil is that it allows file listing and therefore enables you to recursively download whole folders or even the complete archive if you have the space.
Let’s say I want to download all available Landsat 8 and 7 scenes for a part of the Mekong Delta located at Path/Row 124/053. For this I’ll need to execute two commands.

Landsat download from Amazon

Allthough Amazon only mirrors Landsat-8 they do it blazingly fast. Each newly acquired LT8 scene is made available on Amazons Web Service, often within hours after the acquisition. If you are processing data on AWS you can use the bucket directly at s3://landsat-pds/.

Downloading the data is a little trickier than from Google. Each scene is stored in a folder with the naming convention http://landsat-pds.s3.amazonaws.com/L8/PATH/ROW/SCENE_ID/index.html. All bands are available as GeoTiff as well as the MTL metadata files and everything else you’d also get from the USGS Level 1 dataset. While you can list the content of each scenes folder you can not list the content of the folders above that level - so no recurrent listing and downloading.

These are just some of the beautiful projects that are made by possible by open source in combination with open data and I hope the open data policy of Landsat as well as the upcoming Sentinel fleet will spurr a lot more. These are interesting times for remote sensing and other data scientists.