4. Can I automate download of data?

YES , command lines can be particularly useful:
• if you want to download data regularly and automatically (without having to connect manually to the web portal each time)
• if you want to download a large amount of data because each downloading command is volume limited

To achieve this you can build command lines using a python script. It can be launched under different environments: Windows (possibly through Cygwin), Linux and MacOS. See also FAQ 5.

5. How to write and run the script to download data?

The CLS datastore web portal allows you generating a template command line using the python script for downloading data you are interested in. This command line integrates the extraction parameters. In order to first create your command line, you have to navigate as if you wanted to download the data through the web portal (see FAQ 3).

The popup displayed explain how to proceed: you need to have python installed, you need to download the motu-client package and extract the archive. Then, copy and paste the command line written in the popup, and adapt to your local environment the path of your motu-client package, and finally replace username and password by yours (usually proxy parameter are useless). After the run of the command line, you should receive a file data.nc corresponding to your request.

Of course, in order to download the data every day for example, you just need to modify the parameter “date” in your command.

6. Is a download limited in terms of volume?

Volume limitations for a single request have been set up on download mechanisms in order to guaranty good performances. The maximum amount of data you can download is 1 Gigabyte for a single request. If your single request is beyond this size, an error message will be immediately displayed: “The size of the data to download – … Megabytes- exceeds the maximum allowed – 1024 Megabytes”. You can also encounter a “proxy error 502”. This error message appears when a request lasts too long.

In order to overcome the limited volume associated to a single request, you need to use multiple requests (i.e.: a loop within a script command line) hence splitting the data volume into smaller pieces at a time. For example in your script command line, you can change the temporal extraction windows.

7. How to download a large amount of data?

One download request results into one file, so downloading a large amount of data (i.e. more than 1Gb) with a single request is not allowed. Moreover, it is not recommended to manipulate files larger than 1Gb, for usability purpose.

Downloading a very large amount of data, especially for downloading large time coverage is possible through the download service: The simplest way is to perform a loop on dates. You need to split your download requests into several download requests.

For example, below you can find a loop coded in a simple Unix/Linux shell script.

The array DATES shall be adapted to each situation (time coverage of the dataset and your own need). Of course, depending on the way you want to code the loop (python, perl, java…), you will have to adapt the code. The line in blue is a command line to download one time step of data.