High Performance Computing

Cloud storage management

Backing up data from Monsoon to Google Drive is easy. This approach can be used with rclone for Dropbox, and Onedrive (untested).

While we can help you get started with this approach, we cannot support any issues that you have with your data in the cloud storage. You are responsible for the management and retrieval of data from your cloud storage.

We will utilize the rclone utility which is installed on Monsoon as a module: “module load rclone”

In this doc we’ll be focusing on using Google Drive, which has unlimited cloud storage for all at NAU

Click on the following link and on the next page select “Faculty/Staff Google Account Request”: Google Services at NAU

Establish a directory structure in your Google Drive via the web interface. For this doc we will use “backup” as the name of the directory that we will push our data to, from Monsoon. The backup folder is in the root of your Google Drive (one time setup)

Above we are creating a tar file of the large directory and splitting it into 200GB pieces on the fly. You may want smaller pieces depending on how large your directory data set is that you’d like to backup. In this scenario, we are backing up a 3TB directory, so 200GB is a good size.

rclone will start 32 file transfers in parallel. Only two files per second will actually be accepted by Google however due to their rate limiting. Howver, there can be many more files in transit.

this method will prove to be the best as you will achieve the highest combined throughput

rclone will indicate success or failure for the various copy and sync command. To see what has been uploaded to your remote drive, you can easily check what files and directories are there by running an rclone list command: