Credential Setup: Toil Autoscaling with AWS

This post will cover credential setup and the initial launching of a toil autoscaling cluster.

Note: To manage autoscaling, toil mimics a mesos cluster by spinning up an AWS EC2 instance with a docker image of a mesos leader node. This sort of virtual cluster can then spin up mesos worker nodes as needed to manage autoscaling according to resource requirements. This tutorial covers only setting up credentials and initially launching this leader node.

This tutorial was run on Ubuntu 16.04 using Toil version 3.12.0.

First, setup an rsa key for AWS. On Ubuntu, generate your key locally with the command:

On the IAM Dashboard page, choose your account name in the navigation bar, and then choose My Security Credentials.

Expand the Access keys (access key ID and secret access key) section.

Choose Create New Access Key. Then choose Download Key File to save the access key ID and secret access key to a file on your computer. After you close the dialog box, you can’t retrieve this secret access key again.

Now you should have a newly generated “AWS Access Key ID” and “AWS Secret Access Key”. We can now install the AWS CLI and make sure that it has the proper credentials:

TOIL_APPLIANCE_SELF=quay.io/ucsc_cgl/toil:latest – This is optional. It specifies a mesos docker image that we maintain with the latest version of toil installed on it. If you want to use a different version of toil, please specify the image tag you need from: https://quay.io/repository/ucsc_cgl/toil?tag=latest&tab=tags