Setting Up S3 Uploader

If you are running in Amazon AWS, Heron provides out of the box uploader for S3,
the object storage. S3 uploader is useful when the topologies run in a distributed
cluster of Amazon EC2 compute instances. Since S3 replicates the data, it provides
a scalable mechanism for distributing the user topology jars.

S3 Uploader Configuration

You can make Heron use S3 uploader by modifying the uploader.yaml config file specific
for the Heron cluster. You’ll need to specify the following for each cluster:

heron.class.uploader — Indicate the uploader class to be loaded. You should set this
to com.twitter.heron.uploader.s3.S3Uploader

heron.uploader.s3.bucket — Specifies the S3 bucket where the topology jar should be
uploaded.

heron.uploader.s3.access_key — Specify the access key of the AWS account that has
write access to the bucket

heron.uploader.s3.secret_key — Specify the secret access of the AWS account that has
write access to the bucket

Example S3 Uploader Configuration

Below is an example configuration (in uploader.yaml) for a S3 uploader: