Community Contributed Software

Amazon Web Services provides links to these packages as a convenience for our customers,
but software not authored by an "@AWS" account has not been reviewed or screened by AWS.

Please review this software to ensure it meets your needs before using it.

The purpose of this sample application is to help Ruby and Rails developers integrate Amazon SQS using the ActiveMessaging plugin for Rails into applications that require a large amount of 'heavy-lifting' be handled asynchronously and in a scalable manner.In this sample application Amazon SQS enables the heavy-lifting (image watermarking) to be handled not by the Rails application ('Servers') but rather by a decoupled pool of Amazon EC2 instances ('Workers') dedicated to work asynchronously.

Details

About This Sample

Messaging

A Rails application running on a 'Server' EC2 instance uploads
to
Amazon S3 an image submitted through the Rails application's form by
the user. The Rails application puts a job message with details of the
uploaded object into the 'todo' queue from the Rails controller using
the ActiveMessaging Rails plug-in. One or more EC2 'Worker' instances
poll the 'todo' queue for jobs, read the message and download,
watermark the image then upload the new image to Amazon S3. The details
of this new image are added to the original job message and the new
message put into the 'done' SQS queue then the original message in the
'todo' queue is deleted.

The Rails application retrieves messages at regular intervals
from the 'done' queue with the ActiveMessaging 'poller' daemon. An
ActiveRecord model encapsulates a message and the ActiveMessaging
plug-in saves all the SQS messages sent and received. The
Rails application controller reads messages sent and received from the
database rather than directly from the 'done' queue and sends them to
the view that displays jobs. This prevents a situation where a user or
many users update their view to see their submitted and completed jobs
and each refresh or request makes new calls to Amazon SQS. N number of
users each with N refreshes would increase Amazon SQS usage
exponentially.

Figure 1: Media Processing Pipeline with Amazon Web Services

It's worth noting that the format of Amazon SQS messages
created in
this application is shared with that of the 'boto' library from Mitch
Garnaat. The Ruby YAML library is very helpful in working with these
RFC-822 compliant messages.
See Mitch Garnaat's Monster
Muck Mashup - Mass Video Conversion Using AWS

Muck, Heavy Lifting

In this application the 'muck', or 'heavy-lifting' is
demonstrated
by watermarking images. See the 'watermarker.rb' file in the root of
the .ZIP file containing the sample application. Of course your
application might do any other kind of work. If there were intermediate
steps between non-watermarked and watermarked more Amazon SQS queues
might be used. In this sample application there is no intermediate
state and so the job goes directly from the 'todo' queue to the 'done'
queue when the work is performed.

Scaling

The amount of watermarking that can be done asynchronously
without
affecting the Rails application performance at all is increased simply
by starting many more 'Worker' EC2 instances, Amazon SQS ensures that
if there is a job one of the 'Workers' will pick it up and process it.

This code sample uses an SQLite3 database with the Rails
application
- not scalable nor persistent. Using Amazon SimpleDB the Rails
application could be scaled by starting many instances of the Rails
application and all would use Amazon SimpleDB for persisting the
messages.

This application further demonstrates using public key
encryption to
protect the AWS keys that are passed to the Amazon EC2 instances at
launch time. The corresponding private key is bundled into the AMI. On
launching an instance of the AMI, the private key is deleted before
rc.local adds to the SSH
authorized_hosts file the keypair used to launch the
instance .

The ec2-launch-instances reads the user data it associates
with the
instance from the configuration file using the -f switch instead of
from the stdin. Using the configuration file works better from a shell
because of the length and contents of the encrypted, base 64 encoded,
AWS keys cipher text.

Once the AWS keys are decrypted, the keys are put into the
appropriate configuration files (broker.yml, amazon_s3.yml) and either
the Rails 'script/server' and ActiveMessaging 'poller' are launched (if
the 'server' keyword is present in the instance user data), or only the
'watermarker.rb' script is run (if the 'worker' keyword is present in
the instance user data).

See the 'launch.rb' file in the root of the accompanying code
sample .

Prerequisites

You are signed up and active for Amazon S3, SQS, EC2

You can run the EC2 Command Line Tools

(try running 'ec2-describe-instances')

You have followed the Amazon EC2 getting started guide

(you will have a key called gsg-keypair, use this or your preferred
keypair where you see <mykeypair>
in the instructions below)

You have an empty bucket in Amazon S3

(create a new bucket if
you need to)
If
you prefer or wish to later download the code annd run it locally see
the section below titled "Running the Sample Code Locally".

Otherwise this code sample is meant to be run inside the
cloud with it's
accompanying public AMI. You do not need to download the code. Continue
with the section immediately below titled "Running the Sample".

Running the Sample

Create a configuration file for 'Servers'.

Create a file called server.cfg.
Edit and save the file with the below two lines in it (replace <mybucket> with
the name of an empty bucket you own):

server<mybucket>

Download the application's public key that will be used to
encrypt your AWS access and secret access keys (or copy the URL and
download it from your browser)

curl -O https://s3.amazonaws.com/aws-pipeline/aws-pipeline_public.pem

Encrypt a copy of your AWS keys with the aws-pipeline
application's public key, base 64 encode it and append it to your
server.cfg

(Substitute your AWS keys where you see <awsaccesskey>
and <awssecretaccesskey>)
(The aws-pipeline EC2 AMI contains a corresponding private key to
decrypt your AWS keys)
(You are trusting the owner of the AMI)
(Only you can SSH into the instances of this AMI that you will launch)