Details

How to Track Spot Instance Activity with the Spot-Notifications Sample Application

How to Track Spot Instance Activity with the Spot-Notifications Sample Application

Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances provide a way for customers to acquire
computing resources at a significantly lower cost than they would pay for On-Demand or Reserved
instances. Spot Instances are especially useful for computing tasks that involve
extended processing but that are not time critical. Spot is also
useful to add computing resources to a task to speed up processing.
With Spot Instances, the customer specifies the price, or "bid", they are willing to pay
for a
type of EC2 instance,
such as a High-CPU Medium (c1.medium). Then, if the Spot Price, which Amazon sets
based on Amazon's supply of excess capacity and demand for that capacity, is
below the customer's bid price, the customer acquires the instance. When the
Spot Price increases above the customer's bid price, the customer loses the
capacity. For more background information about Spot Instances, see the
AWS website

The Spot Notifications sample application demonstrates a way to monitor
information about your usage of Amazon EC2 Spot Instances. Note, this
application doesn't create or otherwise manage Spot Instance
requests. However, the ability to manage Spot Instances is available through the Amazon EC2 API. For more
information about managing Spot Instances programmatically, see the
Amazon Elastic Compute Cloud (EC2) User Guide
as well as the AWS developer guides for
Java,
.NET,
and PHP.

As distributed, the application enables you to monitor the following.

Amazon EC2 instances currently running in your account, including, but not limited, to Spot Instances.

The current set of Spot Instance requests associated with your account.

The current set of Spot Instance prices within a particular region.

Whenever the status of any these changes, the application sends a notification
using Amazon Simple Notification Service (Amazon SNS). For example, if a new EC2
instance launches in your account, the application sends a notification.

The application monitors activity within a particular AWS region. The region is
specified in the source code (see below). However, note that Spot requests and
Spot Instance pricing are specific to a particular Availability Zone (AZ).
Because regions contain multiple AZs, Spot Instance pricing can vary across a
region, i.e. the same EC2 instance type could have different Spot prices in different AZs.

Because of the number of different types of EC2 instances
and because regions contain multiple AZs, the
number of Spot Instance price points is large.
To manage this complexity, the application enables you to specify
which EC2 instance types and AZs to monitor for Spot Instance price
changes and how large a price change is required in order to trigger a notification.

The following sections provide an overview of how the Spot-Notifications
application is designed and describe how to build, configure, and run
the application.

Design Overview for Spot Notifications

The application is designed around the idea of a monitor. Each monitor tracks
a separate type of data related to Spot Instances. Each of the monitors uses
the EC2 API to retrieve information about the current state of your EC2
resources, such as Spot prices, and persist that information in a SimpleDB
domain. The monitor then periodically retrieves this state information and
compares it to the information that was stored previously. If any of the state
information has changed, the monitor sends a notification.

The application includes three monitors:
Instance, SpotRequest, and SpotPrice.

Instance Monitor

The instance monitor tracks all Amazon EC2 instances associated with the account
being monitored--not only Spot Instances. The instance monitor sends a
notification whenever the state of one of these EC2 instances changes. So, for
example, if you choose to terminate one of these EC2 instances, you will receive
a notification when the instance transitions to the shutting-down state and a
second notification when the instance reaches the terminated state.

SpotRequest Monitor

The SpotRequest monitor tracks Spot requests (as opposed to Spot Instances) and
sends a notification whenever the state of one of your Spot requests changes. For
example, when you first create a Spot request, it enters an Open state.
Therefore, you would receive a notification whenever you create a new Spot
Request. Similarly, if you cancel a Spot Request, it will transition from Open
to Cancelled--and again you would receive a notification.

Here is an example of a notification for the transition from Open to Cancelled.

111122223333sir-aca7a011Amazon.EC2.Request.StateTransitionFROM: open TO: cancelledYour Amazon EC2 Spot Request has had a state transition.

Consider a scenario in which you create a Spot request and it is subsequently
fulfilled. The Spot request would move through the following states.

Open -> Pending -> Running -> Active

This results in your receiving four notifications.

SpotPrice Notifications Monitor

The Spot Price monitor sends notifications if the prices for Spot Instances
changes in a way that you have specified as being significant. For example, you could configure the application to
notify you if the Spot price for t1.micro instances in the us-east-1b AZ changes by more than 20%. More information about
specifying Spot price ranges and thresholds is provided later in this article.

Implementation of Monitors

From an implementation standpoint, the application defines an abstract Monitor class.

public abstract class Monitor extends Thread

The Monitor class is defined in

com\amazonaws\codesamples\notifications\monitor\Monitor.java

Each specific monitor, such as the SpotRequest monitor, is implemented in a
class that derives from Monitor.

public class SpotRequestMonitor extends Monitor

Each of these derived classes is defined in a corresponding subfolder.

At program start, the main function for the application reads a configuration
file, Notifications.properties (described below), to retrieve the list of
monitors. It uses reflection to instantiate each of the monitors and then calls
the start method on each monitor.

The start method in turn makes a call to the run method exposed by
each of the monitors, which starts each of them asynchronously.

The run method in each monitor contains an infinite loop (while( true))
that sleeps for a length of time. Upon waking, it determines if the timeout
specified in Notifications.properties has elapsed. If so, it calls the monitormethod which executes all the tasks associated with that particular monitor. For
example, in the case of the SpotPrice monitor, the monitor method would
determine if there have been any significant changes in Spot Instance prices by retrieving
the current set of pricing data and comparing it to what was stored in SimpleDB. If the
pricing data have changed, the monitor generates a notification.

The run method records the current time so that it can calculate when it should next run
the monitors. Then, before going back to sleep, it executes the updateConfig and
updateMonitorSpecificConfig methods. These methods read from the Notifications.properties file
to update the configuration information used by the application.
This makes it possible for you to change the behavior of the application without
having to restart it. Note, however, that if you want to change which monitors are running, you must
restart the program because it is only at startup that the main method enumerates which monitors to run.

Data is Stored in Amazon SimpleDB

Each monitor uses a separate SimpleDB domain to store the information that it
tracks. For the three included monitors, the application creates the following three
Amazon SimpleDB domains.

Amazon_EC2_Notifications_instance

Amazon_EC2_Notifications_spotrequest

Amazon_EC2_Notifications_spotprice

The monitors store and retrieve information to and from SimpleDB using a set of utility functions
that are defined in

src\com\amazonaws\codesamples\notifications\util\SDBPublisher.java

Monitors use SDBPublisher to Access SimpleDB

In addition, each monitor implements a set of conversion methods that
translate the data between the format in which it is stored in the domain and
the data structures that are used by the monitor. For example, for the Spot request monitor,
these methods are implemented in:

Note that if the application exits, the most-current state
information continues to persist in the SimpleDB domain. When the application is
restarted, each monitor compares the state information at the time it restarts to what
is stored in the domain and generates appropriate notifications for any differences detected.

How to Change the Domain Names

The names of the Amazon SimpleDB domains are configurable by changing a static string in the
source code. The prefix text for the domains is specified in SDPublisher.java:

The suffix text, such as "spotrequest", is provided by the Java file that implements the monitor
itself.

Notifications via Amazon SNS Topics

Each monitor is also associated with an SNS topic. When the monitor detects a
change that requires a notification, it publishes the notification to the
associated SNS topic. The SNS topic for each monitor is specified in
Notifications.properties. You can specify a different SNS topic for each monitor
or use the same topic for all your monitors. You will need to create these
Amazon SNS topics yourself using the AWS Console, the AWS Toolkit for Eclipse,
or programmatically using one of the AWS SDK for Java.

Similarly to SimpleDB, the monitors interact with Amazon SNS using a set of utility functions
defined in

How to Change which Region to Monitor

The Notifications application monitors only a single region. By default, the app
monitors us-east-1. To monitor a different region, add a call to setEndpoint to
the code that creates the Amazon EC2 client. The call is shown in a comment.
Uncomment the code and edit the setEndpoint call to use your desired
region.

The code that creates the Amazon EC2 client is in the init method that is called
by the constructor for each monitor. The init method is also responsible for instantiating
the NotificationPublisher and the SDBPublisher.

For more information about setting regions, go to
AWS Region Selection
in the AWS SDK for Java Developer Guide.

Irrespective of the region that you are monitoring, the SimpleDB domains that
store the Spot Instance data are always provisioned in the us-east-1 region.

Configure, Build, and Run the Application

Because the purpose of the application is to help you monitor and manage
resources in your AWS account, you'll need an AWS account in order use
the application. Also, you will need the AWS SDK for Java because
the application accesses AWS through the class library available in the SDK.
Instructions for setting up an AWS account and the SDK are available in the
AWS SDK for Java Developer Guide.

In the file, specify AWS credentials that have sufficient
permissions to run the application. The credentials you use should be permissioned
to access Amazon EC2, Amazon SNS, and Amazon SimpleDB.

Although your root credentials would have sufficient access, we recommend that
you do not use your root credentials in the AwsCredentials.properties file. Your
root credentials provide complete access to your AWS account including
your billing information. You should always keep your
root credentials as secure as possible and use them only when necessary.

So instead of using your root credentials, we recommend that you create an IAM user with only the permissions
necessary to run the Notifications application.

When you create an IAM user, you have the option of permissioning it with
pre-made permissions templates. For the Notifications application, use the following
templates to specify permissions for Amazon EC2 and Amazon SNS.

EC2 Read Access

SNS Full Access

There is no pre-made template for Amazon SimpleDB. For this service, use the
IAM policy generator to specify the following custom policy for SimpleDB.

This line specifies which monitors should be loaded by the application at
start-up time. You cannot change the set of monitors while the application is
running; instead, you would need to change this setting and then restart
the application.

These first two settings are applicable to all the monitors. In addition,
each monitor has its own section in the Notifications.properties file.
Each monitor-specific section specifies the following properties.
(The snippets below are taken from the Spot request section of the file.)

Path to the Java Class that Implements the Monitor

# Define the path to the class to load
spotrequest.class=com.amazonaws.codesamples.notifications.monitor.spotrequest.SpotRequestMonitor

Amazon SNS topic for the Monitor

# Define the topic you would like to have notifications published to
spotrequest.topic=arn:aws:sns:us-east-1:11112222333:EC2Notifications

By default, the file is configured so that all the monitors
use the same SNS topic. However, you can specify the SNS
topic that a given monitor uses in the Notifications.properties file. By
creating separate SNS topics for each monitor and specifying those topics in
Notifications.properties, you can route different types of notifications to
different topics. This approach can be useful if you want to use
different SNS "endpoints" for different types of notifications. For example, if
the state of one of your EC2 instances changes, you might want to receive the
notification through email, but if there is a significant change in Spot
Instance pricing, you might want to receive the notification as a text message
(SMS).

We recommend that you set a display name for your SNS topic.
The display name appears in the "From:" field for email
notifications and SMS messages. A display name is required for SMS messages.

The AWS Toolkit for Eclipse does not enable you to subscribe to an Amazon SNS
topic using the SMS protocol. However, you can subscribe to SMS from the
AWS Management Console.
For information on using SMS with Amazon SNS, go to the Amazon SNS
Getting Started Guide.

Refresh Period

# Define the refresh period in seconds you would like for the instance.
spotrequest.refresh=60

This is a time period in seconds that specifies how frequently the monitor should retrieve new data.

Each monitor also defines a default refresh interval, which the monitor uses if no
refresh interval for that monitor is specified in Notifications.properties.

The interval is defined by the REFRESH_DEFAULT static variable and retrieved by
the getRefreshDefault function.

In addition to the standard monitor configuration options, for the Spot price
monitor, you need to specify which type of Amazon EC2 instances to monitor, which AZs,
and the magnitude of change required in Spot price in order to trigger a notification.
The corresponding lines in Notifications.properties have the following syntax:

You can specify the magnitude of change in three ways. By specifying "ANY" (text
is not case sensitive), any change in Spot price will trigger a notification. We
do not recommend this configuration because it tends to generate a lot of
notifications.

If you specify a pair of numbers that represent currency, you will receive a
notification if the Spot price moves into or out of this range.
For example, if the pair of numbers is $0.05 and $0.10, the application will
send a notification if the Spot price increases from $0.07 to $0.12.

If you specify a pair of percentages, the monitor determines whether the
Spot price has fallen more than the first percentage or risen more than the
second percentage. If either of these is true, the application sends a notification.

In addition to specifying the magnitude of change that will trigger a
notification, the lines in this section also specify whether the change applies to a
particular class of Spot Instances. In the following example, a notification is
sent only if the price change occurs for Spot Instances in the us-east-1a
availability zone, running on m1.small class Amazon EC2 instances, and running
the Linux operating system.

spotprice.threshold.us-east-1a.m1.small.Linux/UNIX=$0.05,$0.10

Availability Zone

The AVAILABILITY ZONE specifier is formatted as:

REGION+LETTER REPRESENTING AZ

For example:

us-west-1b

Note that the set of AZs for your selected region may not follow
a canonical ordering of letters. For example, the us-west-1 region might have
us-west-1b and us-west-1c, but not us-west-1a.
To see which AZs are available for a given region, go to the following page in
the AWS Management Console:

Also, the mapping between AZ names and actual datacenters (DC) may vary from one account to another.
For example, us-west-1b may map to a given DC for one account, but a
different DC for another account. Therefore, if your configuration for the application specifies
AZs, the configuration may not work as expected if you move it to a different account.

Instance Type

The INSTANCE TYPE is specified using the "API name" for the type of the Amazon EC2
instances to use for the deployment.

Examples are t1.micro for Micro instances or m1.xlarge for Extra Large instances. For
a list of instance types and corresponding API names, see the Amazon EC2
detail page.

Platform

This specification is either the string "Linux/UNIX" or "Windows". If you omit this constraint, the
application will monitor all Spot Instance irrespective of platforms.

How to Build and Run the Application

The application download includes an ant build file (build.xml) that you can use to build the application.
You will need to configure the class path in the ant file so that the paths to the AWS class
libraries match your environment.

To build the application, type:

ant build

This will invoke the build target from the ant file.

As part of the build, the AWSCredentials.properties and Notifications.properties
files are copied to the build output tree.

The application caches the configuration from Notifications.properties in a
SimpleDB domain which is, by default, called
AMAZON_EC2_Notifications_EC2_NOTIFICATIONS_CONFIG. Before running the
application, you should create this domain using a utility that is provided with
the application.

To run the utility to create the configuration domain, type:

ant ConfigUploader

This invokes the ConfigUploader target to run the utility.

To run the application, type:

ant NotificationsApp

This will invoke the NotificationsApp target from the ant file.

If you are using the Eclipse IDE, you can create a new Eclipse project for the
application based off of the ant file. From the Eclipse File menu, point to
New and then click Project.... In the New Project wizard, select Java Project from Existing Ant Buildfile.

Create a New Java Project from the Ant Build File

When the applications first starts, you may receive some logging diagnostics as
the application creates the SimpleDB domains that store the Spot Instance data.