Tutorial: Scaling Container Instances with
CloudWatch Alarms

The following procedures help you to create an Auto Scaling group for an Amazon ECS
cluster. The Auto Scaling
group contains container instances that you can scale up (and down) using CloudWatch
alarms.

Depending on the Amazon EC2 instance types that you use in your clusters, and quantity
of
container instances that you have in a cluster, your tasks have a limited amount of
resources that they can use while running. Amazon ECS monitors the resources available
in the
cluster to work with the schedulers to place tasks. If your cluster runs low on any
of these
resources, such as memory, you are eventually unable to launch more tasks until you
add more
container instances, reduce the number of desired tasks in a service, or stop some
of the
running tasks in your cluster to free up the constrained resource.

In this tutorial, you create a CloudWatch alarm using the MemoryReservation metric
for your cluster. When the memory reservation of your cluster rises above 75% (meaning
that
only 25% of the memory in your cluster is available to for new tasks to reserve),
the alarm
triggers the Auto Scaling group to add another instance and provide more resources
for your tasks
and services.

Prerequisites

This tutorial assumes that you have enabled CloudWatch metrics for your clusters and
services. Metrics are not available until the clusters and services send the metrics
to
CloudWatch, and you cannot create CloudWatch alarms for metrics that do not exist
yet.

Your Amazon ECS container instances require at least version 1.4.0 of the container
agent
to enable CloudWatch metrics. For information about checking your agent version and
updating
to the latest version, see Updating the Amazon ECS Container Agent.

Your Amazon ECS container instances also require ecs:StartTelemetrySession
permission on the IAM role that you launch your container instances with. If you
created your Amazon ECS container instance role before CloudWatch metrics were available
for Amazon ECS,
then you might need to add this permission. For information about checking your Amazon
ECS
container instance role and attaching the managed IAM policy for container instances,
see To check for the
ecsInstanceRole in the IAM console.

Step 1: Create a CloudWatch Alarm for a Metric

After you have enabled CloudWatch metrics for your clusters and services, and the
metrics
for your cluster are visible in the CloudWatch console, you can set alarms on the
metrics. For
more information, see Creating Amazon
CloudWatch Alarms in the Amazon CloudWatch User Guide.

For this tutorial, you create an alarm on the cluster MemoryReservation
metric to alert when the cluster's memory reservation is above 75%.

On the Modify Alarm page, choose the
MemoryReservation metric for the default cluster and choose
Next.

In the Alarm Threshold section, enter a name and
description for your alarm.

Name:memory-above-75-pct

Description:Cluster memory reservation above 75%

Set the threshold and time period requirement to
MemoryReservation greater than 75% for 1 period.

(Optional) Configure a notification to send when the alarm is triggered. You
can also choose to delete the notification if you don't want to configure one
now.

Choose Create Alarm. Now you can use this alarm to
trigger your Auto Scaling group to add a container instance when the memory reservation
is above 75%.

(Optional) You can also create another alarm that triggers when the memory
reservation is below 25%, which you can use to remove a container instance from
your Auto Scaling group.

Step 2: Create a Launch Configuration for an Auto Scaling
Group

Now that you have enabled CloudWatch metrics and created an alarm based on one of
those
metrics, you can create a launch configuration and an Auto Scaling group for your
cluster. For
more information and other configuration options, see the Amazon EC2 Auto Scaling User Guide.

On the Create Auto Scaling Group page, choose
Create a new launch configuration.

On the Choose AMI step of the Create Auto
Scaling Group wizard, choose Community
AMIs.

Choose the ECS-optimized AMI for your Auto Scaling group.

To use the Amazon ECS-optimized Amazon Linux 2 AMI, type amzn2-ami-ecs in the
Search community AMIs field and press the
Enter key. Choose Select next to
the amzn2-ami-ecs-hvm-2.0.20190107-x86_64-ebs AMI.

The current Amazon ECS-optimized Amazon Linux 2 AMI IDs by region are listed below
for reference.

On the Choose Instance Type step of the Create
Auto Scaling Group wizard, choose an instance type for your Auto Scaling
group and choose Next: Configure details.

On the Configure details step of the Create
Auto Scaling Group wizard, enter the following information. The
other fields are optional. For more information, see Creating Launch
Configurations in the Amazon EC2 Auto Scaling User Guide.

IP Address Type: Select the IP address type
option for your container instances. To allow external traffic to be
able to reach your containers, choose Assign a public IP
address to every instance.

(Optional) If you have configuration information to pass to your container
instances with Amazon EC2 user data, choose Advanced Details and
enter your user data in the User data field. For more
information, see Amazon ECS Container Agent Configuration.

Choose Next: Add Storage.

On the Add Storage step of the Create Auto
Scaling Group wizard, make any storage configuration changes
needed for your instances and choose Next: Configure Security
Group.

On the Configure Security Group step of the
Create Auto Scaling Group wizard, select an existing
security group that meets the needs of your containers, or create a new security
group, and choose Review.

Take the action: Enter the number of instances to
add to your cluster when the alarm is triggered.

If you configured an alarm to trigger a group size reduction, set that alarm
in the Decrease Group Size section and specify how many
instances to remove if that alarm is triggered. Otherwise, collapse the
Decrease Group Size section by choosing the
X in the upper-right-hand corner of the section.

Note

If you configure your Auto Scaling group to remove container instances, any tasks
running on the removed container instances are stopped. If your tasks are
running as part of a service, Amazon ECS restarts those tasks on another instance
if the required resources are available (CPU, memory, ports). However, tasks
that were started manually are not restarted automatically.

Choose Review, Create Auto Scaling
Group.

Step 4: Verify and Test your Auto Scaling Group

Now that you've created your Auto Scaling group, you should be able to see your instances
launching in the Amazon EC2 console Instances page. These instances
should register into your Amazon ECS cluster as well after they launch.

To test that your Auto Scaling group is configured properly, create some tasks that
consume a
considerable amount of memory and start launching them into your cluster. After your
cluster exceeds the 75% memory reservation from the CloudWatch alarm for the specified
number
of periods, you should see a new instance launch in the Amazon EC2 console.

Step 5: Cleaning Up

When you have completed this tutorial, you may choose to keep your Auto Scaling group
and
Amazon EC2 instances in service for your cluster. However, if you are not actively
using
these resources, you should consider cleaning them up so your account does not incur
unnecessary charges. You can delete your Auto Scaling group to terminate the Amazon
EC2 instances
within it, but your launch configuration remains intact. You can create a new Auto
Scaling group
with the launch configuration later, if you choose.