Category: Amazon EC2

If you have invested in the creation of “golden” Linux images suitable for your on-premises environment, I have some good news for you.

We have extended our popular VM Import/Export feature with support for multiple Linux distributions and virtualization formats. You can import the golden images that you currently run in VMware, Xen, and Microsoft Hyper-V environments and launch the resulting AMIs (Amazon Machine Images). You can also export Linux images, just like you can do with Windows instances today.

Why Import?You can use this feature in a couple of different ways. If you have invested a lot of time and energy in the creation of a process for the generation, certification, and tracking of golden images, you can now leverage that effort as part of a full or partial migration to the AWS cloud.

You can also import images to the cloud as part of your disaster recovery plan, preferably before disaster strikes (there’s a lot of data transfer involved, so advance planning will pay off in a shorter recovery time).

If you are planning to migrate existing applications and workloads to AWS, you can use VM Import to create Amazon EC2 instances from your VMs, leaving the existing software and setings within the VMs intact.

The DetailsWe are launching with support for 64-bit Linux images in the following formats:

VMware – ESX and VMware Workstation VMDK.

Citrix Xen – VHD

Microsoft Hyper-V VHD

The images must use the Grub bootloader (the legacy version or the newer Grub 2) and a stock kernel. You should make sure that DHCP is enabled, and that any firewall rules (iptables or otherwise) will allow access to the instance once it is imported. The root filesystem must be in ext2, ext3, ext4, Btrfs, JFS, or XFS format; you can have /boot on a separate partition from /, but both must be located on the same disk.

We are supporting the following distributions and versions:

Red Hat Enterprise Linux 5.1 – 6.5

Centos 5.1 – 6.5

Ubuntu 12.04, 12.10, 13.04, 13.10

Debian 6.0.0 – 6.0.8, 7.0.0 – 7.2.0

The import process produces an AMI in HVM format. It can be launched on the following EC2 instance types:

cc1.4xlarge

cc2.8xlarge

cg1.4xlarge

cr1.8xlarge

hi1.4xlarge

hs1.8xlarge

m3.2xlarge

m3.xlarge

Once imported, the AMI resides in a particular AWS Region. You can use the Cross-Region AMI Copy feature to make copies in other Regions as needed.

Talk to UsAs is the case with every new AWS feature, we are looking forward to your suggestions and your feedback in order to help us prioritize further development. Please feel free to leave comments on this blog post or in the EC2 forum.

As you know, we launched our new compute optimized instance family (C3) a few weeks ago, and wow, are we seeing unprecedented demand across all sizes and all Regions! As one of our product managers just told me, these instances are simply “fast in every dimension.” They have a high performance CPU, matched with SSD-based instance storage and EC2’s new enhanced networking capabilities, all at a very affordable price.

We believed that this instance type would be popular, but would not have imagined just how popular they’ve been. The EC2 team took a look back and found that growth in C3 usage to date has been higher than they have seen for any other newly introduced instance type. We’re not talking about some small percentage difference here. It took just two weeks for C3 usage to exceed the level that the former fastest-growing instance type achieved in twenty-two weeks! This is why some of you are not getting the C3 capacity you’re asking for when you request it.

In the face of this growth, we have enlarged, accelerated, and expedited our orders for additional capacity across all Regions. We are working non-stop to get it in-house, and hope to be back to more normal levels of capacity in the next couple of weeks.

Amazon EC2’s Auto Scaling feature gives you the power to build systems that adapt to a workload that varies over time. You can scale out to meet peak demand, and then scale in later to minimize costs.

Today we are adding Auto Scaling support to the AWS Management Console. You can now create launch configurations and Auto Scaling groups with point-and-click ease, and you can bid for Spot Instances when scaling out. You can also initiate scaling operations from the console and you can manage the associated notifications.

Let’s take a tour of the console’s new support for Auto Scaling. The welcome page outlines the benefits and the major steps:

The launch configuration specifies the Amazon Machine Image (AMI), EC2 instance type, EBS storage, security group, and other details needed to launch new instances as part of the scale-up process. The console leads you through the necessary steps, beginning with the selection of the desired AMI:

With the AMI chosen, your next task is to choose the EC2 instance type that will be launched when scaling out:

Then you provide a name for your launch configuration, along with an IAM role, enable CloudWatch detailed monitoring, and request EBS-optimized instances. You can even choose a purchasing option (On- Demand or Spot).

If you decide to use Spot Instances, the console will show you the current price for the selected instance type in each Availability Zone. You can use this information to help you make an informed choice when you enter the maximum price that you want to pay to launch a Spot instance:

You can also request the creation of new EBS disk storage volumes as part of the launch. These volumes can be deleted on termination, or they can be left around. The first option is perfect if you use the EBS volumes for temporary storage; the second would be appropriate if you generate log files on the instance and need to move them to long-term storage after the instance has been terminated.

You can choose to attach an existing Security Group to all newly launched instances, or you can create and customize a new one.

With all of the details specified, now is the time to review them and to create the launch configuration:

As you probably know, the launch configuration provides Auto Scaling with all of the information needed to launch and terrminate EC2 instances as part of scaling operations, but it doesn’t actually launch any instances. To do that you need to create an Auto Scaling group. Click the following button to do this:

The console will lead you through the steps needed to create your Auto Scaling group. You can set the initial size (number of EC2 instances) of the group, along with the desired minimum and maximum size. You can also choose to launch the instances into a particular Virtual Private Cloud (VPC), and you can select the desired Availability Zones.

If you are using the instances to handle incoming HTTP traffic, you can also choose to associate the Auto Scaling group with an Elastic Load Balancer:

The next step is optional. If you are simply using the Auto Scaling group to ensure that a particular number of instances are up and running, you can skip it. If you want the group to vary in size in response to a changing load or to other factors, then you need to set up scaling policies.

Groups that vary in size must have a Scale Out policy and a Scale In policy. These policies are triggered by Amazon CloudWatch alarms. For example, you can activate the policies when the average CPU load (across the Auto Scaling group) rises above or drops below certain thresholds. Or, you can activate them in response to changes in the amount of network traffic to or from the instances in the group. You can even create custom CloudWatch metrics such as “Requests Per Second” and use them to initiate scaling operations.

As you can see, you can choose the actions to be taken, along with the associated quantities (number of EC2 instances) for the scale out and scale in activities:

Each Auto Scaling activity generates an Amazon SNS notification; you can route these to an existing topic or you can create a new topic and subscribe it to one or more email addresses from the console:

After you create the Auto Scaling group, you can watch the scaling history using the console

You can also initiate scale out and scale in operations

This new feature is available in all of the public AWS Regions and you can start using it today. Give it a try, and let me know what you think.

Derek Lyon sent me a really nice guest post to introduce an important new EC2 feature!

— Jeff;

I am happy to announce that Amazon EC2 now supports resource-level permissions for the RunInstances API. This release enables you to set fine-grained controls over the AMIs, Snapshots, Subnets, and other resources that can be used when creating instances and the types of instances and volumes that users can create when using the RunInstances API.

This release is part of a larger series of releases enabling resource-level permissions for Amazon EC2, so lets start by taking a step back and looking at some of the features that we already support.

EC2 Resource-Level Permission So Far In July, we announced the availability of Resource-level Permissions for Amazon EC2. Using the initial set of APIs along with resource-level permissions, you could control which users are allowed to do things like start, stop, reboot, and terminate specific instances, or attach, detach or delete specific volumes.

Since then, we have continued to add support for additional APIs, bringing the total up to 19 EC2 APIs that currently support resource-level permissions, prior to today’s release. The additional functionality that we have added allows you to control things like which users can modify or delete specific Security Groups, Route Tables, Network ACLs, Internet Gateways, Customer Gateways, or DHCP Options Sets.

We also provided the ability to set permissions based on the tags associated with resources. This in turn enabled you to construct policies that would, for example, allow a user the ability to modify resources with the tag environment=development on them, but not resources with the tag environment=production on them.

We have also provided a series of debugging tools, which enable you to test policies by making DryRun API calls and to view additional information about authorization errors using a new STS API, DecodeAuthorizationMessage.

Resource-level Permissions for RunInstances Using EC2 Resource-level Permissions for RunInstances, you now have the ability to control both which resources can be referenced and used by a call to RunInstances, and which resources can be created as part of a call to RunInstances. This enables you to control the use of the following types of items:

The AMI used to run the instance

The Subnet and VPC where the instance will be located

The Availability Zone and Region where the instance and other resources will be created

Any Snapshots used to create additional volumes

The types of instances that can be created

The types and sizes of any EBS volumes created

You can now use resource-level permissions to limit which AMIs a user is permitted to use when running instances. In most cases, you will want to start by tagging the AMIs that you want to whitelist for your users with an appropriate tag, such as whitelist=true. (As part of the whitelisting process, you will also want to limit which users have permission to the tagging APIs, otherwise the user can add or remove this tag.) Next, you can construct an IAM policy for the user that only allows them to use an AMI for running instances if it has your whitelist tag on it. This policy might look like this:

If you want to set truly fine-grained permissions, you can construct policies that combine these elements. This enables you to set fine-grained policies that do things like allow a user to run only m3.xlarge instances in a certain Subnet (i.e. subnet-1a2b3c4d), using a particular Image (i.e. ami-5a6b7c8d) and a certain Security Group (i.e. sg-11a22b33). The applications for these types of policies are far-reaching and we are excited to see what you do with them.

Because permissions are applied at the API level, any users that the IAM policy is applied to will be restricted by the policy you set, including users who run instances using the AWS Management Console, the AWS CLI, or AWS SDKs.

You can find a complete list of the resource types that you can write policies for in the Permissions section of the EC2 API Reference. You can also find a series of sample policies and use cases in the IAM Policies section of the EC2 User Guide.

Many AWS customers run CPU-bound, compute-intensive workloads on Amazon EC2, often using parallel processing frameworks such as Hadoop to distribute work and collect results. This includes batch data processing, analytics, high-performance scientific computing, 3D rendering, engineering, and simulation.

To date these needs have been met by the existing members of our compute-optimized instance families — the C1 and CC2 instance types. When compared to EC2’s general purpose instance types, the instances in this family have a higher ratio of compute power to memory.

Hello C3Today we are introducing the C3 family of droids instances. Compared to C1 instances, the C3 instances provide faster processors, approximately double the memory per vCPU and SSD-based instance storage.

As the newest member of our lineup of compute-optimized instances, the C3’s were designed to deliver high performance at an economical price. The C3 instances feature per-core performance that bests that provided by any of the other EC2 instance types, at a price-performance ratio that will make them a great fit for many compute-intensive workloads.

Use the CoresEach virtual core (vCPU) on a C3 instance type is a hardware Hyper-Thread on a 2.8 GHz Intel Xeon E5-2680v2 (Ivy Bridge) processor. There are five members of the C3 family:

Instance Name

vCPU Count

Total ECU

RAM

Local Storage

Hourly On-Demand

c3.large

2

7

3.75 GiB

2 x 16 GB SSD

$0.15

c3.xlarge

4

14

7 GiB

2 x 40 GB SSD

$0.30

c3.2xlarge

8

28

15 GiB

2 x 80 GB SSD

$0.60

c3.4xlarge

16

55

30 GiB

2 x 160 GB SSD

$1.20

c3.8xlarge

32

108

60 GiB

2 x 320 GB SSD

$2.40

Prices are for Linux instances in US East (Northern Virginia).

ProtocolsIf you launch C3 instances inside of a Virtual Private Cloud and you use an HVM AMI with the proper driver installed, you will also get the benefit of EC2’s new enhanced networking. You will see significantly higher performance (in terms of packets per second), much lower latency, and lower jitter.

Getting TechnicalAs you may have noticed, we are specifying the underlying processor type for new instance types. Armed with this information, you can choose to make use of specialized instructions or to tune your application to exploit other characteristics (e.g. cache behavior) of the actual processor. For example, the processor in the C3 instances supports Intel’s AVX (Advanced Vector Extensions) for efficient processing of vector-oriented data in 256-bit chunks.

Some NumbersIn order to measure the real-world performance of the new C3 instances, we launched a 26,496 core cluster and evaluated it against the most recent Top500 scores. This cluster delivered an Rmax of 484.18 teraflops and would land at position 56 in the June 2013 list. Notably, this is over twice the performance of the last cluster that we submitted to Top500. We also built an 8,192 cluster, which delivered an Rmax of 163.9, putting it at position 210 on the Top500 list.

Launch One NowThe C3 instances are available today in the US East (Northern Virginia), US West (Oregon), EU (Ireland), Asia Pacific (Singapore), Asia Pacific (Tokyo), and Asia Pacific (Sydney) Regions. You can choose to launch C3 instances as On-Demand, Reserved Instances, or Spot Instances.

The I2 instances are optimized for high performance random I/O. They are a great fit for transactional systems and NoSQL databases like Cassandra and MongoDB.

The instances use 2.5 GHz intel Xeon E5-2670v2 processors with Turbo mode enabled. They also benefit EC2’s new enhanced networking. You will see significantly higher performance (in terms of packets per second), much lower latency, and lower jitter when you launch these instances from within a Virtual Private Cloud (VPC).

We’ll be releasing more information at launch time. Here are the preliminary specs to tide you over until then:

Instance Name

vCPU Count

RAM

Instance Storage (SSD)

i2.large

2

15 GiB

1 x 360 GB

i2.xlarge

4

30.5 GiB

1 x 720 GB

i2.2xlarge

8

61 GiB

2 x 720 GB

i2.4xlarge

16

122 GiB

4 x 720 GB

i2.8xlarge

32

244 GiB

8 x 720 GB

The i2.8xlarge instances will be able to deliver 350,000 random read IOPS and 320,000 random write IOPS. Numbers for the other instance types will be proportionally smaller, based on the number of SSD devices associated with the instance.

Once upon a time, enterprises had a straightforward way to give each employee access to a desktop computer. New employees would join the organization and receive a standard-issue desktop, preconfigured with a common set of tools and applications. This one-size-fits all model was acceptable in the early days of personal computing, but not anymore.

Enterprise IT has been engaged in a balancing act in order to meet the needs of a diverse and enlightened user base. They must protect proprietary corporate data while giving employees the ability to work whenever and wherever they want, while using the desktop or mobile device of their choice.

Our new Amazon WorkSpaces product gives Enterprise IT the power to meet this challenge head-on. You, the IT professional, can now provision a desktop computing experience in the cloud for your users. Your users can access the applications, documents, and intranet resources that they need to get their job done, all from the comfort of their desktop computer, laptop, iPad, or Android tablet.

Let’s take a look at the WorkSpaces feature set and use cases. We’ll also take a look at it from the viewpoint of an IT professional, and then we’ll switch roles and see what it looks like from the user’s point of view.

WorkSpaces Feature SetAmazon WorkSpaces provides, as I have already mentioned, a desktop computing experience in the cloud. It is easy to provision and maintain, and can be accessed from a wide variety of client devices.

Each WorkSpaces user can install the client application on the device of their choice. After a quick download, they have access to a complete Windows 7 experience in the cloud, with persistent storage, bundled utilities and productivity applications, and access to files and other resources on the corporate intranet.

The IT professional chooses to supply each user with a given WorkSpaces Bundle. There are four standard bundles. Here are the hardware specifications for each one:

Each user has access to between 50 and 100 GB of persistent AWS storage from their WorkSpace (the precise amount depends on the bundle that was chosen for the user). The persistent storage is backed up to Amazon S3 on a regular basis, where it is stored with 99.99999999% durability and 99.99% availability over the course of a year.

Pricing is on a per-user, per-month basis, as follows:

Standard – $35 / user / month.

Standard Plus – $50 / user / month.

Performance – $60 / month.

Performance Plus – $75 / user / month.

WorkSpaces Use CasesI believe that you will find many ways to put WorkSpaces to use within your organization after you have spent a little bit of time experimenting with it. Here are a few ideas to get you started:

Mobile Workers – Allow users to access their desktops from iPads, Kindles, and Android tablets so that they can be productive while connected and on-the-go.

Secure WorkSpaces – You can meet stringent compliance requirements and still deliver a managed desktop experience to your users.

Students, Seasonal, and Temporary Workers – Provision WorkSpaces on an as-needed basis so that students, seasonal workers, temporary workers, and consultants can access the applications that they need, then simply terminate the WorkSpace when they leave.

Developers – Provide local and remote developers with the tools that they need to have in order to be productive, while ensuring that source code and other intellectual property are protected.

WorkSpaces for the IT ProfessionalLet’s take a look at Amazon WorkSpaces through the eyes of an IT professional tasked with providing cloud desktops to some new employees. All of the necessary tasks can be performed from the WorkSpaces Console:

Start by choosing a WorkSpaces profile:

Add new users by name and email address:

You can provision up to five WorkSpaces at a time. They will be provisioned in less than 20 minutes and invitations will be sent to each user via email.

As the administrator, you can manage all of your organization’s WorkSpaces through the console:

WorkSpaces for the UserOk, now let’s turn the tables and take a look at Amazon WorkSpaces from the user’s point of view!

Let’s say that your administrator has gone through the steps that I outlined above and that a new WorkSpace has been provisioned for you. You will receive an email message like this:

The email will provide you with a registration code and a link to the client download. Download the client to your device, enter the registration code, and start using your WorkSpace:

WorkSpaces delivers a Windows 7 desktop experience:

Persistent storage for the WorkSpace is mapped to the D: drive:

WorkSpaces can also be accessed from iPads, Kindles, and Android tablets. Here’s the desktop experience on the Kindle:

Behind the ScenesIf you already know a thing or two about AWS, you may be wondering what happens when you start to use Amazon WorkSpaces.

A Virtual Private Cloud (VPC) is created as part of the setup process. The VPC can be connected to an on-premises network using a secure VPN connection to allow access to an existing Active Directory and other intranet resources.

WorkSpaces run on Amazon EC2 instances hosted within the VPC. Communication between EC2 and the client is managed by the PCoIP (PC-over-IP) protocol. The client connection must allow TCP and UDP connections on port 4172, along with TCP connections on port 443.

Persistent storage is backed up to Amazon S3 on a regular and frequent basis.

Preview WorkSpacesYou can register now in order to get access to the WorkSpaces preview as soon as it is available.

The AWS Test Drives give you direct and easy access to a wide variety of enterprise solution stacks, all hosted on the AWS Cloud. These labs are available to you to run for a half-day evaluation period at no charge. Each test drive includes a guided video tour and a lab manual, so you will be up and running in a matter of minutes.

We are launching over 30 new Test Drive labs at re:Invent including offerings from Oracle, Microsoft, Infor, Sophos, Accenture, MicroStrategy, and Splunk. You can use these Test Drive labs to learn more about the newest and most sophisticated big data, security, and enterprise products, courtesy of our APN Consulting and Technology partners.

Business Intelligence with QlikViewAre you interested in learning more about Business Intelligence? Check out the QlikView lab from IPC-Global — you will have the QlikView BI client up and running within 10 minutes. You’ll be drilling into data and visualizing the results before you know it. This Test Drive is accessed as a Remote Desktop session and looks like this:

Test Drive Microsoft Applications The Test Drive program includes an entire section devoted to Microsoft products. There are new labs for SQL Server, SharePoint, and Exchange 2013 from Apparatus, 2nd Watch, InfoReliance, Booz Allen Hamilton, SPAN Systems, and Megalogix.

The Exchange Lab from Apparatus allows you to deploy a High Availability (HA) configuration across three AWS Availability Zones and be sending and receiving email within 30 minutes.

Here’s the architecture that you will evaluate when you launch the Apparatus HA Exchange test lab:

More Test DrivesI have talked about just a few of the dozens of Test Drive labs that are now available for you to use. Be sure to check out our Big Data, Microsoft, and Security labs to see our newest labs.

Have you used the AWS Marketplace? You can find, buy, and start using over 800 popular AMIs (Amazon Machine Images) in 24 categories using the Marketplace, with more products added every week.

Today we are making the AWS Marketplace even easier to use by making it accessible from within the EC2 tab of the AWS Management Console. As part of this work, we have also improved the console’s Launch Instance Wizard. Read on to learn more about both of these advances.

Marketplace IntegrationYou can now choose to search or browse the Marketplace when you click the console’s Launch Instance button by selecting the AWS Marketplace tab. You can browse through all 24 categories without having to leave the console:

You can browse through individual categories (I selected “Business Software”):

You can also enter a search term (in this case I searched for “Analytics”):

Once you find the desired package, the console will show you the pricing, system requirements, ratings, and other important information:

The console will use information supplied by the software vendor to recommend an instance type and create a new Security Group.

You can then proceed to adjust (if necessary) the instance type, finalize the other details, and launch the product, all within the AWS Management Console.

Launch Instance Wizard ImprovementsWe have improved the Console’s Launch Instance Wizard to make it even easier for you to launch EC2 instances. Searching for public and private AMIs is now instantaneous and the process of choosing instance types and security groups has been simplified. You can now copy rules from an existing security group to a new one, and there’s an auto-complete feature to streamline the process of tagging instances. Finding snapshots and creating volumes from them as part of the launch process is now faster and easier.

The console now groups related EC2 instance types together to allow you to choose the most suitable one more efficiently:

When you start to type a tag name, a popup will offer to complete it for you:

You can select an existing security group and review the rules within it. You can also copy an existing group to a new one with just one click:

You can now search for EBS volume snapshots (including Public Data Sets) when you add storage as part of an instance launch:

As part of this work we have been modernizing and fine-tuning the overall look and feel of the EC2 console. The remaining pages will be updated in the near future. If you have any suggestions, problems, or complaints, please feel free to leave a comment on this post or in the EC2 forum.

To date, we haven’t been very vocal about the performance that is possible when you combine the EC2 Cluster Instances and EBS. I’d like to change that today!

The EC2 Cluster Instance types (CC2, CR1, CG1, HI1, and HS1) support high-performance (10 gigabit) networking between instances and Elastic Block Storage volumes (EBS). Instances of this type make ideal hosts for high-performance relational and NoSQL databases. They are also great for processing workloads that require high throughput, sequential access to large amounts of data.

You can use EBS Provisioned IOPS volumes to create storage arrays that store up to tens of terabytes and provide up to 48,000 16 kilobyte IOPS when accessed from instances of the types listed above. This is equivalent to 768 megabytes per second of data transfer. You can create storage arrays that span multiple EBS volumes by using mdadm. You can use other parallel I/O techniques as well, as is most appropriate for your application and your database.

The CloudWatch graph below shows twelve EBS volumes in action on a CC2 instance, each provisioned for 4,000 IOPS and delivering a consistent 64 megabytes per second per volume for the duration of the test:

In order to achieve this throughput, the volumes were pre-warmed and optimized for queue depth as described in our EBS Volume Performance document.