We have added two new features to Amazon SNS to give you additional control over the content and delivery of your messages. As a brief reminder, SNS allows you to create named Topics, subscribe to Topics (with delivery via email, HTTP / HTTPS, an SMS message, or to an SQS queue), and to publish messages to Topics.

SNS Delivery PoliciesThe SNS Delivery Policies give you options to control the delivery rate and error handling for each SNS endpoint. You can, for example, use a Delivery Policy to avoid overwhelming a particular endpoint with a sudden barrage of messages.

Delivery Policies can be set for Topics and for the endpoints associated with a particular Topic. Each Delivery Policy contains a Retry Policy and a Throttle Policy. With this release, the policies are effective for the HTTP and HTTPS Subscription types.

The Retry Policy can specify the following options:

minDelayTarget - Minimum delay for a retry.

maxDelayTarget - Maximum delay for a retry.

numNoDelayRetries - Number of retries to be done with no delay (as soon as possible).

numMinDelayRetries - Number of retries to be done at minDelayTarget intervals before initiating the backoff function.

numMaxDelayRetries - Number of retries to be done at maxTargetDelay during the backoff function.

There are default, minimum, and maxium values for each option; see the SNS documentation for more information.

The Throttle Policy can specify one option:

maxReceivesPerSecond - Maximum number of delivery attempts per second per Subscription.

All attempts to deliver a message are based on an "effective Delivery Policy" which combines the default policy, any policy values set for the Topic, and any policy values set for the Subscription endpoint. Values left unspecified at the Subscription level will be inherited from the Topic's Delivery Policy and then from the default policy.

SNS Message FormattingThis feature gives you the ability to publish a message that contains content that is specific to each type of Subscription. You could, for example, send a short message to an SMS endpoint and a longer message to an email endpoint.

To use this feature, you set the new MessageStructure parameter to "json" when you call the SNS publish function. The associated message body must contain a JSON object with a default message body and optional message bodies for other protocols:

Amazon S3 is a great way to store files for the short or for the long term.

If you use S3 to store log files or other files that have a limited lifetime, you probably had to build some sort of mechanism in-house to track object ages and to initiate a bulk deletion process from time to time. Although our new Multi-Object deletion function will help you to make this process faster and easier, we want to go ever farther.

S3's new Object Expiration function allows you to define rules to schedule the removal of your objects after a pre-defined time period. The rules are specified in the Lifecycle Configuration policy that you apply to a bucket. You can update this policy through the S3 API or from the AWS Management Console.

Each rule has the following attributes:

Prefix - Initial part of the key name, (e.g. “logs/”), or the entire key name. Any object in the bucket with a matching prefix will be subject to this expiration rule. An empty prefix will match all objects in the bucket.

Status - Either Enabled or Disabled. You can choose to enable rules from time to time to perform deletion or garbage collection on your buckets, and leave the rules disabled at other times.

Expiration - Specifies an expiration period for the objects that are subject to the rule, as a number of days from the object's creation date.

Id - Optional, gives a name to the rule.

You can define up to 100 expiration rules for each of your Amazon S3 buckets; however, the rules must specify distinct prefixes to avoid ambiguity. After an Object Expiration rule is added, the rule is applied to objects that already exist in the bucket as well as any new objects added to the bucket after the rule is created. We calculate the expiration date for an object by adding that object's creation time to the expiration period and rounding off the resulting time to midnight of that day. If you make a GET or a HEAD request on an object that has been scheduled for expiration, the response will include an x-amz-expiration header that includes this expiration date and the corresponding rule Id.

We evaluate the expiration rules once each day. During this time, based on their expiration dates, any object found to be expired will be queued for removal. You will not be billed for any associated storage for those objects on or after their expiration date. If server access logging has been enabled for that S3 bucket, an S3.EXPIRE.OBJECT record will be generated when an object is removed.

You can use the Object Expiration feature on buckets that are stored using Standard or Reduced Redundancy Storage. You cannot, however, use it in conjunction with S3 Versioning (this is, as they say, for your own protection). You will have to delete all expiration rules for the bucket before enabling versioning on that bucket.

Using Object Expiration rules to schedule periodic removal of objects can help you avoid having to implement processes to perform repetitive delete operations. We recommend that you use Object Expiration for performing recurring deletions that can be scheduled, and use Multi-Object Delete for efficient one-time deletions.

If you look closely at the services and facilities provided by AWS, you'll see that we've chosen to factor architectural components that were once considered elemental (e.g. a server) into multiple discrete parts that you can instantiate and control individually.

For example, you can create an EC2 instance and then attach EBS volumes to it on an as-needed basis. This is more dynamic and more flexible than procuring a server with a fixed amount of storage.

Today we are adding additional flexibility to EC2 instances running in the Virtual Private Cloud. First, we are teasing apart the IP addresses (and important attributes associated with them) from the EC2 instances and calling the resulting entity an ENI, or Elastic Network Interface. Second, we are giving you the ability to create additional ENIs, and to attach a second ENI to an instance (again, this is within the VPC).

Each ENI lives within a particular subnet of the VPC (and hence within a particular Availability Zone) and has the following attributes:

A very important consequence of this new model (and one took me a little while to fully understand) is that the idea of launching an EC2 instance on a particular VPC subnet is effectively obsolete. A single EC2 instance can now be attached to two ENIs, each one on a distinct subnet. The ENI (not the instance) is now associated with a subnet.

Similar to an EBS volume, ENIs have a lifetime that is independent of any particular EC2 instance. They are also truly elastic. You can create them ahead of time, and then associate one or two of them with an instance at launch time. You can also attach an ENI to an instance while it is running (we sometimes call this a "hot attach"). Unless the Delete on Termination flag is set, the ENI will remain alive and well after the instance is terminated. We'll create a ENI for you at launch time if you don't specify one, and we'll set the Delete on Terminate flag so you won't have to manage it. Net-net: You don't have to worry about this new level of flexibility until you actually need it.

You can put this new level of addressing and security flexibility to use in a number of different ways. Here are some that we've already heard about:

Management Network / Backnet - You can create a dual-homed environment for your web, application, and database servers. The instance's first ENI would be attached to a public subnet, routing 0.0.0.0/0 (all traffic) to the VPC's Internet Gateway. The instance's second ENI would be attached to a private subnet, with 0.0.0.0 routed to the VPN Gateway connected to your corporate network. You would use the private network for SSH access, management, logging, and so forth. You can apply different security groups to each ENI so that traffic port 80 is allowed through the first ENI, and traffic from the private subnet on port 22 is allowed through the second ENI.

Multi-Interface Applications - You can host load balancers, proxy servers, and NAT servers on an EC2 instance, carefully passing traffic from one subnet to the other. In this case you would clear the Source/Destination Check Flag to allow the instances to handle traffic that wasn't addressed to them. We expect vendors of networking and security products to start building AMIs that make use of two ENIs.

MAC-Based Licensing - If you are running commercial software that is tied to a particular MAC address, you can license it against the MAC address of the ENI. Later, if you need to change instances or instance types, you can launch a replacement instance with the same ENI and MAC address.

Low-Budget High Availability - Attach a ENI to an instance; if the instance dies launch another one and attach the ENI to it. Traffic flow will resume within a few seconds.

Here is a picture to show you how all of the parts -- VPC, subnets, routing tables, and ENIs fit together:

I should note that attaching two public ENIs to the same instance is not the right way to create an EC2 instance with two public IP addresses. There's no way to ensure that packets arriving via a particular ENI will leave through it without setting up some specialized routing. We are aware that a lot of people would like to have multiple IP addresses for a single EC2 instance and we plan to address this use case in 2012.

Creating AlarmsYou can now create a new CloudWatch alarm from the Monitoring tab of the selected EC2 instance or EBS volume using the Create Alarm button:

Let's say I want to know when the Network Out traffic for my JeffServer instance exceeds 1.5 Megabytes within a 5 minute period (this instance hosts personal blogs for me and several members of my family, along with some other random web sites, none of which see a whole lot of traffic). I chose this number after inspecting the detailed graph for this metric on this instance:

A click of the Create Alarm button takes me to the new Create Alarm dialog. I can choose the metric and the time interval, and I can also choose the notification method:

I can choose to send notifications to an existing SNS topic, create a new topic, or to a list of email addresses. If I choose the latter option, CloudWatch will automatically create an SNS topic with a suitable name and subscribe the email addresses to the topic. The dialog displays the alarm threshold using a red line superimposed on the actual metrics data:

The Monitoring tab now displays a summary of the alarms for the selected instance (highlighting added):

The AWS team is growing rapidly and we're all doing our best to find, interview, and hire the best people for each job. In order to do my part to grow our team, I started to list the most interesting and relevant open jobs at the end of my blog posts. At first I searched our main job site for openings. I'm not a big fan of that site; it serves its purpose but the user interface is oriented toward low-volume searching. I write a lot of blog posts and I needed something better and faster.

Over a year ago I decided to scrape all of the jobs on the site and store them in a SimpleDB domain for easy querying. I wrote a short PHP program to do this. The program takes the three main search URLs (US, UK, and Europe/Asia/South Africa) and downloads the search results from each one in turn. Each set of results consists of a list of URLs to the actual job pages (e.g. Mgr - AWS Dev Support).

Early versions of my code downloaded the job pages sequentially. Since there are now 370 open jobs, this took a few minutes to run and I became impatient. I found Pete Warden's ParallelCurl and adapted my code to use it. I was now able to fetch and process up to 16 job pages at a time, greatly reducing the time spent in the crawl phase.

My code also had to parse the job pages and to handle five different formatting variations. Once the pages were parsed it was easy to write the jobs to a SimpleDB domain using the AWS SDK for PHP.

Now that I had the data at hand, it was time to do something interesting with it. My first attempt at visualization included a tag cloud and some jQuery code to show the jobs that matched a tag:

I was never able to get this page to work as desired. There were some potential scalability issues because all of the jobs were loaded (but hidden) so I decided to abandon this approach.

I gave upon the fancy dynamic presentation and generated a simple static page (stored in Amazon S3, of course) instead, grouping the jobs by city:

My code uses the data stored in the SimpleDB domain to identify jobs that have appeared since the previous run. The new jobs are highlighted in the yellow box at the top of the page.

I set up a cron job on an EC2 instance to run my code once per day. In order to make sure that the code ran as expected, I decided to have it send me an email at the conclusion of the run. Instead of wiring my email address in to the code, I created an SNS (Simple Notification Service) topic and subscribed to it. When SNS added support for SMS last month, I subscribed my phone number to the same topic.

I found the daily text message to be reassuring, and I decided to take it even further. I set up a second topic and published a notification to it for each new job, in human readable, plain-text form.

The next step seemed obvious. With all of this data in hand, I could generate a tweet for each new job. I started to write the code for this and then discovered that I was reinventing a well-rounded wheel! After a quick conversation with my colleague Matt Wood, it turned out that he already had the right mechanism in place to publish a tweet for each new job.

Matt subscribed an SQS queue to my per-job notification topic. He used a CloudWatch alarm to detect a non-empty queue, and used the alarm to fire up an EC2 instance via Auto Scaling. When the queue is empty, a second alarm reduces the capacity of the group, thereby terminating the instance.

Being more clever than I, Matt used an AWS CloudFormation template to create and wire up all of the moving parts:

The instance used to process the new job positions runs a single Ruby script, and is bootstrapped from a standard base Amazon Linux AMI using CloudFormation.

The CloudFormation template passes in a simple bootstrap script using instance User Data, taking advantage of the cloud-init daemon which runs at startup on the Amazon Linux AMI. This in turn triggers CloudFormation’s own cfn-init process, which configures the instance for use based on information in the CloudFormation template.

A collection of packages are installed via the yum and rubygems package managers (including the AWS SDK for Ruby), the processing script is downloaded and installed from S3, and a simple, YAML format configuration file is written to the instance which contains keys, Twitter configuration details and queue names used by the processing script.

At a certain point, we decided to add some geo-sophistication to the process. My code already identified the location of each job, so it was a simple matter to pass this along to Matt's code. Given that I am located in Seattle and he's in Cambridge (UK, not Massachusetts), we didn't want to coordinate any type of switchover. Instead, I simple created another SNS topic and posted JSON-formatted messages to it. This loose coupling allowed Matt to make the switch at a time convenient to him.

So, without any master plan in place, Matt and I have managed to create a clean system for finding, publishing, and broadcasting new AWS jobs. We made use of the following AWS technologies:

Tonight we have a post written by a guest blogger! Alex Dunlap is a Senior Manager on the AWS Team, responsible for Product Management for Amazon CloudFront and Amazon Route 53. Take it away, Alex...

-- Jeff;

Jeff reached out to me recently and asked if I’d be willing to provide a guest post on the AWS blog, to give you a bit of insight into what we’re up to on one of the teams that build and operate AWS services. That sounded like a great idea to me, and I’m happy to be the first of what I’m sure will be many AWS service managers who write a post here to AWS users directly. Today, I’m excited to tell you about a new feature we’re launching: support for larger objects – up to 20 GB - in Amazon CloudFront.

One of the things I like the most about my job is talking with customers and learning how they are using AWS and Amazon CloudFront, and how we can make them even better. Over the last year or so, I’ve heard from many Amazon CloudFront customers, particularly those using Amazon CloudFront for video delivery and for software downloads, that they had a need to deliver objects larger than our previous maximum file size of 5 GB.

When customers need to deliver large objects, there are two sides to the equation: the server side (in this case Amazon CloudFront) and the client side. The client side can be a video player or a file downloader app. There are lots of cool things you can do with both players and downloaders to improve reliability, performance, and user experience, but I’ll leave those for another blog post. Today, I’ll focus on what we did with Amazon CloudFront itself.

One of the first customers who contacted us asking if we could increase our maximum supported object size was GameFly, an online video game rental subscription service with over 1,500 Windows/Mac titles and more than 8,000 console games. GameFly recently announced a new digital PC client (currently in beta) and in anticipation of launching, they needed to ensure that they had a solution that would be able to handle the delivery of larger video game files that can be up to 16 GB each. They wanted to use Amazon CloudFront, but couldn’t with our limit of 5 GB per file. They told us that while they had looked at options such as splitting their files into smaller sizes, the best option for their use case would be if we could simply support larger files.

When faced with a request like this, we ask ourselves: “is there something that makes Gamefly unique? Or are they just the tip of an iceberg of what customers want?” So we reached out to even more customers to ask whether this was an important feature to add as well as to understand the size of the files they needed to deliver. They told us very clearly, yes, bigger objects in Amazon CloudFront were the way to go! 20 Gigabytes emerged as the right size to support – this addressed the needs both for high definition video and for large software download use cases. We then evaluated the technical work, made the decision to allow caching and delivery of files up to 20 GB, and started building the feature. Today, it launches and you can immediately start using Amazon CloudFront to deliver your larger files.

This is the easiest type of feature for customers to use, as it “just works.” There’s no need to configure anything special, you can just start using Amazon CloudFront for objects up to 20 GBs. This works not only with download (HTTP) distributions, but also with streaming (RTMP) of HD video files. And the feature works whether you’re using Amazon S3 as the origin or your own custom origin.

Let us know what you think about this feature, or anything else about Amazon CloudFront– we always love to hear your feedback on our services. Plus, if working in a space like this sounds interesting, we’d like to hear from you as we have several jobs open in Product Management, Software Engineering, Business Development, Sales and Operations. We’ve listed many of the open positions below.

With the paint barely dry on our US West (Oregon) Region, we are now ready to expand again. This time we are going South of the Equator, to Sao Paulo, Brazil. With the opening of this new Region, AWS customers in South and Central America can now enjoy fast, low-latency access to the suite of AWS infrastructure services.

New RegionThe new South America (Sao Paulo) Region supports the following services:

This is our eighth Region, and our first in South America (see the complete AWS Global Infrastructure Map for more information). You can see the full list in the Region menu of the AWS Management Console:

You can launch EC2 instances or store data in the new Region by simply making the appropriate selection from the menu.

New ResourcesPortions of the AWS web site are now available in Portuguese. You can switch languages using the menu in the top right:

We now have an AWS team (sales, marketing, and evangelism to start) in Brazil. Our newest AWS evangelist, Jose Papo, is based in Sao Paulo. He will be writing new editions of the AWS Blog in Portuguese and Spanish.

CustomersWe already have some great customers in Brazil. Here's a sampling:

Peixe Urbano is the leading online discount coupon site in Brazil. They launched on AWS and have scaled to a top 50 site with no capital expenditure.

Gol Airlines (one of the largest in Brazil) uses AWS to provide in-flight wireless service to customers.

The R7 news portal is one of the most popular sites in Brazil. The site makes use of CloudFront, S3, and an auto-scaled array of EC2 instances.

Orama is a financial institution with a mission of providing better access to investments for all Brazilians. They run the majority of their customer relationship systems on AWS.

Itaú Cultural is a non-profit cultural institute. The institute's IT department is now hosting new projects on AWS.

Casa & Video is one of Brazil's largest providers of electronics and home products. They have turned to AWS to handle seasonal spikes in traffic.

Solution ProvidersOur ISV and System Integrator partner ecosystem in Brazil includes global companies such as Accenture, Deloitte, and Infor along with local favorites Dedalus and CI&T.

We have added an SMTP interface to the Amazon Simple Email Service (SES) to make it even easier for you to send transactional or bulk email. We've had a lot of requests for this feature and I am confident that it will be really popular. There's a new SMTP item in the SES tab of the AWS Management Console:

This items leads to a page containing all of the information that you need to have in order to make use of SMTP with SES:

You no longer need to write any code to gain the efficiency and deliverability benefits of SES. Instead, you use the SES tab of the AWS Management Console to create an SMTP user and a set of credentials, and then you use those credentials to configure the email client of your choice:

This simple process also creates an IAM user with the appropriate permissions and policies. Credentials in hand, you can configure your mail client (Thunderbird, Outlook, and so forth) to use SMTP for mail delivery.

You will still need to verify the source ("from") addresses that you plan to use to send email, and you'll also need to request production access in order to increase your sending limits. You can initiate both of these tasks from the console:

The newest version of the SES Developer Guide includes a new section on the SMTP interface, with complete directions for configuring SMTP client apps, packaged software, application code (including a PHP + MySQL example), and server-side mail transfer agents (MTAs) including Sendmail, Postfix, and Exim.

Amazon S3's new Multi-Object Delete gives you the ability to delete up to 1000 objects from an S3 bucket with a single request. If you are planning to delete large numbers of objects from S3 then you can quickly do so by using Multi-Object Delete. You can also delete object versions (in buckets where S3 object versioning has been enabled) by including Version Ids in the request.

Net-Net: Your application will make fewer calls to S3, each call will do more work, and the application will probably run faster as a result. Overall performance gains will vary depending on network latency between your client application and Amazon S3 and the number of objects in the request.

Multi-Object Delete provides you two response modes – verbose and quiet. The default XML response to a Multi-Object Delete request will include a status element (<Deleted> or <Error>) for every object. You can enable Quiet mode in the request if all you need are the <Error> elements.