A number of our customers want to store very large files in Amazon S3 -- scientific or medical data, high resolution video content, backup files, and so forth. Until now, they have had to store and reference the files as separate chunks of 5 gigabytes (GB) or less. So, when a customer wanted to access a large file or share it with others, they would either have to use several URIs in Amazon S3 or stitch the file back together using an intermediate server or within an application.

No more.

We've raised the limit by three orders of magnitude. Individual Amazon S3 objects can now range in size from 1 byte all the way to 5 terabytes (TB). Now customers can store extremely large files as single objects, which greatly simplifies their storage experience. Amazon S3 does the bookkeeping behind the scenes for our customers, so you can now GET that large object just like you would any other Amazon S3 object.

In order to store larger objects you would use the new Multipart Upload API that I blogged about last month to upload the object in parts. This opens up some really interesting use cases. For example, you could stream terabytes of data off of a genomic sequencer as it is being created, store the final data set as a single object and then analyze any subset of the data in EC2 using a ranged GET. You could also use a cluster of EC2 Cluster GPU instances to render a number of frames of a movie in parallel, accumulating the frames in a single S3 object even though each one is of variable (and unknown at the start of rendering) size.

The limit has already been raised, so the race is on to upload the first 5 terabyte object!

The new Mechanical Turk blog will be used to share information with Mechanical Turk Requesters, Workers, and fans of the Mechanical Turk marketplace.

It will include product announcements, how-to guides, best practices, case studies, and examples of how businesses are using Mechanical Turk for everything from transcription to content moderation to data cleansing and testing of search algorithms.

We want to make it easier for developers to build AWS applications on mobile devices. Today we are introducing new SDKs for AWS Development on devices that run the Google's Android and Apple's iOS operating systems (iPhones, iPads, and the iPod Touch).

The AWS SDK for Android and the AWS SDK for iOS provide developers with access to storage (Amazon S3), database (Amazon SimpleDB), and messaging facilities (Amazon SQS and Amazon SNS). Both SDKs are lean and mean to allow you to make the most of the limited memory found on a mobile device. The libraries take care of a number of low-level concerns such as authentication, retrying of requests, and error handling.

Both of the SDKs include libraries, full documentation and some sample code. Both of the libraries are available in source form on GitHub (iOS and Android) and we will be more than happy to accept external contributions.

In order to allow code running on the mobile device to make calls directly to AWS, we've also outlined a number of ways to store and protect the AWS credentials needed to make the calls in our new Credential Management in Mobile Applications document.

I am really looking forward to seeing all sorts of cool and creative (not to mention useful) AWS-powered applications show up on mobile devices in the future. If you build such an app, feel free to leave a comment on this post or send email to awseditor@amazon.com.

If your application needs to process, store, or transmit credit card data, you are probably familiar with the Payment Card Industry Data Security Standard, otherwise known as PCI DSS. This standard specifies best practices and security controls needed to keep credit card data safe and secure during transit, processing, and storage. Among other things, it requires organizations to build and maintain a secure network, protect cardholder data, maintain a vulnerability management program, implement strong security measures, test and monitor networks on a regular basis, and to maintain an information security policy.

I am happy to announce that AWS has achieved validated Level 1 service provider status for PCI DSS. Our compliance to PCI DSS v2 has been validated as compliant by an independent Quality Security Assessor (QSA). AWS's status as a validated Level 1 Service Provider means that merchants and other service providers now have access to a computing platform that been verified to conform to PCI standards. Merchants and services providers with a need to certify against PCD DSS and to maintain their own certification can now leverage the benefits of the AWS cloud and even simplify their own PCI compliance efforts by relying on AWS's status as a validated service provider. Our validation covers the services that are typically used to manage a cardholder environment including the Amazon Elastic Compute Cloud (EC2), the Amazon Simple Storage Service (S3), Amazon Elastic Block Storage (EBS), and the Amazon Virtual Private Cloud (VPC).

Our Qualified Service Assessor has submitted a complete Report on Compliance and a fully executed Attestation of Compliance to Visa as of November 30, 2010. AWS will appear on Visa's list of validated service providers in the near future.

Until recently, it was unthinkable to even consider the possibility of attaining PCI compliance within a virtualized, multi-tenant environment. PCI DSS version 2.0, the newest version of DSS published in late October 2010, did provide guidance for dealing with virtualization but did not provide any guidance around multi-tenant environments. However, even without multi-tenancy guidance, we were able to work with our PCI assessor to document our security management processes, PCI controls, and compensating controls to show how our core services effectively and securely segregate each AWS customer within their own protected environment. Our PCI assessor found our security and architecture conformed with the new PCI standard and verified our compliance.

Even if your application doesn't process, store, or transmit credit card data, you should find this validation helpful since PCI DSS is often viewed as a good indicator of the ability of an organization to secure any type of sensitive data. We expect that our enterprise customers will now consider moving even more applications and critical data to the AWS cloud as a result of this announcement.

Update: Many people have asked us if they need to launch some sort of special PCI compliant environment. They do not need to do so. The entire infrastructure that supports EC2, S3, EBS and VPC is compliant and there is no separate environment or special API to use. Any server or storage object deployed in these services is in a PCI compliant environment, globally.

In 1995 I registered my first domain name and put it online. Back then, registration was expensive and complex. Before you could even register a domain you had to convince at least two of your friends to host the Domain Name Service (DNS) records for it. These days, domain registration is inexpensive and simple. DNS hosting has also been simplified, but it is still a human-powered forms-based process.

Today we are introducing Amazon Route 53, a programmable Domain Name Service. You can now create, modify, and delete DNS zone files for any domain that you own. You can do all of this under full program control—you can easily add and modify DNS entries in response to changing circumstances. For example, you could create a new sub-domain for each new customer of a Software as a Service (SaaS) application. DNS queries for information within your domains will be routed to a global network of 16 edge locations tuned for high availability and high performance.

Route 53 introduces a new concept called a Hosted Zone. A Hosted Zone is equivalent to a DNS zone file. It begins with the customary SOA (Start of Authority) record and can contain other records such as A (IPV4 address), AAAA (IPV6 address), CNAME (canonical name), MX (mail exchanger), NS (name server), and SPF (Sender Policy Framework). You have full control over the set of records in each Hosted Zone.

You start out by creating a new Hosted Zone for a domain. The new zone will contain one SOA record and four NS records. Then you can post batches of changes (additions, deletions, and alterations) to the Hosted Zone. You'll get back a change id for each batch. You can poll Route 53 to verify that the changes in the batch (as identified by the change id) have been propagated to all of the name servers (this typically takes place within 60 seconds).

The zone's status will change from PENDING to INSYNC when all of the changes have been propagated. You can update your domain registration with the new nameservers at this point. Our Route 53 Getting Started Guide contains a complete guide to getting started with a new Hosted Zone.

Each record in a Hosted Zone can refer to AWS or non-AWS resources as desired. This means that you can use Route 53 to provide DNS services for any desired combination of traditional and cloud-based resources, and that you can switch back and forth quickly and easily.

You can access Route 53 using a small set of REST APIs. Toolkit and AWS Management Console support is on the drawing board, as is support for the so-called "Zone Apex" issue.

Route 53 will cost you $1 per month per Hosted Zone, $0.50 per million queries for the first billion queries per month, and $0.25 per million queries after that. Most sites typically see an order of magnitude fewer DNS queries than page views. If your site gets one million page views per month, it would be reasonable to expect about 100,000 DNS queries per month. In other words, one billion queries is a lot of queries and many sites won’t come anywhere near this number. The results of a DNS query are cached by clients. You could set a high TTL (Time to Live) on the records in your Hosted Zone in order to reduce the number of queries and the cost.

Route 53 supports up to 100 Hosted Zones per AWS account. If you need more, simply contact us and we'll be happy to help.

We just added a new BatchDeleteAttributes call to Amazon SimpleDB and you can read all about it in the SimpleDB Developer Guide. This new call will make it easier and quicker for you to delete multiple attributes or multiple items with one request. SimpleDB processes this call more efficiently than it does a series of DeleteAttribute calls, and you'll also reduce the number of requests. You can delete up to 25 attributes or items per call, and you can issue multiple calls in parallel if you are running in a threaded environment.

PS - I'm going to make good use of this call myself. Over a year ago I started a little data collection project on the side and promptly forgot about it. I created several new SimpleDB items every minute and now I have 7 or 8 million of them.

The Amazon CloudWatch team has put together a really impressive set of new features. Too many, in fact, to fit on this page. I've written a series of posts with all of the information. Here's a summary, with links to each post:

Basic Monitoring of Amazon EC2 instances at 5-minute intervals at no additional charge.

You can now use Amazon CloudWatch to monitor your EC2 instances at no additional charge. CPU load, disk I/O, and network I/O metrics are collected at five minute intervals and stored for two weeks. We call this Basic Monitoring.

You can also choose more Detailed Monitoring (one minute intervals) at a cost of $0.015 / per hour per instance.

This change does not affect the existing monitoring of Elastic Load Balancers and Relational Database Instances at one minute intervals; nor does it affect the existing monitoring of Elastic Block Storage volumes at five minute intervals.

Auto Scaling can now take advantage of the instance health information collected by the associated Elastic Load Balancer.

Specifically, once a load balancer determines that an instance is unhealthy (using the health checks that were established when the load balancer was created), Auto Scaling can be instructed to terminate the unhealthy instances and to launch replacements. This is all done within the established Auto Scaling model using a TerminateInstance scaling activity followed by a LaunchInstance activity.

You can opt in to this behavior by setting a single property (HealthCheckType) on the Auto Scaling group associated with the load balancer. You can also use the HealthCheckGracePeriod property to defer health checks for newly created instances so that they have some time to fully initialize and start responding to health checks.

The SetInstanceHealth function can be used to integrate your own health check system with Auto Scaling. You can choose whether or this call honors the grace period or not: if you need to mark an instance unhealthy, by hand, you can ignore the grace period. If you build an automated health check system, you will probably choose to honor it.

When you call DescribeAutoScalingGroups, you'll be able to see which instances are healthy and which are not. The unhealthy instances will, of course, remain in the group for a very short time.

The new CloudWatch Alarms feature allows you to watch CloudWatch metrics and to receive notifications when the metrics fall outside of the levels (high or low thresholds) that you configure. You can attach multiple Alarms to each metric and each one can have multiple actions.

Here's how they relate to each other:

A CloudWatch Alarm is always in one of three states: OK, ALARM, or INSUFFICIENT_DATA. When the metric is within the range that you have defined as acceptable, the Monitor is in the OK state. When it breaches a threshold it transitions to the ALARM state. If the data needed to make the decision is missing or incomplete, the monitor transitions to the INSUFFICIENT_DATA state.

Alarms watch metrics and execute actions by publishing notifications to Amazon SNS topics or by initiating Auto Scaling actions. SNS can deliver notifications using HTTP, HTTPS, Email, or an Amazon SQS queue. Your application can receive these notifications and then act on them in any desired way.

Actions can be set for the transition into each of the three states. The actions happen only on state transitions, and will not be re-executed if the condition persists for hours or days.

You can use the fact that multiple actions are allowed for a Alarm to send an email when a threshold is breached. This will allow you to verify that your scaling or recovery actions are triggered when expected and are working as desired.