It is time to expand the AWS footprint once again, with a new Region in Sydney, Australia. AWS customers in Australia can now enjoy fast, low-latency access to the suite of AWS infrastructure services.

CustomersOver 10,000 organizations in Australia and New Zealand are already making use of AWS. Here's a very small sample:

The Commonwealth Bank of Australia runs customer-facing web applications on AWS as part of a cloud strategy that has been underway for the past five years. The seamless scaling enabled by AWS has allowed their IT department to focus on innovation.

Brandscreen, a fast-growing Australian start-up,has developed a real-time advertising trading platform for the media industry.They use Elastic MapReduce to process vast amounts of data to test out machine learning algorithms. They store well over 1 PB of data in Amazon S3 and add another 10 TB every day.

MYOB uses AWS to host the MYOB Atlas, a simple website builder that enables businesses to be online within 15 minutes. They currently have more than 40,000 small and medium-sized businesses using Atlas on the AWS cloud.

Halfbrick Studios hosts the highly acclaimed Fruit Ninja game on AWS. They use DynamoDB and multiple Availability Zones to host tens of millions of regular players.

AWS Partner NetworkA number members of the AWS Partner Network have been preparing for the launch of the new Region. Here's a sampling (send me email with launch day updates):

Canonical is working to bring the official Ubuntu AMIs to our new Region. The latest supported images for Ubuntu Server 10.04 LTS,
11.10, 12.04 LTS and 12.10 have been migrated over. Daily images have been
turned on for the new region. The Amazon Quickstart list is also populated with
the proper image ID's.

Acquia provides hosted Drupal (again, see my interview with Acquia's Tom Erickson to learn more) to over 2,400 customers. They are working to ensure that their service will be available to customers in the new Region.

ESRI is the leading provider of Geographic Information Systems, with over one million users in more than 350,000 organizations. They are making their ArcGIS platform available in the new Region.

MetaCDN provides global cloud-based content delivery, video encoding and streaming services. They are working to ensure that their video encoding, persistent storage and delivery services will be available to customers in the new Region.

On the GroundIn order to serve enterprises, government agencies, academic institutions, small-to-mid size companies, startups, and developers, we now have offices in Sydney, Melbourne, and Perth. We will be adding a local technical support operation in 2013 as part of our global network of support centers, all accessible through AWS Support.

If you are like me, you spent this past Sunday afternoon looking forward to the landing of the Curiosity rover on Mars. I was able to wrest the remote control away from my family and change to NASA TV in time to watch through the aptly named "seven minutes of terror" as Curiosity performed an intricate series of steps that resulted in a safe, on-target landing.

The AWS team is proud to have helped to bring the video to you and to other like-minded space geeks around the globe. Now that the (red) dust has cleared, we've published a new NASA success story with full information on their AWS setup.

We've been working with NASA for several years, providing support to a number of current and future missions. The data collection and image processing tasks for the earlier Spirit and Opportunity rovers runs on AWS, coordinated by the Amazon Simple Workflow service (read the case study to learn more).

Now that Curiosity is ready to roll, the mission will use AWS to automate and accelerate the analysis of the images that it sends each day. This accelerated analysis will lead to better decision making and an increase in the amount of exploration that Curiosity can do each day.

The Curiosity rover includes a stereoscopic camera. In order to produce a finished image, each pair (left and right) of images must be warped to compensate for perspective, then stereo matched to each other, stitched together, and then tiled into a larger panorama. This complex workflow is difficult to express using a system based on message queues. Instead, the entire process was modeled and implemented using the Amazon Simple Workflow service and the AWS Flow Framework. As described in the case study, the Simple Workflow Service coordinates worker tasks that are spread across EC2 instances and JPL data centers. The worker tasks are able to participate in the workflow without regard to their location.

The AWS team is growing rapidly and we're all doing our best to find, interview, and hire the best people for each job. In order to do my part to grow our team, I started to list the most interesting and relevant open jobs at the end of my blog posts. At first I searched our main job site for openings. I'm not a big fan of that site; it serves its purpose but the user interface is oriented toward low-volume searching. I write a lot of blog posts and I needed something better and faster.

Over a year ago I decided to scrape all of the jobs on the site and store them in a SimpleDB domain for easy querying. I wrote a short PHP program to do this. The program takes the three main search URLs (US, UK, and Europe/Asia/South Africa) and downloads the search results from each one in turn. Each set of results consists of a list of URLs to the actual job pages (e.g. Mgr - AWS Dev Support).

Early versions of my code downloaded the job pages sequentially. Since there are now 370 open jobs, this took a few minutes to run and I became impatient. I found Pete Warden's ParallelCurl and adapted my code to use it. I was now able to fetch and process up to 16 job pages at a time, greatly reducing the time spent in the crawl phase.

My code also had to parse the job pages and to handle five different formatting variations. Once the pages were parsed it was easy to write the jobs to a SimpleDB domain using the AWS SDK for PHP.

Now that I had the data at hand, it was time to do something interesting with it. My first attempt at visualization included a tag cloud and some jQuery code to show the jobs that matched a tag:

I was never able to get this page to work as desired. There were some potential scalability issues because all of the jobs were loaded (but hidden) so I decided to abandon this approach.

I gave upon the fancy dynamic presentation and generated a simple static page (stored in Amazon S3, of course) instead, grouping the jobs by city:

My code uses the data stored in the SimpleDB domain to identify jobs that have appeared since the previous run. The new jobs are highlighted in the yellow box at the top of the page.

I set up a cron job on an EC2 instance to run my code once per day. In order to make sure that the code ran as expected, I decided to have it send me an email at the conclusion of the run. Instead of wiring my email address in to the code, I created an SNS (Simple Notification Service) topic and subscribed to it. When SNS added support for SMS last month, I subscribed my phone number to the same topic.

I found the daily text message to be reassuring, and I decided to take it even further. I set up a second topic and published a notification to it for each new job, in human readable, plain-text form.

The next step seemed obvious. With all of this data in hand, I could generate a tweet for each new job. I started to write the code for this and then discovered that I was reinventing a well-rounded wheel! After a quick conversation with my colleague Matt Wood, it turned out that he already had the right mechanism in place to publish a tweet for each new job.

Matt subscribed an SQS queue to my per-job notification topic. He used a CloudWatch alarm to detect a non-empty queue, and used the alarm to fire up an EC2 instance via Auto Scaling. When the queue is empty, a second alarm reduces the capacity of the group, thereby terminating the instance.

Being more clever than I, Matt used an AWS CloudFormation template to create and wire up all of the moving parts:

The instance used to process the new job positions runs a single Ruby script, and is bootstrapped from a standard base Amazon Linux AMI using CloudFormation.

The CloudFormation template passes in a simple bootstrap script using instance User Data, taking advantage of the cloud-init daemon which runs at startup on the Amazon Linux AMI. This in turn triggers CloudFormation’s own cfn-init process, which configures the instance for use based on information in the CloudFormation template.

A collection of packages are installed via the yum and rubygems package managers (including the AWS SDK for Ruby), the processing script is downloaded and installed from S3, and a simple, YAML format configuration file is written to the instance which contains keys, Twitter configuration details and queue names used by the processing script.

At a certain point, we decided to add some geo-sophistication to the process. My code already identified the location of each job, so it was a simple matter to pass this along to Matt's code. Given that I am located in Seattle and he's in Cambridge (UK, not Massachusetts), we didn't want to coordinate any type of switchover. Instead, I simple created another SNS topic and posted JSON-formatted messages to it. This loose coupling allowed Matt to make the switch at a time convenient to him.

So, without any master plan in place, Matt and I have managed to create a clean system for finding, publishing, and broadcasting new AWS jobs. We made use of the following AWS technologies:

I have made many visits to Japan over the last several years to speak at conferences and to meet with developers. I really enjoy the people, the strong sense of community, and the cuisine.

Over the years I have learned that there's really no substitute for sitting down, face to face, with customers and potential customers. You can learn things in a single meeting that might not be obvious after a dozen emails. You can also get a sense for the environment in which they (and their users or customers) have to operate. For example, developers in Japan have told me that latency and in-country data storage are of great importance to them.

Although I can't share the exact location of the Region with you, I can tell you that private beta testers have been putting it to the test and have reported single digit latency (e.g. 1-10 ms) from locations in and around Tokyo. They were very pleased with the observed latency and performance.

Existing toolkit and tools can make use of the new Tokyo Region with a simple change of endpoints. The documentation for each service lists all of the endpoints for each service.

This offering goes beyond the services themselves. We also have the following resources available:

Put it all together and developers in Japan can now build applications that respond very quickly and that store data within the country.

The JAWS-UG (Japan AWS User Group) is another important resource. The group is headquartered in Tokyo, with regional branches in Osaka and other cities. I have spoken at JAWS meetings in Tokyo and Osaka and they are always a lot of fun. I start the meeting with an AWS update. The rest of the meeting is devoted to short "lightning" talks related to AWS or to a product built with AWS. For example, the developer of the Cacoo drawing application spoke at the initial JAWS event in Osaka in late February. Cacoo runs on AWS and features real-time collaborative drawing.

We've been working with some of our customers to bring their apps to the new Region ahead of the official launch. Here is a sampling:

Zynga is now running a number of their applications here. In fact (I promise I am not making this up) I saw a middle-aged man playing Farmville on his Android phone on the subway when I was in Japan last month. He was moving sheep and fences around with rapid-fire precision!

The enStratus cloud management and governance tools support the new region.

All of the enStratus AMIs are available. The tools feature a fully localized user interface (Cloud Manager, Cluster Manager, User Manager, and Report) that can display text in English, Japanese, Korean, Traditional Chinese, and French.

enStratus also provides local currency support and can display estimated operational costs in JPY (Japan / Yen) and a number of other currencies.

Sekai Camera is a very cool augmented reality application for iPhones and Android devices. It uses the built-in camera on each device to display a tagged, augmented version of what the camera is looking at. Users can leave "air tags" at any geographical location. The application is built on AWS and makes use of a number of services including EC2, S3, SimpleDB, SQS, and Elastic Load Balancing. Moving the application to the Tokyo Region will make it even more responsive and interactive.

Cloudworks is a management tool for AWS built in Japan, and with a Japanese language user interface. It includes a daily usage report, scheduled jobs, and a history of all user actions. It also supports AWS Identity and Access Management (IAM) and copying of AMIs from region to region.

Browser 3Gokushi is a well-established RPG (Role-Playing Game) that is now running in the new region.

Here's some additional support that came in after the original post:

RightScale has been supporting our Japanese customers during the months leading up to today's release. You can read more about what they do to move to a new region in their newest blog post: RightScale Global: Next Stop Japan.

I'm very pleased to invite you all to join the AWS team in London, for our first Tech Summit of 2011. We'll take a quick, high level tour of the Amazon Web Services cloud platform before diving into the technical detail of how to build highly available, fault tolerant systems, host databases and deploy Java applications with Elastic Beanstalk.

We're also delighted to be joined by three expert customers who will be discussing their own, real world use of our services:

Do you own the rights to some interesting structured or semi-structured data? If so, you may find AWS-powered WebServius to be of interest. They make it easy for you to monetize your data by providing you the ability to sell access via pay-per-use data access APIs and a bulk download process.

Their system is optimized for use with data stored in Amazon SimpleDB. WebServius handles each aspect of monetization process including developer signup, API key management, usage metering, quote enforcement, usage-based pricing, and billing. In short, all of the messy and somewhat mundane details that you must address before you can start making money from you data. You can access the data in three forms (normal and simplified XML formats or JSON).

As a data vendor you have a lot of flexibility with your pricing. You can price your data per row and/or by column. WebServius offers a free plan for low-traffic (up to 50 subscribers or 10,000 calls to the access APIs, rate-limited to 5 calls per second) access and several usage and revenue-based pricing plans for higher traffic and/or paid access to data.

You can see WebServius in action at several sites including Mergent (historical securities pricing, company fundamentals and executives, annual reports, and corporate actions and dividends), Retailigence (retail intelligence and product data), and Compass Marketing Solutions (rich data on over 16 million business establishments in the US).

We just added a new BatchDeleteAttributes call to Amazon SimpleDB and you can read all about it in the SimpleDB Developer Guide. This new call will make it easier and quicker for you to delete multiple attributes or multiple items with one request. SimpleDB processes this call more efficiently than it does a series of DeleteAttribute calls, and you'll also reduce the number of requests. You can delete up to 25 attributes or items per call, and you can issue multiple calls in parallel if you are running in a threaded environment.

PS - I'm going to make good use of this call myself. Over a year ago I started a little data collection project on the side and promptly forgot about it. I created several new SimpleDB items every minute and now I have 7 or 8 million of them.