Search This Blog

Troubles in US-East, migrating to US-West

I've been having increasing amounts of problems with my servers in the US-East-1 of Amazon Web Services. Fearful that this may be some fallout or aftershock of the Great EC2 Outage of 2011, I quickly sprung into action, migrating my servers over to US-West-1 (located in California).

It's important to note, however, that this had almost no impact on any of my production systems, as I already base my core services in two regions. These services continued to operate taking the full load of the system while us-east was having problems.

However, I took this as an opportunity to review my disaster recovery plan, and build up a new image that automatically updates from bitbucket repositories using an SSH key on reboot.

I launched my old instance, updated it quickly, and then used the following commands to bundle it from within the instance (as root):

Note that your bucket must be in the same region that you're bundling from. This means that if you want to actually migrate that manifest to another region, you have to use the ec2-migrate-manifest tool:

And then you can re-upload it to a us-west-1 bucket using the ec2-upload-bundle command above. When you do this, you will be asked to make sure you want to upload to the other region, since it's not in the current region. Just say "y" for this since you've already migrated it using the ec2-migrate-manifest tool. This will take significantly longer since you're submitting cross-region

Lastly, you'll need to register this image with EC2 (or one copy in each region if you uploaded it to each region). I like to do this using boto:

Popular Posts

Ever wonder how sites like battle.net support things like this in Google Chrome?

Well I did, so I did a little bit of digging. It turns out Google Chrome supports an open standard called Open Search. This format is relatively simple, and very easy to add to your own site. I just added it to some of our systems in under 5 minutes.

Adding OpenSearch to your site is incredibly simple, you just have to add a simple tag to your index HTML page, and add a simple XML file that it points to. The link tag looks like this:
<link rel="search" type="application/opensearchdescription+xml" href="http://my-site.com/opensearch.xml" title="MySite Search" />

For a while, I have been creating command line tools provided right with boto which I used to manage AWS. Recently, others have become interested in these tools as well, and I've seen several other contributors adding to these tools to make them even more useful to others. One recent submission by Ales Zoulek added some nice features to my list_instances command, which I use on a regular basis to list out the instances that are currently active for my account in EC2.

Amazon now lets you add Tags to EC2 objects such as Instances and Snapshots. This allows you to actually "Name" your EC2 instance, as well as add some metadata that could be used for AMI initialization, etc. Ales added the ability to list these tags by name within the list_instances command line application:

Last week, Amazon announced the launch of a new product, DynamoDB. Within the same day, Mitch Garnaat quickly released support for DynamoDB in Boto. I quickly worked with Mitch to add on some additional features, and work out some of the more interesting quirks that DynamoDB has, such as the provisioned throughput, and what exactly it means to read and write to the database.

One very interesting and confusing part that I discovered was how Amazon actually measures this provisioned throughput. When creating a table (or at any time in the future), you set up a provisioned amount of "Read" and "Write" units individually. At a minimum, you must have at least 5 Read and 5 Write units partitioned. What isn't as clear, however, is that read and write units are measured in terms of 1KB operations. That is, if you're reading a single value that's 5KB, that counts as 5 Read units (same with Write). If you choose to operate in eventually consistent mode, you'r…