CL-EC2 - An Interface to Amazon's Elastic Compute Cloud

Introduction

CL-EC2 is a Common Lisp library for communicating with Amazon's EC2 and CloudWatch Query APIs. It was originally written as a foundation for Saffron Technology's Lisp-based Sierra management environment. As such, CL-EC2 does not implement the entire EC2 Query API, but rather a fairly large subset that may be used to manipulate/query images, instances, volumes, snapshots, etc. In other words, most of the useful Query API is present, at least from my perspective, but more should be included to finish off the entire API.

So far, CL-EC2 has only been tested on SBCL (I use version 1.0.30). There is a minor implementation dependency (for retrieving environment variables) that has been abstracted, so porting CL-EC2 itself should be very easy. However, the library does have dependencies on a few other open-source packages that must be supported by your Lisp platform of choice:

Documentation

Right now, generally the source code is the documentation. If you look at api.lisp and cloudwatch.lisp you'll see functions that mimic, in a Lispy fashion, those aspects of the EC2 Query API supported by CL-EC2. As the project evolves documentation will improve.

To help get you started, below is a code fragment from our Sierra environment that uses the most complicated CL-EC2 interface -- running an AMI instance. Once you've seen the code in api.lisp you'll see these operations are straightforward. The only thing to note here is that the :USER-DATA keyword parameter must be a hash table that represents key/value pairs; EC2:RUN-INSTANCES takes this table and transforms it into a comma-separated list of these pairs. I have no illusions that this approach is ideal; it simply reflects the needs of our Sierra environment, and the fact that the library was not originally intended for general consumption. This is one of probably several weak areas in the API that will change as the project evolves.

I apologize for the lack of written documentation, especially code examples. As the project moves forward I'll try and correct this.

Environment Variables

There are two environment variables you'll need to set before using the library: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. If you've any familiarity at all with EC2 you'll know what these are. Optionally, you may also set AWS_DEFAULT_AVAILABILITY_ZONE to one of the valid EC2 availability zones, if you typically use just one (as we do right now).

EC2 Query Signing

The current version of CL-EC2 uses Signature Version 2 for its query signing. There is also an implementation of Version 1; however, Amazon has apparently deprecated this version and they claim at some point it will no longer be supported.

Examples of Usage

Here are a few examples taken from our Sierra environment, which was written atop CL-EC2. Note that Sierra-specific functions are not included in CL-EC2.

Mailing Lists

Download

You can download the source code as a gzip'd tarball here. Use ASDF to build/install it.

Contributions

I would be thrilled to accept contributions from users of CL-EC2. There are aspects of the EC2 API that are unimplemented because I've had no need for them; current function interfaces are likely
not ideal for the general community; and there are probably bugs I haven't encountered yet. At some point, if contributions begin coming in, I'll get the source code into Subversion.

A Few Words on Sierra

SaffronSierra is a developer Platform-as-a-Service for Saffron Technology's Natural Intelligence Platform, including SaffronMemoryBase. Using Associative Memory technology, SaffronMemoryBase simplifies data analysis and helps uncover the "World of Experience" that lies within different kinds of data. By leveraging the SaffronMemoryBase REST APIs, it's possible to quickly find connections among people, places, things and events in data, along with their frequency counts.

Project members

David E. Young (dyoung at saffrontech dot com)

Release History and Change Log

2010-02-25 David E. Young
* Release 0.3.
* Added some additional slots to AMI-INSTANCE (architecture, private and public ip addresses).
* Added some examples to the web page.
2010-02-24 David E. Young
* Release 0.2.
* Borrowed Drakma's URL-ENCODE function and modified it to conform to the Amazon Version 2 signing specification by encoding
spaces as %20 rather than '+'.
2010-02-23 David E. Young
* Release 0.1.
* Implements EC2 API version "2009-11-30".
* Implements CloudWatch API version "2009-05-15".

Todo

The remaining pieces of the EC2 Query API need to be implemented.

The CL-EC2 API needs to be scrubbed and made accommodating, where necessary, to the general EC2 community.

If there ends up being enough interest, get the source code into Subversion.

There's an odd bug somewhere, either in CL-EC2 or on the Amazon side, that is exhibited by CW:LIST-METRICS. This query used to work just fine until about a week ago; then it began failing with a DelegationFailure code from EC2. Curiously, the Cloudwatch scratchpad tool provided by Amazon behaves the same way now (although it also used to work).