Menu

Adding to an SQS Queue Using AWS Lambda and a Serverless API Endpoint

Last October, I heard this crazy idea from a guy at a startup event. The gist: run an application without setting up any servers. Just set up your code on AWS (Amazon Web Services) Lambda, and set up your API using AWS API Gateway. When an API endpoint is hit, you respond by executing code in AWS Lambda. Yeah. No servers. Just use the services AWS has to complete the request (of which there are many). I was thinking, "Yeah, right, that's just nuts."

After about 4 months, I found a use case to give it a try. I wanted to set up an endpoint, and do some processing on the payload, and then put some of the data into SQS (Simple Queue Service) for processing by another service. If successful, we'd be able to shut off 2 servers and do this small task in the cloud.

Note: This looks like a long, complex article, but it's easy as pie. There are just a lot of pictures.

The project files are available on Github:

Here are the components, and what they do:

AWS API Gateway - API hosting and management service

AWS Lambda - execute code without provisioning server resources

AWS SQS (Simple Queue Service)

AWS IAM (Identity & Access Management)

This tutorial assumes you already have an AWS account, and you know a little something about SQS.

Creating the API

The name here is not part of your API url. Give it a nice name, and click Create API.

Adding a resource

Now click Create Resource. The resource creates the url path in the API.

This is how you build the url. So if you want /v1/something you need to make a v1 resource and then a something resource under that.

Since I just need an endpoint called /message, I'll make a message resource.

Enter your resource name and click Create Resource.

Now that the resource has been made, you'll see the path on the left:

The resource is nothing without a way to interact with it. I'll make a POST for posting a message to the /message endpoint.

Click the Create Method button. You'll get a little HTTP method dropdown under the resource on the left.

Select POST, and click the Checkbox to save it. Now we are presented with the options to handle the invocation of the resource using that method.

For this, I'm going to use a Lambda Function.

I'm a big fan of us-east-1 so I chose it as my region. Then a little message pops up since I don't have any lambdas yet. Click the link in the message.

Creating a Lambda

Don't worry too much about this part. It's super easy.

You'll see a big list of blueprints to start with, and you're free to select one that fits your purpose. There wasn't one for my needs so I clicked skip.

Now we'll configure the function.

Give the function a name. My lambda function will insert the API payload into a queue. So I called it enqueue.

You'll need to choose the language to write the function in. Currently AWS Lambda supports:

Node.js

Java version 8

Python 2.7

I'm using Python. Many more languages are on the roadmap.

About Python module support

I'm not going to get into too much detail here, but there are many packages supported. This includes the boto3 package, which is the go-to Python package for interacting with AWS APIs. One of the great things is that you don't need to specify any of your AWS keys, since you're working within the context of AWS.

Provide the code for your function in the text area of the page. Use the editor if your code does not require custom libraries. If you need custom libraries, you can upload your code and libraries as a .ZIP file.

You can see I'm importing boto3 and json, both included in AWS.
I set up a connection to my SQS queue, called "test-messages".
The final line encodes the event payload and adds it to the queue.

The important parts of this are:

lambda_handler - this is the function we must define that will be called when a trigger event happens. The trigger event in this case is the call to our API endpoint.

event - the payload received in the API call

context - runtime information about the call

You'll notice I'm not specifying any AWS keys. As I mentioned above:

One of the great things is that you don't need to specify any of your AWS keys, since you're working within the context of AWS.

The code in place:

Setting up the IAM Role

The second part of making the Lambda function is creating a special role for your Lambda so that it can run AWS actions on your behalf. You don't make an IAM user for this. You make a role. This page has more info, but the key part is the second paragraph:

Regardless of how your Lambda function is invoked, AWS Lambda always executes the function. At the time you create a Lambda function, you specify an IAM role that AWS Lambda can assume to execute your Lambda function on your behalf. This role is also referred to as the execution role. If your Lambda function accesses other AWS resources during execution (for example, to create an object in an Amazon S3 bucket, to read an item from a DynamoDB table, or to write logs to CloudWatch Logs), you need to grant the execution role permissions for the specific actions that you want to perform using your Lambda function.

We'll handle that now. Below where you specified the lambda code, you'll see this section:

Do as I've done here, and choose lambda_basic_execution (the recommended option).

It provides a a very basic security policy that only covers logging the Lambda activity to CloudWatch.

Edit the policy and add additional permissions as needed, as I've done here to allow for queue insertion:

Testing the API

Deploy the API

Let's take this live!

Click the Deploy API button.

I made a stage called production. For business purposes, you will likely have a development stage, testing stage, pre-production stage, and production stage. Each of this will have its own URL, and you can deploy changes to your API to one stage without affecting the other stages.

Once you click Deploy, your API is up and ready for traffic.

The base URL of your new API endpoint is in blue at the top.

The API url will have this pattern:

https://YOUR_API_ID.execute-api.THE_REGION.amazonaws.com/STAGE_NAME

To get the path of your endpoint to hit, click to open the tree on the left.

If you get null as a response, it's probably fine. Just check to make sure the Lambda did what you expected. In my case I checked the queue to make sure the payload from the call was in it.

If you get an error about a missing authorization token, make sure to re-deploy the API to pick up any changes you've made. It is a general error, so it can also mean you're hitting a 404.

For a very simple Lambda for testing a GET call, add another method to your resource, with this as the Lambda:

def lambda_handler(event, context):
return 'hello there'

Then hit that endpoint with a GET (by just putting the url in your browser) and you'll see "hello there"

API Keys

By default, your API is open to the public (if they know the full url path). You can lock it down by creating API keys and setting the API to always require a key. I won't cover that here.

Caveats

Lambda Container Initialization

When a Lambda function starts up, it is run in a container, so it can take up to 2 seconds to execute because the container has to start running. At scale, you shouldn't run in to this problem, because the container will not shut down immediately. More API calls soon after will keep the container "warm" so it will be very responsive. But an idle function will close down after a few minutes. Then you would run into that container start lag on the next invocation. If this is a problem and you have sporadic traffic, you can set up a process to periodically hit your endpoint to keep it warm.

Pricing

You're only billed for the time your Lambda function is running. You are billed in 100 ms increments. There is a free tier, so it's extremely affordable, but keep your eye on the bills. AWS Lambda Pricing

Want to Know More?

This AWS webinar from December 15, 2015 was a nice introduction to the services listed above: