in Cloud it all happens

AWS Lambda SQS events with Serverless Framework

On 28 JUN 2018 Amazon announced adding support for SQS events in Lambda.
This greatly simplifies development of serverless message-driven systems.
Previously, if we wanted to listen on SQS events in Lambda, we would need to poll for messages on some time intervals.
Lambda would have to be triggered by cron scheduler to check if any new messages appeared in SQS queue.
If there was nothing new then it was a waste of resources and money of course.

Now we do not need scheduler anymore and Lambda function will be automatically invoked when new message appears in SQS queue.

If you have not heard of this framework yet then you can have a look at very good documentation here.

SIDE NOTE:
At the time of writing this post the SQS events are not yet supported by released Serverless Framework ver. 1.27.3 but there is a PR ready so this should be added to the next release.
If you are impatient like me, you can pull the branch from GitHub and build it yourself:

npm install

Then you can use bin/serverless script to create and deploy project.

I can imagine many use-cases for message-driven architecture, e.g. in order processing system a message could be sent to another service after an order was placed by a customer . Customer would be immediately notified about order submitted, while the whole processing of the order could happen asynchronously (in message-driven manner).

I tried to keep this example as simple as possible and not to dive deep into any specific business scenario.

We will have only two functions:

sender - will be triggered by REST API and submit a new message to SQS queue

receiver - will process messages from SQS queue.

Let’s create a new project from aws-nodejs template:

serverless create --template aws-nodejs

This would generate two files:

serverless.yml - a defintion of new service stack on AWS (using CloudFromation underneath)

handler.js - a demo function

We can rename handler.js to receiver.js and slightly update it.

After updating the generated sources for your demo they would looks as follows:

Let me explain you what is happening here.
First, we defined our service name sqs-triggers-demo and specified in which region we want to create it (us-east-1).
Next, we need to give access to send messages to SQS queue, which will be used by sender function.

Queue name MyQueue is hard-coded here. In production code we would probably like to pass it from environment properties.
I have also hard-coded an account ID for simplicity, but we should also pass it as a property. However, syntax in this case is a bit ugly and would distract readers from the main subject. You can have a look here how to parametrize it.

Then we specify two functions: sender and receiver and handlers for them which will be coded in files sender.js and receiver.js.
Each of them will have one function named handler.

The receiver function has an SQS event defined. It will be triggered from MyQueue.
It needs to have an arn of the queue defined. In this case it’s specified by logical ID, but it could be done also this way:

Here we’re not doing more than parsing an event received from SQS and extracting HTTP body passed from sender function:

var body = event.Records[0].body;

The event contains an array Records which is an array of SQS messages. The size depends on the batch size specified for SQS event on Lambda function (batchSize in serverless.yml) and the number of events in the queue.

Then I am expecting the HTTP body to have text parameter and print it out:

console.log("text: ", JSON.parse(body).text);

Surely, it will fail if there is no such parameter, but we will test it with the right data :)

Now, we can deploy the application and print info to get the endpoint URL:

You may wonder what’s the case with Lambda scaling when we fload SQS with messages.
Amazon documentation explains it:

For Lambda functions that process Amazon SQS queues, AWS Lambda will automatically scale the polling on the queue until the maximum concurrency level is reached, where each message batch can be considered a single concurrent unit. AWS Lambda’s automatic scaling behavior is designed to keep polling costs low when a queue is empty while simultaneously enabling you to achieve high throughput when the queue is being used heavily.Here is how it works:

When an Amazon SQS event source mapping is initially enabled, or when messages first appear after a period without traffic, Lambda begins polling the Amazon SQS queue using five clients, each making long poll requests in parallel.

Lambda monitors the number of inflight messages, and when it detects that this number is increasing, it will increase the polling frequency by 20 ReceiveMessage requests per minute and the function concurrency by 60 calls per minute. As long as the queue remains busy, scale up continues until at least one of the following occurs:

The per-function concurrency limit of the function attached to the SQS queue (if any) has been reached.

Note

Account-level limits are impacted by other functions in the account, and per-function concurrency applies to all events sent to a function. For more information, see Managing Concurrency.When AWS Lambda detects that the number of inflight messages is decreasing, it will decrease the polling frequency by 10 ReceiveMessage requests per minute and decrease the concurrency used to invoke your function by 30 calls per minute.

Fortunately, there is a way to limit concurrency per lambda function with reservedConcurrency property.
E.g. if we wanted to allow only one thread per receiver function, the function definition would looks like that :