Menu

AWS Step Functions with Serverless

12 January 2017

Step Functions were introduced a month ago at the 2016 re:Invent conference. They offer a way to string together Lambda functions via a number of standard programming constructs. While they certainly offer benefits over gluing things together on your own, this article will discuss the gaps that you'll need to overcome to use Step Functions with Serverless apps.

Step Function Overview

First, lets start with the basics.

As mentioned, Step Functions are a way to glue together AWS Lambda functions. Step Functions provide several programming concepts such as serial execution, parallel execution, and exception handling. The neat thing about Step Functions are that they provide the building blocks often used when composing code. They provide infrastructure code that most programmers often write themselves and provide benefits such as error and state reporting along the way. These are nice things to have without having to do any additional work. As a result, I've started using them to replace traditional coding patterns.

Creating a Service

Let us use a simple example of downloading an assets (such as the current Bitcoin price from Coin Desk's price index), parsing that JSON result, and then persisting that parsed information into some data store (such as DynamoDB).

So... that's pretty ugly. That function violates the Single Responsibility Principle. This can certainly be improved by pulling out functionality into individual methods, but at the end of the day it still executes under a single Lambda context, which violates the SRP for Lambda functions.

Lets take an evolution to a better design. Breaking each function into individual Lambdas.

Individual Lambdas

Breaking apart the functionality into individual Lambdas makes a lot of sense. For starters, we get a single Lambda function for each logical piece of code. This has several benefits:

function reuse across various pieces of your application

logging and duration tracking for each functional piece of code

testing of each part in isolation is now possible

smaller footprint means you get faster loading and execution time... which translates into reduced cost

enable VPC access for only parts of code that require it

So what does it look like to split our functions? Well, lets start with the first piece, making the HTTP request to a feed.

This method will accept the input object that was generated by the parsing function and insert it into the database. Pretty cool.

Now, you may be saying to yourself, how do we glue all of this together? And you would be correct.

The downside to breaking apart these function is that we need to consider orchestration. At the 2015 re:Invent there were numerous talks on patterns for building complex applications. To list a few:

direct Lambda invocation (Lambda A blocks on call to Lambda B)

using SNS/SQS to pass events and messages

using Kinesis to pass data

using S3 to pass data

I won't focus on these because frankly, we have a solution in this article for handling the orchestration... Step Functions. They prevent us from having to write orchestration code. So let's get started with a Step Function that calls our three Lambda Functions.

We now have the pieces to create a Step Function.

Creating a Step Function

Before you can create one though, you will need to create a new IAM role that has the ability to invoke Lambda functions as shown below:

The state machine definition species the ARN for each of the Lambda functions that we previously created. You may have noticed that REGION and ACCOUNT_ID are placeholders for the actual values. You will need to modify the above template to match the actual ARNs of your functions.

At this point you can manually execute the step function by creating it through the user interface. This is a good way to test and verify that things are working.

If you're like me, you may feel dirty using the AWS UI to perform code related tasks. Unfortunately there are a few short comings.

Step Function Interactions

Step Functions aren't yet fully baked. There are some issues that make working with them challenging.

Command Line Access

Fortunately, the AWS command line tools do support step functions! You can access them via

aws stepfunctions

You'll be able to list, create, invoke, and delete Step Functions from the command line.

You will need to include the latest version of the aws-sdk as part of your function so that you have access to the sdk invocation functions. Unfortunately, the Node Lambda functions include SDK version 2.6.9 which does not have StepFunction support. This adds almost 3.5MB to your deployment.

With all that you can finally code your service to invoke your Step Function:

The above function simply invokes the named state machine. You can pass input into the state machine. In this example, it simply passes in an empty object.

If you invoke this Lambda function, it will trigger the State Machine. The Lambda will immediately complete once the State Machine is started.

Conclusion

Step Functions hold promise for simplifying Lambda interaction. There are a few hurdles that need to be jumped through, but considering this technologies is less than a month old, it's safe to assume there will be some improvements! Hopefully this article gets you started on the path of using them.