Introduction to AWS State Machines for decoupling Lambda functions

If you’re familiar with AWS (Amazon Web Services), you should be familiar with Amazon Lambda functions where you can write your serverless code. However, if you want to perform some heavy work, Lambda functions are not the most preferred choice since there are multiple limitations with regards to memory and execution time. Because of that, you might tend to use EC2 (or ECS) instances and deploy your code, because of the power it provides. Even though your code is logically decoupled (or rather less coupled), it would still be physically coupled because they run on the same machine.

If you can illustrate your algorithm in a more decoupled manner, for instance as a flowchart diagram, here is your chance to use AWS Step Functions.

What is a Step Function?

Step Function is a methodology which enables you to coordinate between your distributed micro-services (i.e. Lambda) and can easily be represented by a visual workflow.

So, we can describe a Step Function as a properly arranged micro-services which still are running independently in order to achieve a common end-goal. (You can also imagine them as small building blocks such as LEGOs).

Here are some special features and benefits you gain by using Step Functions.

You can visualize your algorithm/process in a visual form consisting of different steps/states.

You don’t have to explicitly call the next step(s) from your micro-service code.

This enables you to easily plug new micro-services and scale faster.

Also, it makes your micro-services more independent and individually testable.

Instead of (or in addition to) monitoring individual micro-services, now you can monitor them as a part of a single process.

Which means, you can monitor the final outcome of the process along with the individual micro-services.

When to use a Step Function?

When you can clearly see interdependent micro-services calling one another creating a logical flow of work.

When you can clearly see that your micro-services consist of several independent functions which can be separated into new micro-services.

When not to use a Step Function

When you don’t see a logical dependency between the input/outputs of your micro-services.

For more information on Step Functions, please follow the below links.

Getting Started

To ease the explanation, let’s think of an example scenario where you can apply a Step Function.

Simple Image Classifier

Introduction

We need to build a system which will classify a given image into either of the categories of Human, Animal or Building. It will do some dark magic (i.e. Image Processing and Machine Learning) to give a category to the image with a probability.

If the probability is less than 75%, we should fail.

If it’s a Human, it should try to process the image further and try to detect if it’s a Male, Female, Unidentifiable.

If the image is of an Animal, it should provide the closest animal type (i.e. Cat, Dog, Horse etc.).

If it’s a building, there is no further processing is needed.

If the given image doesn’t fall into either of those three categories, it should throw an error.

Upon successfully extracting the information, it should store the information in an RDS instance.

Visual Representation.

Regardless of using a Step Function or not, it’s always better to illustrate this flow as a diagram.

Logical Data Flow of the Business Requirement

Note: We will NOT talk about how to implement the logic behind classifiers.

According to the above diagram, we can create four Lambda functions for the states.

Download the image & call the generic image classifier.

Call the gender classifier.

Call the animal classifier.

Store data to RDS.

However, the Fail state shouldn’t really have code level implementation, because it represents a failure state of the flow (State Machine).

AWS Step Function

We can consider the above diagram as a representation of an AWS Step Function, where each of the nodes represents a State and the relationship between them is defined using a State Machine.

State Machine

If you’re familiar with the concept of State Machines in Computer Science, you’ll already know the basics of how an AWS State Machine works. A State Machine (specifically, a Finite State Machine) will take an input and process it using different steps according to the value passed from the previous step.

Make sure that you use the input for a state/task considering the output of the previous state.

Testing the Step Function

After you deploy your Lambdas and then the State Machine, testing is pretty straightforward. You can either use the AWS Web Console or any SDKs to call a Step Function.

In the web console, go to the Step Function page and then you will see “Start Execution” button. When you click that, you will be prompted to provide the desired input and then click on “Start Execution”.

Summary

Step Functions are a very powerful way of creating coordination between your micro-services (Lambdas). If you can visualize your business flow as a state machine, the best way is to separate the responsibilities into different Lambdas, if you already haven’t. Then you can declare a State Machine to do the coordination between them. It provides you the flexibility of replacing Lambda function easily without affecting others and enables us to individually test our Lambdas as well.