How to use AWS CLI within a Lambda function (aws s3 sync from Lambda)

January 19, 2019 · 6 min read

A step-by-step process to enable AWS CLI within an AWS Lambda function.

In general, when you want to use AWS CLI in Lambda, it’s best to call AWS APIs directly by using the appropriate SDK from your function’s code. But there’s one specific use case that is easier to do with the AWS CLI than API: aws s3 sync. This post will show how to enable AWS CLI in the Lambda execution environment.

Motivation

If you want to copy files from S3 to the Lambda environment, you’d need to recursively traverse the bucket, create directories, and download files. This is not fun to build and debug.

Instead, the same procedure can be accomplished with a single-line AWS CLI command s3 sync that syncs the folder to a local file system. It also allows additional flags like --exclude to restrict what gets synced.

However, the Lambda execution environment doesn’t have the AWS CLI pre-installed and neither can you install it using pip.

Solution

To get the CLI into Lambda, we will create a Lambda Layer that contains all the necessary dependencies and make it available to the function.

Install AWS CLI in a local virtual environment

Package AWS CLI and all its dependencies to a zip file

Create a Lambda Layer

Test if it works

Step-by-step guide

Note: These instructions were created for MacOS and Linux. If you run Windows, instructions will still work but you might need to replace bash variables with hardcoded directory names.

AWS CLI is a collection of Python scripts that use the boto3 library. For this guide to work, you’ll need Python3 and pip installed on your local machine. You will also need virtualenv. Refer to the installation guide if you don’t have virtualenv installed.

You can verify if you already have these tools by checking their version from the terminal.

If you value efficiency, steps 1 through 7 are also available as a shell script in my github repo aws-cli-lambda.

git clone https://github.com/ilyabezdelev/aws-cli-lambda.git

Let’s get going.

1. Prepare a directory & environment variables

Open the terminal, create a working directory for this exercise and cd into it.

Run the following commands in the terminal.

# Automatically detects python version (only works for python3.x)exportPYTHON_VERSION=`python3 -c 'import sys; version=sys.version_info[:3]; print("{0}.{1}".format(*version))'`# Temporary directory for the virtual environmentexportVIRTUAL_ENV_DIR="awscli-virtualenv"# Temporary directory for AWS CLI and its dependenciesexportLAMBDA_LAYER_DIR="awscli-lambda-layer"# The zip file that will contain the layerexportZIP_FILE_NAME="awscli-lambda-layer.zip"

2. Create a virtual environment

To get all AWS CLI dependencies conveniently stored in a separate directory, we’ll create an isolated virtual environment using virtualenv and activate it.

Your terminal prompt will now indicate that you’re in a virtual envioronment. The prompt should be prepended with (awscli-virtualenv) like this:

(awscli-virtualenv) mymac:bin username$

3. Install AWS CLI in the virtual environment

The following command will install AWS CLI and all its dependencies to the virtual environment.

pip install awscli

4. Change the path to python

The AWS CLI executable is the file named aws and its first line provides the path to the Python interpreter. Since we’re using a virtual environment, it will have a path local to the environment we created: #!/PATH_TO_WORKING_DIR/awscli-virtualenv/bin/python3.7.

We need to replace this path with the path to the Python interpreter in the Lambda execution environment, which is #!/var/lang/bin/python.

To replace the path to Python in the aws executable, you can run this command on MacOS.

sed -i '' "1s/.*/\#\!\/var\/lang\/bin\/python/" aws

… or this command on Linux, which has slightly different syntax because the implementation of sed on MacOS differs from that on Linux.

sed -i "1s/.*/\#\!\/var\/lang\/bin\/python/" aws

Alternatively, you can open aws in a text editor and replace the first line by hand.

5. Deactive virtualenv

Now we’ll deactivate the virtual environment.

deactivate

The terminal prompt should have switched back to its normal state (no more (awscli-virtualenv) in the prompt).

6. Package AWS CLI and its depenedencies in a zip file

The following set of commands will prepare a temporary directory for packaging AWS CLI, copy all dependencies, and zip them.

# Changes current directory back to where it startedcd ../..
# Creates a temporary directory to store AWS CLI and its dependencies
mkdir ${LAMBDA_LAYER_DIR}# Changes the current directory into the temporary directorycd${LAMBDA_LAYER_DIR}# Copies aws and its dependencies to the temp directory
cp ../${VIRTUAL_ENV_DIR}/bin/aws .
cp -r ../${VIRTUAL_ENV_DIR}/lib/python${PYTHON_VERSION}/site-packages/ .
# Zips the contents of the temporary directory
zip -r ../${ZIP_FILE_NAME} *

7. Clean up

Remove awscli-virtualenv and awscli-lambda-layer directories manually or with the following commands.

8. Test

Go to the Lambda console, click on Layers in the left navigation and follow the wizard to create a layer. Upload awscli-lambda-layer.zip.

Create a Lambda function using the same version of Python that was used for packaging AWS CLI. Set timeout to 15 seconds and memory limit to 512 MB (I found AWS CLI to be a little too slow in functions with less than 512 MB of memory).

After the function is created, in Designer, click on Layers, click Add layer and select the layer you’ve just created.

Paste the following code and execute the function by clicking the Test button (create a test event from the HelloWorld template when asked). The function should run in under 5 seconds and return AWS CLI’s version in logs.