AWS Lambda – running python bundles and arbitrary executables

In a previous post, I mentioned using Amazon Linux EC2 to create AWS Lambda compatible packages. While this works, another way to create packages that can run on AWS Lambda is to create them locally via a Docker Amazon Linux image . One downside I’ve found to this method is that sometimes these images are incompatible with some of the system files in the Lambda runtime, but at the time of writing this, I found the docker-lambda project to both create compatible lambda linux images as well as a great way to shorten lambda development cycles by emulating a lambda environment you can invoke locally.

To start, here are the instructions to build a Python 3.6 docker lambda image (of course, make sure you have Docker installed):

git clone https://github.com/lambci/docker-lambda.git # clone project from gitcd docker-lambda/# go to project directory
npm install# install project node.js dependenciescd python3.6/build # go to the python Dockerfile build
docker build . # build the image as per instructions in # the Dockerfile (takes time...)
docker images # show docker images, note the id of the built image
docker tag 32e7f5244861 lambci/python3.6:build # name and tag the built docker image using its id
docker run -it lambci/python3.6:build /bin/bash# create a new container based on new image and# run it interactively (/bin/bash command is needed# because CMD ["/bin/bash"] is not included as the# last line in the Dockerfileexit# leave docker container
docker ps-a# locate the newly created container from the above# above command, and note the name given to it
docker start -i vibrant_heyrovsky # resume interactive session with the container# using the container name found above

So, now you have a console to a compatible Amazon Linux shell. To create lambda functions, you basically zip all the relevant files and upload to AWS lambda and after that, you can remotely invoke the required function on Lambda .

My current method will be to have two console windows – one is the above console to the docker bash, and another is a console of the host operating system (whatever OS you are running Docker on). This way, you can easily zip the lambda packages in the Docker console, and then copy them from your OS console (and from there upload them to AWS Lambda)

Now that we have a local Lambda-compatible environment, let’s create an actual AWS user that will be used to upload and run the packages that we’ll create in our local Lambda-compatible Docker container.

To run the following, make sure you first have the AWS CLI installed on your OS.

Let’s create our lambda user using the above CLI. Of course, the assumption is that you already have a credentials file in your .aws directory which enable you to do the next part. If not, you’ll need to create a user with the appropriate privileges from the AWS IAM console, get that user’s aws key id and aws secret, then locally run aws configure and follow the instructions. This will create your initial credentials file.

We’ll now create a user that we’ll use for AWS lambda. The information here is based on this excellent simple tutorial with some minor changes to suit this one.

The above will return the role identifier as an Amazon Resource Name (ARN), for example: arn:aws:iam::716980512849:role/basic_lambda_role . You’ll need this ARN whenever you create a new lambda function so hold on to it.

We now have all the ingredients to create, update and invoke AWS Lambda functions. We’ll do that later, but first – let’s get back to creating the code package that is required when creating a lambda function. The code package is just a zip file which contains all your code and its dependencies that are uploaded to lambda when you create or update your lambda function. The next section will explain how to do this.

We’ll start with creating and invoking a python package that has some dependencies, and then show how to create a package that can run arbitrary executables on AWS Lambda

Creating a local Python 3.6 package

So now, let’s make a package example that will return the current time in Seoul. To do this, we’ll install a python module named arrow, but we’ll install it in a local directory since we need to package our code with this python module. To do this, open your docker console that is running the lambda compatible environment and:

cd /var/task # move to the base lambda directory in the docker image
mkdir arrowtest # Create a directory for the lambda package we're going to make
cd arrowtest # move in to the directory
pip install arrow -t ./ # install the arrow python library in this directory
ls # take a look at what has been added

Ok, so we have the python file with the lambda function, we have the dependencies, now all we need to do is zip the contents of the entire directory and add this zip file as a parameter to the lambda function creation.

This would work, however with larger Python libraries, you might want to remove certain files that aren’t being used by you python code and would just waste space on lambda. My rather primitive but effective method for doing this is cloning the complete directory and start removing files that seem pointless until something breaks, and then I put them back and try other things until I’m happy with the size reduction. In the cloned directory, I actually rename directories before removing them as it’s easier to run the script after renaming and rename them back if we see that the directory is needed by the script.

Installed python libraries can contain many directories and files of different types. There are python files, binary dynamic libraries (usually with .so extensions) and others. Knowing what these are can help decide what can be removed to make the zipped package more lean. In this example, the directory sizes are a non issue, but other python libraries can get much larger.

an example of some stuff I deleted

rm -rf *.dist-info
rm -rf *.egg-info
rm six.py
rm -rf dateutil # we're not making use of this - it's just wasting space
# test that the script is still working after all we've deleted
python arrowtest.py test
du -hd1 # we're down to 332K from 1.2MB and the script still works.

now, let’s package this directory in a zip file. if you don’t have zip installed on your docker container yet then

yum install zip

and now after removing unneeded files and dependencies, let’s pack our directory:

zip -r package.zip .

now that we have the package on the docker container, let’s copy it to our OS from our OS console:

docker cp vibrant_heyrovsky:/var/task/arrowtest_clone/package.zip .

(replace vibrant_heyrovsky with the name of your docker image).

So we have a zipped package that we tested on docker – let’s create a lambda function from this package and invoke it (replace arn:aws:iam::716980512849:role/basic_lambda_role with your own ARN):