[gdal-dev] Use GDAL on AWS Lambda

[gdal-dev] Use GDAL on AWS Lambda

Hello,

I am looking for any help from anyone who has experience with using GDAL on
AWS Lambda. I am trying to deploy a python function which uses GDAL onto AWS
Lambda. At the moment I am using Serverless to package and deploy my
function. I am at a roadblock because I can't figure out how to get all the
dependencies into my deployment package. My Serverless function packages the
dependencies listed in a requirements.txt using the
serverless-python-requirements plugin and dockerizePip: true option. At
deployment, I get the error:

Serverless: Installing requirements of requirements.txt in .serverless…
Serverless: Docker Image: lambci/lambda:build-python3.6
Error --------------------------------------------------
The directory ‘/.cache/pip/http’ or its parent directory is not owned by the
current user and the cache has been disabled. Please check the permissions
and owner of that directory. If executing pip with sudo, you may want sudo’s
-H flag.
The directory ‘/.cache/pip’ or its parent directory is not owned by the
current user and caching wheels has been disabled. check the permissions and
owner of that directory. If executing pip with sudo, you may want sudo’s -H
flag.
Command “python setup.py egg_info” failed with error code 1 in
/tmp/pip-install-yo9exd91/pygdal/
For debugging logs, run again after setting the “SLS_DEBUG=*” environment
variable.

I have already updated setuptools to the latest version and even included
the latest version of setuptools in requirements.txt. I think the problem is
that at deployment, Serverless does not pick up all the non-python
dependencies of GDAL. I don't know how to work around this.

I am looking for any help from anyone who has experience with using GDAL on
AWS Lambda. I am trying to deploy a python function which uses GDAL onto AWS
Lambda. At the moment I am using Serverless to package and deploy my
function. I am at a roadblock because I can't figure out how to get all the
dependencies into my deployment package. My Serverless function packages the
dependencies listed in a requirements.txt using the
serverless-python-requirements plugin and dockerizePip: true option. At
deployment, I get the error:

Serverless: Installing requirements of requirements.txt in .serverless…
Serverless: Docker Image: lambci/lambda:build-python3.6
Error --------------------------------------------------
The directory ‘/.cache/pip/http’ or its parent directory is not owned by the
current user and the cache has been disabled. Please check the permissions
and owner of that directory. If executing pip with sudo, you may want sudo’s
-H flag.
The directory ‘/.cache/pip’ or its parent directory is not owned by the
current user and caching wheels has been disabled. check the permissions and
owner of that directory. If executing pip with sudo, you may want sudo’s -H
flag.
Command “python setup.py egg_info” failed with error code 1 in
/tmp/pip-install-yo9exd91/pygdal/
For debugging logs, run again after setting the “SLS_DEBUG=*” environment
variable.

I have already updated setuptools to the latest version and even included
the latest version of setuptools in requirements.txt. I think the problem is
that at deployment, Serverless does not pick up all the non-python
dependencies of GDAL. I don't know how to work around this.

Re: Use GDAL on AWS Lambda

This repo is based on Sean Gillies's wheel build for Fiona, Rasterio, and Shapely (https://github.com/sgillies/frs-wheel-builds). I believe it has instructions, but the general idea is that it uses an old linux Docker image to build GDAL wheels for a few different Python versions. Once those are built, I use the pip --target option to install my Lambda code, any required Python modules, and the GDAL wheel into a local directory. Then I just zip up that directory, upload to S3, and run my Lambda function. The zip ends up being about 30 MB.

For example, to install the GDAL wheel into the local directory, I would do something like this:
pip install --target ./lambda_zip_dir ./wheels/GDAL-2.3.0-cp36-cp36m-manylinux1_x86_64.whl

I am looking for any help from anyone who has experience with using GDAL on
AWS Lambda. I am trying to deploy a python function which uses GDAL onto AWS
Lambda. At the moment I am using Serverless to package and deploy my
function. I am at a roadblock because I can't figure out how to get all the
dependencies into my deployment package. My Serverless function packages the
dependencies listed in a requirements.txt using the
serverless-python-requirements plugin and dockerizePip: true option. At
deployment, I get the error:

Serverless: Installing requirements of requirements.txt in .serverless…
Serverless: Docker Image: lambci/lambda:build-python3.6
Error --------------------------------------------------
The directory ‘/.cache/pip/http’ or its parent directory is not owned by the
current user and the cache has been disabled. Please check the permissions
and owner of that directory. If executing pip with sudo, you may want sudo’s
-H flag.
The directory ‘/.cache/pip’ or its parent directory is not owned by the
current user and caching wheels has been disabled. check the permissions and
owner of that directory. If executing pip with sudo, you may want sudo’s -H
flag.
Command “python setup.py egg_info” failed with error code 1 in
/tmp/pip-install-yo9exd91/pygdal/
For debugging logs, run again after setting the “SLS_DEBUG=*” environment
variable.

I have already updated setuptools to the latest version and even included
the latest version of setuptools in requirements.txt. I think the problem is
that at deployment, Serverless does not pick up all the non-python
dependencies of GDAL. I don't know how to work around this.

Re: Use GDAL on AWS Lambda

A bit late to the party, but there's also geolambda which works in a similar manner as the above solutions. It's got the latest version of GDAL and dependencies, Python 2.7 and 3.6 and packaging scripts for packaging up your code for deployment to lambda: