Serverless Screenshot service with Lambda

Recently a client requested a feature which involved screenshots of random urls. Now, there are several services out there which will do this for you.
Most of these services have interesting rest api’s and pricing models. I really wanted to develop something with Serverless, and took this as an opportunity to check things out. This will run on the Amazon services (eg: Lambda).

Quick installation 🚀

If you just want to launch the service yourself, you can use this magic button which will setup everything for you in your AWS account through the magic of CloudFormation:

Examples

Some example screenshots (these are scaled down, click to see original):

Note: The BBC one actually has some unicode fonts near the bottom, works like a charm :)

Overview

We’re going to create a service to which we can POST a url, it will capture and store a screenshot of the given url for us, and return a url where we can download the screenshot. Also, we want to have different thumbnail sizes of every screenshot, and we want to be able to list all the available thumbnail sizes of a url.

The architecture of what we’re going to build, looks like this:

Architecture for this project

The app posts to Api Gateway, which will trigger a Lambda function, which will take the screenshot, and upload it to the bucket. It will then return the url for the created screenshot.

When the file gets uploads (putObject) to the S3 bucket, it will trigger the Create Thumbnails function, which will take the screenshot as input, and use ImageMagick to create several thumbnail versions of the given image, and upload those to the same bucket as well.

Lastly, there will be a function which the app can call, to get a list of all available thumbnail sizes. The App can use the returned urls to present them to the end user, and their browser will download it through CloudFront from S3.

Getting Started

This is my first project in NodeJS, so bear with me (I’m more of a Python guy), any tips or PR’s are greatly appreciated 😄

When you download the project, you can just run a npm install, to get all the requirements installed, and get going. If you want to follow along from scratch, read on. Be sure to install the latest Serverless version (npm install -g serverless).

We start the project by creating a new project (in the current directory):

The handler.js file will contain our functions, and with event.json we can simulate our calls. I later moved this to an events directory, so I could simulate multiple events for the other calls.

Let’s first take a look at the serverless.yml file, which contains the configuration for our Service.

Provider configuration

First, we need to setup our provider configuration, in this case, we’ll use aws, with nodejs4.3. We also add an api key, so we can securely use our api, and add some iam rules to allow the functions to list and upload to the S3 bucket.

provider:name:awsruntime:nodejs4.3stage:dev# We want to lock down the ApiGateway, so we can control who can use the apiapiKeys:-thumbnail-api-key# We need to give the lambda functions access to list and write to our bucket, it needs:# - to be able to 'list' the bucket# - to be able to upload a file (PutObject)iamRoleStatements:-Effect:"Allow"Action:-"s3:ListBucket"-"s3:PutObject"Resource:{"Fn::Join":["",["arn:aws:s3:::",{"Ref":"ThumbnailBucket"}]]}

Functions

We need to create two functions. One for taking a screenshot (called, you guessed it: takeScreenshot), one for creating a thumbnail of the screenshots (createThumbnails), and one for listing the available screenshot sizes (listScreenshots).

functions:takeScreenshot:handler:handler.take_screenshottimeout:15# our screenshots can take a while sometimesevents:-http:path:screenshotsmethod:post# Marking the function as private will require a valid api-keyprivate:truerequest:# we want to mark the url param as requiredparameters:querystrings:url:truelistScreenshots:handler:handler.list_screenshotstimeout:15events:-http:path:screenshotsmethod:getprivate:truerequest:parameters:querystrings:url:truecreateThumbnails:handler:handler.create_thumbnailsevents:# this event type will also create the screenshots bucket and trigger# the lambda function every time a file is uploaded (ObjectCreated)-s3:bucket:${self:custom.bucket_name}event:s3:ObjectCreated:*

Custom Resources

We also need a CloudFront distribution to serve our screenshots. This is not supported by Serverless out of the box, so we need to create a custom resource for it. We also define some Output’s, so we can use these later on.

resources:Outputs:ScreenshotBucket:Description:"Screenshotbucketname"Value:${self:custom.bucket_name}CloudFrontUrl:Description:"CloudFronturl"Value:{"Fn::GetAtt":"CloudFrontEndpoint.DomainName"}Resources:# Create an endpoint for the S3 bucket in CloudFront# this configuration basically just sets up a forwarding of requests# to S3, and forces all traffic to httpsCloudFrontEndpoint:Type:AWS::CloudFront::DistributionProperties:DistributionConfig:Enabled:TrueDefaultCacheBehavior:TargetOriginId:ScreenshotBucketOriginViewerProtocolPolicy:redirect-to-httpsForwardedValues:QueryString:TrueOrigins:-Id:ScreenshotBucketOriginDomainName:${self:custom.bucket_name}.s3.amazonaws.comCustomOriginConfig:OriginProtocolPolicy:http-only

Plugin variables

We use the stage-variables plugin, so we can set stage variables in ApiGateway, which we can then use in our Lambda functions. You can configure them in the custom section. This also allows us to use CloudFormation references inside our Lambda functions. CloudFormation will fill in the values with the actual output for us.

custom:# change this, so it's unique for your setupbucket_name:${self:provider.stage,opt:stage}-${env:USER}-screenshots# these variables are passed on through ApiGateway, to be used# in your functions in the event.stageVariablesstageVariables:bucketName:${self:custom.bucket_name}endpoint:{"Fn::Join":["",["https://",{"Fn::GetAtt":"CloudFrontEndpoint.DomainName"},"/"]]}

Defining functions

In the handler.js file, you should define your functions. In our case, it will hold three functions: take_screenshot, list_screenshots and create_thumbnails. You can use the handler definition in in the functions section of serverless.yml to optionally use different files if you want to separate them. The actual implementations of the functions are in the repository.
The functions will always receive three variables: event, context and cb. The event will contain the event which triggered the Lambda (in our case, either an ApiGateway call, or the S3 bucket event). The context variable contains the Lambda context to find out how much memory you have, the platform, etc. The cb (callback) can be used to signal for error or success.

Event variables

Depending from which service you receive an event, you will receive different events. In the case of our Api Gateway events, we get a lot of stuff, headers, query parameters, stage variables, etc. In our case, we’re only using event.query and event.stageVariables, but there is also event.headers, event.body, etc. The mapping to the event is defined by Serverless, and the default mapping (and fields) can be seen in the repository.

Extra binaries

For this project, we will need some extra binaries (PhantomJS in particular), which take the screenshots. We also use ImageMagick, but that is provided by AWS by default in the Lambda image, so we don’t package that separately.
Serverless will package any extra files in the project directory automatically. So adding extra binaries is as simple as just creating a directory, and adding the files.

If you need (compiled) binaries, you can use the Amazon Lambda AMI to spin up an EC2 instance (or use this easy docker container), compile your own binaries, and copy them over to your project. The project repository already contains the binaries you can use.

Deploying

Now that we have everything in place, we can deploy the application. It’s a good idea to use different deploy environments, so let’s start with a dev environment: sls deploy -s dev. This will zip everything together (including the binaries), create a deployment bucket, upload the zip file, and update the cloudformation template with all the resources that we need.

sls deploy -s dev --verbose

(the --verbose will show you all the CloudFormation output during deployment, very nice!).

Using the Service

Now that everything is deployed, we can test our service. You can use your favorite http client like curl or postman for this.
As we configured our api’s to use api keys, you need to provide a valid api key with each request, just add a head x-api-key with each request, and it should work. You can grab the api keys from the Api Gateway console, or from the Serverless output with sls info.

Pricing

So what will this cost you? If every call would take around 10 seconds (which is a safe bet, usually they are way faster). You could make 250.000 calls per month in the free tier, which wouldn’t cost you a thing.

So creating at least 100.000 screenshots per month, with 15 thumbnails per screenshot would be absolutely free.

Conclusion

Taking screenshots is something which is perfectly fine to do with Lambda. There is no need to setup queuing, batch workers, etc. Lambda can handle the screenshots, thumbnails and storage. Whenever a request comes in, it will automatically spin up, auto scaling to whatever is your need.
Serverless makes it really easy to setup, configure and deploy your (micro)services, and now that it’s using CloudFormation, it’s really easy to extend your services with other AWS services you might need in your project.