We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we'll assume that you are happy to receive all cookies from this website. If you would like to change your preferences you may do so by following the instructions here

Paid-by-Request Content Delivery with AWS

06.01.2016

Mathias Münscher

What happened? AWS has a new HTTPS gateway

With the release of the AWS API Gateway in July 2015, Amazon has recently opened the gates to server-less web-apps, utilizing Lambda service as their computation component. This opens an exiting perspective for scalable microservices, as AWS offers a wide range of other applications services such as storage, transactional mails, messaging, queuing etc. The real game changer is the pricing model though: you only pay for what is transferred, stored or computed, not for the provisioning of some resource.

What's the story? Content delivery with opt-in

Our client offers static content to the public with a steep demand during it's introduction phase. While the demand is dropping again after a short period, the content still needs to be served for much longer.

The content is essentially static. However, the consumer may adapt it slightly and send it by mail to other customers. For legal reasons the user needs to prove ownership of the email address used with a double opt-in via mail.

What we need

What we have (AWS Service)

logic to control the workflow

Lambda

REST-API

API Gateway

store the content

S3

web delivery

Cloudfront CDN

serve emails

SES

Concept: The workflow with AWS

Overview elements

Left The user operates a browser and an email client.

Right The amazon cloud with its services.

Load the frontend

User He/she sends a request for a static page that points at Cloudfront.

Process the user input

Frontend The static content uses a little JS app that allows the user to adapt the content.

Frontend The respective data is send to the Api-gateway.

API Gateway The gateway validates and transforms the data (into json) before it's passed on to the Lambda function

Lambda Processes the input: Loads, manipulates and saves further data from S3

Double opt-in step 1

Lambda Subsequently, Lambda initiates the double opt-in as it triggers an email to the user

SES The AWS emailing service is called only with the essential data for the email.

Double opt-in step 2

User He/She receives an email with a verification link.

Api-Gateway Again, pre-process and passes on the data to Lambda

Lambda completes the double opt-in and executes the follow-up logic.

Lambda triggers sending of emails to a third party as the user is now authenticated - in order to prevent a manipulation of the authentication links or requiring a database for tokens we pass data between these steps encoded as signed JSON Web Tokens.

Implementation

Lambda function

Lambda functions may be written in Javascript (which we choose), Python or Java 8. AWS provides the necessary SDK. The SDK includes the API to all AWS services.

The code is executed through a dedicated handler function which gets passed an event as well as a context object. The first contains the input data (in this case provided by the API Gateway). The latter contains the execution context (fail- / success-handler, etc.)

The code has essentially the same power as any Node.js process. It can load, save, process data with the SDK or any other Node library. It only interacts with the container to

receive input

return output

return an error

However, there are more options (e.g. get meta-infos). For a quick start we suggest to have a look at Tim B's blog entry on developing with Lambda and grunt.

CORS As API Gateway requires https, which in turn requires certificate management we decided to use the AWS domain for API access. Since our static content origins from a different domain we needed to enable CORS on the API Gateway.

Output

Static error response

A lambda function fails either checked (containing a deliberate error code) or subsequently to an unhandled exception (containing the error code Process exited before completing the request). In both cases we choose to return a http 404 response to avoid revealing any information about the system.

The gateway uses a regex to assign the http response. For checked fails Lambda returns a "404" string. Hence the regex is simply 404|Process exited before completing request

To suppress any values in the response body we add a mapping template with content-type application/json and a template that only hold the mandatory json brackets: {}. An empty template or a single space would not work.

HTTP Redirect When the the user follows the verification link we redirect the request to a static response.

The gateway maps the error code only, what forces Lambda to fail in order to redirect. This is not an intuitive way, but we haven't found anything better, yet.

Lambda fails with context.done("301-redirect-distribution-sent");

The api-gateway maps respectively and returns a 301 response.

Eventually we add a static location header (as output mapping is not yet possible for a header template). Hence, if you need to redirect dynamically, you're stuck at the moment!

Error prone: It is possible to set different location for your stages by manually changing the value before you deploy.

S3 and Cloudfront CDN

S3 itself comes with basic web-delivery. Particular, you can set to every file:

Content-Type To resolve files in a REST-style manner we leave out the file-extension:http://www.domain.tld/something/about/ returns the path /something/about. To do this, we need to specifically set the content-type to e.g. text/html for the file

Access Rights needs to set to public-read

TTL/Caching might be set to max-age=0 for staging purposes

Cloudfront links a domain to one/multiple S3 buckets. We use Cloudformation to setup a CDN for staging and production:

Linked Buckets

Allowed REST MethodsGET, HEAD

Path Pattern that resolve to assets or user manipulated content

Static Error Pages are set to a /static/.. path

Default root file typically index.html

SES

Sending limitations

Amazon handles SES (for legal reasons) more restrictive than the its other services. SES implements two quotas which are region- and recipient-specific. You start out in the SES Sandbox and may apply for increased limits. The effects of exceeding the limits are detailed e.g. in the SES Blog.

Lambda: subscribing to topic Currently the gateway may only map input to a custom event if the Content-type header is set to application/json. This is not the case for SNS message, so we passed the input directly to Lambda. Subsequently the event object equals the SNS message JSON. We extract and call the value of event.SubscribeURL.

Lambda: handling messages We attempt to send the email using the SDK. If we get a throttling exception we return a 503 response to SNS.

Summary

Pay-Per-Request We consider it the concept's most revolutionary aspect that you pay for what you actually need. This is a business-side friendly perspective, even if the actual cost calculation is still quite difficult.

API-Gateway inflexible, yet The configuration of the API-Gateway is currently quite complex and hard to automate. However, we expect CloudFormation support in the foreseeable future.