Tracking and Analyzing Emails Using Webhooks

Our application is driven by calls to action delivered to project backers via email. We send ~2 million emails a month with 98%+ deliverability.

Given the important role email plays in our business, we wanted to have better technical and behavioral insights into what was going on with our sent emails. We wanted to know more about our email at the project and backer level as well as what kind of impacts our emails have on conversion rates. Here are some of the steps we took to get a better picture:

Webhooks vs on-the-fly querying

We use Mailgun for sending email. Their platform provides two main ways to understand what your email is doing:

HTTP APIs with rolled up statistics

Event-based webhooks

We weren’t sure of our use cases for the data yet, so we decided that storing the data in our database would be the most flexible path forward.

Sending headers via ActionMailer callbacks

We instrumented the mailers of interest to include additional SMTP headers including the backer_id and project_id. ActionMailercallbacks allowed us a convenient hook into the email-sending lifecycle that looks like the below wrapper:

With this setup, we could query for emails sent to a particular backer or roll up statistics for a given project. This setup has given us much better insight into the emails we send to each user and the higher-level effect our emails are having on each particular project.

Questions we can now answer

With the above setup we can interrogate our system about email deliverability. Here are some specific queries we’re running:

Backer level deliverability

backer.sent_emails.group(:status).count

Project level deliverability

project.sent_emails.group(:status).count

Project and email type deliverability

project.sent_emails.group(:subject, :status).count

Email deliverability across our system

SentEmail.group(:status).count

Having this data in our system means we can now surface relevant statistics to our project creators about what our application is doing and how emails affect their business.

A couple of optimization improvements

With the above implementation we are seeing response times averaging around ~30ms with reasonable P99 outlier response times. Although there are a handful of improvements we could make (see below), we are cautious not to pre-optimize and have been happy with this setup.

We are continuing to monitor things. In the event of performance issues, here are a couple ideas we are considering for the future:

Enqueueing background jobs for later asynchronous processing in the webhook instead of doing Postgres queries in-band. We would be trading a Postgres query for a Redis query here.

Extracting the controller action into middleware and reducing the amount of the Rails stack required to process the action.