Join the Queues Using Messaging and SendGrid

When building applications there are often many tasks you require your user to perform that can take up quite a lot of time. Processing avatar images, preparing large files for download or importing data from a series of connected accounts are all tasks that, whilst they might be essential, should feel quick and seamless for the user.

Handling tasks such as these inside application logic may be fine at first, but at a certain scale, if you’re making users wait whilst their account information imports because you’ve got several processes tied up doing it, they’re going to start considering their options elsewhere and hey buddy, those options don’t include you.

One of the ways to achieve this is to pass off this work to some background processes, often called ‘workers,’ that decide what they’re doing based on information they get from a message queue.

How message queues work

This calls for a diagram (explanation below):

In their most simple form a message queue behaves like so:

Queue has a message written to it by your application. Messages are often a small payload of JSON or XML.

Messages persist until they are picked up and actioned by another script, this could be running as a CRON job.

Once complete the message should be removed from the queue.

Do you ever write yourself reminders for later on a notepad just so they’re out of the way but you won’t forget them? That’s exactly the same as using a message queue in software. Think of it in that way and there are a tonne of practical uses for them.

Writing and retrieving messages

For these examples I’m going to use HP’s Cloud Messaging Service because it’s the most recent cloud based queue I’ve come across. Don’t worry if you’re not using that though, there are many similarities between queues and messaging-as-a-service (MSGaaS) providers in terms of RESTful operation so you’ll find these translate to pretty much any other service.

Above, we pass the same authentication headers as before, only this time to the ‘messages’ endpoint of our newly created ‘my_queue’. Additionally we’re sending over some JSON as our message packet that defines a task to be completed (import_checkins) and an additional ID to use when performing that task.

Getting a task from the queue

Let’s say our service imports check-in data from Foursquare for a user on a daily basis and then sends it to them as a daily digest email via SendGrid. We would have a CRON job that runs every 24 hours calling a script called import_checkins.rb. This script would be reading from the message queue, grabbing tasks marked as ‘import_checkins’ and getting check-in data for the FourSquare ID specified.

Each task would be processed by looping through each of the returned messages. Once the script completes its work, it’s good form to tidy up afterwards so it should delete the message from the queue. In the case of HP’s Cloud Messaging Service, once messages are retrieved they are also removed leaving the latest message at the HEAD of the response. Nifty.

Note: If you’re using other queues you may need to issue a DELETE request to remove a message from the queue once it’s been dealt with

Uses with SendGrid

If you’re sending email that doesn’t need to get to the user right away; a daily digest, activities, job listings, something that sits between transactional and newsletter style mail – using a message queue to process that backlog for users could be a great way to go.

Sure, you could absolutely achieve this just by hitting your database, but consider how lightweight and out of the way a message queue might feel in comparison if all you need do is write out a task and an ID in JSON to a queue for processing later.

I also think that there’s great scope for working with our Events Notifications endpoint here as well. Particularly if you want to store that information but not process it right away.

Writing these to a queue in real time will be much faster and less process intensive than trying to receive and process them in real time.

My recommendation is that as you scale out your application, consider breaking out lengthy tasks or lower priority pieces of work out into queues and deal with them later. Think of it this way, if it’s small and doesn’t need to be done right now, queue it up. Your servers and more importantly, your users will thank you for it

Author: Martyn Davies

Martyn Davies is a Developer Evangelist at SendGrid and a creative developer based in London. He has worked in technology for over 14 years with a background in both the music industry and technology. A serial hackathon organiser, mentor and startup advisor, you’ll find him presenting, demoing, hacking and chatting at hack days, conferences and meetups in the UK & Europe on a regular basis.