Search This Blog

Loading...

Friday, January 11, 2013

IronMQ Push Queues - Reliable Message Delivery for the Cloud

IronMQ Push Queues adds notification capabilities to your queues allowing you to send messages to other systems in a scalable, flexible and cost-effective way. You simply add subscribers, which are HTTP endpoints, to your IronMQ queues then any message you post to your queue will be reliably delivered to the subscribers.

Why Push Queues?

There are many reasons where pushing a message is useful, here are a few big ones.

Let your application do the work

Instead of having processes running that pick up messages from your queue for processing, you can let your application process the queue messages simply by defining an endpoint/route in your application. When your queue receives a message, IronMQ will post the message to your application endpoint, then your application can process it. This makes it really easy to build workers since you don't have to think about much beyond your application.

More efficient for lightly used queues

Push Queues eliminate the need to poll your queues for messages, which was the only option until now. Polling for messages is fine and effective if the queue is heavily utilized and there are always messages on the queue, but can be wasteful when the queue isn’t used that much. For instance, if you pushed one message onto the queue every hour, but you need to get that message immediately, then you may poll every second which means you would check the queue and get nothing back 3,599 every hour.

More reliable message delivery if you use IronMQ as an intermediary between systems

Another reason to use push queues is to ensure reliable delivery of messages. For instance, if your application posts to an HTTP endpoint and that endpoint was down or the request failed for some reason, then what? Either the message is lost or you'd have to build in some new retry logic, logging and reporting to ensure that the message didn't just get lost in the void. But if you post the message to IronMQ and let IronMQ deliver the message, you can ensure safe delivery to the destination because IronMQ will retry failed requests and if retries don’t work, you can find which requests have failed and do something about it. IronMQ provides a much more robust solution for communicating between systems.

If you have provide webhook support in your service, consider using IronMQ to deliver those messages so you can ensure reliable delivery.

Two different types of push queues: unicast and multicast.

Unicast pushes out to a single random subscriber. This is a worker type pattern where you can have a pool of processes (subscribers), but you want your messages processed once and only once by one subscriber.

Multicast will push to all subscribers. This is a pub/sub pattern which delivers messages to all subscribers/endpoints.

The type of push queue is defined with the "push_type" parameter when you update your queue to be a push queue. Default is "multicast".

Retries

If a subscriber doesn’t return a 2XX status code, IronMQ will retry pushing the message to the subscriber for the number of retries you specify using the “retries” parameter, with a delay between each try that you also specify with the “retries_delay” parameter. The default "retries" is 3 and "retries_delay" is 60 seconds.

Checking Status

You can check the push status for each message you post and it will tell you the status and status code (http response code) for each subscriber/endpoint. For example

Example Usage

Here's an example how to use push queues. It consists of a simple ruby web app with two endpoints that are setup to receive POST requests. Here's how to test it out (you'll need an Iron.io account and a Heroku account for this).

1) Setup a simple application on Heroku to receive messages

git clone https://github.com/treeder/push_queue_demo_app.git

cd push_queue_demo_app

heroku create

heroku addons:add iron_cache (we need this for simple storage for the demo)

git push heroku master

heroku logs --tail (do this so you can watch the logs and see the incoming requests)

Grab the URL you'll see in the output of the above command and open it in your browser just to ensure it's running. It should say "I like getting messages from IronMQ!" The full code for this simple app can be found here: https://github.com/treeder/push_queue_demo_app

2) Create a queue with subscribers

Now we'll create a push queue on IronMQ with two subscribers. The subscribers are two different endpoints to the application above so you'll need the URL you got from heroku above. Here is a curl snippet to set subscribers on a queue (if the queue doesn't exist, it will be created automatically):

You'll need to replace TOKEN and PROJECT_ID with your Iron.io token and project_id and change the base url's to point to your heroku application that you got from above.

3) Post a message to the queue

Now that we've changed the queue to a push queue with multicast and set a couple subscribers on it, let's push a message to the queue and both of the endpoints on your application will receive the messages. Here is a curl example to post a message to the queue:

You should get a curl response that looks like this:

Now refresh your app in the browser and you should see that it received the messages. And you can see the requests in the heroku logs too.

4) Check delivery status (Optional)

If you want the detailed status of the delivery to each of your subscribers, you can check that too. In the curl example below, you'll need to exchange MESSAGE_ID with the id that was returned in the response above when you posted a message.

This should return a response like this:

Conclusion

Well there you have it folks, reliable message delivery built for the cloud. Please try it out and let us know what you think, we'd love your feedback.