Search This Blog

Wednesday, July 24, 2013

You can now add queues and workers as subscribers to your IronMQ queues so now a single message post can be pushed onto multiple queues and/or start IronWorker tasks. This is often referred to as "fan-out" and opens up a lot of interesting possibilities such as:

Multiple consumers of a single message.

Better decoupling by adding queues as subscribers that different consumers can consume without the producer of the messages having to know about it.

Better performance since you only need to push one message then let IronMQ deal with distributing it to multiple queues.

Here's a simple diagram showing this pattern:

In this example, the queue called "Q1" has three other queues as subscribers, Q2, Q3, and Q4. Whenever a message is put on Q1, that message will be duplicated and pushed to all of it's subscribers which in this care are the three other queues.

How To Add Queues and Workers as Subscribers

Using the new subscriber types is the same way you do with http subscribers, but the URL scheme is different. For instance, to post to an IronMQ queue, you can add a subscriber with a URL like:

ironmq:///my_awesome_queue

This URL references a queue called "my_awesome_queue" and if this is one of your subscribers, a message put on the parent queue will be pushed to "my_awesome_queue" as well.

And if you want an IronWorker task to start, you can use an IronWorker url like:

ironworker:///my_awesome_worker

You pass these in to the queue update endpoint or the add/remove subscribers endpoints with json like the following:
That would set 5 subscribers on your queue, 3 subscribers are other queues, 1 is a worker and 1 is a normal http endpoint subscriber.

Conclusion

Being able to fan-out messages into other queues enables a lot of interesting messaging scenarios and we hope it can help solve some of your messaging problems. Give it a try and let us know what you think, we love feedback!

Tuesday, July 23, 2013

One of the most common feature requests we get for IronWorker is the ability to view logs in real-time while your workers are running. Until now, you had to wait for your IronWorker tasks to finish before being able to view the log and it can be pretty frustrating to try to debug your workers if you have to wait for a task to complete to see the log — especially if it's a task that takes a long time to finish.

Now you can send your log output to any syslog endpoint so you can see it in real-time. You can run your own syslog server with something like syslogd or Splunk, or you can use a "logging as a service" provider such as Papertrail or Loggly. I'll show you how to use Papertrail in this post.

How to Setup Real-time Remote Logging with Papertrail

First get your Papertrail log destination host/port:

Then go into your project settings page in HUD and enter the syslog URL, for example: udp://logs.papertrailapp.com:12345.

Click Update and that's it! Now all the output from your workers will be sent to both your IronWorker logs as well as streamed in real-time to the provided UDP address.

Here's a post of ours on using IronMQ to increase scale that recently appeared on the Rackspace Developers DevOps blog. Definitely worth checking out if you're looking to scale out background processing and quicken the user response loop.

Using IronMQ for Delayed Processing and Increasing Scaleby Paddy Foran

It’s an established pattern to use message queues when building scalable, extensible, and resilient systems, but a lot of developers are still unsure how to go about actually implementing message queues in their architectures. Worse, the number of queuing solutions makes it hard for developers to get a grasp on exactly what a queue is, what it does, and what each solution brings to the table.

At Iron.io, we’re building IronMQ, a queuing solution we’ve developed specifically to meet the specific needs of today’s cloud architectures. In this post, we wanted to detail how to use queues in your applications and highlight a couple of unique capabilities that IronMQ provides (and which are not found in RabbitMQ and other non-native cloud queues).

One of the things that queuing does really, really well is getting work out of the way. Queues are built to be fast ways to make data available for other processes. That means that you can do more with your data, without making your customer wait. When it comes to response times every second matters, so only critical processing should take place within the immediate response loop. Queues let you do processing on data and perform non-immediate tasks without adding to your response time.

Patterns, Code Samples, and a Demo Application

The post goes through several patterns for increasing scale by pushing tasks to the background and includes a number of code samples.

There's also a demo application which includes basic model where workers are just polling a queue as well as more sophisticated patterns that push to multiple queues to handle different tasks (send an email, post to social media, create thumbnails, update user records, etc.)
The code for the demo application can be found here on GitHub.

It's a great post for developers looking at scaling applications. But don't take our word for it. Check it out on the Rackspace Developers DevOps blog.

What makes Yii even cooler is that it now has Iron.io services integrated into it via Yiiron. IronMQ, IronWorker, and IronCache are all supported via this package. In terms of IronMQ, this means that adding high-performance cloud-based message queuing is about as simple as adding config items to point to the IronMQ endpoints. (It's the same in regards to an elastic task queue with IronWorker and a key/value data store with IronCache.)

We'd love to be able to take credit for this integration but it was all the doing of John Eskilsson (aka br0sk on GitHub). John is a freelance systems architect and web developer. He's currently working for Edeva AB, an innovative Swedish company working on intelligent traffic solutions. They make extensive use of IronMQ and IronWorker, moving data from the speed bumps to the central system and for performing analytical calculations and backups.

Yiiron and the Work of John Eskilsson

Configuring IronMQ

1) Create an Iron.io account
If you don’t already have an iron.io account please sign up for a free account at Iron.io.

Go to the Iron.io dashboard (hud.iron.io/dashboard) and create a new project.

When the project is created, click the key icon and take a note of the token and the project id.

2) Install Yiiron into your app
Unzip the yiiron.zip file that you downloaded from the Yiiron Yii extension page and put all the files in the extensions directory. It would look something like this:

/var/www/myapp/protected/extensions/yiiron

Note: You can also install the the extension using Composer. It uses the new Yii Composer installer. Remember to set the composer flag for the extension to false if you used the zip file and set it to true if you have installed using composer. This is needed since the iron.io classes will be found in different locations based on how it was installed.

Tuesday, July 9, 2013

We're happy to announce that IronMQ is now available in the Rackspace Chicago (ORD) region. A year and a half ago, we launched IronMQ on Rackspace in their Dallas / Fort Worth (DFW) region and today we're expanding our footprint into another Rackspace data center.

Our mission for IronMQ is to be the message queue for the cloud, meaning we want to make it the best MQ around and make it available everywhere. Adding the Rackspace ORD region brings us one step closer to accomplishing that mission.

The ORD instance is running the latest version of IronMQ, version 3, which has all the latest and greatest features. Also, it's running on Rackspace's next generation servers (OpenStack) backed by SSD drives for highest performance.

Providing Choice and Enabling New Solutions

Providing service on multiple cloud infrastructure providers (AWS and Rackspace so far) and multiple regions within those providers not only gives our users more choice, but it also enables reliability and failover scenarios that would be otherwise extremely hard to pull off. Not only can you failover to different regions, but you can failover to an entirely different cloud provider! For instance, let's say AWS has a major outage (I know, it's pretty rare, but it does happen). You can failover to use one of our Rackspace IronMQ regions with very little effort, it's simply a different host in your clients.

Providing choice and enabling solutions to scenarios like this is something only IronMQ can provide.