Give Codeship’s CI/CD Platform a Try

Want to learn more?

You’re always striving to give your users a better experience when they use your website or application. One of the most important ways to achieve this is by giving them quick server response times. In this article, we’ll explore how to use Rails Active Job to enable us to do this through the use of a queuing system. You can also use queues to help normalize traffic spikes or load on the server, allowing work to be done when the server is less busy.

Active Job was first included in Rails 4.2 as a way to standardize the interface to a number of queueing options which already existed. The most common queues used within Rails applications are Sidekiq, Resque, and Delayed Job.

Active Job allows your Rails app to work with any one of them (as well as with other queues) through a single standard interface. For the full list of which backends you can use with Rails Active Job, refer to this page. It’s also important to see which features are supported by which queueing system; some don’t support delayed jobs for example.

Adding an item to the queue

If you want to process the job as soon as possible, you can use the perform_later method. As soon as a worker is available it will process the job.

UpdateUserStatsJob.perform_later user

Queueing for later

If you would rather have the job performed a week from now, some queue backends allow you to pass additional time parameters when adding a job.

UserReminderJob.set(wait: 1.week).perform_later user

What goes in a job class?

The job class is where you put the code that will be executed by the queue. There is a perform method which is called and sent whatever parameters were sent when the job was first enqueued (when you called the perform_later method).

Using Rails Active Job with Sidekiq and Resque

Both Sidekiq and Resque rely on having Redis installed, which is where they store the items in the queue. To use either of these, I recommend following the instructions found on the Resque GitHub page or the Sidekiq wiki.

We will need to tell Active Job which queue we are using, which can be done in the application config file. In this example, we’ll be working with Sidekiq.

Sidekiq and Resque both come with web interfaces to view information about the workers and which jobs are in the queue. Sidekiq is more efficient and quick than Resque but requires that your Ruby code is thread-safe. Also, even though this is somewhat down to my own personal preference, Sidekiq has a nicer web interface than Resque does.

Common Patterns for Queueing

There are a number of common patterns or types of jobs that you want to process in the queue. The basic rule I follow is to ask whether it needs to happen right now and/or if it might take a long time to process.

If it has to happen right now (for example, whether someone’s credit card information is correct), you’ll more than likely have to bite the bullet and process it before the response can go back to the user. Even still, you should think about the user experience by displaying a message letting them know that you’re processing their information and it may take a little while.

Sending email

Sending email is the most common task that can and should be done in the background job. There is no reason to send emails immediately (before the response is rendered), and I always move all emails to the queue. Even if the email server responds in 100ms, that’s still 100ms that you’re making your user wait when they don’t need to.

Sending emails via a background job is super simple with Active Job, mainly because it comes built-in to ActionMailer.

By changing the method deliver_now to deliver_later, Active Job will automatically send this email asynchronously in the queue.

UserMailer.welcome(@user).deliver_later

Processing images

Images can take a while to be processed. This is especially true if you have a few (or more) different styles and sizes that need to be created. Luckily, both Paperclip and CarrierWave have additional gems which can help them process these images in the queue rather than at the time of uploading.

For Delayed Paperclip, you simply call an additional method letting it know what you would like to process in the background, and the gem will handle the rest. You can even have it process some styles right away, while other styles get processed in the queue.

This would allow us to show the :small image right away, while the :medium and :large images are done in the background.

User uploaded content

Often when you have user uploaded content it needs to be processed. This may be a CSV file that needs to be imported into the system, an image which needs to have thumbnails generated, or a video that needs to be processed.

A large CSV file may take a few minutes to process, in which time the browser’s connection may time out. I’ve taken to processing most data uploads asynchronously in the queue.

The process I use is as follows:

Accept the file and upload it to S3 (or wherever you are storing user generated content).

Add a job to the queue to process this file.

The user will immediately see a success page letting them know that their file has been submitted for processing.

The worker will download the file, process it, and mark it as having been processed.

Another thing to keep in mind is that you will want to store a report of the import in the database. This may include any records that couldn’t be processed due to invalid data. What I do is create a second error file for each import that the user can download.

Generating reports

Large reports can often take longer to generate than you want your user to wait for. You also might not want to put this sort of load on your app servers. You can generate a report in the queue and then email a link to the user to be able to download it when it is ready. I’ve seen this be incredibly useful when producing reports for the accounting department, which often needs to download reports with millions of records in them.

The flow for generating this type of report is as follows:

Allow user to specify which report they wish to generate along with all of its filters.

Add a job to the queue to produce this report.

The user will immediately see a page or notification letting them know that their report has been submitted for processing and how they can expect to receive it.

The user will either be notified within the user interface of the website/app that the file is ready to download, and/or they will receive an email with a link to download the finished report.

Talking with external APIs

External APIs can be flaky and slow, and your users’ experience should not depend on them whenever possible. Take this example below where we use their IP address to find out some geo information about them using the Telize API. It generally responds in 200ms to 500ms, which, added to your current response time, can make a large difference. This is something that can wait to be done, especially when used for reporting purposes. Even though this is showing IP geo information, all external APIs should be treated the same way. Talk to them in the background if at all possible.

Here we perform the actual work to be done. This code doesn’t implement actually logging the geo info to a log or database. It makes a real remote call to the API to show how long requests like this can take.

In our Rails logs we can see what’s happening. It enqueues the job with the argument "::1", performs the job right away (because we are using the Inline queue), outputs some debug info from our class, and then lets us know when the job is finished. It also shows that it took 572.39ms.

Notifying others of changes

When a user creates new content (for example, they tweet something), you often have to let others know of that change. Determining who to notify can be a difficult (slow) process, and there is no reason to slow down the experience of the user who is creating this content.

If the tweet is created successfully, you can add a job in the controller to notify users that were mentioned or who follow this user.

GlobalID for Object Serialization

You’ll notice in the last example that I actually just passed the entire tweet object to the worker. It used to be quite common to have to pass the tweet ID and then query for that tweet once inside the worker, but GlobalID allows us to pass the entire object and handles the serialization and deserialization for us.

In Conclusion

Active Job is a great addition to Rails. It won’t get you out of having to learn how to best use the queue backend that you end up going with, but it will provide a clean and single interface for adding jobs and processing those jobs, no matter the backend. If you’re starting a new Rails project or adding a queueing system to an existing one, definitely think about using Active Job rather than talking directly to the queue.

Using queues can increase your website usability (by lowering response times), provide more consistent response times and server loads (by spreading the heavy lifting over various servers and workers), and open new doors to what your website can do (by allowing more complex processing out of the user request/response flow).

Subscribe via Email

Over 60,000 people from companies like Netflix, Apple, Spotify and O'Reilly are reading our articles. Subscribe to receive a weekly newsletter with articles around Continuous Integration, Docker, and software development best practices.

We promise that we won't spam you. You can unsubscribe any time.

Join the Discussion

Leave us some comments on what you think about this topic or if you like to add something.