Background processing is a must-be in any bigger project. There’s no way to process everything in a lifetime of a single request. It’s such common, that I venture to say that each and every one of us made at least one background worker. Today I would like to tell you a bit about reentrancy.

What is reentrancy

In computing, a computer program or subroutine is called reentrant if it can be interrupted in the middle of its execution and then safely called again (“re-entered”) before its previous invocations complete execution. The interruption could be caused by an internal action such as a jump or call, or by an external action such as a hardware interrupt or signal. Once the re-entered invocation completes, the previous invocations will resume correct execution.

So basically it means, that if our worker crashes, we can execute him again and everything will be just fine. No database states fixing, no cleanups, no resetting – nothing. Just re-executing worker task.

How many workers do you have like this? ;) I must admit: I’ve created non-reentrant workers many times and many times I wish I didn’t.

Why our workers should be reentrant and why it’s not an overhead to make them like this

Making workers reentrant, especially at the beginning will take you more time than creating a “standard” one. “This will create an overhead” you might think. This might be true but… it’s not. If you have many workers that are constantly doing something, and some of them crash, reentrancy will save you a lot of time. It allows you to just fix the issue and rerun tasks again, without having to worry about anything else. Without it, you would spend some time fixing database structure, cleaning things up, performing requests to external APIs from production console and other non-programming related stuff. I guarantee that you will waste much more time doing this, than writing your workers well in a first place.

How to make my workers reentrant?

It’s not so hard as you might thing, although sometimes it can be a bit tricky. Of course every worker is somehow unique but there are some general rules that you can use.

Transactional, non-API example test case

The easiest stuff is with non-API, transaction-only workers, that calculate and update some data:

class ScoreWorker
include Sidekiq::Worker
def perform(user_id)
user = User.find(user_id)
user.update(status: 'calculating')
# This can take a while...
user.calculate_score!
user.update(status: 'calculated')
end
end

If something would happen when calculate_score! is executed, we would end up with a user with endless “calculating” state. The easiest way to fix this, is to use ActiveRecord::Base.transaction block:

class ScoreWorker
include Sidekiq::Worker
def perform(user_id)
ActiveRecord::Base.transaction do
user = User.find(user_id)
user.update(status: 'calculating')
# This can take a while...
user.calculate_score!
user.update(status: 'calculated')
end
end
end

If anything goes wrong, we will get back to where we started. Unfortunately this approach has one huge disadvantage: user status is changed in transaction, so until it is committed, we won’t have the ‘calculating’ status (at least if you don’t have dirty reads).

Non-transactional, non-API example test case

Approach presented below can be also used to improve previous example. Let’s imagine we don’t have transactional DB and that every operation is performed separately. We need to catch an exception, rewind everything back and then just reraise error:

class ScoreWorker
include Sidekiq::Worker
def perform(user_id)
user = User.find(user_id)
# State machine is always nice :)
user.calculating_score!
# This can take a while...
user.calculate_score!
user.calculated_score!
rescue e
# Reset everything so it can be processed again later
# We "if" in case error was raised in the first line
user.reset_score! if user
raise e
end
end

Of course this will not prevent us from DB failures, but in my experience, workers tend to fail mostly not because of database issues but because of some problems in the app (worker) logic.

Non-transactional, API example test case

What about external API interactions? When we change something remote, we cannot just simply “unchange” stuff. Let’s say that we have a charging mechanism, that makes a call (charge) and then it sends an invoice to this user:

How can we provide reentrancy for worker like that? What will happen when user.send_invoice_confirmation! fails? We cannot charge user again for the same month. This means, that we cannot execute this worker task again. We might check whether or not user has been charged: