The only mandatory parameter is QUEUE. To make a worker monitor multiple queues, separate the names with a comma, e.g. : “achievement,notification”. The order matters, and achievement will always be polled before notification. All the jobs in achievement will be executed before checking the notification queue.

Setting QUEUE to * will poll all the queues, in alphabetical order.

Workers have to be started in the CLI. You can’t ceate worker through a browser, because :

Why not directly pass the parameters to PHP ?

Answer #1

Answer #2

Because it let us define the parameters we only want, in the order we want.

If PHP was reading the parameters, he will parse them in a defined order, like regular function parameters (function($QUEUE, $INTERVAL, $COUNT)). You will then be unable to set $COUNT without setting $INTERVAL beforehand. There’s still a way to do so, but it’ll require some extra code. When processing hundred of jobs per second, there’s no room for extra code.

Daemon-ify the workers

It’s great, your workers are running … as long as you keep your terminal window open. As you see, the php command hogs up the whole terminal window. When you stop the script, either by closing the window or with Ctlr+C, the worker stops too. We’ll need to bring out the as-old-as-the-world trick : append a & at the end.

QUEUE=notification php resque.php &

Above command will daemon-ify the php command, and liberate your terminal. But if you were using verbose mode, all the output will be lost. We’ll need to save that output somewhere before daemonizing the worker.

It’ll also be nice to thrown in a nohup, enabling the command to keep running even if the user logged out.

Possible permissions issues:

All files created by your workers (running under user A) can’t be read by the rest of you php code (running under user B).

Workers don’t have permissions to create or edit files created by the rest of your application.

Let’s play

Let’s finish this section with various examples.

For simplification purpose, the permissions, file logging and daemonization part sudo nohup -u USER, >> /path/to/your/logfile.log 2>&1 & are dropped from the following examples.

To create a worker polling the default queue each 10 seconds:

INTERVAL=10 QUEUE=default php resque.php

To create 5 workers polling the default queue each 5 seconds:

QUEUE=default COUNT=5 php resque.php

INTERVAL parameter is not needed, as 5 seconds is already the default value.

To create a worker polling the queues achievement and notification:

QUEUE=achievement,notification php resque.php

Remember, queue name order defines their priority. achievement will always be checked before notification.

To create a worker polling all the existing queues, in alphabetical order:

QUEUE=* php resque.php

If your Redis is located at a different address:

QUEUE=default REDIS_BACKEND=192.168.1.56:6380 php resque.php

To pass your application autoloader:

QUEUE=default APP_INCLUDE=/path/to/autoloader.php php resque.php

Autoloader importance will be developed later.

Tip You can log each php resque.php output to a different log file.

Ensure your workers were created successfully

Since you’re piping the output, you’ll not know immediately if your worker creation failed. A way to verify that is to monitor your log file: a successful worker creation should output *** Starting worker YOURHOSTNAME:PID:queuename.

Sometimes, you could see more process than you have workers. This happens only when you have the pcntl extension. It means that the worker forks another process to execute the job.

Forking

On certain platforms, when a Resque worker reserves a job it immediately forks a child process. The child processes the job then exits. When the child has exited successfully, the worker reserves another job and repeats the process.

If Resque workers processed jobs themselves, it’d be hard to whip them into shape. Let’s say one is using too much memory: you send it a signal that says “shutdown after you finish processing the current job,” and it does so. It then starts up again – loading your entire application environment. This adds useless CPU cycles and causes a delay in queue processing.

Plus, what if it’s using too much memory and has stopped responding to signals?

Thanks to Resque’s parent / child architecture, jobs that use too much memory release that memory upon completion. No unwanted growth.

And what if a job is running too long? You’d need to kill -9 it then start the worker again. With Resque’s parent / child architecture you can tell the parent to forcefully kill the child then immediately start processing more jobs. No startup delay or wasted cycles.

The parent / child architecture helps us keep tabs on what workers are doing, too. By eliminating the need to kill -9 workers we can have parents remove themselves from the global listing of workers. If we just ruthlessly killed workers, we’d need a separate watchdog process to add and remove them to the global listing – which becomes complicated.

Pausing and stopping a worker

To stop a worker, simply kill his process. You can find the process id (PID) in the worker name (YOURHOSTNAME:PID:queuename). You can also get your workers PID by running ps u | grep resque.php. This method will not tell you which process correspond to which worker though.

Soft stopping workers

To wait for all the jobs in the middle of processing to finish, before stopping the worker, use QUIT signal.

kill -QUIT YOUR-WORKER-PID

We can also use the corresponding signal number:

kill -3 YOUR-WORKER-PID

Kill only the child, but keep the worker

kill -USR1 YOUR-WORKER-PID # or kill -30 YOUR-WORKER-PID

Pause worker

kill -USR2 YOUR-WORKER-PID # or kill -31 YOUR-WORKER-PID

Resume a paused worker

kill -CONT YOUR-WORKER-PID # or kill -19 YOUR-WORKER-PID

Load distribution

You don’t necessary have to queue and execute jobs on the same server. You can perfectly queue jobs from a server A, and have the worker from a server B executing them. You’ll need to install php-resque and all your jobs classes on server B.

Next Time …

Now that you know how to start and stop a worker, in part 5, we’ll talk about jobs, and how to write job classes your workers can understand.

Background jobs are jobs that are executed outside the main flow of your program, and usually handled by a queue system. This first tutorial of the serie will introduce what’s a background job and its importance. Part 1 : Introduction Part 2 : Queue system Part 3 : Installation Part 4 : Worker Part 5 […]

As said in part 2, we’ll use php-resque for our queue system. In this part, I’ll explain how to install all the tools needed to run php-resque, a port of Resque. Resque (pronounced like “rescue”) is a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later. Obviously, we’ll […]

When the resque worker is running, there is a default interval of 5 seconds. So when there are 100’s of 1000’s of jobs in the queue, will it process all of them first and then take a nap for 5 seconds before checking if there are newer tasks? or will it take a nap at every 5 seconds even if there are pending tasks?

wa0x6e

All. Each time a worker polls a queue, it process all the jobs inside the queue, then go back to sleep

madqus

I’m encountering a problem. Each time I run ps u | grep resque.php it gives me different pid. If I try kill the current process -bash: kill: (20915) – No such process. PID is keep increasing each time. Would appreciate any clue.

madqus

@wa0x6e:disqus Is this a issue as you see or Am I lack of proper understanding of workers.

wa0x6e

What OS are you running on ? PID should remain the same, and does not change. PID change means that the worker somehow died, and something restarted it. Do you have any tools to monitor and automatically restart process, such as monit or god ?

madqus

Its ubuntu. I don’t have monitor running yet. You can see the pattern. Every time I run grep for resque.php it increases.

wa0x6e

From your screenshot, the worker is not running. The process displayed is the grep command you just executed.

Remember that ‘ps u’ will also display the process for `grep resque.php`, so that’s normal the pid change each time

madqus

Holly. Thanks!! Had to drop screenshot. Anyway I guess above answer is clear enough to anyone else without screenshot.

Sazzad Tushar Khan

It was working fine. But after restarting the vagrant box today again its not getting the QUEUE. And is saying, ‘Set QUEUE env var containing the list of queues to work.’