As it was explained in the first part of the article, queues are great to defer the processing long tasks, allowing to provide faster user interfaces so users do not have to wait for those tasks to complete.

Read this article to learn how to efficiently process queued tasks in PHP and keep track of the results.

Contents

Introduction

The Benefits of Queueing Processing Tasks

Processing a Queue

A Real Life Example

Queuing Tasks

The Queue Processor Worker

Improving the Performance of the Queue Processing

Conclusion

Introduction

As it was explained in the first part of this article, you can "postpone" the execution of a long task by sending it to a queue. That will help to to speed up your application for the end user as he does not have to wait for your task to complete.

The task is queued, your users have better experience with your sites or apps. The only thing left to implement is to process the tasks in the queue doing the actual work.

The Benefits of Queueing Processing Tasks

Now you may wonder if are there any other benefits of using a queue, apart from a quick response of such an application? Sure, the benefits are:

You can see the stats of processing: how many succeed and fail

You can get the exact time the tasks take in average

You can track the errors of processing, figure out the reason and then "replay" the failed entries. This can be very handy if your team mate changed the password to a service your task relies on, or you run out of credits and need to fund the account

Some complex rules can be applied to processing, for example, if you send queued emails via Gmail mail server, it allows to send only 2000 emails per day, so your queues processors may take care of this easily

Processing can take place only when it is the best time for your servers, for instance, when it is night or when there is an Internet connection or mobile phone signal

You can process tasks in a batch, for instance, establish a single connection to your STMP server and send a hundred queued emails within a single session

Processing a Queue

If you remember "The Imitation Game" movie, there were special workers,— a group of ladies, who were given each a deck of papers to decode, and as soon as a paper was decoded, it was given back to the manager. That is a perfect example of "a worker" or "queue processor".

So in this article you willl learn how to create a queue processor. The processor of the queue or just "worker" is a small script that:

Runs regularly, usually as from the cron program, for example, every 5 minutes,

Takes N jobs from the queue

Process the jobs of this batch one by one

Reports the results: marks the jobs as processed, or logs the error message for failed ones so you could analyze it later

This is a simplified example of code, as it processes a single job per run, and the step number 4, report the results, is not implemented yet.

Anyway, this simple piece of code can do the job already, and you can run in using cron every 5 minutes using a crontab configuration like this (if you just saved the code above as queue-worker.php file):

status field equals "queued" value by default— that means that a job is queued and is waiting for the processing. When we take an element from the queue to process, we change this field value to "processing". If the processing was successful, we set this value to "done", otherwise to "failed" and log the error message.

From my experience it is best to log the timestamp when the item was queued and the timestamp of the last status change. This is saved in created_at and updated_at fields accordingly. When we just create a new record in this table, we set created_at to equal to current time. And every time the record is updated (say, we change the status field), the updated_at field gets updated automatically — it's a trick of MySQL for TIMESTAMP fields that have "ON UPDATE CURRENT_TIMESTAMP" setting.

Queuing Tasks

Ok, we have a table to queue our jobs. Now let's write the code to add an item to the queue. It depends on your implementation and framework, but it's really likely that you need to pass the database handler object reference to the constructor:

Improving the Performance of the Queue Processing

There are a few ways to improve this system. For instance if you need to process more than 1 item per run, like for instance 10 or 20, items you need to change Queue::getItem() method, pass a $number parameter as the number of required jobs to process, and change "LIMIT 1" to "LIMIT $number" in the SQL query.

Conclusion

As you can see, making a queue and implement its processing is not a super complex task. You just need to implement it once, but then it allows you to kill many birds by one stone, so you can use the solution as many times as necessary.

If you liked this article or you have a question about processing queues of pending tasks, post a comment here.