ABSTRACT

The Redis::JobQueue package is a set of Perl modules which allows creation of a simple job queue based on Redis server capabilities.

DESCRIPTION

The main features of the package are:

Supports the automatic creation of job queues, job status monitoring, updating the job data set, obtaining a consistent job from the queue, removing jobs, and the classification of possible errors.

Contains various reusable components that can be used separately or together.

Provides an object oriented API.

Support of storing arbitrary job-related data structures.

Simple methods for organizing producer, worker, and consumer clients.

Atributes

id

An id that uniquely identifies the job, scalar.

queue

Queue name in which job is placed, scalar.

expire

For how long (seconds) job data structures will be kept in memory.

status

Job status, scalar. See Redis::JobQueue::JobEXPORT section for the list of pre-defined statuses. Can be also set to any arbitrary value.

workload, result

User-set data structures which will be serialized before stored in Redis server. Suitable for passing large amounts of data.

*

Any other custom-named field passed to "constructor" or "update_job" method will be stored as metadata scalar in Redis server. Suitable for storing scalar values with fast access (will be serialized before stored in Redis server).

EXPORT

None by default.

The following additional constants, defining defaults for various parameters, are available for export:

Redis::JobQueue::MAX_DATASIZE

Maximum size of the data stored in workload, result: 512MB.

DEFAULT_SERVER

Default address of the Redis server - 'localhost'.

DEFAULT_PORT

Default port of the Redis server - 6379.

DEFAULT_TIMEOUT

Maximum time (in seconds) to wait for a new job from the queue, 0 - unlimited.

4 - Command failed because its execution requires more than allowed memory, set in maxmemory.

E_JOB_DELETED

5 - Job's data was removed.

E_REDIS

6 - Other error on Redis server.

CONSTRUCTOR

new( redis => $server, timeout => $timeout )

Creates a new Redis::JobQueue object to communicate with Redis server. If invoked without any arguments, the constructor new creates and returns a Redis::JobQueue object that is configured with the default settings and uses local redis server.

load_job( $job )

Loads job data from the Redis server. The argument is either job ID or a Redis::JobQueue::Job object.

Method returns the object corresponding to the loaded job. Returns undef if the job is not found on the Redis server.

The following examples illustrate uses of the load_job method:

$job = $jq->load_job( $id );
# or
$job = $jq->load_job( $job );

get_next_job( queue => $queue_name, $blocking => 1 )

Selects the job identifier which is at the beginning of the queue.

get_next_job takes arguments in key-value pairs. You can specify a queue name or a reference to an array of queue names. Queues from the list are processed in random order.

By default, each queue is processed in a separate request with the result returned immediately if a job is found (waiting) in that queue. If no waiting job found, returns undef. In case optional blocking argument is true, all queues are processed in a single request to Redis server and if no job is found waiting in queue(s), get_next_job execution will be paused for up to timeout seconds or until a job becomes available in any of the listed queues.

Job queue data structures are created automatically when job is placed in the queue and deleted when all jobs are removed from the queue.

DEPENDENCIES

In order to install and use this package it's recommended to use Perl version 5.010 or later. Some modules within this package depend on other packages that are distributed separately from Perl. We recommend that you have the following packages installed before you install Redis::JobQueue package:

BUGS AND LIMITATIONS

By design Redis::JobQueue uses freeze before storing job data on Redis (workload, result containers and the custom-named fields). This ensures that among other things, UTF8-encoded strings are safe when passed this way. The other main string job data fields (id, queue, job, status, message) are not processed in any way and passed to Redis as-is. They are designed as an easy and fast way for software developer to store some internal / supplemental data among job details.

The use of maxmemory-police all* in the redis.conf file could lead to a serious (but hard to detect) problem as Redis server may delete the job queue objects.

It may not be possible to use this module with the cluster of Redis servers because full name of some Redis keys may not be known at the time of the call to Lua script ('EVAL' or 'EVALSHA' command). Redis server may not be able to forward the request to the appropriate node in the cluster.

We strongly recommend using of maxmemory option in the redis.conf file if the data set may be large.

The Redis::JobQueue package was written, tested, and found working on recent Linux distributions.