README.md

Thoonk

Thoonk is a persistent (and fast!) system for push feeds, queues, and jobs which
leverages Redis. Thoonk.py is the Python implementation of Thoonk, and is
interoperable with other versions of Thoonk (currently Thoonk.js for node.js).

Feed Types

Feed

The core of Thoonk is the feed. A feed is a subject that you can publish items
to (string, binary, json, xml, whatever), each with a unique id (assigned or
generated). Other apps and services may subscribe to your feeds and recieve
new/update/retract notices on your feeds. Each feed persists published items
that can later be queried. Feeds may also be configured for various behaviors,
such as max number of items, default serializer, friendly title, etc.

Queue

Queues are stored and interacted with in similar ways to feeds, except instead
of publishes being broadcast, clients may do a "blocking get" to claim an item,
ensuring that they're the only one to get it. When an item is delivered, it is
deleted from the queue.

Queues are useful for direct message passing.

Sorted Feed

Sorted feeds are unbounded, manually ordered collections of items. Sorted feeds
behave similarly to plain feeds except that items may be edited in place or
inserted in arbitrary order.

Job

Jobs are like Queues in that one client claims an item, but that client is also
required to report that the item is finished or cancel execution. Failure to to
finish the job in a configured amount of time or canceling the job results in
the item being reintroduced to the available list. Unlike queues, job items are
not deleted until they are finished.

Jobs are useful for distributing load, ensuring a task is completed regardless
of outages, and keeping long running tasks away from synchronous interfaces.

Installation

In theory, if I publish the package correctly, then this should work.

pip install thoonk

Requirements

Thoonk requires the redis-py package. I strongly recommend installing
the hiredis package as well, since it should significantly increase performance.

Since the redis package is undergoing refactoring with backwards incompatible
changes, be sure to use version 2.2.4.

pip install redis==2.2.4
pip install hiredis

Running the Tests

After checking out a copy of the source code, you can run

python testall.py

from the main directory.

Using Thoonk

Initializing

Listen mode is off by default. Turn it on if you want to get live events.

Publishing To a Queue

Popping a Queue

Using a Job Feed

A job feed is a queue of individual jobs; there is no inherent relationship between jobs from the
same job feed.

job = thoonk.job('job_feed')

Publishing a Job

Creating new jobs is done by putting them in the job queue. New items are placed at the end of the
queue, but it is possible to insert high priority jobs at the front.

job.put('job contents')
job.put('priority job', priority=job.HIGH)

Claiming a Job

Workers may pull jobs from the queue by claiming them using the get() method. The default behaviour
is to block indefinitely while waiting for a job, but a timeout value in seconds may be supplied
instead.

data = job.get()
timed_data = job.get(timeout=5)

Cancelling a Job Claim

Cancelling a job is done by a worker who has claimed the job. Cancellation relinquishes the claim to
the job and puts it back in the queue to be given to another worker.

job.cancel('job id')

Stalling a Job

Stalling a job removes it from the queue to prevent it from executing, but does not completely
delete it. A stalled job is effectively paused, waiting for whatever issue that required the
stall to be resolved.

job.stall('job id')

Retrying a Stalled Job

Once a job jas been stalled, it can be retried once the issue requiring stalling has been
resolved.

job.retry('job id')

Retracting a Job

Retracting a job completely removes it from the queue, preventing it from being executed.

job.retract('job id')

Finishing a Job

Finishing a job can be done in three ways. The first is as a simple acknowledgment that the task
has been completed.

job.finish('job id')

The second is when there is result data that should be returned to the job owner. The result=True
parameter is used since it is possible for None to be an actual job result.

job.finish('job id', 'result contents', result=True)

It may also be desired to only keep job results around for a short period of time. In which case,
a timeout parameter may be added.

job.finish('job id', 'result contents', result=True, timeout=5)

Check Job Results

Checking the result of a job can require knowing what the original job request actually was.
Thus, the get_result method will return both values.