django-gearman-commands is set of Django management commands
and base classes aiming to simplify developing and submitting
tasks to Gearman job server from Django.

About Gearman

Gearman, as stated on project website, provides ‘a generic application framework to farm out work to other machines or processes that are better suited to do the work’.
Practically, Gearman is a daemon, service, running on TCP port and waiting for Clients wishing to get job done and Workers who handle and process the jobs.
Gearman - anagram for “Manager” itself does exactly what manager does - accepts requests from Clients and distribute them to Workers.

Note that single Gearman Job Server in illustration above is only meant to simplify illustration.
Even Gearman itself is really stable, in production it is recommended to run at least 2 job servers
for redundancy.

Gearman has API for writing Clients and Workers for a lot of environments - Python, Ruby, PHP, Java, Perl, C and probably some others.
In practice, you can write workers in same or different languages.
For example, if you have PHP application, you can use Gearman PHP API but your workers can be written in Python.
Gearman is designed with flexibility in mind.

Why django-gearman-commands

For Python, there is a great python-gearman 2.x API by Yelp - https://github.com/Yelp/python-gearman.
python-gearman API allows you to write Clients, Workers and perform some additional administration tasks
in Python without assuming you use some specific web application framework.

The main issue solved by django-gearman-commands is writing Workers in a way they are aware of your Django application.

Gearman Worker is standalone script running on background and waiting for job requests.
In theory, you write Python script implementing worker and logic of job and that’s it.
In practice, your worker needs to use Django facilities.
When you use virtualenv (which you should), you also need to think of imports, DJANGO_SETTINGS_MODULE
and all that jazz in your Worker script.

Django provides a way to write standalone, console-based scripts - Commands - https://docs.djangoproject.com/en/dev/howto/custom-management-commands/
Similar to ‘./manage.py syncdb’ or ‘./manage.py migrate’, it is possible to write your own commands invoked by ‘./manage.py mycommand’.
This is recommended way to write scripts in Django project which are not run from web application itself.
Django Commands can be called from console or from Django application itself, which makes them quite flexible
for both manual command-line invocation or programmatic invocation.

What django-gearman-commands provides

Base Class for your Workers

django_gearman_commands.GearmanWorkerBaseCommand is base class for writing Gearman Workers
as Django commands. No need to hassle with low-level python-gearman API, just inherit GearmanWorkerBaseCommand,
override task_name and do_job and you are ready to go.

Submitting Jobs Easily

django-gearman-commands provides ‘gearman_submit_job’ command that can be used to submit new jobs
to gearman. Instead of writing your own class to submit jobs and handle job arguments,
invoke ./manage.py gearman_submit_job task_name job_data

Gearman Administration Overview

Gearman itself provides administration functions returning version of Gearman server, active workers
and list of registered tasks with their relations to workers, running and pending jobs.
Simply run ./manage.py gearman_server_info to get current status of Gearman servers.
If you want to output that information yourself, you can use django_gearman_commands.GearmanServerInfo class.

Getting started

Setup

So you have your Django application and want to install django-gearman-commands.
django-gearman-commands is standard Django app which provides no models, views or urls,
only few classes, custom management commands and tests.

There is only one new dependency to add to your app (except Django) - python-gearman API:

Writing workers

django_gearman_commands.GearmanWorkerBaseCommand is base class for your custom django commands acting like Gearman workers.
You should write custom command for each specific task.

Suppose we want to write worker to import some complex data which can take a long time.
Create file ‘gearman_worker_data_import.py’ in your Django app management/commands directory
with following content:

By default, jobs are submitted in background and ‘gearman_submit_job’ does not wait for job to finish.
You can override this with ‘–foreground’ CLI option. See ‘./manage.py gearman_submit_job –help’.
If you did everything right, your worker method ‘your_code_performing_job_logic()’ should be now running in background.

This method is fine if you want to run job manually or from cron.
For example, if you want to run data_import for cron every 5 minutes, you can add something like this to cron:

However, in lot of cases, you want to run job on-demand, for example in some Django view, user makes some action
and you want to run job immediately - sending email, importing data or anything else you need and don’t want to block
user’s web request until task is completed.
Django can call custom management commands programatically, via django.core.management.call_command method:

By using job submission wrapper Command ‘gearman_submit_job’,
you are now able submit jobs from console, cron and your app with same API.

Task namespaces

Sometimes it can be useful to distinguish between jobs submitted with same task name from several applications connected
to same gearman servers. For example, you may have several instances of same django project deployed for individual
clients.
In that case, you can add GEARMAN_CLIENT_NAMESPACE to your django settings to uniquely identify tasks
submitted by project:

GEARMAN_CLIENT_NAMESPACE = 'MyCustomer1'

Gearman server info

gearman_server_info outputs current status of Gearman servers.
If you installed prettytable dependency, here is how output looks like:

This will start Gearman server compiled in /home/gearman/earmand/gearmand/gearmand with SQLite persistent queue on localhost.
Of course, your variables may vary.

Supervisor + Workers

You can create single .conf file for all workers relevant to single application.
This will create process ‘group’ and allows you to reload of all workers related to some application
at once when you redeploy new code.

For example, create ‘myapp.conf’ in /etc/supervisor/conf.d with all workers relevant to ‘myapp’::

Contributing to django-gearman-commands

Contributions are welcome.
If possible, please use following workflow:

find out what is bothering you

check Issues page if problem is not already discussed

fork django-gearman-commands

fix it in your fork and add test to tests/__init__.py

add yourself as Contributor to ‘Authors and Contributors’

and make Pull Request with description what change is supposed to do

Running tests

Tests are located in tests/__init__.py file.
There is a wrapper ‘runtests.py’ in root directory to setup Django environment with minimal dependencies and run tests.
The point is to allow testing of django-gearman-commands during development without full-blown Django application:

$ python runtests.py

As you can read from runtests.py, tests expect Gearman server running on localhost on standard port 4730.