We wanted to have a simple platform independent solution for managing the
coordination and distribution of batched execution across our analysis
platforms. restq solved our wants into a system that could:

segregate execution based on a category or type (realm),

manage priorities of job execution (ordered queues),

enqueue, check-out, and expiry time based (almost FIFO) dequeuing of jobs
from a realm.

status of jobs remaining against arbitrary tag indices.

zero configuration for the concepts talked about above.

What’s in restq:

An implementation of the execution management system described above.

A RESTful web API that exposes complete control over the execution
management system.

A command line interface accessible in the shell through the entry point
‘restq’. The CLI makes it trivial to kick off a restq server. It also
implements a set of commands which allow users to enqueue and dequeue commands into
a realm. This makes it super trivial to deploy scheduled execution jobs
across a pool of servers.

For additional tips / tricks with this restq feel free to post a question at
the github restq/issues page.

Time to remove pwd from our realm… We’re done with this argument and we no
longer require it for execution. You will notice that the fun tag no longer
exists in the realm as it was only attached to pwd.

> restq remove arg pwd
>

The default lease time for a dequeue of an argument is 600s. After this
expiry time, ‘ls -lah’ will once again be available for dequeue.

> restq pull
ls -lah

How to distribute a shell script for execution

Add ‘work.sh’ script into the default realm.

> restq add --file=work.sh "chmod +x work.sh; ./work.sh"

Now when this job is dequeued using the restq cli, the path ‘./work.sh’ will
be written to using the data read from the original ‘work.sh’ and the
arguments will be written out to stdout.

> eval "`restq pull`"

The following is an example of a script that could be deployed across multiple
machines to continuously pull and execute jobs that have been added into the
default realm.