README.md

DEPRECATED!

This library is no longer maintained. The functionality of the library has moved into SciLuigi.

Luigi's Monkey Wrench

This is a small library (50 LOC exactly, as of Feb 12) that intends to make writing Luigi workflows that use a lot of shell commands
(which is common e.g. in bioinformatics) a tad easier by allowing to define workflow tasks with a simple shell command pattern, and
dependencies by using a simple single-assignment patter for specifying how tasks inputs depend on each other's outputs, like so:

But let's go though this example in a bit more detail, to see what we are really doing:

import luigi
from luigis_monkey_wrench import*# Yes, we write the workflow definition inside a normal luigi task ...classMyWorkFlow(WorkflowTask):
# ... and do this by setting up the dependency graph and (letting the workflow# task depend on it, by) returning the last task in the dependecy graph in the# workflow task's requires() function:defrequires(self):
# Create tasks by initializing ShellTasks, and giving# the shell tasks to execute to the cmd parameter.# File names are given in a this special form (including <>):# <i:INPUT_NAME># <o:OUTPUT_NAME:OUTPUT_FILENAME># Output file names can also include the filename of an input:# <o:some_output:<i:some_input>.some_extension># One can also just replace the extension, or ending, of the input# filename, in the output file name, using the following syntax:# <o:OUTPUT_NAME:<i:INPUT_NAME:OLD_EXTENSION|NEW_EXTENSION>># E.g, to create <filename>.csv as output from <filename>.txt, we do:# <o:some_output:<i:some_input:.txt|.csv>>
hejer = shell('echo hej > <o:hejfile:hej.txt>')
fooer = shell('cat <i:hejfile> | sed "s/hej/foo/g" > <o:foofile:<i:hejfile:.txt|.foo>>')
# Define the workflow "dependency graph" by telling how outputs# from tasks are re-used in inputs of other tasks
fooer.inports['hejfile'] = hejer.outport('hejfile')
# Return the last task in the workflowreturn fooer
# We finally make this file into an executable python file, and let luigi take of the running# which will, among many other cool things, mean that we get a nice command line interface# generated for us:if__name__=='__main__':
luigi.run()