Description

When talking of parallel processing, some task requires a substantial
set-up time. This is the case of Natural Language Processing (NLP) tasks
such as classification, where models need to be loaded into memory. In
these situations, we can not start a new process for every data set to
be handled, but the system needs to be ready to process new incoming
data. This talk will look at job queue systems, with particular focus on
gearman. We will see how we are using it at Synthesio for NLP tasks; how
to set up workers and clients, make it redundant and robust, monitor its
activity and adapt to demand.