Performance can be significantly improved in different contexts by making
small optimizations on the dask graph before calling the scheduler.

The dask.optimization module contains several functions to transform graphs
in a variety of useful ways. In most cases, users won’t need to interact with
these functions directly, as specialized subsets of these transforms are done
automatically in the dask collections (dask.array, dask.bag, and
dask.dataframe). However, users working with custom graphs or computations
may find that applying these methods results in substantial speedups.

In general, there are two goals when doing graph optimizations:

Simplify computation

Improve parallelism.

Simplifying computation can be done on a graph level by removing unnecessary
tasks (cull), or on a task level by replacing expensive operations with
cheaper ones (RewriteRule).

Parallelism can be improved by reducing
inter-task communication, whether by fusing many tasks into one (fuse), or
by inlining cheap operations (inline, inline_functions).

Below, we show an example walking through the use of some of these to optimize
a task graph.

Here we’re counting the occurrence of the words 'orange, 'apple', and
'pear' in the list of words, formatting an output string reporting the
results, printing the output, then returning the output string.

To perform the computation, we pass the dask graph and the desired output keys
to a scheduler get function:

>>> fromdask.threadedimportget>>> outputs=['print1','print2']>>> results=get(dsk,outputs)word list has 2 occurrences of apple, out of 7 wordsword list has 2 occurrences of orange, out of 7 words>>> results('word list has 2 occurrences of orange, out of 7 words', 'word list has 2 occurrences of apple, out of 7 words')

As can be seen above, the scheduler computed only the requested outputs
('print3' was never computed). This is because the scheduler internally
calls cull, which removes the unnecessary tasks from the graph. Even though
this is done internally in the scheduler, it can be beneficial to call it at
the start of a series of optimizations to reduce the amount of work done in
later steps:

Looking at the task graph above, there are multiple accesses to constants such
as 'val1' or 'val2' in the dask graph. These can be inlined into the
tasks to improve efficiency using the inline function. For example:

>>> fromdask.optimizationimportinline>>> dsk2=inline(dsk1,dependencies=dependencies)>>> results=get(dsk2,outputs)word list has 2 occurrences of apple, out of 7 wordsword list has 2 occurrences of orange, out of 7 words

Now we have two sets of almost linear task chains. The only link between them
is the word counting function. For cheap operations like this, the
serialization cost may be larger than the actual computation, so it may be
faster to do the computation more than once, rather than passing the results to
all nodes. To perform this function inlining, the inline_functions function
can be used:

>>> fromdask.optimizationimportinline_functions>>> dsk3=inline_functions(dsk2,outputs,[len,str.split],... dependencies=dependencies)>>> results=get(dsk3,outputs)word list has 2 occurrences of apple, out of 7 wordsword list has 2 occurrences of orange, out of 7 words

Now we have a set of purely linear tasks. We’d like to have the scheduler run
all of these on the same worker to reduce data serialization between workers.
One option is just to merge these linear chains into one big task using the
fuse function:

>>> fromdask.optimizationimportfuse>>> dsk4,dependencies=fuse(dsk3)>>> results=get(dsk4,outputs)word list has 2 occurrences of apple, out of 7 wordsword list has 2 occurrences of orange, out of 7 words

Fused linear tasks together to ensure they run on the same worker using fuse.

As stated previously, these optimizations are already performed automatically
in the dask collections. Users not working with custom graphs or computations
should rarely need to directly interact with them.

These are just a few of the optimizations provided in dask.optimization. For
more information, see the API below.

For context based optimizations, dask.rewrite provides functionality for
pattern matching and term rewriting. This is useful for replacing expensive
computations with equivalent, cheaper computations. For example, dask.array
uses the rewrite functionality to replace series of array slicing operations
with a more efficient single slice.

The interface to the rewrite system consists of two classes:

RewriteRule(lhs,rhs,vars)

Given a left-hand-side (lhs), a right-hand-side (rhs), and a set of
variables (vars), a rewrite rule declaratively encodes the following
operation:

lhs->rhsiftaskmatcheslhsovervariables

RuleSet(*rules)

A collection of rewrite rules. The design of RuleSet class allows for
efficient “many-to-one” pattern matching, meaning that there is minimal
overhead for rewriting with multiple rules in a rule set.

The RewriteRule objects describe the desired transformations in a
declarative way, and the RuleSet builds an efficient automata for applying
that transformation. Rewriting can then be done using the rewrite method:

The whole task is traversed by default. If you only want to apply a transform
to the top-level of the task, you can pass in strategy='top_level' as shown:

# Transforms whole task>>>rs.rewrite((sum,[(add,3,3),(mul,3,3)]))(sum,[(mul,3,2),(pow,3,2)])# Only applies to top level, no transform occurs>>>rs.rewrite((sum,[(add,3,3),(mul,3,3)]),strategy='top_level')(sum,[(add,3,3),(mul,3,3)])

The rewriting system provides a powerful abstraction for transforming
computations at a task level. Again, for many users, directly interacting with
these transformations will be unnecessary.

Some optimizations take optional keyword arguments. To pass keywords from the
compute call down to the right optimization, prepend the keyword with the name
of the optimization. For example to send a keys= keyword argument to the
fuse optimization from a compute call, use the fuse_keys= keyword:

Dask defines a default optimization strategy for each collection type (Array,
Bag, DataFrame, Delayed). However different applications may have different
needs. To address this variability of needs, you can construct your own custom
optimization function and use it instead of the default. An optimization
function takes in a task graph and list of desired keys and returns a new
task graph.

defmy_optimize_function(dsk,keys):new_dsk={...}returnnew_dsk

You can then register this optimization class against whichever collection type
you prefer and it will be used instead of the default scheme.

This trades parallelism opportunities for faster scheduling by making tasks
less granular. It can replace fuse_linear in optimization passes.

This optimization applies to all reductions–tasks that have at most one
dependent–so it may be viewed as fusing “multiple input, single output”
groups of tasks into a single task. There are many parameters to fine
tune the behavior, which are described below. ave_width is the
natural parameter with which to compare parallelism to granularity, so
it should always be specified. Reasonable values for other parameters
with be determined using ave_width if necessary.

Parameters:

dsk: dict

dask graph

keys: list or set, optional

Keys that must remain in the returned dask graph

dependencies: dict, optional

{key: [list-of-keys]}. Must be a list to provide count of each key
This optional input often comes from cull

ave_width: float (default 2)

Upper limit for width=num_nodes/height, a good measure of
parallelizability

max_width: int

Don’t fuse if total width is greater than this

max_height: int

Don’t fuse more than this many levels

max_depth_new_edges: int

Don’t fuse if new dependencies are added after this many levels

rename_keys: bool or func, optional

Whether to rename the fused keys with default_fused_keys_renamer
or not. Renaming fused keys can keep the graph more understandable
and comprehensive, but it comes at the cost of additional processing.
If False, then the top-most key will be used. For advanced usage, a
function to create the new name is also accepted.

The right-hand-side of the rewrite rule. If it’s a task, variables in
rhs will be replaced by terms in the subject that match the variables
in lhs. If it’s a function, the function will be called with a dict
of such matches.

vars: tuple, optional

Tuple of variables found in the lhs. Variables can be represented as
any hashable object; a good convention is to use strings. If there are
no variables, this can be omitted.

Examples

Here’s a RewriteRule to replace all nested calls to list, so that
(list, (list, ‘x’)) is replaced with (list, ‘x’), where ‘x’ is a
variable.

Here’s a more complicated rule that uses a callable right-hand-side. A
callable rhs takes in a dictionary mapping variables to their matching
values. This rule replaces all occurrences of (list, ‘x’) with ‘x’ if
‘x’ is a list itself.