The ProgressBar class builds on the scheduler callbacks described above to
display a progress bar in the terminal or notebook during computation. This can
be a nice feedback during long running graph execution. It can be used as a
context manager around calls to get or compute to profile the
computation:

The CacheProfiler class is used to profile dask execution at the scheduler
cache level. During execution it records the following information for each
task:

Key

Task

Size metric

Cache entry time in seconds since the epoch

Cache exit time in seconds since the epoch

Where the size metric is the output of a function called on the result of each
task. The default metric is to count each task (metric is 1 for all tasks).
Other functions may be used as a metric instead through the metric keyword.
For example, the nbytes function found in cachey can be used to measure
the number of bytes in the scheduler cache:

As an example to demonstrate using the diagnostics, we’ll profile some linear
algebra done with dask.array. We’ll create a random array, take its QR
decomposition, and then reconstruct the initial array by multiplying the Q and
R components together. Note that since the profilers (and all diagnostics) are
just context managers, multiple profilers can be used in a with block:

The results from the Profiler object. This shows the execution time for
each task as a rectangle, organized along the y-axis by worker (in this case
threads). Similar tasks are grouped by color, and by hovering over each task
one can see the key and task that each block represents.

The results from the ResourceProfiler object. This shows two lines, one
for total CPU percentage used by all the workers, and one for total memory
usage.

The results from the CacheProfiler object. This shows a line for each
task group, plotting the sum of the current metric in the cache against
time. In this case it’s the default metric (count), and the lines represent
the number of each object in the cache at time. Note that the grouping and
coloring is the same as for the Profiler plot, and that the task
represented by each line can be found by hovering over the line.

From these plots we can see that the initial tasks (calls to
numpy.random.random and numpy.linalg.qr for each chunk) are run
concurrently, but only use slightly more than 100% CPU. This is because the
call to numpy.linalg.qr currently doesn’t release the global interpreter
lock, so those calls can’t truly be done in parallel. Next, there’s a reduction
step where all the blocks are combined. This requires all the results from the
first step to be held in memory, as shown by the increased number of results in
the cache, and increase in memory usage. Immediately after this task ends, the
number of elements in the cache decreases, showing that they were only needed
for this step. Finally, there’s an interleaved set of calls to dot and
sum. Looking at the CPU plot shows that these run both concurrently and in
parallel, as the CPU percentage spikes up to around 350%.

Schedulers based on dask.local.get_async (currently
dask.get, dask.threaded.get, and dask.multiprocessing.get)
accept five callbacks, allowing for inspection of scheduler execution.

The callbacks are:

start(dsk)

Run at the beginning of execution, right before the state is initialized.
Receives the dask graph.

start_state(dsk,state)

Run at the beginning of execution, right after the state is initialized.
Receives the dask graph and scheduler state.

pretask(key,dsk,state)

Run every time a new task is started. Receives the key of the task to be
run, the dask graph, and the scheduler state.

posttask(key,result,dsk,state,id)

Run every time a task is finished. Receives the key of the task that just
completed, the result, the dask graph, the scheduler state, and the id of
the worker that ran the task.

finish(dsk,state,errored)

Run at the end of execution, right before the result is returned. Receives
the dask graph, the scheduler state, and a boolean indicating whether or not
the exit was due to an error.

Custom diagnostics can be created either by instantiating the Callback
class with the some of above methods as keywords or by subclassing the
Callback class.
Here we create a class that prints the name of every key as it’s computed:

fromdask.callbacksimportCallbackclassPrintKeys(Callback):def_pretask(self,key,dask,state):"""Print the key of every task as it's started"""print("Computing: {0}!".format(repr(key)))

These results can be visualized in a bokeh plot using the visualize
method. Note that this requires bokeh to be installed.

>>> prof.visualize()

You can activate the profiler globally

>>> prof.register()

If you use the profiler globally you will need to clear out old results
manually.

>>> prof.clear()

Note that when used as a context manager data will be collected throughout
the duration of the enclosed block. In contrast, when registered globally
data will only be collected while a dask scheduler is active.

The default is to count each task (metric is 1 for all tasks). Other
functions may used as a metric instead through the metric keyword. For
example, the nbytes function found in cachey can be used to measure
the number of bytes in the cache.