By swapping out in-memory numpy arrays with in-memory sparse arrays we can
reuse the blocked algorithms of Dask.array to achieve parallel and distributed
sparse arrays.

The blocked algorithms in Dask.array normally parallelize around in-memory
numpy arrays. However, if another in-memory array library supports the NumPy
interface then it too can take advantage of dask.array’s parallel algorithms.
In particular the sparse array library
satisfies a subset of the NumPy API and works well with, and is tested against,
Dask.array.

We can convert each of these chunks of NumPy arrays into a sparse.COO array.

importsparses=x.map_blocks(sparse.COO)

Now our array is composed not of many NumPy arrays, but rather of many
sparse arrays. Semantically this does not change anything. Operations that
work will work identically (assuming that the behavior of numpy and
sparse are identical) but performance characteristics and storage costs may
change significantly

Any in-memory library that copies the NumPy ndarray interface should work here.
The sparse library is a minimal
example. In particular an in-memory library should implement at least the
following operations:

A concatenate function matching the interface of np.concatenate.
This must be registered in dask.array.core.concatenate_lookup.

All ufuncs must support the full ufunc interface, including dtype= and
out= parameters (even if they don’t function properly)

All reductions must support the full axis= and keepdims= keywords
and behave like numpy in this respect

The array class should follow the __array_priority__ protocol and be
prepared to respond to other arrays of lower priority.

If dot support is desired, a tensordot function matching the
interface of np.tensordot should be registered in
dask.array.core.tensordot_lookup.

The implementation of other operations like reshape, transpose, etc.
should follow standard NumPy conventions regarding shape and dtype. Not
implementing these is fine; the parallel dask.array will err at runtime if
these operations are attempted.

Dask.array supports mixing different kinds of in-memory arrays. This relies
on the in-memory arrays knowing how to interact with each other when necessary.
When two arrays interact the functions from the array with the highest
__array_priority__ will take precedence (for example for concatenate,
tensordot, etc.).