Sometimes you have Dask Application you want to deploy completely on YARN,
without having a corresponding process running on an edge node. This may come
up with production applications deployed automatically, or long running jobs
you don’t want to consume edge node resources.

To handle these cases, dask-yarn provides a CLI Docs that can be used to
submit applications to be run on the YARN cluster asynchronously. There are
three commands that may be useful here:

This is because the script won’t be run until the cluster is already created -
at that point configuration passed to the YarnCluster constructor
won’t be useful. Cluster configuration is instead passed via the dask-yarnsubmit CLI (note that as before, the cluster can be
scaled dynamically after creation).