When the dynamic resource allocation feature is enabled, an application's use of executors
is dynamically adjusted based on workload. This means that an application can relinquish
resources when the resources are no longer needed, and request them later when there is more
demand. This feature is particularly useful if multiple applications share resources in your
Spark cluster.

Dynamic resource allocation is available for use by the Spark Thrift server and general
Spark jobs.

Note

Dynamic Resource Allocation does not work with Spark Streaming.

You can configure dynamic resource allocation at either the cluster or the job
level:

Cluster level:

On an Ambari-managed cluster, the Spark Thrift server uses dynamic resource
allocation by default. The Thrift server increases or decreases the number of
running executors based on a specified range, depending on load. (In addition, the
Thrift server runs in YARN mode by default, so the Thrift server uses resources
from the YARN cluster.) The associated shuffle service starts automatically, for
use by the Thrift server and general Spark jobs.

On a manually installed cluster, dynamic resource allocation is not enabled by
default for the Thrift server or for other Spark applications. You can enable and
configure dynamic resource allocation and start the shuffle service during the
Spark manual installation or upgrade process.