For advanced use only. A string to be inserted into spark-conf/spark-env.sh for this role only.

spark-conf/spark-env.sh_role_safety_valve

false

Logs

Display Name

Description

Related Name

Default Value

API Name

Required

History Server Log Directory

The log directory for log files of the role History Server.

log_dir

/var/log/spark

log_dir

false

History Server Logging Threshold

The minimum log level for History Server logs

INFO

log_threshold

false

History Server Maximum Log File Backups

The maximum number of rolled log files to keep for History Server logs. Typically used by log4j or logback.

10

max_log_backup_index

false

History Server Max Log Size

The maximum size, in megabytes, per log file for History Server logs. Typically used by log4j or logback.

200 MiB

max_log_size

false

Monitoring

Display Name

Description

Related Name

Default Value

API Name

Required

Enable Health Alerts for this Role

When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting
eventserver_health_events_alert_threshold

true

enable_alerts

false

Enable Configuration Change Alerts

When set, Cloudera Manager will send alerts when this entity's configuration changes.

false

enable_config_alerts

false

Process Swap Memory Thresholds

The health test thresholds on the swap memory usage of the process.

Warning: Any, Critical: Never

process_swap_memory_thresholds

false

Role Triggers

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health
system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:

triggerName(mandatory) - The name of the trigger. This value must be unique for the specific role.

streamThreshold(optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition
fires. By default set to 0, and any stream returned causes the condition to fire.

enabled(optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.

expressionEditorConfig(optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the
Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:[{"triggerName": "sample-trigger",
"triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for
more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

[]

role_triggers

true

File Descriptor Monitoring Thresholds

The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.

Warning: 50.0 %, Critical: 70.0 %

spark_history_server_fd_thresholds

false

History Server Host Health Test

When computing the overall History Server health, consider the host's health.

true

spark_history_server_host_health_enabled

false

History Server Process Health Test

Enables the health test that the History Server's process state is consistent with the role configuration

true

spark_history_server_scm_health_enabled

false

Unexpected Exits Thresholds

The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window
configuration for the role.

Performance

If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.

rlimit_fds

false

Ports and Addresses

Display Name

Description

Related Name

Default Value

API Name

Required

History Server WebUI Port

The port of the history server WebUI

history.port

18088

history_server_web_port

true

Resource Management

Display Name

Description

Related Name

Default Value

API Name

Required

Cgroup CPU Shares

Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be
given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.

cpu.shares

1024

rm_cpu_shares

true

Cgroup I/O Weight

Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host
experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.

blkio.weight

500

rm_io_weight

true

Cgroup Memory Hard Limit

Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages
charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default
processes not managed by Cloudera Manager will have no limit.

memory.limit_in_bytes

-1 MiB

rm_memory_hard_limit

true

Cgroup Memory Soft Limit

Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages
charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use
a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.

memory.soft_limit_in_bytes

-1 MiB

rm_memory_soft_limit

true

Suppressions

Display Name

Description

Related Name

Default Value

API Name

Required

Suppress Configuration Validator: CDH Version Validator

Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.

Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Environment Advanced
Configuration Snippet (Safety Valve) parameter.

false

role_config_suppression_spark_history_server_role_env_safety_valve

true

Suppress Health Test: File Descriptors

Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when
computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

false

role_health_suppression_spark_history_server_file_descriptor

true

Suppress Health Test: Host Health

Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the
overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

false

role_health_suppression_spark_history_server_host_health

true

Suppress Health Test: Process Status

Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing
the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

false

role_health_suppression_spark_history_server_scm_health

true

Suppress Health Test: Swap Memory Usage

Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when
computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

false

role_health_suppression_spark_history_server_swap_memory_usage

true

Suppress Health Test: Unexpected Exits

Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when
computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

false

role_health_suppression_spark_history_server_unexpected_exits

true

master

Advanced

Display Name

Description

Related Name

Default Value

API Name

Required

Master Environment Advanced Configuration Snippet (Safety Valve)

For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of
this role except client configuration.

SPARK_MASTER_role_env_safety_valve

false

Master Logging Advanced Configuration Snippet (Safety Valve)

For advanced use only, a string to be inserted into log4j.properties for this role only.

log4j_safety_valve

false

Automatically Restart Process

When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.

For advanced use only. A string to be inserted into spark-conf/spark-env.sh for this role only.

spark-conf/spark-env.sh_role_safety_valve

false

Logs

Display Name

Description

Related Name

Default Value

API Name

Required

Master Log Directory

The log directory for log files of the role Master.

log.dir

/var/log/spark

log_dir

false

Master Logging Threshold

The minimum log level for Master logs

INFO

log_threshold

false

Master Maximum Log File Backups

The maximum number of rolled log files to keep for Master logs. Typically used by log4j or logback.

10

max_log_backup_index

false

Master Max Log Size

The maximum size, in megabytes, per log file for Master logs. Typically used by log4j or logback.

200 MiB

max_log_size

false

Monitoring

Display Name

Description

Related Name

Default Value

API Name

Required

Enable Health Alerts for this Role

When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting
eventserver_health_events_alert_threshold

true

enable_alerts

false

Enable Configuration Change Alerts

When set, Cloudera Manager will send alerts when this entity's configuration changes.

false

enable_config_alerts

false

Process Swap Memory Thresholds

The health test thresholds on the swap memory usage of the process.

Warning: Any, Critical: Never

process_swap_memory_thresholds

false

Role Triggers

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health
system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:

triggerName(mandatory) - The name of the trigger. This value must be unique for the specific role.

streamThreshold(optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition
fires. By default set to 0, and any stream returned causes the condition to fire.

enabled(optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.

expressionEditorConfig(optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the
Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:[{"triggerName": "sample-trigger",
"triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for
more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

[]

role_triggers

true

File Descriptor Monitoring Thresholds

The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.

Warning: 50.0 %, Critical: 70.0 %

spark_master_fd_thresholds

false

Master Host Health Test

When computing the overall Master health, consider the host's health.

true

spark_master_host_health_enabled

false

Master Process Health Test

Enables the health test that the Master's process state is consistent with the role configuration

true

spark_master_scm_health_enabled

false

Unexpected Exits Thresholds

The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window
configuration for the role.

Warning: Never, Critical: Any

unexpected_exits_thresholds

false

Unexpected Exits Monitoring Period

The period to review when computing unexpected exits.

5 minute(s)

unexpected_exits_window

false

Other

Display Name

Description

Related Name

Default Value

API Name

Required

Additional Master args

Additional arguments for the master

additional.args

master_additional_args

false

Master Address

Overrides the address where the Master will listen for connections. Note: Spark will not work if the wildcard (0.0.0.0) address is
used here.

Performance

If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.

rlimit_fds

false

Ports and Addresses

Display Name

Description

Related Name

Default Value

API Name

Required

Master Port

The port of the master

server.port

7077

master_port

true

Master WebUI Port

The port of the master WebUI

webui.port

18080

master_webui_port

true

Resource Management

Display Name

Description

Related Name

Default Value

API Name

Required

Cgroup CPU Shares

Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be
given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.

cpu.shares

1024

rm_cpu_shares

true

Cgroup I/O Weight

Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host
experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.

blkio.weight

500

rm_io_weight

true

Cgroup Memory Hard Limit

Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages
charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default
processes not managed by Cloudera Manager will have no limit.

memory.limit_in_bytes

-1 MiB

rm_memory_hard_limit

true

Cgroup Memory Soft Limit

Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages
charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use
a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.

memory.soft_limit_in_bytes

-1 MiB

rm_memory_soft_limit

true

Suppressions

Display Name

Description

Related Name

Default Value

API Name

Required

Suppress Configuration Validator: CDH Version Validator

Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.

Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when
computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

false

role_health_suppression_spark_master_file_descriptor

true

Suppress Health Test: Host Health

Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the
overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

false

role_health_suppression_spark_master_host_health

true

Suppress Health Test: Process Status

Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing
the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

false

role_health_suppression_spark_master_scm_health

true

Suppress Health Test: Swap Memory Usage

Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when
computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

false

role_health_suppression_spark_master_swap_memory_usage

true

Suppress Health Test: Unexpected Exits

Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when
computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

For advanced use only, a string to be inserted into spark-conf/spark-env.sh. Applies to configurations
of all roles in this service except client configuration.

spark-conf/spark-env.sh_service_safety_valve

false

Monitoring

Display Name

Description

Related Name

Default Value

API Name

Required

Enable Service Level Health Alerts

When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer
setting eventserver_health_events_alert_threshold

true

enable_alerts

false

Enable Configuration Change Alerts

When set, Cloudera Manager will send alerts when this entity's configuration changes.

false

enable_config_alerts

false

Service Triggers

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the
health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:

triggerName(mandatory) - The name of the trigger. This value must be unique for the specific service.

streamThreshold(optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition
fires. By default set to 0, and any stream returned causes the condition to fire.

enabled(optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.

expressionEditorConfig(optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the
Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:[{"triggerName": "sample-trigger",
"triggerExpression": "IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad", "streamThreshold": 10, "enabled": "true"}]See the trigger rules documentation for
more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark (Standalone) Service Advanced
Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.

For advanced use only. A string to be inserted into spark-conf/spark-env.sh for this role only.

spark-conf/spark-env.sh_role_safety_valve

false

Logs

Display Name

Description

Related Name

Default Value

API Name

Required

Worker Log Directory

The log directory for log files of the role Worker.

log.dir

/var/log/spark

log_dir

false

Worker Logging Threshold

The minimum log level for Worker logs

INFO

log_threshold

false

Worker Maximum Log File Backups

The maximum number of rolled log files to keep for Worker logs. Typically used by log4j or logback.

10

max_log_backup_index

false

Worker Max Log Size

The maximum size, in megabytes, per log file for Worker logs. Typically used by log4j or logback.

200 MiB

max_log_size

false

Monitoring

Display Name

Description

Related Name

Default Value

API Name

Required

Enable Health Alerts for this Role

When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting
eventserver_health_events_alert_threshold

true

enable_alerts

false

Enable Configuration Change Alerts

When set, Cloudera Manager will send alerts when this entity's configuration changes.

false

enable_config_alerts

false

Process Swap Memory Thresholds

The health test thresholds on the swap memory usage of the process.

Warning: Any, Critical: Never

process_swap_memory_thresholds

false

Role Triggers

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health
system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:

triggerName(mandatory) - The name of the trigger. This value must be unique for the specific role.

streamThreshold(optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition
fires. By default set to 0, and any stream returned causes the condition to fire.

enabled(optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.

expressionEditorConfig(optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the
Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:[{"triggerName": "sample-trigger",
"triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for
more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

[]

role_triggers

true

File Descriptor Monitoring Thresholds

The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.

Warning: 50.0 %, Critical: 70.0 %

spark_worker_fd_thresholds

false

Worker Host Health Test

When computing the overall Worker health, consider the host's health.

true

spark_worker_host_health_enabled

false

Worker Process Health Test

Enables the health test that the Worker's process state is consistent with the role configuration

true

spark_worker_scm_health_enabled

false

Unexpected Exits Thresholds

The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window
configuration for the role.

Warning: Never, Critical: Any

unexpected_exits_thresholds

false

Unexpected Exits Monitoring Period

The period to review when computing unexpected exits.

5 minute(s)

unexpected_exits_window

false

Other

Display Name

Description

Related Name

Default Value

API Name

Required

Total Java Heap Sizes of Worker's Executors in Bytes

Memory available to the Worker's Executors. This is the maximum sum total of all the Executors' Java heap sizes on this Worker node.
Passed to Java -Xmx. Measured in bytes.

Performance

If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.

rlimit_fds

false

Ports and Addresses

Display Name

Description

Related Name

Default Value

API Name

Required

Worker Port

The port of the worker

server.port

7078

worker_port

true

Worker WebUI Port

The port of the worker WebUI

webui.port

18081

worker_webui_port

true

Resource Management

Display Name

Description

Related Name

Default Value

API Name

Required

Cgroup CPU Shares

Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be
given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.

cpu.shares

1024

rm_cpu_shares

true

Cgroup I/O Weight

Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host
experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.

blkio.weight

500

rm_io_weight

true

Cgroup Memory Hard Limit

Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages
charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default
processes not managed by Cloudera Manager will have no limit.

memory.limit_in_bytes

-1 MiB

rm_memory_hard_limit

true

Cgroup Memory Soft Limit

Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages
charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use
a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.

memory.soft_limit_in_bytes

-1 MiB

rm_memory_soft_limit

true

Suppressions

Display Name

Description

Related Name

Default Value

API Name

Required

Suppress Configuration Validator: CDH Version Validator

Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.

Whether to suppress configuration warnings produced by the built-in parameter validation for the Work directory parameter.

false

role_config_suppression_work_directory

true

Suppress Parameter Validation: Additional Worker args

Whether to suppress configuration warnings produced by the built-in parameter validation for the Additional Worker args
parameter.

false

role_config_suppression_worker_additional_args

true

Suppress Health Test: File Descriptors

Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when
computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

false

role_health_suppression_spark_worker_file_descriptor

true

Suppress Health Test: Host Health

Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the
overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

false

role_health_suppression_spark_worker_host_health

true

Suppress Health Test: Process Status

Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing
the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

false

role_health_suppression_spark_worker_scm_health

true

Suppress Health Test: Swap Memory Usage

Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when
computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

false

role_health_suppression_spark_worker_swap_memory_usage

true

Suppress Health Test: Unexpected Exits

Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when
computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.

If this documentation includes code, including but not limited to, code examples, Cloudera makes this available to you under the terms of the Apache License, Version 2.0, including any required
notices. A copy of the Apache License Version 2.0 can be found here.