Monitoring YARN Applications

The YARN Applications page displays information about the YARN jobs that are running and have run in your cluster. You can filter the jobs by time period and by specifying simple filtering expressions.

Export a JSON file with the query results that you can use for further analysis.

Click Export.

Configuring YARN Application Monitoring

You can configure the visibility of the YARN application monitoring results.

For information on how to configure whether admin and non-admin users can view all applications, only that user's applications, or no applications, see Configuring Application Visibility.

Results Tab

Jobs are ordered with the most recent at the top. Each job has summary and detail information. A job summary includes start and end timestamps, query (if the job is part of a Hive query)
name, pool, job type, job ID, and user. For example:

A running job displays a progress bar under the start timestamp:

Use the Actions drop-down menu to the right of each job listing to do the following. (Not all
options display, depending on the type of job.)

Filtering Jobs

You filter jobs by selecting a time range and specifying a filter expression in the search box.

You can use the Time Range Selector or a duration link ( ) to set the time range.
(See Time Line for details).

Filter Expressions

Filter expressions specify which entries should display when you run the filter. The simplest expression consists of three components:

Attribute - Query language name of the attribute.

Operator - Type of comparison between the attribute and the attribute value. Cloudera Manager supports the standard comparator operators =, !=, >, <, >=, <=, and RLIKE. (RLIKE performs regular expression matching as specified in the Java Pattern class documentation.) Numeric values can be compared with all operators. String values can be
compared with =, !=, and RLIKE. Boolean values can be compared with = and
!=.

Value - The value of the attribute. The value depends on the type of the attribute. For a Boolean value, specify either true or false. When specifying a string value, enclose the value in double quotes.

You create compound filter expressions using the AND and OR operators. When more than one operator is used in an expression, AND is evaluated first, then OR. To change the order of
evaluation, enclose subexpressions in parentheses.

Compound Expressions

To find all the jobs issued by the root user that ran for longer than ten seconds, use the expression:

user = "root" AND application_duration >= 100000.0

To find all the jobs that had more than 200 maps issued by users Jack or Jill, use the expression:

maps_completed >= 200.0 AND (user = "Jack" OR user = "Jill")

Choosing and Running a Filter

Do one of the following:

Select a Suggested or Recently Run Filter

Click the to the right of the Search button to
display a list of sample and recently run filters, and select a filter. The filter text displays in the text box.

Construct a Filter from the Workload Summary Attributes

Optionally, click Select Attributes to display a dialog box where you can chose attributes to display in the Workload
Summary section. Select the checkbox next to one or more attributes and click Close. Only attributes that support filtering appear in the Workload Summary section. See the Attributes table.

The attributes display in the Workload Summary section along with values or ranges of values that you can filter on. The values and ranges display as
links with checkboxes. Select one or more checkboxes to add the range or value to the query. Click a link to run a query on that value or range. For example:

Type a Filter

Start typing or press Spacebar in the text box. As you type, filter attributes matching the typed letter display. If you press Spacebar, standard filter attributes display. These suggestions are part of typeahead, which helps build valid queries. For information about the attribute name and supported
values for each field, hover over the field in an existing query.

Select an attribute and press Enter.

Press Spacebar to display a drop-down list of operators.

Select an operator and press Enter.

Specify an attribute value in one of the following ways:

For attribute values that support typeahead, press Spacebar to display a drop-down list of values and press Enter.

Type a value.

Click in the text box and press Enter or click Search. The list displays the results that match the specified
filter. If the histograms are showing, they are redrawn to show only the values for the selected filter. The filter is added to the Recently Run list.

Filter Attributes

Filter attributes, their names as they are displayed in Cloudera Manager, their types, and descriptions, are enumerated below.
Note:Only attributes where the Supports Filtering? column
value is TRUE appear in the Workload Summary section.

Attributes

Display Name

(Attribute Name)

Type

Supports Filtering?

Description

Allocated Memory

(allocated_mb)

NUMBER

FALSE

The sum of memory in MB allocated to the application's running containers. Called 'allocated_mb' in searches.

Allocated Memory Seconds

(allocated_memory_seconds)

NUMBER

TRUE

The amount of memory the application has allocated (megabyte-seconds). Called 'allocated_memory_seconds' in
searches.

Allocated VCores

(allocated_vcores)

NUMBER

FALSE

The sum of virtual cores allocated to the application's running containers. Called 'allocated_vcores' in searches.

Allocated VCore Seconds

(allocated_vcore_seconds)

NUMBER

TRUE

The amount of CPU resources the application has allocated (virtual core-seconds). Called 'allocated_vcore_seconds' in
searches.

Application ID

(application_id)

STRING

FALSE

The ID of the YARN application. Called 'application_id' in searches.

Application State

(state)

STRING

TRUE

The state of this YARN application. This reflects the ResourceManager state while the application is running and the
JobHistory Server state after the application has completed. Called 'state' in searches.

Application Tags

(application_tags)

STRING

FALSE

A list of tags for the application. Called 'application_tags' in searches.

Application Type

(application_type)

STRING

TRUE

The type of the YARN application. Called 'application_type' in searches.

Bytes Read

(bytes_read)

BYTES

TRUE

Bytes read. Called 'bytes_read' in searches.

Bytes Written

(bytes_written)

BYTES

TRUE

Bytes written. Called 'bytes_written' in searches.

Combine Input Records

(combine_input_records)

NUMBER

TRUE

Combine input records. Called 'combine_input_records' in searches.

Combine Output Records

(combine_output_records)

NUMBER

TRUE

Combine output records. Called 'combine_output_records' in searches.

Committed Heap

(committed_heap_bytes)

BYTES

TRUE

Total committed heap usage. Called 'committed_heap_bytes' in searches.

Completed Maps and Reduces

(tasks_completed)

NUMBER

TRUE

The number of completed map and reduce tasks in this MapReduce job. Called 'tasks_completed' in searches. Available
only for running jobs.

CPU Allocation

(vcores_millis)

NUMBER

TRUE

CPU allocation. This is the sum of 'vcores_millis_maps' and 'vcores_millis_reduces'. Called 'vcores_millis' in
searches.

CPU Time

(cpu_milliseconds)

MILLISECONDS

TRUE

CPU time. Called 'cpu_milliseconds' in searches.

Data Local Maps

(data_local_maps)

NUMBER

TRUE

Data local maps. Called 'data_local_maps' in searches.

Data Local Maps Percentage

(data_local_maps_percentage)

NUMBER

TRUE

The number of data local maps as a percentage of the total number of maps. Called 'data_local_maps_percentage' in
searches.

Diagnostics

(diagnostics)

STRING

FALSE

Diagnostic information on the YARN application. If the diagnostic information is long, this may only contain the
beginning of the information. Called 'diagnostics' in searches.

Duration

(application_duration)

MILLISECONDS

TRUE

How long YARN took to run this application. Called 'application_duration' in searches.

Executing

(executing)

BOOLEAN

FALSE

Whether the YARN application is currently running. Called 'executing' in searches.

Failed Map and Reduce Attempts

(failed_tasks_attempts)

NUMBER

TRUE

The number of failed map and reduce attempts for this MapReduce job. Called 'failed_tasks_attempts' in searches.
Available only for failed jobs.

Failed Map Attempts

(failed_map_attempts)

NUMBER

TRUE

The number of failed map attempts for this MapReduce job. Called 'failed_map_attempts' in searches. Available only for
running jobs.

Failed Maps

(num_failed_maps)

NUMBER

TRUE

Failed maps. Called 'num_failed_maps' in searches.

Failed Reduce Attempts

(failed_reduce_attempts)

NUMBER

TRUE

The number of failed reduce attempts for this MapReduce job. Called 'failed_reduce_attempts' in searches. Available
only for running jobs.

Failed Reduces

(num_failed_reduces)

NUMBER

TRUE

Failed reduces. Called 'num_failed_reduces' in searches.

Failed Shuffles

(failed_shuffle)

NUMBER

TRUE

Failed shuffles. Called 'failed_shuffle' in searches.

Failed Tasks

(num_failed_tasks)

NUMBER

TRUE

The total number of failed tasks. This is the sum of 'num_failed_maps' and 'num_failed_reduces'. Called
'num_failed_tasks' in searches.

Total fallow slots time. This is the sum of 'fallow_slots_millis_maps' and 'fallow_slots_millis_reduces'. Called
'fallow_slots_millis' in searches.

File Bytes Read

(file_bytes_read)

BYTES

TRUE

File bytes read. Called 'file_bytes_read' in searches.

File Bytes Written

(file_bytes_written)

BYTES

TRUE

File bytes written. Called 'file_bytes_written' in searches.

File Large Read Operations

(file_large_read_ops)

NUMBER

TRUE

File large read operations. Called 'file_large_read_ops' in searches.

File Read Operations

(file_read_ops)

NUMBER

TRUE

File read operations. Called 'file_read_ops' in searches.

File Write Operations

(file_write_ops)

NUMBER

TRUE

File write operations. Called 'file_large_write_ops' in searches.

Garbage Collection Time

(gc_time_millis)

MILLISECONDS

TRUE

Garbage collection time. Called 'gc_time_millis' in searches.

HDFS Bytes Read

(hdfs_bytes_read)

BYTES

TRUE

HDFS bytes read. Called 'hdfs_bytes_read' in searches.

HDFS Bytes Written

(hdfs_bytes_written)

BYTES

TRUE

HDFS bytes written. Called 'hdfs_bytes_written' in searches.

HDFS Large Read Operations

(hdfs_large_read_ops)

NUMBER

TRUE

HDFS large read operations. Called 'hdfs_large_read_ops' in searches.

HDFS Read Operations

(hdfs_read_ops)

NUMBER

TRUE

HDFS read operations. Called 'hdfs_read_ops' in searches.

HDFS Write Operations

(hdfs_write_ops)

NUMBER

TRUE

HDFS write operations. Called 'hdfs_write_ops' in searches.

Hive Query ID

(hive_query_id)

STRING

FALSE

If this MapReduce job ran as a part of a Hive query, this field contains the ID of the Hive query. Called
'hive_query_id' in searches.

Hive Query String

(hive_query_string)

STRING

TRUE

If this MapReduce job ran as a part of a Hive query, this field contains the string of the query. Called
'hive_query_string' in searches.

Hive Sentry Subject Name

(hive_sentry_subject_name)

STRING

TRUE

If this MapReduce job ran as a part of a Hive query on a secured cluster using impersonation, this field contains the
name of the user that initiated the query. Called 'hive_sentry_subject_name' in searches.

Input Directory

(input_dir)

STRING

TRUE

The input directory for this MapReduce job. Called 'input_dir' in searches.

Input Split Bytes

(split_raw_bytes)

BYTES

TRUE

Input split bytes. Called 'split_raw_bytes' in searches.

Killed Map and Reduce Attempts

(killed_tasks_attempts)

NUMBER

TRUE

The number of map and reduce attempts that were killed by user(s) for this MapReduce job. Called
'killed_tasks_attempts' in searches. Available only for killed jobs.

Killed Map Attempts

(killed_map_attempts)

NUMBER

TRUE

The number of map attempts killed by user(s) for this MapReduce job. Called 'killed_map_attempts' in searches.
Available only for running jobs.

Killed Reduce Attempts

(killed_reduce_attempts)

NUMBER

TRUE

The number of reduce attempts killed by user(s) for this MapReduce job. Called 'killed_reduce_attempts' in searches.
Available only for running jobs.

Launched Map Tasks

(total_launched_maps)

NUMBER

TRUE

Launched map tasks. Called 'total_launched_maps' in searches.

Launched Reduce Tasks

(total_launched_reduces)

NUMBER

TRUE

Launched reduce tasks. Called 'total_launched_reduces' in searches.

Map and Reduce Attempts in NEW State

(new_tasks_attempts)

NUMBER

TRUE

The number of map and reduce attempts in NEW state for this MapReduce job. Called 'new_tasks_attempts' in searches.
Available only for running jobs.

Map Attempts in NEW State

(new_map_attempts)

NUMBER

TRUE

The number of map attempts in NEW state for this MapReduce job. Called 'new_map_attempts' in searches. Available only
for running jobs.

Map Class

(mapper_class)

STRING

TRUE

The class used by the map tasks in this MapReduce job. Called 'mapper_class' in searches. You can search for the mapper
class using the class name alone, for example 'QuasiMonteCarlo$QmcMapper', or the fully qualified classname, for example, 'org.apache.hadoop.examples.QuasiMonteCarlo$QmcMapper'.

The percentage of maps completed for this MapReduce job. Called 'map_progress' in searches. Available only for running
jobs.

Maps Completed

(maps_completed)

NUMBER

TRUE

The number of map tasks completed as a part of this MapReduce job. Called 'maps_completed' in searches.

Map Slots Time

(slots_millis_maps)

MILLISECONDS

TRUE

Total time spent by all maps in occupied slots. Called 'slots_millis_maps' in searches.

Maps Pending

(maps_pending)

NUMBER

TRUE

The number of maps waiting to be run for this MapReduce job. Called 'maps_pending' in searches. Available only for
running jobs.

Maps Running

(maps_running)

NUMBER

TRUE

The number of maps currently running for this MapReduce job. Called 'maps_running' in searches. Available only for
running jobs.

Maps Total

(maps_total)

NUMBER

TRUE

The number of Map tasks in this MapReduce job. Called 'maps_total' in searches.

Memory Allocation

(mb_millis)

NUMBER

TRUE

Total memory allocation. This is the sum of 'mb_millis_maps' and 'mb_millis_reduces'. Called 'mb_millis' in
searches.

Merged Map Outputs

(merged_map_outputs)

NUMBER

TRUE

Merged map outputs. Called 'merged_map_outputs' in searches.

Name

(name)

STRING

TRUE

Name of the YARN application. Called 'name' in searches.

Oozie Workflow ID

(oozie_id)

STRING

FALSE

If this MapReduce job ran as a part of an Oozie workflow, this field contains the ID of the Oozie workflow. Called
'oozie_id' in searches.

Other Local Maps

(other_local_maps)

NUMBER

TRUE

Other local maps. Called 'other_local_maps' in searches.

Other Local Maps Percentage

(other_local_maps_percentage)

NUMBER

TRUE

The number of other local maps as a percentage of the total number of maps. Called 'other_local_maps_percentage' in
searches.

Output Directory

(output_dir)

STRING

TRUE

The output directory for this MapReduce job. Called 'output_dir' in searches.

Pending Maps and Reduces

(tasks_pending)

NUMBER

TRUE

The number of maps and reduces waiting to be run for this MapReduce job. Called 'tasks_pending' in searches. Available
only for running jobs.

Physical Memory

(physical_memory_bytes)

BYTES

TRUE

Physical memory. Called 'physical_memory_bytes' in searches.

Pig Script ID

(pig_id)

STRING

FALSE

If this MapReduce job ran as a part of a Pig script, this field contains the ID of the Pig script. Called 'pig_id' in
searches.

Pool

(pool)

STRING

TRUE

The name of the resource pool in which this application ran. Called 'pool' in searches. Within YARN, a pool is referred
to as a queue.

Progress

(progress)

NUMBER

TRUE

The progress reported by the application. Called 'progress' in searches.

Rack Local Maps

(rack_local_maps)

NUMBER

TRUE

Rack local maps. Called 'rack_local_maps' in searches.

Rack Local Maps Percentage

(rack_local_maps_percentage)

NUMBER

TRUE

The number of rack local maps as a percentage of the total number of maps. Called 'rack_local_maps_percentage' in
searches.

Reduce Attempts in NEW State

(new_reduce_attempts)

NUMBER

TRUE

The number of reduce attempts in NEW state for this MapReduce job. Called 'new_reduce_attempts' in searches. Available
only for running jobs.

Reduce Class

(reducer_class)

STRING

TRUE

The class used by the reduce tasks in this MapReduce job. Called 'reducer_class' in searches. You can search for the
reducer class using the class name alone, for example 'QuasiMonteCarlo$QmcReducer', or fully qualified classname, for example, 'org.apache.hadoop.examples.QuasiMonteCarlo$QmcReducer'.

Reduce CPU Allocation

(vcores_millis_reduces)

NUMBER

TRUE

Reduce CPU allocation. Called 'vcores_millis_reduces' in searches.

Reduce Input Groups

(reduce_input_groups)

NUMBER

TRUE

Reduce input groups. Called 'reduce_input_groups' in searches.

Reduce Input Records

(reduce_input_records)

NUMBER

TRUE

Reduce input records. Called 'reduce_input_records' in searches.

Reduce Memory Allocation

(mb_millis_reduces)

NUMBER

TRUE

Reduce memory allocation. Called 'mb_millis_reduces' in searches.

Reduce Output Records

(reduce_output_records)

NUMBER

TRUE

Reduce output records. Called 'reduce_output_records' in searches.

Reduce Progress

(reduce_progress)

NUMBER

TRUE

The percentage of reduces completed for this MapReduce job. Called 'reduce_progress' in searches. Available only for
running jobs.

Reduces Completed

(reduces_completed)

NUMBER

TRUE

The number of reduce tasks completed as a part of this MapReduce job. Called 'reduces_completed' in searches.

Reduce Shuffle Bytes

(reduce_shuffle_bytes)

BYTES

TRUE

Reduce shuffle bytes. Called 'reduce_shuffle_bytes' in searches.

Reduce Slots Time

(slots_millis_reduces)

MILLISECONDS

TRUE

Total time spent by all reduces in occupied slots. Called 'slots_millis_reduces' in searches.

Reduces Pending

(reduces_pending)

NUMBER

TRUE

The number of reduces waiting to be run for this MapReduce job. Called 'reduces_pending' in searches. Available only
for running jobs.

Reduces Running

(reduces_running)

NUMBER

TRUE

The number of reduces currently running for this MapReduce job. Called 'reduces_running' in searches. Available only
for running jobs.

Reduces Total

(reduces_total)

NUMBER

TRUE

The number of reduce tasks in this MapReduce job. Called 'reduces_total' in searches.

Running Containers

(running_containers)

NUMBER

FALSE

The number of containers currently running for the application. Called 'running_containers' in searches.

Running Map and Reduce Attempts

(running_tasks_attempts)

NUMBER

TRUE

The number of map and reduce attempts currently running for this MapReduce job. Called 'running_tasks_attempts' in
searches. Available only for running jobs.

Running Map Attempts

(running_map_attempts)

NUMBER

TRUE

The number of running map attempts for this MapReduce job. Called 'running_map_attempts' in searches. Available only
for running jobs.

Running MapReduce Application Information Retrieval Duration.

(running_application_info_retrieval_time)

NUMBER

TRUE

How long it took, in seconds, to retrieve information about the MapReduce application.

Running Maps and Reduces

(tasks_running)

NUMBER

TRUE

The number of maps and reduces currently running for this MapReduce job. Called 'tasks_running' in searches. Available
only for running jobs.

Running Reduce Attempts

(running_reduce_attempts)

NUMBER

TRUE

The number of running reduce attempts for this MapReduce job. Called 'running_reduce_attempts' in searches. Available
only for running jobs.

Total slots time. This is the sum of 'slots_millis_maps' and 'slots_millis_reduces'. Called 'slots_millis' in
searches.

Spilled Records

(spilled_records)

NUMBER

TRUE

Spilled Records. Called 'spilled_records' in searches.

Successful Map and Reduce Attempts

(successful_tasks_attempts)

NUMBER

TRUE

The number of successful map and reduce attempts for this MapReduce job. Called 'successful_tasks_attempts' in
searches. Available only for successful jobs.

Successful Map Attempts

(successful_map_attempts)

NUMBER

TRUE

The number of successful map attempts for this MapReduce job. Called 'successful_map_attempts' in searches. Available
only for running jobs.

Successful Reduce Attempts

(successful_reduce_attempts)

NUMBER

TRUE

The number of successful reduce attempts for this MapReduce job. Called 'successful_reduce_attempts' in searches.
Available only for running jobs.

Total Maps and Reduces Number

(total_task_num)

NUMBER

TRUE

The number of map and reduce tasks in this MapReduce job. Called 'tasks_total' in searches. Available only for running
jobs.

Total Tasks

(total_launched_tasks)

NUMBER

TRUE

The total number of tasks. This is the sum of 'total_launched_maps' and 'total_launched_reduces'. Called
'total_launched_tasks' in searches.

Tracking Url

(tracking_url)

STRING

FALSE

The MapReduce application tracking URL.

Uberized Job

(uberized)

BOOLEAN

FALSE

Whether this MapReduce job is uberized - running completely in the ApplicationMaster. Called 'uberized' in searches.
Available only for running jobs.

Unused Memory Seconds

(unused_memory_seconds)

NUMBER

TRUE

The amount of memory the application has allocated but not used (megabyte-seconds). This metric is available only from
CDH 5.7 onwards and is calculated hourly if container usage metric aggregation is enabled. Called 'unused_memory_seconds' in searches.

Unused VCore Seconds

(unused_vcore_seconds)

NUMBER

TRUE

The amount of CPU resources the application has allocated but not used (virtual core-seconds). This metric is available
only from CDH 5.7 onwards and is calculated hourly if container usage metric aggregation is enabled. Called 'unused_vcore_seconds' in searches.

User

(user)

STRING

TRUE

The user who ran the YARN application. Called 'user' in searches.

Virtual Memory

(virtual_memory_bytes)

BYTES

TRUE

Virtual memory. Called 'virtual_memory_bytes' in searches.

Work CPU Time

(cm_cpu_milliseconds)

MILLISECONDS

TRUE

Attribute measuring the sum of CPU time used by all threads of the query, in milliseconds. Called 'work_cpu_time' in
searches. For Impala queries, CPU time is calculated based on the 'TotalCpuTime' metric. For YARN MapReduce applications, this is calculated from the 'cpu_milliseconds' metric.

Sending Diagnostic Data to Cloudera for YARN Applications

You can send diagnostic data collected from YARN applications, including metadata, configurations, and log data, to Cloudera Support for analysis. Include a support ticket number if
one exists to enable Cloudera Support to address the issue more quickly and efficiently. To send YARN application diagnostic data, perform the following steps:

From the YARN page in Cloudera Manager, click the Applications menu.

On the upper right, above the list of YARN applications, click the button Collect Diagnostics Data.

Note:

You can also click the down arrow on the right-hand end of the row which contains the application you want to collect diagnostics on and select Collect
Diagnostics Data:

In the Send YARN Applications Diagnostic Data dialog box, provide the following information:

If applicable, the Cloudera Support ticket number of the issue being experienced on the cluster.

Optionally, add a comment to help the support team understand the issue.

If this documentation includes code, including but not limited to, code examples, Cloudera makes this available to you under the terms of the Apache License, Version 2.0, including any required
notices. A copy of the Apache License Version 2.0 can be found here.