OpsCenter 4.0 User Guide

JavaScript must be enabled in order to use this site.

Please enable JavaScript in your browser and refresh the page.

Pending task metrics for cluster operations

Pending tasks for the following metrics indicate a backup of cluster operational processes
such as those maintaining node consistency, system schemas, fault detection, and inter-node
communications. Pending tasks for resource-intensive operations (such as repair, bootstrap
or decommission) are normal and expected while that operation is in progress, but should
continue decreasing at a steady rate in a healthy cluster.

Manual repair tasks pending

The number of operations still to be completed when you run anti-entropy repair on a node.
It will only show values greater than 0 when a repair is in progress. Repair is a
resource-intensive operation that is executed in stages: comparing data between replicas,
sending changed rows to the replicas that need to be made consistent, deleting expired
tombstones, and rebuilding row indexes and bloom filters. Tracking the state of this metric
can help you determine the progress of a repair operation. It is not unusual to see a large
number of pending tasks when a repair is running, but you should see the number of tasks
progressively decreasing.

Gossip tasks pending

Cassandra uses a protocol called gossip to discover location and state information about
the other nodes participating in a Cassandra cluster. In Cassandra, the gossip process runs
once per second on each node and exchanges state messages with up to three other nodes in
the cluster. Gossip tasks pending shows the number of gossip messages and acknowledgments
queued and waiting to be sent or received. The optimal number of pending gossip tasks is 0
(or at most a very small number). A value greater than 0 indicates possible network problems
(see network traffic for indications of network health).

Hinted handoff pending

While a node is offline, other nodes in the cluster will save hints about rows that were
updated during the time the node was unavailable. When a node comes back online, its
corresponding replicas will begin streaming the missed writes to the node to catch it up.
The hinted handoff pending metric tracks the number of hints that are queued and waiting to
be delivered once a failed node is back online again. High numbers of pending hints are
commonly seen when a node is brought back online after some down time. Viewing this metric
can help you determine when the recovering node has been made consistent again. Hinted
handoff is an optional feature of Cassandra. Hints are saved for a configurable period of
time (an hour by default) before they are dropped. This prevents a large accumulation of
hints caused by extended node outages.

Internal responses pending

The number of pending tasks from various internal tasks such as nodes joining and leaving
the cluster.

Migrations pending

The number of pending tasks from system methods that have modified the schema. Schema
updates have to be propagated to all nodes, so pending tasks for this metric can manifest in
schema disagreement errors.

Miscellaneous tasks pending

The number of pending tasks from other miscellaneous operations that are not ran
frequently.

Request response pending

The progress of rows of data being streamed from the receiving node. Streaming of data
between nodes happens during operations such as bootstrap and decommission when one node
sends large numbers of rows to another node.

Streams pending

The progress of rows of data being streamed from the sending node. Streaming of data
between nodes happens during operations such as bootstrap and decommission when one node
sends large numbers of rows to another node.