Cluster endpoint descriptions

Usage details

Review ACL information for an endpoint

To check Access Control List (ACL) properties for an endpoint, append /acl to the path. For more information see Access Control List in the REST API User Manual.

Authentication and Authorization

Username and password authentication is required for access to endpoints and REST operations.

Splunk users must have role and/or capability-based authorization to use REST endpoints. Users with an administrative role, such as admin, can access authorization information in Splunk Web. To view the roles assigned to a user, select Settings > Access controls and click Users. To determine the capabilities assigned to a role, select Settings > Access controls and click Roles.

App and user context

Typically, knowledge objects, such as saved searches or event types, have an app/user context that is the namespace. For more information about specifying a namespace, see Namespace in the REST API User Manual.

Splunk Cloud limitations

If you have a managed Splunk Cloud deployment with search head clustering and index clustering, the REST API supports access to the search head only. You can use the REST API to interact with the search head in your deployment. Using the REST API to access any other cluster member nodes is not supported. For example, index cluster management endpoints are not applicable to Splunk Cloud deployments.

Only valid for peer nodes in a cluster. The time, in seconds, that a peer attempts to send a heartbeat to the master

heartbeat_timeout

Only valid for the master node in a cluster configuration. The time, in seconds, before a master considers a peer down. Once a peer is down, the master initiates steps to replicate buckets from the dead peer to its live peers. Defaults to 60 seconds.

master_uri

Valid only for nodes configured as a peer or searchhead.

URI of the cluster master to which this node connects.

max_peer_build_load

The number of jobs that a peer can have in progress at any time that make the bucket searchable.

Valid only for nodes configured as peers. The address on which a peer is available for accepting replication data. This is useful in the cases where a peer host machine has multiple interfaces and only one of them can be reached by another splunkd instance.

register_search_address

IP address that advertises this indexer to search heads.

rep_cxn_timeout

Low-level timeout, in seconds, for establishing a connection for replicating data.

rep_max_rcv_timeout

Maximum cumulative time, in seconds, for receiving acknowledgement data from peers. Defaults to 600s.

rep_max_send_timeout

Maximum time, in seconds, for sending replication slice data between cluster nodes. Defaults to 600s.

rep_rcv_timeout

Low-level timeout, in seconds, for receiving data between cluster nodes.

Determines how many copies of raw data are created in the cluster. This could be less than the number of cluster peers.

Must be greater than 0 and greater than or equal to the search factor. Defaults to 3.

replication_port

TCP port to listen for replicated data from another cluster member.

replication_use_ssl

Indicates whether to use SSL when sending replication data.

restart_timeout

Only valid for nodes configured as a master. The amount of time, in seconds, the master waits for a peer to come back when the peer is restarted (to avoid the overhead of trying to fix the buckets that were on the peer). Defaults to 600 seconds.

Note: This only works if the peer is restarted from Splunk Web.

search_factor

Only valid for nodes configured as a master. Determines how many searchable copies of each bucket to maintain. Must be less than or equal to replication_factor and greater than 0. Defaults to 2.

secret

Secret shared among the nodes in the cluster to prevent any arbitrary node from connecting to the cluster. If a peer or searchhead is not configured with the same secret as the master, it is not able to communicate with the master.

Only valid for peer nodes in a cluster. Time, in seconds, that a peer attempts to send a heartbeat to the master

heartbeat_timeout

Number

Only valid for the master node in a cluster configuration. Time, in seconds, before a master considers a peer down. Once a peer is down, the master initiates steps to replicate buckets from the dead peer to its live peers. Defaults to 60 seconds.

master_uri

URI

Valid only for nodes configured as a peer or searchhead. URI of the cluster master to which this node connects.

max_peer_build_load

Number

The number of jobs that a peer can have in progress at any time that make the bucket searchable.

Valid only for nodes configured as peers. The address on which a peer is available for accepting replication data. This is useful in the cases where a peer host machine has multiple interfaces and only one of them can be reached by another splunkd instance.

register_search_address

N/A

IP address that advertises this indexer to search heads.

rep_cxn_timeout

Number

Low-level timeout, in seconds, for establishing a connection for replicating data.

rep_max_rcv_timeout

Number

Maximum cumulative time, in seconds, for receiving acknowledgement data from peers. Defaults to 600s.

rep_max_send_timeout

Number

Maximum time, in seconds, for sending replication slice data between cluster nodes. Defaults to 600s.

rep_rcv_timeout

Number

Low-level timeout, in seconds, for receiving data between cluster nodes.

Only valid for nodes configured as a master. Determines how many copies of raw data are created in the cluster. This could be less than the number of cluster peers. Must be greater than 0 and greater than or equal to the search factor. Defaults to 3.

replication_port

Number

TCP port to listen for replicated data from another cluster member.

replication_use_ssl

Number

Indicates whether to use SSL when sending replication data.

restart_timeout

Number

Only valid for nodes configured as a master. The amount of time, in seconds, the master waits for a peer to come back when the peer is restarted (to avoid the overhead of trying to fix the buckets that were on the peer). Defaults to 600 seconds.

Note: This only works if the peer is restarted from Splunk Web.

search_factor

Number

Only valid for nodes configured as a master. Determines how many searchable copies of each bucket to maintain. Must be less than or equal to replication_factor and greater than 0. Defaults to 2.

secret

N/A

Secret shared among the nodes in the cluster to prevent any arbitrary node from connecting to the cluster. If a peer or searchhead is not configured with the same secret as the master, it is not able to communicate with the master. Corresponds to pass4SymmKey setting in server.conf.

cluster/master/buckets/{bucket_id}/freeze

Set the bucket's state to frozen. The frozen state may not persist after a cluster master restart unless one of the peers has set the frozen state. A POST to this endpoint does not set the bucket's state to frozen on peers.

Note: Use this endpoint with caution. It is recommended to test the endpoint in a test cluster prior to use on an actual bucket.

cluster/master/buckets/{bucket_id}/remove_from_peer

If the request causes the cluster to lose its complete state, the cluster will engage in fixup activities. This may result in another copy of the same bucket appearing on this peer. If, however, the specified bucket is frozen, the cluster does not attempt any fixup activities.

Caution: Using this endpoint will cause irreversible data loss. It is recommended to test the endpoint on a test-cluster prior to use on an actual bucket.

Authentication and Authorization
Requires the admin role or indexes_edit capability.

POST

Delete this bucket from specified peer. Set bucket state to frozen

Request parameters

Name

Type

Description

peer (required)

GUID

Peer GUID

Returned values
None. If the peer parameter is missing from the request, an error message is returned.

cluster/master/control/control/roll-hot-buckets

This endpoint forces a specified bucket in an indexer cluster to roll from hot to warm. Pass the bucket id (bid) to the master node. The master instructs the origin peer for that bucket to roll its copy. In turn, the origin peer tells all the replicating peers to roll their copies

You might discover a bucket that is stuck in fixup and needs to be rolled using logs, Splunk Web, or either of the following two endpoints.

cluster/master/control/default/cancel_bundle_push

Cancels and resets the bundle push operation. Use this endpoint when the cluster master does not receive a validation response from the cluster peer due to an error. For more information, see Configuration bundle issues.

cluster/master/fixup

Access a list of buckets on a specific fixup priority level. Bucket fixups are processed in order of priority level. See Request parameters below for priority level details.

When you access a particular fixup level, buckets may appear in it even though they do not need fixup at this level. Initially, each bucket requiring fixup is added to all levels, even though it might only require processing in a subset of all levels. As the bucket is processed through a level, it is deleted from that level.

GET

Required. Fixup priority level. Use one of the following level values, listed in order of priority.

streaming : Hot buckets that need to be rolled or have their size committed.

data_safety : Buckets without at least two rawdata copies.

generation : Buckets without a primary copy.

replication_factor : Buckets without replication factor number of copies.

search_factor : Buckets without search factor number of copies.

checksum_sync : Level for syncing a bucket's delete files across all peers that have this bucket. Syncing is determined based on the checksum of all of the delete files.

index

String

Optional. Index name.

Returned values
For each bucket in the specified fixup level, the response includes the following details for the initial time when the bucket went into the fixup level and the latest time that the bucket was checked.

GET

The amount of time, in seconds, the cluster master will wait for a peer in primary decommission status to finish primary reassignment

and restart, during a searchable rolling restart with timeouts. Only valid for rolling_restart=searchable_force. Default value is 180. Max accepted value is 1800.

maintenance_mode

Indicates if the cluster is in maintenance mode. Happens during rolling restart, bundle push, and other maintenance activities.

messages

Array of messages from server.

multisite

Indicates if multisite is enabled for this master. Make sure you set site parameters on the peers if you set this to true. Defaults to false.

peers

Object containing all the peers in the cluster. For each peer, the label, site and status are provided.

restart_inactivity_timeout

The amount of time, in seconds, that the master waits for a peer to restart and rejoin the cluster before it considers the restart a failure and proceeds to restart other peers. A value of zero (0) means that the master waits indefinitely for a peer to restart. Only valid for rolling_restart=searchable_force. Default is 600secs.

POST

Configure this server as a cluster searchhead node.

Request parameters

Name

Type

Description

name

String

Required. The URI of the master node in the cluster.

secret

String

Required. Secret shared among the nodes in the cluster to prevent any arbitrary node from connecting to the cluster. If a peer or searchhead is not configured with the same secret as the master, it is not able to communicate with the master.

POST

Update cluster search head node configuration.

Request parameters

Name

Type

Description

master_uri

String

The URI of the master node in the cluster for which this searchhead is configured.

secret

String

Secret shared among the nodes in the cluster to prevent any arbitrary node from connecting to the cluster. If a peer or searchhead is not configured with the same secret as the master, it is not able to communicate with the master.

Corresponds to pass4SymmKey setting in server.conf.

Returned values
None

cluster/slave/buckets

https://<host>:<mPort>/services/cluster/slave/buckets

Access cluster peers bucket configuration.

GET

List cluster peers bucket configuration.

Request parameters

Name

Type

Description

generation_id

String

The generation ID for this peer. For each generation, the master server in a cluster configuration assigns generation IDs. A generation identifies which copies of a cluster's buckets are primary and therefore can participate in a search.

GET

List peer specified bucket information.

Request parameters

Name

Type

Description

generation_id

String

The generation ID for this peer. For each generation, the master server in a cluster configuration assigns generation IDs. A generation identifies which copies of a cluster's buckets are primary and therefore can participate in a search.

Returned values

Name

Description

checksum

Used internally to identify this bucket.

earliest_time

Indicates the time of the earliest event in this bucket.

generation_id

The generation ID for this peer.

generations

A sparse list of generation id to bucket primacy for the given peer.

latest_time

Indicates the time for the latest event in this bucket.

search_state

Indicates if the bucket is Searchable or Unsearchable.

status

Indicates the status of this bucket. One of the following values.

Complete

The copy of this bucket contains the full complement of information.

StreamingSource

The copy of this bucket is sending data to peer nodes for replication.

StreamingTarget

The copy of this bucket is receiving replicated data.

NonStreamingTarget

This copy of a warm bucket replication is in progress. Once replication is complete, the status changes to Complete.

StreamingError

The copy of this bucket encountered errors while streaming data.

PendingTruncate

The master asked the peer to truncate this copy of the bucket to a certain size and is waiting for confirmation.

PendingDiscard

The master asked the peer to discard this copy of the bucket (for whatever reason, and is waiting for confirmation.

cluster/slave/control/control/re-add-peer

Set the peer to re-add itself to the master. This syncs the peer's state, including its in-memory bucket state, to the master. By default, this resets the peer's primary bucket copies and the master reassigns them across the cluster. To keep the peer's existing primary bucket copies, use the optional clearMasks=false parameter.

This endpoint can be useful when the master and the peer have a state mismatch, for example when bucket information is not in sync between them.

POST

Re-add the cluster indexer to the cluster master.

Request parameters

Name

Type

Default

Description

clearMasks

Boolean. Use true or false.

true

Optional. Indicates whether the master should reassign all primary bucket copies across all peers. The default true value prompts the master to reassign all primary bucket copies across all peers. Use false to re-add the peer but keep the existing primary bucket copies.

cluster/slave/control/control/set_manual_detention

If you have Splunk Enterprise, you can use this endpoint to put the peer node in manual detention mode or take the peer out of this mode. In manual detention, the peer does not serve as a replication target. Detention helps slow the growth of disk usage on the peer.

Application usage
This parameter compares the baseline between the current instance, on which the GET request is made, with the baseline of other members. From each of the other members, the system retrieves the oldest changeset that is not more than 23 hours old and therefore safe from purging. The system then tries to find that changeset in the current instance's local changeset repository. If the changeset is found in the local repository, then the current instance and the member share a baseline.

Establishing a shared baseline between a captain and members is a prerequisite for successful configuration replication.

Name

Description

check_share_baseline

One of the following values is returned for each of the other members.

Yes: The current instance shares a baseline with this node.

No: The current instance node does not share a baseline with this node.

Connection error: The current instance cannot contact this node. A warning is logged with additional details.

server_name

Name for the member whose baseline is being compared to the current instance.

A Number of unpublished changes key is returned with one of the following values.

Name

Description

0

All changes on this cluster member have been pushed to the captain. There are no unpublished changes on this member.

0 (This instance is the captain)

This message is returned when requesting unpublished status on the captain. The captain is always in sync with itself, so there are no unpublished changes.

[Number greater than 0]

The number unpublished local changes on this member. Changes are held until the next replication occurs. The node is still healthy in this case.

No captain is available

The search head cluster does not currently have a captain.

Missing common baseline with the captain

This member might be out of sync with the captain if this message persists after several replication periods.

This message can also appear during a transition period, for example, when a captain is switched or a member is manually resynced. On a healthy search head cluster, the unpublished value should return to a numeric value after one replication period.

GET

dispatch_time - The UTC time of dispatch for the joberrormsg - If the job failed, capturing the reason for failurepeer - GUID of the member that the job was sent tosid - the search id of this attemptsuccess - a boolean for success/failure of the job

job_state

Job State can be SCHEDULED/DISPATCHED/COMPLETED. A SCHEDULED job has been received by the captain from the scheduler to schedule. A DISPATCHED job has started to run on a remote member. A COMPLETED job has finished running on the remote member.

saved_search

The name of the saved-search from the associated savedsearches.conf file.

GET

dispatch_time - The UTC time of dispatch for the joberrormsg - If the job failed, capturing the reason for failurepeer - GUID of the member that the job was sent tosid - the search id of this attemptsuccess - a boolean for success/failure of the job

job_state

Job State can be SCHEDULED/DISPATCHED/COMPLETED. A SCHEDULED job has been received by the captain from the scheduler to schedule. A DISPATCHED job has started to run on a remote member. A COMPLETED job has finished running on the remote member.

saved_search

The name of the saved-search from the associated savedsearches.conf file.

Only valid for member nodes in a searchhead cluster. The time, in seconds, that a member attempts to send a heartbeat to the captain

heartbeat_timeout

Only valid for the captain node in a searchhead cluster configuration. The time, in seconds, before a captain considers a member down. Once a member is down, the captain initiates steps to replicate artifacts from the dead member to its live members. Defaults to 60 seconds.

Valid only for nodes configured as members. The address on which a member is available for accepting replication data. This is useful in the cases where a member host machine has multiple interfaces and only one of them can be reached by another splunkd instance.

rep_cxn_timeout

Low-level timeout, in seconds, for establishing a connection for replicating data.

rep_max_rcv_timeout

Maximum cumulative time, in seconds, for receiving acknowledgement data from members. Defaults to 600s.

rep_max_send_timeout

Maximum time, in seconds, for sending replication slice data between searchhead cluster nodes. Defaults to 600s.

Determines how many copies of raw data are created in the searchhead cluster. This could be less than the number of searchhead cluster members.

Must be greater than 0 and greater than or equal to the search factor. Defaults to 3.

replication_port

TCP port to listen for replicated data from another searchhead cluster member.

replication_use_ssl

Indicates whether to use SSL when sending replication data.

restart_timeout

Only valid for nodes configured as a captain. The amount of time, in seconds, the captain waits for a member to come back when the member is restarted (to avoid the overhead of trying to fix the artifacts that were on the member). Defaults to 600 seconds.

Note: This only works if the member is restarted from Splunk Web.

secret

Secret shared among the nodes in the searchhead cluster to prevent any arbitrary node from connecting to the searchhead cluster. If a member or searchhead is not configured with the same secret as the captain, it is not able to communicate with the captain.

shcluster/member/control/control/set_manual_detention

Put the search head cluster member in manual detention mode or take the search head cluster member out of this mode. When a search head cluster member is in manual detention, it does not accept new search jobs, including both scheduled and ad-hoc searches. Existing search jobs run to completion. It also participates in cluster administration operations with the exception of artifact replication.

Enter your email address, and someone from the documentation team will respond to you:

Send me a copy of this feedback

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »