Each forwarder queries the master node for a list of all peer nodes in the cluster. It then uses load balancing to forward data to the set of peer nodes. In the case of a multisite cluster, a forwarder can query the master for a list of all peers on a single site.

How indexer discovery works

Briefly, the process works like this:

1. The peer nodes provide the master with information on their receiving ports.

3. The master transmits the peer nodes' URIs and receiving ports to the forwarders.

4. The forwarders send data to the set of nodes provided by the master.

In this way, the forwarders stay current with the state of the cluster, learning of any peers that have joined or left the cluster and updating their set of receiving peers accordingly.

In the case of a multisite cluster, each forwarder can identify itself as a member of a site. In that case, the master transmits a list of all peer nodes for that site only, and the forwarder limits itself to load balancing across that site. See Use indexer discovery in a multisite cluster.

Note: If the master goes down, the forwarders will rely on their most recent list of available peer nodes. However, the list does not persist through a forwarder restart. Therefore, if a forwarder restarts while the master is down, it will not have a list of peer nodes and will not be able to forward data, resulting in potential data loss. Similarly, if a forwarder starts up for the first time, it must wait for the master to return before it can get a list of peers.

Configure indexer discovery

These are the main steps for setting up connections between forwarders and peer nodes, using indexer discovery:

1. Configure the peer nodes to receive data from forwarders

In order for a peer to receive data from forwarders, you must configure the peer's receiving port. One way to specify the receiving port is to edit the peer's inputs.conf file. For example, this setting in inputs.conf sets the receiving port to 9997:

Note: When using indexer discovery, each peer node can have only a single configured receiving port. The port can be configured for either splunktcp or splunktcp-ssl, but not for both. You must use the same method for all peer nodes in the cluster: splunktcp or splunktcp-ssl.

You can simplify peer input configuration by deploying a single, identical inputs.conf file across all the peers. The receiving port that you specify in the common copy of inputs.conf will supersede any ports you enable on each individual peer. For details on how to create and deploy a common inputs.conf across all peers, see Update common peer configurations and apps.

2. Configure the master node to enable indexer discovery

The pass4SymmKey attribute specifies the security key used with communication between the master and the forwarders. Its value must be the same for all forwarders and the master node. It can have a different value from the pass4SymmKey attribute used for communication between the master and the cluster nodes, which is set in the [clustering] stanza, as described in Configure the security key.

The polling_rate attribute (optional) provides a means to adjust the rate at which the forwarders poll the master for the latest list of peer nodes. Its value must be an integer between 1 and 10. The default is 10. See Adjust the frequency of polling.

In the [indexer_discovery:<name>] stanza, the <name> references the <name> set in the indexerDiscovery attribute in the [tcpout:<target_group>] stanza.

The pass4SymmKey attribute specifies the security key used with communication between the master and the forwarders. Its value must be the same for all forwarders and the master node. You must explicitly set this value for each forwarder.

The <master_uri> is the URI and management port for the master node. For example: "https://10.152.31.202:8089".

In the [tcpout:<target_group>] stanza, set the indexerDiscovery attribute, instead of the server attribute that you would use to specify the receiving peer nodes if you were not enabling indexer discovery. With indexer discovery, the forwarders get their list of receiving peer nodes from the master, not from the server attribute. If both attributes are set, indexerDiscovery takes precedence.

b. Enable indexer acknowledgment for each forwarder

Note: This step is required to ensure end-to-end data fidelity. If that is not a requirement for your deployment, you can skip this step.

To ensure that the cluster receives and indexes all incoming data, you must turn on indexer acknowledgment for each forwarder.

To configure indexer acknowledgment, set the useACK attribute in each forwarder's outputs.conf, in the same stanza where you set the indexerDiscovery attribute:

Use indexer discovery in a multisite cluster

In multisite clustering, the cluster is partitioned into sites, typically based on the location of the cluster nodes. See Multisite indexer clusters. When using indexer discovery with multisite clustering, you can optionally configure each forwarder to be site-aware, so that it forwards data to peer nodes only on a single specified site.

Important: When you use indexer discovery with multisite clustering, you must assign a site-id to all forwarders, whether or not you want the forwarders to be site-aware. If you want a forwarder to be site-aware, you assign it a site-id for a site in the cluster, such as "site1," "site2," and so on. If you do not want a forwarder to be site-aware, you assign it the special site-id of "site0". When a forwarder is assigned "site0", it will forward to peers across all sites in the cluster.

Add this stanza to the forwarder's server.conf file:

[general]
site = <site-id>

Note the following:

You must assign a <site-id> to each forwarder sending data to a multisite cluster. This must either be a valid site in the cluster or the special value "site0".

If you want the forwarder to send data only to peers at a specific site, assign the id for that site, such as "site1."

If you want the forwarder to send data to all peers, across all sites, assign a value of "site0".

If you do not assign any id, the forwarder will not send data to any peer nodes.

Caution: If you assign a forwarder to a specific site and that site goes down, the forwarder will not fail over to another site. It will stop forwarding data if there are no indexers available on the site.

Use weighted load balancing

When you enable indexer discovery, the forwarders always stream the incoming data across the set of peer nodes, using load balancing to switch the data stream from node to node. This operates in a similar way to how forwarders without indexer discovery use load balancing, but with some key differences. In particular, you can enable weighted load balancing.

In weighted load balancing, the forwarders take each peer's disk capacity into account when they load balance the data. For example, a peer with a 400GB disk receives approximately twice the data of a peer with a 200GB disk.

Important: The disk capacity refers to the total amount of local disk space on the peer, not the amount of free space.

How weighted load balancing works

Weighted load balancing behaves similarly to normal forwarder load balancing. The autoLBFrequency attribute in the forwarder's outputs.conf file still determines how often the data stream switches to a different indexer. However, when the forwarder selects the next indexer, it does so based on the relative disk capacities. The selection itself is random but weighted towards indexers with larger disk capacities.

In other words, the forwarder uses weighted picking. So, if the forwarder has an autoLBFrequency set to 60, then every sixty seconds, the forwarder switches the data stream to a new indexer. If the load balancing is taking place across two indexers, one with a 500GB disk and the other with a 100GB disk, the indexer with the larger disk is five times as likely to be picked at each switching point.

Enable weighted load balancing

The indexerWeightByDiskCapacity attribute is set to false by default. To enable weighted load balancing, you must set it to true.

Change the advertised disk capacity for an indexer

In some cases, you might want weighted load balancing to treat an indexer as though it has a lower disk capacity than it actually has. You can use the advertised_disk_capacity attribute to accomplish this. For example, if you set that attribute to 50 (signifiying 50%) on an indexer with a 500GB disk, weighted load balancing will proceed as though the actual disk capacity was 250GB.

You set the advertised_disk_capacity attribute in the indexer's server.conf file:

[clustering]
advertised_disk_capacity = <integer>

Note the following:

The advertised_disk_capacity attribute indicates the percentage that will be applied to the indexer's actual disk capacity before it sends the capacity to the master. For example, if set to 50 on an indexer with a 500GB disk, the indexer tells the master that the disk capacity is 250GB.

The value can vary from 10 to 100.

The default is 100.

Adjust the frequency of polling

Forwarders poll the master at regular intervals to receive the most recent list of peers. In this way, they become aware of any changes to the set of available peers and can modify their forwarding accordingly. You can adjust the rate of polling.

The frequency of polling is based on the number of forwarders and the value of the polling_rate attribute, configured in the master's server.conf file. The polling interval for each forwarder follows this formula:

To configure polling_rate, add the attribute to the [indexer_discovery] stanza in server.conf on the master:

[indexer_discovery]
polling_rate = <integer>

Note the following:

The polling_rate attribute must be an integer between 1 and 10.

The default is 10.

Configure indexer discovery with SSL

You can configure indexer discovery with SSL. The process is nearly the same as configuring without SSL, with just a few additions and changes:

1. Configure the peer nodes to receive data from forwarders over SSL.

2. Configure the master node to enable indexer discovery.

3. Configure the forwarders for SSL.

The steps below provide basic configuration information only, focusing on the differences when configuring for SSL. For full details on indexer discovery configuration, see Configure indexer discovery.

1. Configure the peer nodes to receive data from forwarders over SSL

Edit each peer's inputs.conf file to specify the receiving port and to configure the necessary SSL settings:

Enter your email address, and someone from the documentation team will respond to you:

Send me a copy of this feedback

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »