Decommission a site in a multisite indexer cluster

To decommission a site, you reconfigure several site-specific attributes.

Decommission a site

Caution: Before proceeding, be aware of these issues:

If a peer on the decommissioned site contains any buckets that were created before the peer was part of a cluster, such buckets exist only on that peer and therefore will be lost when the site is decommissioned.

Similarly, if the decommissioned site started out as a single-site cluster before it became part of a multisite cluster, any buckets that were created when it was a single-site cluster exist only on that site and will be lost when the site is decommissioned.

Once the master restarts at the end of the decommissioning process, it will commence bucket fix-up activities to return the cluster to a complete state. This can take a considerable amount of time, particularly if the decommissioned site held a large number of origin buckets.

Prerequsites

The cluster must meet these conditions before you decommission one of its sites:

The cluster must be in a complete state.

The master node cannot be on the site that you are planning to decommission. If it is, move the master to to a new site, following the guidelines in Handle master site failure.

The site_replication_factor attribute must be configured so that at least one copy of each bucket resides on a site not due for decommissioning. For example, in a two-site cluster, a valid configuration would be site_replication_factor = origin:1,total:2.

The site_search_factor attribute must be configured so that at least one searchable copy of each bucket resides on a site not due for decommissioning. For example, in a two-site cluster, a valid configuration would be site_search_factor = origin:1,total:2.

If you need to reconfigure site_replication_factor or site_search_factor so that all buckets have copies on other sites, you must then wait until the master completes fix-up activities and returns the cluster to a complete state before proceeding with the decommissioning.

Steps

For each search head on the site, either disable the search head or change its site attribute to specify a remaining site. For example, to change the search head's site to site2:

Restart the master. This step causes the attribute changes to take effect. It also removes the master from maintenance mode and starts fix-up activities for peers on the remaining sites.

Note: When the master restarts, the peers from the decommissioned site will unsuccessfully try to rejoin the cluster. Ignore the resulting messages.

Run splunk stop on each peer on the decommissioned site.

To verify that that the decommissioning was successful, look at the top of the master dashboard. It should state that both the search factor and the replication factor are met. When both are met, the cluster is in a complete state, indicating that the decommissioning was successful. See View the master dashboard.

Note: Because site decommissioning typically involves a large amount of bucket fix-up activity, it can take a considerable amount of time for the cluster to return to its complete state.

Reconfigure attributes

When you decommission a site, you must make changes to several site-specific attributes in server.conf on the master node:

available_sites: Remove the decommissioned site from the site list for this attribute.

site_replication_factor and site_search_factor: If the decommissioned site is an explicit site in either of these attributes, remove it and otherwise reconfigure the attribute as necessary.

You must restart the master node after you change any of these attributes.

Map the decommissioned site

When a site gets decommissioned, the origin bucket copies for that site remain bound to the decommissioned site until you map the site to a remaining, active site. This can make the cluster unable to meet its replication or search factors.

To deal with this issue, you can map decommissioned sites to active sites. The bucket copies for which a decommissioned site is the origin site will then be replicated to the active site specified by the mapping, allowing the cluster to again meet its replication and search factors.

Note: The site_replication_factor and site_search_factor attributes determine the cluster's number of origin bucket copies.

Syntax

To map a site that you are planning to decommission to a remaining site, use the site_mappings attribute in server.conf. Set this attribute on the master node only, using this syntax:

site_mappings = <comma-separated string>

Note the following:

The <comma-separated string> contains mappings from decommissioned sites to the remaining, active sites. These mappings can be of two types:

<decommissioned_site_id>:<active_site_id>. For example, site2:site3, where site2 is a decommissioned site and site3 is an active site. This is called an explicit mapping. There can be multiple explicit mappings.

default_mapping:<active_site_id>. For example, default_mapping:site4, where site4 is an active site. There can be at most one default mapping. It is recommended that you always include a default mapping, as fallback for an incorrect or missing explicit mapping.

In the case of <decommissioned_site_id>:<active_site_id>, the origin bucket copies in <decommissioned_site_id> will get replicated from remaining sites to peers on the <active_site_id>. This allows the cluster to meet the requirements of its replication and search factors.

In the case of default_mapping:<active_site_id>, the origin bucket copies for any decommissioned site without an explicit mapping will get replicated to <active_site_id>.

If an active site in a mapping is itself later decommissioned, its previous mappings must be remapped to a currently active site. For example, in the case of site2:site3, if site3 is itself decommissioned, you must replace the previous mapping, site2:site3, with a new set of mappings that map both site2 and site3 to an active site, such as site4, using the string site2:site4,site3:site4.

Restart the master node after changing this attribute.

Examples

These examples assume a cluster that originally had five sites, site1 through site5.

"site_mappings = site2:site3" This configuration maps a decommissioned site2 to an active site3. The mapping causes origin bucket copies for site2 to replicate to site3. There is no default site mapping.

"site_mappings = site1:site3,default_mapping:site4" This configuration maps a decommissioned site1 to site3 and maps all other decommissioned sites to site4. The mapping causes origin bucket copies for the decommissioned sites to replicate to their respective mapped sites.

"site_mappings = default_mapping:site5" This configuration maps all decommissioned sites to site5. The mapping causes origin bucket copies for all decommissioned sites to replicate to site5.

Decommission a site in a multisite indexer cluster

Enter your email address, and someone from the documentation team will respond to you:

Send me a copy of this feedback

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »