Use rolling upgrade

Splunk Enterprise version 7.1.0 and later supports rolling upgrade for indexer clusters. Rolling upgrade lets you perform a phased upgrade of indexer peers with minimal interruption to your ongoing searches. You can use rolling upgrade to minimize search disruption when upgrading peer nodes to a new version of Splunk Enterprise.

Requirements and considerations

Review the following requirements and considerations before you initiate a rolling upgrade:

Rolling upgrade only applies to upgrades from version 7.1.x to a later version of Splunk Enterprise.

The cluster master and all peer nodes must be running version 7.1.0 or later. For upgrade instructions, see Upgrade an indexer cluster.

All search heads and search head clusters must be running version 7.1.0 or later.

Do not attempt any clustering maintenance operations, such as rolling restart, bundle pushes, or node additions, during upgrade.

How a rolling upgrade works

When you initiate a rolling upgrade, you select a peer and take it offline. During the offline process, the master reassigns bucket primaries to other peers to retain the searchable state, and the peer completes any in-progress searches within a configurable timeout. See The fast offline process.

After the master shuts down the peer, you perform the software upgrade and bring the peer back online, at which point the peer rejoins the cluster. You repeat this process for each peer node until the rolling upgrade is complete.

A rolling upgrade behaves in the following ways:

Peer upgrades occur one at a time based on the default search factor of SF=2. With SF=3 or greater, you can upgrade the search factor minus one number of peers at a time. For example, with SF=3 you can upgrade two peers at a time.

The peer waits for any in-progress searches to complete. It will wait up to a maximum time period determined by the decommission_search_jobs_wait_secs attribute in server.conf. The default of 180 seconds is enough time for the majority of searches to complete in most cases.

Rolling upgrades apply to both historical searches and real-time searches.

In-progress searches that take longer than the default 180 seconds might generate incomplete results and a corresponding error message. If you have a scheduled search that must complete, either increase the decommission_search_jobs_wait_secs value or do not perform a rolling upgrade within the search's timeframe.

Before you perform a rolling upgrade, make sure the search_retry attribute in the [search] stanza of limits.conf is set to false (the default). Setting this to true might cause searches that take longer than the decommission_search_jobs_wait_secs value to generate duplicate or partial results without an error message.

Disable deferred scheduled searches

By default, scheduled searches are deferred until after the rolling upgrade finishes, based on the default defer_scheduled_searchable_idxc = true setting in savedsearches.conf.

The behavior of deferred scheduled searches depends on the scheduling mode: Real-time scheduled searches are skipped entirely; continuous scheduled searches stop until after upgrade is complete. See Real-time scheduling and continuous scheduling.

To disable the defer_scheduled_searchable_idxc setting so that scheduled searches are not deferred:

On the search head, edit $SPLUNK_HOME/etc/system/local/savedsearches.conf.

Enter your email address, and someone from the documentation team will respond to you:

Send me a copy of this feedback

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »