How to upgrade a distributed Splunk Enterprise environment

Distributed Splunk Enterprise environments vary widely. Some have multiple indexers or search heads, some have search head pools, and others have indexer- and search-head clusters. These types of environments present challenges over upgrading single-instance installations.

Determine the upgrade procedure to follow for your type of environment

Depending on the kind of distributed environment you have, you might have to follow separate instructions to complete the upgrade. This topic provides guidance on how to upgrade distributed environments that do not have any clustered elements like index- or search-head clusters. It also has information on how to upgrade environments that use the deprecated search head pool feature. Environments with clustered elements, such as indexer clusters and search head clusters, have different upgrade procedures in different topics.

To upgrade a distributed environment that has a search head pool or does not have any clustered elements, follow the procedures in this topic.

If you have additional questions about upgrading your distributed Splunk Enterprise environment, log a case at the Splunk Support Portal.

Cross-version compatibility between distributed components

While there is some range in compatibility between various Splunk software components, they work best when they are all at a specific version. If you have to upgrade one or more components of a distributed deployment, you should confirm that the components you upgrade remain compatible with the components that you don't.

Test apps prior to the upgrade

Before you upgrade a distributed environment, confirm that Splunk apps work on the version of Splunk Enterprise that you want to upgrade to. You must test apps if you want to upgrade a distributed environment with a search head pool, because search head pools use shared storage space for apps and configurations.

When you upgrade, the migration utility warns of apps that need to be copied to shared storage for pooled search heads when you upgrade them. It does not copy them for you. You must manually copy updated apps, including apps that ship with Splunk Enterprise (such as the Search app) - to shared storage during the upgrade process. Failure to do so can cause problems with the user interface after you complete the upgrade.

On a reference machine, install the full version of Splunk Enterprise that you currently run.

Install the apps on this instance.

Access the apps to confirm that they work as you expect.

Upgrade the instance.

Access the apps again to confirm that they still work.

If the apps work as you expect, move them to the appropriate location during the upgrade of your distributed environment:

If you use non-pooled search heads, move the apps to $SPLUNK_HOME/etc/apps on each search head during the search head upgrade process.

If you use pooled search heads, move the apps to the shared storage location where the pooled search heads expect to find the apps.

If your distributed environment has pooled search heads, the process to upgrade the environment becomes significantly more complex. If your organization has restrictions on downtime, use a maintenance window to perform this upgrade.

Following are the key concepts to upgrade this kind of environment.

Pooled search heads must be enabled and disabled as a group.

The version of Splunk Enterprise on all pooled search heads must be the same.

You must test apps and configurations that the search heads use prior to upgrading the search head pool.

If you have additional concerns about this guidance here, you can log a case through the Splunk Support Portal.

To upgrade a distributed Splunk environment with multiple indexers and pooled search heads:

Prepare the upgrade

See "Configure search head pooling" in the Distributed Search manual for instructions on how to enable and disable search head pooling on each search head.

Confirm that any apps that the pooled search heads use will work on the upgraded version of Splunk Enterprise, as described in "Test your apps prior to the upgrade" in this topic.

If you use a deployment server in your environment, disable it temporarily. This prevents the server from distributing invalid configurations to your other components.

Upgrade your deployment server, but do not restart it.

Designate a search head in your search head pool to upgrade as a test for functionality and operation.

For the remainder of these instructions, refer to that search head as "Search Head #1."

Note: You must remove search heads from a search head pool temporarily before you upgrade them. This must be done for several reasons:

To prevent changes to the apps and user objects hosted on the search head pool shared storage.

To stop the inadvertent migration of local apps and system settings to shared storage during the upgrade.

To ensure that you have a valid local configuration to use as a fallback, should a problem occur during the upgrade.

If problems occur as a result of the upgrade, search heads can be temporarily used in a non-pooled configuration as a backup.

Upgrade the search head pool

Caution: Remove each search head from the search head pool before you upgrade it, and add it back to the pool after you upgrade. While you don't need to confirm operation and functionality of each search head, only one search head at a time can be up during the upgrade phase.

Bring down all of the search heads in your environment. At this point, searching capability becomes unavailable, and remains unavailable until you restart all of the search heads after upgrading.

Place the confirmed working apps in the search head pool shared storage area.

Remove Search Head #1 from the search head pool.

Upgrade Search Head #1.

Restart Search Head #1.

Test the search head for operation and functionality. In this case, "operation and functionality" means that the instance starts and that you can log into it. It does not mean that you can use apps or objects hosted on shared storage. It also does not mean distributed searches will run correctly.

If the upgraded Search Head #1 functions as desired, bring it down.

Copy the apps and user preferences from the search head to the shared storage.

Add the search head back to the search head pool.

Restart the search head.

Upgrade the remaining search heads in the pool with this procedure, one by one.

Restart the search heads

After you have upgraded the last search head in the pool, restart all of them.

Test all search heads for operation and functionality across all of the apps and user objects that are hosted on the search head pool.

Enter your email address, and someone from the documentation team will respond to you:

Send me a copy of this feedback

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »