Troubleshoot add-ons

Stuck? Try these techniques to troubleshoot add-ons built by Splunk.

Useful searches

Search the internal index for logs specific to the add-on. For the source, use a wild-carded version of the brief name of the technology that the add-on supports. The source should usually match the last element in the add-on folder name. See the examples below.

Add-on

Folder name

Search

Splunk Add-on for Amazon Web Services

Splunk_TA_aws

index = _internal source=*aws*

Splunk Add-on for Cisco IPS

Splunk_TA_cisco-ips

index = _internal source=*cisco-ips*

Splunk Add-on for Java Management Extensions

Splunk_TA_jmx

index = _internal source=*jmx*

Splunk Add-on for ServiceNow

Splunk_TA_snow

index = _internal source=*snow*

Once you have narrowed to the proper type of logs to find issues, here are some troubleshooting techniques that might be useful for a given scenario.

Keyword searching

Add useful keywords to your search, like "ERROR" or the name of the service or server that you are trying to connect to.

Find unusual logs

Finding rare events is easier when all the similar things are hidden. If you add

| dedup punct

to the end of a search, it will temporarily reduce repetitive "noise" and make unusual lines stand out.

Find transactions

Some data sources log in a repetitive pattern across many entries. If a common field value is available, such as *pid*, you can use

| transaction pid

at the end of a search to collapse these patterns into single events. That makes them easier to analyze with tools like stats or dedup. Transaction can use multiple fields as well:

Increase timeout settings

If you are configuring accounts or inputs in Splunk Web and you experience errors or trouble saving, go to $SPLUNK_HOME/etc/system/local/web.conf and change your timeout settings as shown below.

[settings]
splunkdConnectionTimeout = 300

Turn on debug logging

Some add-ons allow you to configure the log level in the add-on setup page. Toggle this to DEBUG to review more verbose logs to help you find the root cause. If the add-on that you are using does not support this through the setup in Splunk Web, it may be possible to configure this in the configuration files instead. Check the documentation for each add-on for guidance.

Check if the add-on is intended to be visible or not

Some add-ons have a custom configuration UI that you can use in Splunk Web to set up a connection to your data and configure inputs. Other add-ons do not have a custom UI, and instead rely on the Splunk platform's native data collection and configuration capabilities. If the add-on you are using does not have any custom views, or if it does but you want to turn those views off on certain nodes of your distributed deployment to prevent confusion, make sure that visibility of the add-on is set to False.

If you are trying to launch or load views for an add-on in Splunk Web and you are experiencing results you do not expect, the add-on may not be intended to be visible. You can determine this by looking in the add-on's default/app.conf file. If you see visible = false in the ui stanza, this add-on is not intended to be treated as a visible app in Splunk Web on any nodes of your architecture.

To turn off visibility of an add-on:

Go to Apps > Manage Apps.

Find the row for the add-on you are working with and click Edit properties.

Check for conflicting configurations

Some add-ons are intended to replace older versions of add-ons with different package names. In these cases, you should remove the old, differently named package of the add-on when you install the newer version to avoid conflicts in your configuration files. Review the release notes for specific add-ons to find upgrade guides to walk you through this process.

For example, if a workflow action is not behaving as expected, try running btool.

The output will show which configurations came from which app. They should all originate from the add-on you are troubleshooting. Other conflicting configurations may be causing the unexpected behavior. For example, Splunk Web attempts to render the workflow action result as Splunk view instead of as an external site.

Communication with external systems

Many Splunk developed add-ons that have modular inputs use a third-party API to communicate with an external system. You need to ensure that communication is allowed by your networking infrastructure. If you use a firewall in your environment, you need to open the correct firewall port. For an SSL enabled API, that port is typically 443. For example, the Splunk Add-on for F5 BIG-IP requires port 443 to be open on your firewall to enable F5 BIG-IP to communicate with the iControl API over SSL. If the correct port is not open, you may get "Destination unreachable" errors.

Prevent event data loss

When a Splunk add-on with modular inputs is collecting data on a heavy forwarder and the forwarder is stopped, event data can be lost. Workaround: If you need to stop a forwarder for any reason, first disable all add-ons that might be collecting data on it, and wait for at least 10 seconds before stopping the forwarder.

On Windows, disable an add-on with modular inputs before upgrading it

On Windows, upgrading an add-on with modular inputs may fail due to some processes that keep running in the background and prevent some files from being overwritten during the upgrade. As a workaround, disable an add-on with modular inputs before upgrading it and enable it after upgrade is complete.

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »