Enable debug logging

The Splunk platform internal logging levels are DEBUG INFO WARN ERROR FATAL (from most to least verbose). This topic gives a few popular options for how you might want to put Splunk into debug mode.

Be warned, Splunk software debug mode is extremely verbose. All the extra chatter might obscure something that might have helped you diagnose your problem. Running Splunk software in debug mode for an extended time makes your internal log files unwieldy. Running debug mode is not recommended on production systems.

Enable debug logging on all of splunkd.log

Splunk software has a debugging parameter (--debug) that you can use when starting Splunk software from the CLI in *nix. This command outputs logs to $SPLUNK_HOME/var/log/splunk/splunkd.log. To enable debug logging from the command line:

Navigate to $SPLUNK_HOME/bin.

Stop Splunk, if it is running.

Save your existing splunkd.log file by moving it to a new filename, like splunkd.log.old.

Specific areas can be enabled to collect debugging details over a longer period with minimal performance impact. See the category settings in the file $SPLUNK_HOME/etc/log.cfg to set specific log levels without enabling a large number of categories as with --debug. Restart Splunk after changing this file.

Important: Changes to $SPLUNK_HOME/etc/log.cfg are overwritten if you upgrade your version of Splunk software.

Note: Not all messages marked WARN or ERROR indicate actual problems with Splunk software; some indicate that a feature is not being used.

Note also that this option is not available on Windows. To enable debugging on Splunk software running on Windows, enable debugging on a specific processor in Splunk Web or using log.cfg.

Enable debug logging for a specific processor within splunkd.log

In Splunk Web

You can enable these DEBUG settings via Splunk Web if you have admin privileges. Navigate to Settings > Server settings > Server logging. Search for the processor names using the text box. Click on the processor name to change the logging level to DEBUG. You do not need to restart Splunk. In fact, these changes will not persist if you restart the Splunk instance.

In log.cfg

If you want the processors to be in DEBUG on startup, or if you want to turn on debugging for a few processors or for a lightweight forwarder (with no Splunk Web), create or edit a $SPLUNK_HOME/etc/log-local.cfg file to override changes in $SPLUNK_HOME/etc/log.cfg.

In $SPLUNK_HOME/etc/log.cfg, find the category.* entry that relates to the processor you are interested in, and copy the line to log-local.cfg with INFO or WARN modified to DEBUG. There will not always be an existing entry for the processor you are interested in, and it may take some digging through the logs or documentation to find the correct one.

For example, to see how often Splunk software is updating or retrieving progress-tracking records fora particular file, put 'FileInputTracker' in DEBUG. Update the existing entry to read

category.FileInputTracker=DEBUG

Or for investigating problems monitoring files, use the FileInputTracker and selectProcessor categories.

Restart the Splunk platform. Now every time Splunk software checks the inputs file, it will be recorded in $SPLUNK_HOME/var/log/splunk/splunkd.log. Remember to change these settings back when you are finished investigating.

If a default level is not specified for a category, the logging level defaults to your rootCategory setting.

Note: Leave category.loader at INFO. This is what gives us our build and system info.

To change the maximum size of a log file before it rolls, change the maxFileSize value (in bytes) for the desired file:

This leaves everything else as it was, which means only the debug messages you want are generated. Putting rootCategory into DEBUG mode makes the dispatch directories huge, so it is not a good choice for long-running debug.

log-local.cfg

You can put log.cfg settings into a local file, log-local.cfg file, residing in the same directory as log.cfg. The settings in log-local.cfg take precedence. And unlike log.cfg, the log-local.cfg file doesn't get overwritten on upgrade.

With endpoints

You can access a debugging endpoint that shows status information about monitored files:

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »