What Splunk software logs about itself

Splunk software keeps track of its activity by logging to various files in $SPLUNK_HOME/var/log/splunk.

The Splunk platform internal log files are rolled based on size. You can change the default log rotation size by editing $SPLUNK_HOME/etc/log.cfg.

Search these files in Splunk Web by typing:

index=_internal

Search for errors and warnings by typing:

index=_internal (log_level=error OR log_level=warn*)

Internal logs

Here is a list, with descriptions, of the internal logs in $SPLUNK_HOME/var/log/splunk. The internal logs from Splunk software are useful for troubleshooting or metric analysis.

Note that some log files are not created until your Splunk platform instance uses them, for example crawl.log.

Log file name

Useful for?

audit.log

Information about user activity, most interestingly about a user logging in (or failing to log in), modifying a setting, updating a lookup, or running a search. For example, if you're looking for information about a saved search, audit.log matches the name of a saved search (savedsearch_name) with its search ID (search_id), user, and time. With the search_id, you can look up that particular search elsewhere, like in the search dispatch directory. Read about audit events in the Securing Splunk Manual. Audit.log is the only file indexed to _audit.

Beginning with Splunk 5, no longer used. Read about intentions in Developing Views and Apps for Splunk Web.

license_audit.log

No longer used.

license_usage.log

Indexed volume in bytes per pool, index, source, source type, and host. Available only on a Splunk license master.

license_usage_summary.log

Daily indexed volume in bytes per pool, stack, and host. Available only on a license master. Indexed to _telemetry. See Share data in Splunk Enterprise in the Admin Manual.

metrics.log

Contains periodic snapshots of Splunk performance and system data, including information about CPU usage by internal processors and queue usage in Splunk's data processing. The metrics.log file is a sampling of the top ten items in each category in 30 second intervals, based on the size of _raw. It can be used for limited analysis of volume trends for data inputs. For more information about metrics.log, see About metrics.log and Work with metrics.log in this manual.

migration.log

A log of events during install and migration. Specifies which files were altered during upgrade.

Python events within Splunk. Useful for debugging REST endpoints, communication with splunkd, PDF Report Server App, Splunk Web display issues, sendmail (email alerts), and scripted inputs. With web_service.log, one of the few Splunk logs that uses "WARNING" instead of "WARN" for second most verbose logging level.

remote_searches.log

Messages from StreamedSearch channel. This code is executed on the search peers when a search head makes a search request. So this file contains useful information on indexers regarding searches they're participating in.

scheduler.log

All actions (successful or unsuccessful) performed by the splunkd search and alert scheduler. Typically, this shows scheduled search activity.

searches.log

Beginning with Splunk 5, no longer used. Instead, use the following search syntax: | history. This shows all the searches that have been run, plus stats for the searches.

searchhistory.log

No longer used.

splunkd.log

The primary log written to by the Splunk server. May be requested by Splunk Support for troubleshooting purposes. Any stderr messages generated by scripted inputs, scripted search commands, and so on, are logged here.

splunkd_access.log

Any action done from splunkd through the UI is logged here, including splunkweb, the CLI, all POST GET actions, deleted saved searches, and other programs accessing the REST endpoints. Also logs the time taken to respond to the requests. Search job artifacts logged here include size of data returned with search. sourcetype="splunkd_access"

splunkd_stderr.log

The Unix standard error device for the server. Typically this contains (for *nix) times of healthy start and stop events, as well as various errors like exceptions, assertions, and errors generated by libraries and the operating system.

splunkd_stdout.log

The Unix standard output device for the server.

splunkd_ui_access.log

Starting in 6.2, contains a significant portion of the types of events that used to be logged in web_access.log.

splunkd-utility.log

This log is written to by the prereq-checking utils splunkd clone-prep-clear-config, splunkd validatedb, splunkd check-license, splunkd check-transforms-keys, and splunkd rest (for offline CLI). Each util logs Splunk version, some basic config, and current OS limits like max number of threads, and then messages specific to the util. Consult this log file when splunkd didn't start.

web_access.log

Requests made of Splunk Web, in an Apache access_log format. Much of the types of events logged here are logged in splunkd_ui_access.log starting in 6.2.

web_service.log

Primary log written by splunkweb. Records actions made by splunkweb. This and python.log are the only logs that, in second most verbose logging level, write messages with "WARNING" instead of Splunk log files' usual "WARN."

Introspection logs

Splunk Enterprise platform instrumentation refers to data that your Splunk Enterprise deployment logs in the _introspection index. It gathers data about your Splunk instance and operating system and writes it to log files that you can search later to aid in troubleshooting a variety of problems. You can also view the data at REST endpoints.

Splunk platform search logs

The Splunk platform also creates search logs. These are not indexed to _internal.

Each search has its own directory for all information specific to the search, including a search log. The directory for each search is named with (among other parameters) the search_id. This search_id is matched to a human-readable search name in audit.log. The search directory for a recent search is in $SPLUNK_HOME/var/run/splunk/dispatch/. In addition to the search.log file, a search directory contains other information about the search job, including the results. A search directory for a completed search is referred to as a search artifact.

If you have any long-running real-time searches, you might want to adjust the maximum size of your search logs. These logs are rotated when they reach a default maximum size of 10 MB. Splunk software keeps up to five of them for each search, so the total log size for a search can conceivably grow as large as 30 MB.

Most searches are unlikely to generate logs anywhere near 10 MB in size. But size can become an issue if you have ongoing real-time searches.

Included data models

Splunk Enterprise comes with several sample data models. These data models are constructed from the internal logs from Splunk Enterprise. By interacting with them, you can learn about Splunk software log files and about data models in one fell swoop.

Comments

Hi Noukie972, thank you for asking us about this again. We haven't documented those channels, but I've passed your question along and asked relevant people to take a fresh look at the request. Thanks again for posting.

Andrewb splunk, Splunker

November 12, 2018

Hi
Sorry to re-ask : Is there any documentation describing the ~950 various log channels available under Server settings > Server logging? It would help admins understand which settings should be changed when troubleshooting their system.
Or where can i find a documentation for the catagories loggin for internal and search log
It's will be helpful

Noukie972

November 5, 2018

Hi Sschohn,
We don't have documentation available for that, but a ticket has been filed requesting this information for the top 100 or so channels. It's SPL-128271.

Jlaw splunk, Splunker

October 11, 2017

Hello. Is there any documentation describing the ~950 various log channels available under Server settings > Server logging? It would help admins understand which settings should be changed when troubleshooting their system. Thanks.

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »