Configuration file precedence

Splunk software uses configuration files to determine nearly every aspect of its behavior. A Splunk platform deployment can have many copies of the same configuration file. These file copies are usually layered in directories that affect either the users, an app, or the system as a whole.

When editing configuration files, it is important to understand how Splunk software evaluates these files and which ones take precedence.

When incorporating changes, Splunk software does the following to your configuration files:

It merges the settings from all copies of the file, using a location-based prioritization scheme.

When different copies have conflicting attribute values (that is, when they set the same attribute to different values), it uses the value from the file with the highest priority.

It determines the priority of configuration files by their location in its directory structure, according to whether the file is located in a system, app, or user directory, in that order. To determine priority among the collection of apps directories, Splunk uses ASCII sort order. Files in an apps directory named "A" have a higher priority than files in an apps directory named "B", and so on.

Note: Besides resolving configuration settings amongst multiple copies of a file, Splunk software sometimes needs to resolve settings within a single file. For information on how Splunk software determines precedence within a single props.conf file, see "Attribute precedence within a single props.conf file".

About configuration file context

How precedence is determined depends upon the context of the file.

App or user context versus global context

To determine priority among copies of a configuration file, Splunk software first determines the directory scheme.

Splunk software uses two main schemes of directory precedence.

App or user: Some activities, like searching, take place in an app or user context. The app and user context is vital to search-time processing, where certain knowledge objects or actions might be valid only for specific users in specific apps.

Global: Activities like indexing take place in a global context. They are independent of any app or user. For example, configuration files that determine monitoring behavior occur outside of the app and user context and are global in nature.

Cluster peer configuration context

There's also an expanded precedence order for cluster peer node global configurations. This is because some configuration files, like indexes.conf, must be identical across peer nodes.

To keep them consistent, files are managed from the cluster master, which distributes them to the peer nodes so that all peer nodes contain the same versions of the files. These files have the highest precedence in a cluster peer's configuration, which is explained in the
next section.

When consuming a global configuration, such as inputs.conf, Splunk first uses the attributes from any copy of the file in system/local. Then it looks for any copies of the file located in the app directories, adding any attributes found in them, but ignoring attributes already discovered in system/local. As a last resort, for any attributes not explicitly assigned at either the system or app level, it assigns default values from the file in the system/default directory.

Note: As the next section describes, cluster peer nodes have an expanded order of precedence.

Precedence for cluster peer nodes

For cluster peer nodes, the global context considers some additional peer-specific ("slave-app") directories. These directories contain apps and configurations that are identical across all peer nodes. Here is the expanded precedence order for cluster peers:

With cluster peers, custom settings common to all the peers (those in the slave-app local directories) have the highest precedence.

Precedence order within app or user context

When there's an app/user context, directory priority descends from user to app to system:

1. User directories for current user -- highest priority 2. App directories for currently running app (local, followed by default) 3. App directories for all other apps (local, followed by default) -- for exported settings only 4. System directories (local, followed by default) -- lowest priority

An attribute in savedsearches.conf, for example, might be set at all three levels: the user, the app, and the system. Splunk will always use the value of the user-level attribute, if any, in preference to a value for that same attribute set at the app or system level.

How app directory names affect precedence

Note: For most practical purposes, the information in this subsection probably won't matter, but it might prove useful if you need to force a certain order of evaluation or for troubleshooting.

To determine priority among the collection of apps directories, Splunk uses ASCII sort order. Files in an apps directory named "A" have a higher priority than files in an apps directory named "B", and so on. Also, all apps starting with an uppercase letter have precedence over any apps starting with a lowercase letter, due to ASCII sort order. ("A" has precedence over "Z", but "Z" has precedence over "a", for example.)

In addition, numbered directories have a higher priority than alphabetical directories and are evaluated in lexicographic, not numerical, order. For example, in descending order of precedence:

Note: When determining precedence in the app/user context, directories for the currently running app take priority over those for all other apps, independent of how they're named. Furthermore, other apps are only examined for exported settings.

Summary of directory precedence

Putting this all together, the order of directory priority, from highest to lowest, goes like this:

Important: Within the slave-apps/[local|default] directories, the special _cluster subdirectory has a higher precedence than any app subdirectories starting with a lowercase letter (for example, anApp). However, it has a lower precedence than any apps starting with an uppercase letter (for example, AnApp). This is due to the location of the underscore ("_") character in the ASCII sort order.

Important: In the app/user context, all configuration files for the currently running app take priority over files from all other apps. This is true for the app's local and default directories. So, if the current context is app C, Splunk evaluates both $SPLUNK_HOME/etc/apps/C/local/* and $SPLUNK_HOME/etc/apps/C/default/* before evaluating the local or default directories for any other apps. Furthermore, Splunk only looks at configuration data for other apps if that data has been exported globally through the app's default.meta file, as described in this topic on setting app permissions.

Also, note that /etc/users/ is evaluated only when the particular user logs in or performs a search.

Example of how attribute precedence works

This example of attribute precedence uses props.conf. The props.conf file is unusual, because its context can be either global or app/user, depending on when Splunk is evaluating it. Splunk evaluates props.conf at both index time (global) and search time (apps/user).

Assume $SPLUNK_HOME/etc/system/local/props.conf contains this stanza:

[source::/opt/Locke/Logs/error*]
sourcetype = fatal-error

and $SPLUNK_HOME/etc/apps/t2rss/local/props.conf contains another version of the same stanza:

The line merging attribute assignments in t2rss always apply, as they only occur in that version of the file. However, there's a conflict with the sourcetype attribute. In the /system/local version, the sourcetype has a value of "fatal-error". In the /apps/t2rss/local version, it has a value of "t2rss-error".

Since this is a sourcetype assignment, which gets applied at index time, Splunk uses the global context for determining directory precedence. In the global context, Splunk gives highest priority to attribute assignments in system/local. Thus, the sourcetype attribute gets assigned a value of "fatal-error".

List of configuration files and their context

As mentioned, Splunk decides how to evaluate a configuration file based on the context that the file operates within, global or app/user. Generally speaking, files that affect data input, indexing, or deployment activities are global; files that affect search activities usually have a app/user context.

The props.conf and transforms.conf files can be evaluated in either a app/user or a global context, depending on whether Splunk is using them at index or search time.

Troubleshooting configuration precedence and other issues

Splunk's configuration file system supports many overlapping configuration files in many different locations. The price of this level of flexibility is that figuring out which value for which configuration option is being used in your Splunk installation can sometimes be quite complex. If you're looking for some tips on figuring out what configuration setting is being used in a given situation, read "Use btool to troubleshoot configurations" in the Troubleshooting Manual.

Comments

For clarity, "Splunk uses ASCII sort order." probably needs a little more precision. How about:
Splunk uses 7-bit ASCII sort order for filename precedence, and pays no attention to collation or language settings. Extended and non-ASCII characters in filenames are ignored. (are they?)

Mikecee

September 19, 2015

Rfrey - I assume your main concern here is precedence in the so-called app/user, aka search-time, context, described in this topic. The search head transmits the search-time settings to the search peers via the knowledge bundle.<br /><br />The search peers then use the contents of the knowledge bundle to execute searches on behalf of their search head. In so doing, they follow the app/user precedence. So, for example, a knowledge bundle setting in $SPLUNK_HOME/etc/users/*, for the user running the search, will have precedence over a conflicting setting in $SPLUNK_HOME/etc/apps/Current_running_app/local/* in the bundle.<br /><br />The use of mounted bundles doesn't change this. It just means that the search peer will access the conf files from the mounted bundle rather than from a downloaded copy of the bundle. Similarly, search head pooling doesn't affect the set of files that the search peers receive from the search heads. The contents of the knowledge bundle remain the same in both cases.

Sgoodman

February 6, 2015

Can you elaborate on how file precedence works in relation to mounted bundles and search head pooling, specifically if there is a clear order of precedence based off of location on the file system? I am aware that you can use btool to determine how things are being applied but in a design phase that is a very trial and error approach to determining which copy of each configuration file should host which settings in a distributed environment.

Enter your email address, and someone from the documentation team will respond to you:

Send me a copy of this feedback

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »