Configuration parameters and the data pipeline

Each phase of the data pipeline relies on different configuration file parameters. Knowing which phase uses a particular parameter allows you to identify where in your Splunk deployment topology you need to set the parameter.

What the data pipeline looks like

How Splunk Enterprise components correlate to phases of the pipeline

One or more Splunk Enterprise components can perform each of the pipeline phases. For example, a universal forwarder, a heavy forwarder, or an indexer can perform the input phase.

Data only goes through each phase once, so each configuration belongs on only one component, specifically, the first component in the deployment that handles that phase. For example, say you have data entering the system through a set of universal forwarders, which forward the data to an intermediate heavy forwarder, which then forwards the data onwards to an indexer. In that case, the input phase for that data occurs on the universal forwarders, and the parsing phase occurs on the heavy forwarder.

Where to set a configuration parameter depends on the components in your specific deployment. For example, you set parsing parameters on the indexers in most cases. But if you have heavy forwarders feeding data to the indexers, you instead set parsing parameters on the heavy forwarders. Similarly, you set search parameters on the search heads, if any. But if you aren't deploying dedicated search heads, you set the search parameters on the indexers.

How configuration parameters correlate to phases of the pipeline

This is a non-exhaustive list of configuration parameters and the pipeline phases that use them. By combining this information with an understanding of which Splunk component in your particular deployment performs each phase, you can determine where to configure each setting.

For example, if you are using universal forwarders to consume inputs, you need to configure inputs.conf parameters on the forwarders. If, however, your indexer is directly consuming network inputs, you need to configure those network-related inputs.conf parameters on the indexer.

Input phase

inputs.conf

props.conf

CHARSET

NO_BINARY_CHECK

CHECK_METHOD

sourcetype

wmi.conf

regmon-filters.conf

Parsing phase

props.conf

LINE_BREAKER, SHOULD_LINEMERGE, BREAK_ONLY_BEFORE_DATE, and all other line merging settings

TZ, DATETIME_CONFIG, TIME_FORMAT, TIME_PREFIX, and all other time extraction settings and rules

TRANSFORMS* which includes per-event queue filtering, per-event index assignment, per-event routing. Applied in the order defined

SEDCMD*

MORE_THAN*, LESS_THAN*

INDEXED_EXTRACTIONS

transforms.conf

stanzas referenced by a TRANSFORMS* clause in props.conf

LOOKAHEAD, DEST_KEY, WRITE_META, DEFAULT_VALUE, REPEAT_MATCH

datetime.xml

Indexing phase

props.conf

SEGMENTATION*

indexes.conf

segmenters.conf

Search phase

props.conf

EXTRACT*

REPORT*

LOOKUP*

KV_MODE

FIELDALIAS*

rename

transforms.conf

stanzas referenced by a REPORT* clause in props.conf

filename, external_cmd, and all other lookup-related settings

FIELDS, DELIMS

MV_ADD

lookup files in the lookups folders

search and lookup scripts in the bin folders

search commands and lookup scripts

savedsearches.conf

eventtypes.conf

tags.conf

commands.conf

alert_actions.conf

macros.conf

fields.conf

transactiontypes.conf

multikv.conf

Other configuration settings

There are some settings that don't work well in a distributed Splunk environment. These tend to be exceptional and include:

props.conf

CHECK_FOR_HEADER, LEARN_MODEL, maxDist. These are created in the parsing phase, but they require generated configurations to be moved to the search phase configuration location.

Enter your email address, and someone from the documentation team will respond to you:

Send me a copy of this feedback

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »