Introspection endpoint descriptions

Get information about the system.

Note: Username and password authentication is required for most endpoints and REST operations. Additional capability or role-based authorization may also be required, particularly for POST or DELETE operations.

data/index-volumes

https://<host>:<mPort>/services/data/index-volumes

Description

Provides information about the volume (logical drives) in use by Splunk Enterprise.

The default update period is 10 minutes, as defined by the collectionPeriodInSecs attribute in the $SPLUNK_HOME/etc/apps/introspection_generator_addon/default/server.conf file.

At least one observation period must pass after Splunk Enterprise startup for valid endpoint data to be available. The observation period is defined in the $SPLUNK_HOME/etc/system/default/server.conf file:

data/index-volumes/{name}

Description

The default update period is 10 minutes, as defined by the collectionPeriodInSecs attribute in the $SPLUNK_HOME/etc/apps/introspection_generator_addon/default/server.conf file.

At least one observation period must pass after Splunk Enterprise startup for valid endpoint data to be available. The observation period is defined in the $SPLUNK_HOME/etc/system/default/server.conf file:

GET data/indexes method detail

Request parameters

Response data keys

Indicates whether all data retreived from the index is proper UTF8. If enabled (set to True), degrades indexing performance.

This is a global setting, not a per index setting.

blockSignSize

Controls how many events make up a block for block signatures.

If this is set to 0, block signing is disabled for this index.

A recommended value is 100.

blockSignatureDatabase

The index that stores block signatures of events.

This is a global setting, not a per index setting.

coldPath

Filepath to the cold databases for the index.

coldPath_expanded

Absoute filepath to the cold databases.

coldToFrozenDir

Destination path for the frozen archive. Used as an alternative to a coldToFrozenScript. Splunk Enterprise automatically puts frozen buckets in this directory.

Bucket freezing policy is as follows:

New style buckets (4.2 and on): removes all files but the rawdata

To thaw, run splunk rebuild <bucket dir> on the bucket, then move to the thawed directory

Old style buckets (Pre-4.2): gzip all the .data and .tsidx files

To thaw, gunzip the zipped files and move the bucket into the thawed directory

If both coldToFrozenDir and coldToFrozenScript are specified, coldToFrozenDir takes precedence.

coldToFrozenScript

Path to the archiving script.

See the POST parameter description for details.

compressRawdata

This value is ignored. splunkd process always compresses raw data.

currentDBSizeMB

Total size, in MB, of data stored in the index. The total incudes data in the home, cold and thawed paths.

defaultDatabase

If no index destination information is available in the input data, the index shown here is the destination of such data.

disabled

Indicates if the index is disabled.

enableRealtimeSearch

Indicates if this is a real-time search.

This is a global setting, not a per index setting.

frozenTimePeriodInSecs

Number of seconds after which indexed data rolls to frozen. Defaults to 188697600 (6 years).

Freezing data means it is removed from the index. If you need to archive your data, refer to coldToFrozenDir and coldToFrozenScript parameter documentation.

homePath

An absolute path that contains the hot and warm buckets for the index.

homePath_expanded

An absolute filepath to the hot and warm buckets for the index.

indexThreads

Number of threads used for indexing.

This is a global setting, not a per index setting.

isInternal

Indicates if this is an internal index (for example, _internal, _audit).

lastInitTime

Last time the index processor was successfully initialized.

This is a global setting, not a per index setting.

maxConcurrentOptimizes

The number of concurrent optimize processes that can run against a hot bucket.

This number should be increased if instructed by Splunk Support. Typically the default value should suffice.

maxDataSize

The maximum size in MB for a hot DB to reach before a roll to warm is triggered. Specifying "auto" or "auto_high_volume" causes Splunk Enterprise to autotune this parameter (recommended). Use "auto_high_volume" for high volume indexes (such as the main index); otherwise, use "auto". A "high volume index" is typically one that gets over 10GB of data per day.

"auto" sets the size to 750MB.

"auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems.

Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable number ranges anywhere from 100 - 50000. Any number outside this range should be approved by Splunk Support before proceeding.

If you specify an invalid number or string, maxDataSize is auto-tuned.

Note: The precise size of your warm buckets may vary from maxDataSize, due to post-processing and timing issues with the rolling policy.

maxHotBuckets

Maximum hot buckets that can exist per index. Defaults to 3.

When maxHotBuckets is exceeded, Splunk Enterprise rolls the least recently used (LRU) hot bucket to warm. Both normal hot buckets and quarantined hot buckets count towards this total. This setting operates independently of maxHotIdleSecs, which can also cause hot buckets to roll.

maxHotIdleSecs

Maximum life, in seconds, of a hot bucket. Defaults to 0. A value of 0 turns off the idle check (equivalent to INFINITE idle time).

If a hot bucket exceeds maxHotIdleSecs, Splunk Enterprise rolls it to warm. This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll.

Note: If set too small, you can get an explosion of hot/warm buckets in the filesystem. The system sets a lower bound implicitly for this parameter at 3600, but this is an advanced parameter that should be set with care and understanding of the characteristics of your data.

maxMemMB

The amount of memory, in MB, allocated for indexing.

This is a global setting, not a per index setting.

maxMetaEntries

Sets the maximum number of unique lines in .data files in a bucket, which may help to reduce memory consumption. If set to 0, this setting is ignored (it is treated as infinite).

If exceeded, a hot bucket is rolled to prevent further increase. If your buckets are rolling due to Strings.data hitting this limit, the culprit may be the punct field in your data. If you do not use punct, it may be best to simply disable this (see props.conf.spec in $SPLUNK_HOME/etc/system/README).

There is a small time delta between when maximum is exceeded and bucket is rolled. This means a bucket may end up with epsilon more lines than specified, but this is not a major concern unless excess is significant.

maxRunningProcessGroups

Maximum number of processes that the indexer fires off at a time.

This is a global setting, not a per index setting.

maxTime

ISO8601 format timestamp of the newest event time in the index.

maxTotalDataSizeMB

The maximum size of an index, in MB.

maxWarmDBCount

The maximum number of warm buckets. If this number is exceeded, the warm bucket/s with the lowest value for their latest times are moved to cold.

memPoolMB

Determines how much memory is given to the indexer memory pool.

This is a global setting, not a per-index setting.

minRawFileSyncSecs

Can be either an integer (or "disable"). Some filesystems are very inefficient at performing sync operations, so only enable this if you are sure it is needed

During this period, uncompressed slices are left on disk even after they are compressed. Then splunkd forces a filesystem sync of the compressed journal and removes the accumulated uncompressed files.

If 0 is specified, splunkd forces a filesystem sync after every slice completes compressing. Specifying "disable" disables syncing entirely: uncompressed slices are removed as soon as compression is complete.

minTime

ISO8601 format timestamp of the oldest event time in the index.

partialServiceMetaPeriod

Related to serviceMetaPeriod. By default it is turned off (zero).

If set, it enables metadata sync every <integer> seconds, but only for records where the sync can be done efficiently in-place, without requiring a full re-write of the metadata file. Records that require full re-write are be sync'ed at serviceMetaPeriod.

partialServiceMetaPeriod specifies, in seconds, how frequently it should sync. Zero means that this feature is turned off and serviceMetaPeriod is the only time when metadata sync happens.

If the value of partialServiceMetaPeriod is greater than serviceMetaPeriod, this setting has no effect.

quarantineFutureSecs

Events with timestamp of quarantineFutureSecs newer than "now" that are dropped into quarantine bucket. Defaults to 2592000 (30 days).

This is a mechanism to prevent main hot buckets from being polluted with fringe events.

quarantinePastSecs

Events with timestamp of quarantinePastSecs older than "now" are dropped into quarantine bucket. Defaults to 77760000 (900 days).

This is a mechanism to prevent the main hot buckets from being polluted with fringe events.

rawChunkSizeBytes

Target uncompressed size in bytes for individual raw slice in the rawdata journal of the index. Defaults to 131072 (128KB). 0 is not a valid value. If 0 is specified, rawChunkSizeBytes is set to the default value.

Note: rawChunkSizeBytes only specifies a target chunk size. The actual chunk size may be slightly larger by an amount proportional to an individual event size.

Warning: This is an advanced parameter. Only change it if instructed to do so by Splunk Support.

rotatePeriodInSecs

Rotation period, in seconds, that specifies how frequently to check:

If a new hot bucket needs to be created.

If there are any cold buckets that should be frozen.

If there are any buckets that need to be moved out hot and cold DBs, due to size constraints.

serviceMetaPeriod

Defines how frequently metadata is synced to disk, in seconds. Defaults to 25 (seconds).

You may want to set this to a higher value if the sum of your metadata file sizes is larger than many tens of megabytes, to avoid the hit on I/O in the indexing fast path.

summarize

If true, leaves out certain index details, which provides a faster response.

suppressBannerList

List of indexes for which we suppress "index missing" warning banner messages.

This is a global setting, not a per index setting.

sync

Specifies the number of events that trigger the indexer to sync events.

This is a global setting, not a per index setting.

syncMeta

When true, a sync operation is called before file descriptor is closed on metadata file updates. This functionality improves integrity of metadata files, especially in regards to operating system crashes/machine failures.

Note: Do not change this parameter without the input of Splunk Support.

thawedPath

An absolute path that contains the thawed (resurrected) databases for the index.

Suggestion for the Splunk Enterprise bucket rebuild process for the size of the time-series (tsidx) file to make.

Caution: This is an advanced parameter. Inappropriate use of this parameter causes splunkd to not start if rebuild is required. Do not set this parameter unless instructed by Splunk Support.

Default value, auto, varies by the amount of physical RAM on the host

less than 2GB RAM = 67108864 (64MB) tsidx

2GB to 8GB RAM = 134217728 (128MB) tsidx

more than 8GB RAM = 268435456 (256MB) tsidx

Values other than "auto" must be 16MB-1GB. Highest legal value (of the numerical part) is 4294967295

You can specify the value using a size suffix: "16777216" or "16MB" are equivalent.

coldPath

String

An absolute path that contains the colddbs for the index. The path must be readable and writable. Cold databases are opened as needed when searching. May be defined in terms of a volume definition (see volume section below).

Required. Splunk Enterprise does not start if an index lacks a valid coldPath.

coldToFrozenDir

String

Destination path for the frozen archive. Use as an alternative to a coldToFrozenScript. Splunk Enterprise automatically puts frozen buckets in this directory.

Bucket freezing policy is as follows:

New style buckets (4.2 and on): removes all files but the rawdata

To thaw, run splunk rebuild <bucket dir> on the bucket, then move to the thawed directory

Old style buckets (Pre-4.2): gzip all the .data and .tsidx files

To thaw, gunzip the zipped files and move the bucket into the thawed directory

If both coldToFrozenDir and coldToFrozenScript are specified, coldToFrozenDir takes precedence

coldToFrozenScript

String

Path to the archiving script.

If your script requires a program to run it (for example, python), specify the program followed by the path. The script must be in $SPLUNK_HOME/bin or one of its subdirectories.

Splunk Enterprise ships with an example archiving script in $SPLUNK_HOME/bin called coldToFrozenExample.py. Splunk DOES NOT recommend using this example script directly. It uses a default path, and if modified in place any changes are overwritten on upgrade.

Splunk recommends copying the example script to a new file in bin and modifying it for your system. Most importantly, change the default archive path to an existing directory that fits your needs.

If your new script in bin/ is named myColdToFrozen.py, set this key to the following:

By default, the example script has two possible behaviors when archiving:

For buckets created from version 4.2 and on, it removes all files except for rawdata. To thaw: cd to the frozen bucket and type splunk rebuild ., then copy the bucket to thawed for that index. We recommend using the coldToFrozenDir parameter unless you need to perform a more advanced operation upon freezing buckets.

For older-style buckets, we simply gzip all the .tsidx files. To thaw: cd to the frozen bucket and unzip the tsidx files, then copy the bucket to thawed for that index

When enabled, you do not have to wait until buckets are repaired to start Splunk. However, you might observe a slight performance degratation.

frozenTimePeriodInSecs

Number

188697600

Number of seconds after which indexed data rolls to frozen. Defaults to 188697600 (6 years).

Freezing data means it is removed from the index. If you need to archive your data, refer to coldToFrozenDir and coldToFrozenScript parameter documentation.

homePath

String

An absolute path that contains the hot and warm buckets for the index.

Required. Splunk Enterprise does not start if an index lacks a valid homePath.

CAUTION: Path MUST be readable and writable.

maxBloomBackfillBucketAge

Number

30d

Valid values are: Integer[m|s|h|d]

If a warm or cold bucket is older than the specified age, do not create or rebuild its bloomfilter. Specify 0 to never rebuild bloomfilters.

For example, if a bucket is older than specified with maxBloomBackfillBucketAge, and the rebuilding of its bloomfilter started but did not finish, do not rebuild it.

maxConcurrentOptimizes

Number

6

The number of concurrent optimize processes that can run against a hot bucket.

This number should be increased if instructed by Splunk Support. Typically the default value should suffice.

maxDataSize

Number

auto

The maximum size in MB for a hot DB to reach before a roll to warm is triggered. Specifying "auto" or "auto_high_volume" causes Splunk Enterprise to autotune this parameter (recommended).Use "auto_high_volume" for high volume indexes (such as the main index); otherwise, use "auto". A "high volume index" would typically be considered one that gets over 10GB of data per day.

"auto" sets the size to 750MB.

"auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems.

Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable number ranges anywhere from 100 - 50000. Any number outside this range should be approved by Splunk Support before proceeding.

If you specify an invalid number or string, maxDataSize is auto-tuned.

Note: The precise size of your warm buckets may vary from maxDataSize, due to post-processing and timing issues with the rolling policy.

maxHotBuckets

Number

3

Maximum hot buckets that can exist per index. Defaults to 3.

When maxHotBuckets is exceeded, Splunk Enterprise rolls the least recently used (LRU) hot bucket to warm. Both normal hot buckets and quarantined hot buckets count towards this total. This setting operates independently of maxHotIdleSecs, which can also cause hot buckets to roll.

maxHotIdleSecs

Number

0

Maximum life, in seconds, of a hot bucket. Defaults to 0.

If a hot bucket exceeds maxHotIdleSecs, Splunk Enterprise rolls it to warm. This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll. A value of 0 turns off the idle check (equivalent to INFINITE idle time).

Note:I f you set this too small, you can get an explosion of hot/warm buckets in the filesystem. The system sets a lower bound implicitly for this parameter at 3600, but this is an advanced parameter that should be set with care and understanding of the characteristics of your data.

maxMemMB

Number

5

The amount of memory, expressed in MB, to allocate for buffering a single tsidx file into memory before flushing to disk. Defaults to 5. The default is recommended for all environments.

IMPORTANT: Calculate this number carefully. Setting this number incorrectly may have adverse effects on your systems memory and/or splunkd stability/performance.

maxMetaEntries

Number

1000000

Sets the maximum number of unique lines in .data files in a bucket, which may help to reduce memory consumption. If set to 0, this setting is ignored (it is treated as infinite).

If exceeded, a hot bucket is rolled to prevent further increase. If your buckets are rolling due to Strings.data hitting this limit, the culprit may be the punct field in your data. If you do not use punct, it may be best to simply disable this (see props.conf.spec in $SPLUNK_HOME/etc/system/README).

There is a small time delta between when maximum is exceeded and bucket is rolled. This means a bucket may end up with epsilon more lines than specified, but this is not a major concern unless excess is significant.

maxTimeUnreplicatedNoAcks

Number

300

Upper limit, in seconds, on how long an event can sit in raw slice. Applies only if replication is enabled for this index. Otherwise ignored.

If there are any acknowledged events sharing this raw slice, this paramater does not apply. In this case, maxTimeUnreplicatedWithAcks applies.

Highest legal value is 2147483647. To disable this parameter, set to 0.

Note: this is an advanced parameter. Understand the consequences before changing.

maxTimeUnreplicatedWithAcks

Number

60

Upper limit, in seconds, on how long events can sit unacknowledged in a raw slice. Applies only if you have enabled acks on forwarders and have replication enabled (with clustering).

Note: This is an advanced parameter. Make sure you understand the settings on all forwarders before changing this. This number should not exceed ack timeout configured on any forwarder, and should actually be set to at most half of the minimum value of that timeout. You can find this setting in outputs.conf readTimeout setting under the tcpout stanza.

To disable, set to 0, but this is NOT recommended. Highest legal value is 2147483647.

maxTotalDataSizeMB

Number

500000

The maximum size of an index (in MB). If an index grows larger than the maximum size, the oldest data is frozen.

maxWarmDBCount

Number

300

The maximum number of warm buckets. If this number is exceeded, the warm bucket/s with the lowest value for their latest times is moved to cold.

During this period, uncompressed slices are left on disk even after they are compressed. Then splunkd forces a filesystem sync of the compressed journal and removes the accumulated uncompressed files.

If 0 is specified, splunkd forces a filesystem sync after every slice completes compressing. Specifying "disable" disables syncing entirely: uncompressed slices are removed as soon as compression is complete.

Note: Some filesystems are very inefficient at performing sync operations, so only enable this if you are sure it is needed

minStreamGroupQueueSize

Number

2000

Minimum size of the queue that stores events in memory before committing them to a tsidx file.

Caution: Do not set this value, except under advice from Splunk Support.

namerequired

String

The name of the index to create.

partialServiceMetaPeriod

Number

0

Related to serviceMetaPeriod. If set, it enables metadata sync every <integer> seconds, but only for records where the sync can be done efficiently in-place, without requiring a full re-write of the metadata file. Records that require full re-write are be sync'ed at serviceMetaPeriod.

partialServiceMetaPeriod specifies, in seconds, how frequently it should sync. Zero means that this feature is turned off and serviceMetaPeriod is the only time when metadata sync happens.

If the value of partialServiceMetaPeriod is greater than serviceMetaPeriod, this setting has no effect.

By default it is turned off (zero).

processTrackerServiceInterval

Number

1

Specifies, in seconds, how often the indexer checks the status of the child OS processes it launched to see if it can launch new processes for queued requests. Defaults to 15.

If set to 0, the indexer checks child process status every second.

Highest legal value is 4294967295.

quarantineFutureSecs

Number

2592000

Events with timestamp of quarantineFutureSecs newer than "now" are dropped into quarantine bucket. Defaults to 2592000 (30 days).

This is a mechanism to prevent main hot buckets from being polluted with fringe events.

quarantinePastSecs

Number

77760000

Events with timestamp of quarantinePastSecs older than "now" are dropped into quarantine bucket. Defaults to 77760000 (900 days).

This is a mechanism to prevent the main hot buckets from being polluted with fringe events.

rawChunkSizeBytes

Number

131072

Target uncompressed size in bytes for individual raw slice in the rawdata journal of the index. Defaults to 131072 (128KB). 0 is not a valid value. If 0 is specified, rawChunkSizeBytes is set to the default value.

Note: rawChunkSizeBytes only specifies a target chunk size. The actual chunk size may be slightly larger by an amount proportional to an individual event size.

WARNING: This is an advanced parameter. Only change it if you are instructed to do so by Splunk Support.

How frequently (in seconds) to check if a new hot bucket needs to be created. Also, how frequently to check if there are any warm/cold buckets that should be rolled/frozen.

serviceMetaPeriod

Number

25

Defines how frequently metadata is synced to disk, in seconds. Defaults to 25 (seconds).

You may want to set this to a higher value if the sum of your metadata file sizes is larger than many tens of megabytes, to avoid the hit on I/O in the indexing fast path.

syncMeta

Boolean

true

When true, a sync operation is called before file descriptor is closed on metadata file updates. This functionality improves integrity of metadata files, especially in regards to operating system crashes/machine failures.

Note: Do not change this parameter without the input of a Splunk Support.

thawedPath

String

An absolute path that contains the thawed (resurrected) databases for the index.

Cannot be defined in terms of a volume definition.

Required. Splunk Enterprise does not start if an index lacks a valid thawedPath</codePath>.

Note: Do not change this parameter without the input of Splunk Support.

tstatsHomePath

String

Location to store datamodel acceleration TSIDX data for this index. Restart splunkd after changing this parameter.

If specified, it must be defined in terms of a volume definition.

Caution: Path must be writable.

Default value: volume:_splunk_summaries/$_index_name/tstats

warmToColdScript

String

Path to a script to run when moving data from warm to cold.

This attribute is supported for backwards compatibility with Splunk Enterprise versions older than 4.0. Contact Splunk support if you need help configuring this setting.

Caution: Migrating data across filesystems is now handled natively by splunkd. If you specify a script here, the script becomes responsible for moving the event data, and Splunk-native data migration is not used.

Response data keys

Name

Description

assureUTF8

Boolean value indicating wheter all data retreived from the index is proper UTF8.

If enabled (set to True), degrades indexing performance

Can only be set globally.

blockSignSize

Controls how many events make up a block for block signatures.

If this is set to 0, block signing is disabled for this index.

A recommended value is 100.

blockSignatureDatabase

The index that stores block signatures of events.

This is a global setting, not a per index setting.

bucketRebuildMemoryHint

Suggestion for the Splunk Enterprise bucket rebuild process for the size of the time-series (tsidx) file to make.

coldPath

Filepath to the cold databases for the index.

coldPath_expanded

Absoute filepath to the cold databases.

coldToFrozenDir

Destination path for the frozen archive. Used as an alternative to a coldToFrozenScript. Splunk Enterprise automatically puts frozen buckets in this directory.

Bucket freezing policy is as follows:

New style buckets (4.2 and on): removes all files but the rawdata

To thaw, run <code>splunk rebuild <bucket dir> on the bucket, then move to the thawed directory

Old style buckets (Pre-4.2): gzip all the .data and .tsidx files

To thaw, gunzip the zipped files and move the bucket into the thawed directory

If both coldToFrozenDir and coldToFrozenScript are specified, coldToFrozenDir takes precedence.

coldToFrozenScript

Path to the archiving script.

See the POST parameter description for details.

compressRawdata

This value is ignored. splunkd process always compresses raw data.

currentDBSizeMB

Total size, in MB, of data stored in the index. The total incudes data in the home, cold and thawed paths.

defaultDatabase

If no index destination information is available in the input data, the index shown here is the destination of such data.

enableOnlineBucketRepair

Indicates whether to run asynchronous "online fsck" bucket repair, which runs in a process concurrently with Splunk.

enableRealtimeSearch

Indicates if this is a real-time search.

This is a global setting, not a per index setting.

frozenTimePeriodInSecs

Number of seconds after which indexed data rolls to frozen. Defaults to 188697600 (6 years).

Freezing data means it is removed from the index. If you need to archive your data, refer to coldToFrozenDir and coldToFrozenScript parameter documentation.

homePath

An absolute path that contains the hot and warm buckets for the index.

homePath_expanded

An absolute filepath to the hot and warm buckets for the index.

indexThreads

Number of threads used for indexing.

This is a global setting, not a per index setting.

isInternal

Indicates if this is an internal index (for example, _internal, _audit).

lastInitTime

Last time the index processor was successfully initialized.

This is a global setting, not a per index setting.

maxBloomBackfillBucketAge

If a bucket (warm or cold) is older than this, Splunk Enterprise does not create (or re-create) its bloom filter.

maxConcurrentOptimizes

The number of concurrent optimize processes that can run against a hot bucket.

This number should be increased if instructed by Splunk Support. Typically the default value should suffice.

maxDataSize

The maximum size in MB for a hot DB to reach before a roll to warm is triggered. Specifying "auto" or "auto_high_volume" causes Splunk Enterprise to autotune this parameter (recommended). Use "auto_high_volume" for high volume indexes (such as the main index); otherwise, use "auto". A "high volume index" is typically one that gets over 10GB of data per day.

"auto" sets the size to 750MB.

"auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems.

Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable number ranges anywhere from 100 - 50000. Any number outside this range should be approved by Splunk Support before proceeding.

If you specify an invalid number or string, maxDataSize is auto-tuned.

Note: The precise size of your warm buckets may vary from maxDataSize, due to post-processing and timing issues with the rolling policy.

maxHotBuckets

Maximum hot buckets that can exist per index. Defaults to 3.

When maxHotBuckets is exceeded, Splunk Enterprise rolls the least recently used (LRU) hot bucket to warm. Both normal hot buckets and quarantined hot buckets count towards this total. This setting operates independently of maxHotIdleSecs, which can also cause hot buckets to roll.

maxHotIdleSecs

Maximum life, in seconds, of a hot bucket. Defaults to 0. A value of 0 turns off the idle check (equivalent to INFINITE idle time).

If a hot bucket exceeds maxHotIdleSecs, Splunk Enterprise rolls it to warm. This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll.

Note: If set too small, you can get an explosion of hot/warm buckets in the filesystem. The system sets a lower bound implicitly for this parameter at 3600, but this is an advanced parameter that should be set with care and understanding of the characteristics of your data.

maxMemMB

The amount of memory, in MB, allocated for indexing.

This is a global setting, not a per index setting.

maxMetaEntries

Sets the maximum number of unique lines in .data files in a bucket, which may help to reduce memory consumption. If set to 0, this setting is ignored (it is treated as infinite).

If exceeded, a hot bucket is rolled to prevent further increase. If your buckets are rolling due to Strings.data hitting this limit, the culprit may be the punct field in your data. If you do not use punct, it may be best to simply disable this (see props.conf.spec in $SPLUNK_HOME/etc/system/README).

There is a small time delta between when maximum is exceeded and bucket is rolled. This means a bucket may end up with epsilon more lines than specified, but this is not a major concern unless excess is significant.

maxTime

UNIX timestamp of the newest event time in the index.

maxTimeUnreplicatedNoAcks

Upper limit, in seconds, on how long an event can sit in raw slice. Applies only if replication is enabled for this index. Otherwise ignored.

If there are any acknowledged events sharing this raw slice, this paramater does not apply. In this case, maxTimeUnreplicatedWithAcks applies.

Highest legal value is 2147483647. To disable this parameter, set to 0.

Note: this is an advanced parameter. Understand the consequences before changing.

maxTimeUnreplicatedWithAcks

Upper limit, in seconds, on how long events can sit unacknowledged in a raw slice. Applies only if you have enabled acks on forwarders and have replication enabled (with clustering).

Note: This is an advanced parameter. Make sure you understand the settings on all forwarders before changing this. This number should not exceed ack timeout configured on any forwarder, and should actually be set to at most half of the minimum value of that timeout. You can find this setting in outputs.conf readTimeout setting under the tcpout stanza.

To disable, set to 0, but this is NOT recommended. Highest legal value is 2147483647.

maxTotalDataSizeMB

The maximum size of an index, in MB.

maxWarmDBCount

The maximum number of warm buckets. If this number is exceeded, the warm bucket/s with the lowest value for their latest times are moved to cold.

memPoolMB

Determines how much memory is given to the indexer memory pool.

This is a global setting, not a per-index setting.

minRawFileSyncSecs

Can be either an integer (or "disable"). Some filesystems are very inefficient at performing sync operations, so only enable this if you are sure it is needed

During this period, uncompressed slices are left on disk even after they are compressed. Then splunkd forces a filesystem sync of the compressed journal and removes the accumulated uncompressed files.

If 0 is specified, splunkd forces a filesystem sync after every slice completes compressing. Specifying "disable" disables syncing entirely: uncompressed slices are removed as soon as compression is complete.

minStreamGroupQueueSize

Minimum size of the queue that stores events in memory before committing them to a tsidx file.

minTime

UNIX timestamp of the oldest event time in the index.

partialServiceMetaPeriod

Related to serviceMetaPeriod. By default it is turned off (zero).

If set, it enables metadata sync every <integer> seconds, but only for records where the sync can be done efficiently in-place, without requiring a full re-write of the metadata file. Records that require full re-write are be sync'ed at serviceMetaPeriod.

partialServiceMetaPeriod specifies, in seconds, how frequently it should sync. Zero means that this feature is turned off and serviceMetaPeriod is the only time when metadata sync happens.

If the value of partialServiceMetaPeriod is greater than serviceMetaPeriod, this setting has no effect.

processTrackerServiceInterval

How often, in seconds, the indexer checks the status of the child OS processes it launched to see if it can launch new processes for queued requests.

quarantineFutureSecs

Events with timestamp of quarantineFutureSecs newer than "now" are dropped into quarantine bucket. Defaults to 2592000 (30 days).

This is a mechanism to prevent main hot buckets from being polluted with fringe events.

quarantinePastSecs

Events with timestamp of quarantinePastSecs older than "now" are dropped into quarantine bucket. Defaults to 77760000 (900 days).

This is a mechanism to prevent the main hot buckets from being polluted with fringe events.

rawChunkSizeBytes

Target uncompressed size in bytes for individual raw slice in the rawdata journal of the index. Defaults to 131072 (128KB). 0 is not a valid value. If 0 is specified, rawChunkSizeBytes is set to the default value.

Note: rawChunkSizeBytes only specifies a target chunk size. The actual chunk size may be slightly larger by an amount proportional to an individual event size.

Warning: This is an advanced parameter. Only change it if instructed to do so by Splunk Support.

If there are any buckets that need to be moved out hot and cold DBs, due to size constraints.

serviceMetaPeriod

Defines how frequently metadata is synced to disk, in seconds. Defaults to 25 (seconds).

You may want to set this to a higher value if the sum of your metadata file sizes is larger than many tens of megabytes, to avoid the hit on I/O in the indexing fast path.

suppressBannerList

List of indexes for which we suppress "index missing" warning banner messages.

This is a global setting, not a per index setting.

sync

Specifies the number of events that trigger the indexer to sync events.

This is a global setting, not a per index setting.

syncMeta

When true, a sync operation is called before file descriptor is closed on metadata file updates. This functionality improves integrity of metadata files, especially in regards to operating system crashes/machine failures.

Note: Do not change this parameter without the input of Splunk Support.

DELETE data/indexes/{name} method detail

Request parameters

None

Response data keys

None

Application usage

Before executing this operation, look through all inputs.conf files (on your indexer and on any forwarders sending data to the indexer) and make sure that none of the stanzas are directing data to the index you plan to delete.

In other words, if you want to delete an index called "nogood," make sure the attribute/value pair index=nogood does not appear in any input stanzas. Once the index is deleted, Splunk Enterprise discards any data sent to that index.

For information on deleting indexes and deleting data from indexes, refer to Remove data from Splunk Enterprise in the Splunk Managing Indexers and Clusters manual.

Response data keys

Indicates whether all data retreived from the index is proper UTF8. If enabled (set to True), degrades indexing performance.

This is a global setting, not a per index setting.

blockSignSize

Controls how many events make up a block for block signatures.

If this is set to 0, block signing is disabled for this index.

A recommended value is 100.

blockSignatureDatabase

The index that stores block signatures of events.

This is a global setting, not a per index setting.

bloomfilterTotalSizeKB

Total size of all bloom filter files, in KB.

coldPath

Filepath to the cold databases for the index.

coldPath_expanded

Absoute filepath to the cold databases.

coldToFrozenDir

Destination path for the frozen archive. Used as an alternative to a coldToFrozenScript. Splunk Enterprise automatically puts frozen buckets in this directory.

Bucket freezing policy is as follows:

New style buckets (4.2 and on): removes all files but the rawdata

To thaw, run splunk rebuild <bucket dir> on the bucket, then move to the thawed directory

Old style buckets (Pre-4.2): gzip all the .data and .tsidx files

To thaw, gunzip the zipped files and move the bucket into the thawed directory

If both coldToFrozenDir and coldToFrozenScript are specified, coldToFrozenDir takes precedence.

coldToFrozenScript

Path to the archiving script.

See the POST parameter description for details.

compressRawdata

This value is ignored. splunkd process always compresses raw data.

currentDBSizeMB

Total size, in MB, of data stored in the index. The total incudes data in the home, cold and thawed paths.

defaultDatabase

If no index destination information is available in the input data, the index shown here is the destination of such data.

disabled

Indicates if the index is disabled.

enableRealtimeSearch

Indicates if this is a real-time search.

This is a global setting, not a per index setting.

frozenTimePeriodInSecs

Number of seconds after which indexed data rolls to frozen. Defaults to 188697600 (6 years).

Freezing data means it is removed from the index. If you need to archive your data, refer to coldToFrozenDir and coldToFrozenScript parameter documentation.

homePath

An absolute path that contains the hot and warm buckets for the index.

homePath_expanded

An absolute filepath to the hot and warm buckets for the index.

indexThreads

Number of threads used for indexing.

This is a global setting, not a per index setting.

isInternal

Indicates if this is an internal index (for example, _internal, _audit).

lastInitTime

Last time the index processor was successfully initialized.

This is a global setting, not a per index setting.

maxConcurrentOptimizes

The number of concurrent optimize processes that can run against a hot bucket.

This number should be increased if instructed by Splunk Support. Typically the default value should suffice.

maxDataSize

The maximum size in MB for a hot DB to reach before a roll to warm is triggered. Specifying "auto" or "auto_high_volume" causes Splunk Enterprise to autotune this parameter (recommended). Use "auto_high_volume" for high volume indexes (such as the main index); otherwise, use "auto". A "high volume index" is typically one that gets over 10GB of data per day.

"auto" sets the size to 750MB.

"auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems.

Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable number ranges anywhere from 100 - 50000. Any number outside this range should be approved by Splunk Support before proceeding.

If you specify an invalid number or string, maxDataSize is auto-tuned.

Note: The precise size of your warm buckets may vary from maxDataSize, due to post-processing and timing issues with the rolling policy.

maxHotBuckets

Maximum hot buckets that can exist per index. Defaults to 3.

When maxHotBuckets is exceeded, Splunk Enterprise rolls the least recently used (LRU) hot bucket to warm. Both normal hot buckets and quarantined hot buckets count towards this total. This setting operates independently of maxHotIdleSecs, which can also cause hot buckets to roll.

maxHotIdleSecs

Maximum life, in seconds, of a hot bucket. Defaults to 0. A value of 0 turns off the idle check (equivalent to INFINITE idle time).

If a hot bucket exceeds maxHotIdleSecs, Splunk Enterprise rolls it to warm. This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll.

Note: If set too small, you can get an explosion of hot/warm buckets in the filesystem. The system sets a lower bound implicitly for this parameter at 3600, but this is an advanced parameter that should be set with care and understanding of the characteristics of your data.

maxMemMB

The amount of memory, in MB, allocated for indexing.

This is a global setting, not a per index setting.

maxMetaEntries

Sets the maximum number of unique lines in .data files in a bucket, which may help to reduce memory consumption. If set to 0, this setting is ignored (it is treated as infinite).

If exceeded, a hot bucket is rolled to prevent further increase. If your buckets are rolling due to Strings.data hitting this limit, the culprit may be the punct field in your data. If you do not use punct, it may be best to simply disable this (see props.conf.spec in $SPLUNK_HOME/etc/system/README).

There is a small time delta between when maximum is exceeded and bucket is rolled. This means a bucket may end up with epsilon more lines than specified, but this is not a major concern unless excess is significant.

maxRunningProcessGroups

Maximum number of processes that the indexer fires off at a time.

This is a global setting, not a per index setting.

maxTime

UNIX timestamp of the newest event time in the index.

maxTotalDataSizeMB

The maximum size of an index, in MB.

maxWarmDBCount

Maximum number of warm buckets.

memPoolMB

Determines how much memory is given to the indexer memory pool.

This is a global setting, not a per-index setting.

minRawFileSyncSecs

Can be either an integer (or "disable"). Some filesystems are very inefficient at performing sync operations, so only enable this if you are sure it is needed

During this period, uncompressed slices are left on disk even after they are compressed. Then splunkd forces a filesystem sync of the compressed journal and removes the accumulated uncompressed files.

If 0 is specified, splunkd forces a filesystem sync after every slice completes compressing. Specifying "disable" disables syncing entirely: uncompressed slices are removed as soon as compression is complete.

minTime

UNIX timestamp of the oldest event time in the index.

numBloomfilters

The number of bloom filters created for this index.

numHotBuckets

The number of hot buckets created for this index.

numWarmBuckets

The number of warm buckets created for this index.

partialServiceMetaPeriod

Related to serviceMetaPeriod. By default it is turned off (zero).

If set, it enables metadata sync every <integer> seconds, but only for records where the sync can be done efficiently in-place, without requiring a full re-write of the metadata file. Records that require full re-write are be sync'ed at serviceMetaPeriod.

partialServiceMetaPeriod specifies, in seconds, how frequently it should sync. Zero means that this feature is turned off and serviceMetaPeriod is the only time when metadata sync happens.

If the value of partialServiceMetaPeriod is greater than serviceMetaPeriod, this setting has no effect.

quarantineFutureSecs

Events with timestamp of quarantineFutureSecs newer than "now" that are dropped into quarantine bucket. Defaults to 2592000 (30 days).

This is a mechanism to prevent main hot buckets from being polluted with fringe events.

quarantinePastSecs

Events with timestamp of quarantinePastSecs older than "now" are dropped into quarantine bucket. Defaults to 77760000 (900 days).

This is a mechanism to prevent the main hot buckets from being polluted with fringe events.

rawChunkSizeBytes

Target uncompressed size in bytes for individual raw slice in the rawdata journal of the index. Defaults to 131072 (128KB). 0 is not a valid value. If 0 is specified, rawChunkSizeBytes is set to the default value.

Note: rawChunkSizeBytes only specifies a target chunk size. The actual chunk size may be slightly larger by an amount proportional to an individual event size.

Warning: This is an advanced parameter. Only change it if instructed to do so by Splunk Support.

rotatePeriodInSecs

Rotation period, in seconds, that specifies how frequently to check:

If a new hot bucket needs to be created.

If there are any cold buckets that should be frozen.

If there are any buckets that need to be moved out hot and cold DBs, due to size constraints.

serviceMetaPeriod

Defines how frequently metadata is synced to disk, in seconds. Defaults to 25 (seconds).

You may want to set this to a higher value if the sum of your metadata file sizes is larger than many tens of megabytes, to avoid the hit on I/O in the indexing fast path.

summarize

If true, leaves out certain index details, which provides a faster response.

suppressBannerList

List of indexes for which we suppress "index missing" warning banner messages.

This is a global setting, not a per index setting.

sync

Specifies the number of events that trigger the indexer to sync events.

This is a global setting, not a per index setting.

syncMeta

When true, a sync operation is called before file descriptor is closed on metadata file updates. This functionality improves integrity of metadata files, especially in regards to operating system crashes/machine failures.

Note: Do not change this parameter without the input of Splunk Support.

thawedPath

An absolute path that contains the thawed (resurrected) databases for the index.

Suggestion for the Splunk Enterprise bucket rebuild process for the size of the time-series (tsidx) file to make.

Caution: This is an advanced parameter. Inappropriate use of this parameter causes splunkd to not start if rebuild is required. Do not set this parameter unless instructed by Splunk Support.

Default value, auto, varies by the amount of physical RAM on the host

less than 2GB RAM = 67108864 (64MB) tsidx

2GB to 8GB RAM = 134217728 (128MB) tsidx

more than 8GB RAM = 268435456 (256MB) tsidx

Values other than "auto" must be 16MB-1GB. Highest legal value (of the numerical part) is 4294967295

You can specify the value using a size suffix: "16777216" or "16MB" are equivalent.

coldToFrozenDir

String

Destination path for the frozen archive. Use as an alternative to a coldToFrozenScript. Splunk Enterprise automatically puts frozen buckets in this directory.

Bucket freezing policy is as follows:

New style buckets (4.2 and on): removes all files but the rawdata

To thaw, run splunk rebuild <bucket dir> on the bucket, then move to the thawed directory

Old style buckets (Pre-4.2): gzip all the .data and .tsidx files

To thaw, gunzip the zipped files and move the bucket into the thawed directory

If both coldToFrozenDir and coldToFrozenScript are specified, coldToFrozenDir takes precedence

coldToFrozenScript

String

Path to the archiving script.

If your script requires a program to run it (for example, python), specify the program followed by the path. The script must be in $SPLUNK_HOME/bin or one of its subdirectories.

Splunk Enterprise ships with an example archiving script in $SPLUNK_HOME/bin called coldToFrozenExample.py. Splunk DOES NOT recommend using this example script directly. It uses a default path, and if modified in place any changes are overwritten on upgrade.

Splunk recommends copying the example script to a new file in bin and modifying it for your system. Most importantly, change the default archive path to an existing directory that fits your needs.

If your new script in bin/ is named myColdToFrozen.py, set this key to the following:

By default, the example script has two possible behaviors when archiving:

For buckets created from version 4.2 and on, it removes all files except for rawdata. To thaw: cd to the frozen bucket and type splunk rebuild ., then copy the bucket to thawed for that index. We recommend using the coldToFrozenDir parameter unless you need to perform a more advanced operation upon freezing buckets.

For older-style buckets, we simply gzip all the .tsidx files. To thaw: cd to the frozen bucket and unzip the tsidx files, then copy the bucket to thawed for that index

When enabled, you do not have to wait until buckets are repaired to start Splunk. However, you might observe a slight performance degratation.

frozenTimePeriodInSecs

Number

188697600

Number of seconds after which indexed data rolls to frozen. Defaults to 188697600 (6 years).

Freezing data means it is removed from the index. If you need to archive your data, refer to coldToFrozenDir and coldToFrozenScript parameter documentation.

maxBloomBackfillBucketAge

Number

30d

Valid values are: Integer[m|s|h|d]

If a warm or cold bucket is older than the specified age, do not create or rebuild its bloomfilter. Specify 0 to never rebuild bloomfilters.

For example, if a bucket is older than specified with maxBloomBackfillBucketAge, and the rebuilding of its bloomfilter started but did not finish, do not rebuild it.

maxConcurrentOptimizes

Number

6

The number of concurrent optimize processes that can run against a hot bucket.

This number should be increased if instructed by Splunk Support. Typically the default value should suffice.

maxDataSize

Number

auto

The maximum size in MB for a hot DB to reach before a roll to warm is triggered. Specifying "auto" or "auto_high_volume" causes Splunk Enterprise to autotune this parameter (recommended).Use "auto_high_volume" for high volume indexes (such as the main index); otherwise, use "auto". A "high volume index" would typically be considered one that gets over 10GB of data per day.

"auto" sets the size to 750MB.

"auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems.

Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable number ranges anywhere from 100 - 50000. Any number outside this range should be approved by Splunk Support before proceeding.

If you specify an invalid number or string, maxDataSize is auto-tuned.

Note: The precise size of your warm buckets may vary from maxDataSize, due to post-processing and timing issues with the rolling policy.

maxHotBuckets

Number

3

Maximum hot buckets that can exist per index. Defaults to 3.

When maxHotBuckets is exceeded, Splunk Enterprise rolls the least recently used (LRU) hot bucket to warm. Both normal hot buckets and quarantined hot buckets count towards this total. This setting operates independently of maxHotIdleSecs, which can also cause hot buckets to roll.

maxHotIdleSecs

Number

0

Maximum life, in seconds, of a hot bucket. Defaults to 0.

If a hot bucket exceeds maxHotIdleSecs, Splunk Enterprise rolls it to warm. This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll. A value of 0 turns off the idle check (equivalent to INFINITE idle time).

Note:I f you set this too small, you can get an explosion of hot/warm buckets in the filesystem. The system sets a lower bound implicitly for this parameter at 3600, but this is an advanced parameter that should be set with care and understanding of the characteristics of your data.

maxMemMB

Number

5

The amount of memory, expressed in MB, to allocate for buffering a single tsidx file into memory before flushing to disk. Defaults to 5. The default is recommended for all environments.

IMPORTANT: Calculate this number carefully. Setting this number incorrectly may have adverse effects on your systems memory and/or splunkd stability/performance.

maxMetaEntries

Number

1000000

Sets the maximum number of unique lines in .data files in a bucket, which may help to reduce memory consumption. If set to 0, this setting is ignored (it is treated as infinite).

If exceeded, a hot bucket is rolled to prevent further increase. If your buckets are rolling due to Strings.data hitting this limit, the culprit may be the punct field in your data. If you do not use punct, it may be best to simply disable this (see props.conf.spec in $SPLUNK_HOME/etc/system/README).

There is a small time delta between when maximum is exceeded and bucket is rolled. This means a bucket may end up with epsilon more lines than specified, but this is not a major concern unless excess is significant.

maxTimeUnreplicatedNoAcks

Number

300

Upper limit, in seconds, on how long an event can sit in raw slice. Applies only if replication is enabled for this index. Otherwise ignored.

If there are any acknowledged events sharing this raw slice, this paramater does not apply. In this case, maxTimeUnreplicatedWithAcks applies.

Highest legal value is 2147483647. To disable this parameter, set to 0.

Note: this is an advanced parameter. Understand the consequences before changing.

maxTimeUnreplicatedWithAcks

Number

60

Upper limit, in seconds, on how long events can sit unacknowledged in a raw slice. Applies only if you have enabled acks on forwarders and have replication enabled (with clustering).

Note: This is an advanced parameter. Make sure you understand the settings on all forwarders before changing this. This number should not exceed ack timeout configured on any forwarder, and should actually be set to at most half of the minimum value of that timeout. You can find this setting in outputs.conf readTimeout setting under the tcpout stanza.

To disable, set to 0, but this is NOT recommended. Highest legal value is 2147483647.

maxTotalDataSizeMB

Number

500000

The maximum size of an index (in MB). If an index grows larger than the maximum size, the oldest data is frozen.

maxWarmDBCount

Number

300

The maximum number of warm buckets. If this number is exceeded, the warm bucket/s with the lowest value for their latest times are moved to cold.

During this period, uncompressed slices are left on disk even after they are compressed. Then splunkd forces a filesystem sync of the compressed journal and removes the accumulated uncompressed files.

If 0 is specified, splunkd forces a filesystem sync after every slice completes compressing. Specifying "disable" disables syncing entirely: uncompressed slices are removed as soon as compression is complete.

Note: Some filesystems are very inefficient at performing sync operations, so only enable this if you are sure it is needed

minStreamGroupQueueSize

Number

2000

Minimum size of the queue that stores events in memory before committing them to a tsidx file.

Caution: Do not set this value, except under advice from Splunk Support.

partialServiceMetaPeriod

Number

0

Related to serviceMetaPeriod. If set, it enables metadata sync every <integer> seconds, but only for records where the sync can be done efficiently in-place, without requiring a full re-write of the metadata file. Records that require full re-write are be sync'ed at serviceMetaPeriod.

partialServiceMetaPeriod specifies, in seconds, how frequently it should sync. Zero means that this feature is turned off and serviceMetaPeriod is the only time when metadata sync happens.

If the value of partialServiceMetaPeriod is greater than serviceMetaPeriod, this setting has no effect.

By default it is turned off (zero).

processTrackerServiceInterval

Number

1

Specifies, in seconds, how often the indexer checks the status of the child OS processes it launched to see if it can launch new processes for queued requests. Defaults to 15.

If set to 0, the indexer checks child process status every second.

Highest legal value is 4294967295.

quarantineFutureSecs

Number

2592000

Events with timestamp of quarantineFutureSecs newer than "now" are dropped into quarantine bucket. Defaults to 2592000 (30 days).

This is a mechanism to prevent main hot buckets from being polluted with fringe events.

quarantinePastSecs

Number

77760000

Events with timestamp of quarantinePastSecs older than "now" are dropped into quarantine bucket. Defaults to 77760000 (900 days).

This is a mechanism to prevent the main hot buckets from being polluted with fringe events.

rawChunkSizeBytes

Number

131072

Target uncompressed size in bytes for individual raw slice in the rawdata journal of the index. Defaults to 131072 (128KB). 0 is not a valid value. If 0 is specified, rawChunkSizeBytes is set to the default value.

Note: rawChunkSizeBytes only specifies a target chunk size. The actual chunk size may be slightly larger by an amount proportional to an individual event size.

WARNING: This is an advanced parameter. Only change it if you are instructed to do so by Splunk Support.

How frequently (in seconds) to check if a new hot bucket needs to be created. Also, how frequently to check if there are any warm/cold buckets that should be rolled/frozen.

serviceMetaPeriod

Number

25

Defines how frequently metadata is synced to disk, in seconds. Defaults to 25 (seconds).

You may want to set this to a higher value if the sum of your metadata file sizes is larger than many tens of megabytes, to avoid the hit on I/O in the indexing fast path.

syncMeta

Boolean

true

When true, a sync operation is called before file descriptor is closed on metadata file updates. This functionality improves integrity of metadata files, especially in regards to operating system crashes/machine failures.

Note: Do not change this parameter without the input of a Splunk Support.

Note: Do not change this parameter without the input of Splunk Support.

tstatsHomePath

String

Location to store datamodel acceleration TSIDX data for this index. Restart splunkd after changing this parameter.

If specified, it must be defined in terms of a volume definition.

Caution: Path must be writable.

Default value: volume:_splunk_summaries/$_index_name/tstats

warmToColdScript

String

Path to a script to run when moving data from warm to cold.

This attribute is supported for backwards compatibility with Splunk Enterprise versions older than 4.0. Contact Splunk support if you need help configuring this setting.

Caution: Migrating data across filesystems is now handled natively by splunkd. If you specify a script here, the script becomes responsible for moving the event data, and Splunk-native data migration are not used.

Response data keys

Name

Description

assureUTF8

Boolean value indicating wheter all data retreived from the index is proper UTF8.

If enabled (set to True), degrades indexing performance

Can only be set globally.

blockSignSize

Controls how many events make up a block for block signatures.

If this is set to 0, block signing is disabled for this index.

A recommended value is 100.

blockSignatureDatabase

The index that stores block signatures of events.

This is a global setting, not a per index setting.

bucketRebuildMemoryHint

Suggestion for the Splunk Enterprise bucket rebuild process for the size of the time-series (tsidx) file to make.

coldPath

Filepath to the cold databases for the index.

coldPath_expanded

Absoute filepath to the cold databases.

coldToFrozenDir

Destination path for the frozen archive. Used as an alternative to a coldToFrozenScript. Splunk Enterprise automatically puts frozen buckets in this directory.

Bucket freezing policy is as follows:

New style buckets (4.2 and on): removes all files but the rawdata

To thaw, run splunk rebuild <bucket dir> on the bucket, then move to the thawed directory

Old style buckets (Pre-4.2): gzip all the .data and .tsidx files

To thaw, gunzip the zipped files and move the bucket into the thawed directory

If both coldToFrozenDir and coldToFrozenScript are specified, coldToFrozenDir takes precedence.

coldToFrozenScript

Path to the archiving script.

See the POST parameter description for details.

compressRawdata

This value is ignored. splunkd process always compresses raw data.

currentDBSizeMB

Total size, in MB, of data stored in the index. The total incudes data in the home, cold and thawed paths.

defaultDatabase

If no index destination information is available in the input data, the index shown here is the destination of such data.

enableOnlineBucketRepair

Indicates whether to run asynchronous "online fsck" bucket repair, which runs in a process concurrently with Splunk.

enableRealtimeSearch

Indicates if this is a real-time search.

This is a global setting, not a per index setting.

frozenTimePeriodInSecs

Number of seconds after which indexed data rolls to frozen. Defaults to 188697600 (6 years).

Freezing data means it is removed from the index. If you need to archive your data, refer to coldToFrozenDir and coldToFrozenScript parameter documentation.

homePath

An absolute path that contains the hot and warm buckets for the index.

homePath_expanded

An absolute filepath to the hot and warm buckets for the index.

indexThreads

Number of threads used for indexing.

This is a global setting, not a per index setting.

isInternal

Indicates if this is an internal index (for example, _internal, _audit).

lastInitTime

Last time the index processor was successfully initialized.

This is a global setting, not a per index setting.

maxBloomBackfillBucketAge

If a bucket (warm or cold) is older than this, Splunk Enterprise does not create (or re-create) its bloom filter.

maxConcurrentOptimizes

The number of concurrent optimize processes that can run against a hot bucket.

This number should be increased if instructed by Splunk Support. Typically the default value should suffice.

maxDataSize

The maximum size in MB for a hot DB to reach before a roll to warm is triggered. Specifying "auto" or "auto_high_volume" causes Splunk Enterprise to autotune this parameter (recommended). Use "auto_high_volume" for high volume indexes (such as the main index); otherwise, use "auto". A "high volume index" is typically one that gets over 10GB of data per day.

"auto" sets the size to 750MB.

"auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems.

Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable number ranges anywhere from 100 - 50000. Any number outside this range should be approved by Splunk Support before proceeding.

If you specify an invalid number or string, maxDataSize is auto-tuned.

Note: The precise size of your warm buckets may vary from maxDataSize, due to post-processing and timing issues with the rolling policy.

maxHotBuckets

Maximum hot buckets that can exist per index. Defaults to 3.

When maxHotBuckets is exceeded, Splunk Enterprise rolls the least recently used (LRU) hot bucket to warm. Both normal hot buckets and quarantined hot buckets count towards this total. This setting operates independently of maxHotIdleSecs, which can also cause hot buckets to roll.

maxHotIdleSecs

Maximum life, in seconds, of a hot bucket. Defaults to 0. A value of 0 turns off the idle check (equivalent to INFINITE idle time).

If a hot bucket exceeds maxHotIdleSecs, Splunk Enterprise rolls it to warm. This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll.

Note: If set too small, you can get an explosion of hot/warm buckets in the filesystem. The system sets a lower bound implicitly for this parameter at 3600, but this is an advanced parameter that should be set with care and understanding of the characteristics of your data.

maxMemMB

The amount of memory, in MB, allocated for indexing.

This is a global setting, not a per index setting.

maxMetaEntries

Sets the maximum number of unique lines in .data files in a bucket, which may help to reduce memory consumption. If set to 0, this setting is ignored (it is treated as infinite).

If exceeded, a hot bucket is rolled to prevent further increase. If your buckets are rolling due to Strings.data hitting this limit, the culprit may be the punct field in your data. If you do not use punct, it may be best to simply disable this (see props.conf.spec in $SPLUNK_HOME/etc/system/README).

There is a small time delta between when maximum is exceeded and bucket is rolled. This means a bucket may end up with epsilon more lines than specified, but this is not a major concern unless excess is significant.

maxTime

UNIX timestamp of the newest event time in the index.

maxTimeUnreplicatedNoAcks

Upper limit, in seconds, on how long an event can sit in raw slice. Applies only if replication is enabled for this index. Otherwise ignored.

If there are any acknowledged events sharing this raw slice, this paramater does not apply. In this case, maxTimeUnreplicatedWithAcks applies.

Highest legal value is 2147483647. To disable this parameter, set to 0.

Note: this is an advanced parameter. Understand the consequences before changing.

maxTimeUnreplicatedWithAcks

Upper limit, in seconds, on how long events can sit unacknowledged in a raw slice. Applies only if you have enabled acks on forwarders and have replication enabled (with clustering).

Note: This is an advanced parameter. Make sure you understand the settings on all forwarders before changing this. This number should not exceed ack timeout configured on any forwarder, and should actually be set to at most half of the minimum value of that timeout. You can find this setting in outputs.conf readTimeout setting under the tcpout stanza.

To disable, set to 0, but this is NOT recommended. Highest legal value is 2147483647.

maxTotalDataSizeMB

The maximum size of an index, in MB.

maxWarmDBCount

The maximum number of warm buckets. If this number is exceeded, the warm bucket/s with the lowest value for their latest times are moved to cold.

memPoolMB

Determines how much memory is given to the indexer memory pool.

This is a global setting, not a per-index setting.

minRawFileSyncSecs

Can be either an integer (or "disable"). Some filesystems are very inefficient at performing sync operations, so only enable this if you are sure it is needed

During this period, uncompressed slices are left on disk even after they are compressed. Then splunkd forces a filesystem sync of the compressed journal and removes the accumulated uncompressed files.

If 0 is specified, splunkd forces a filesystem sync after every slice completes compressing. Specifying "disable" disables syncing entirely: uncompressed slices are removed as soon as compression is complete.

minStreamGroupQueueSize

Minimum size of the queue that stores events in memory before committing them to a tsidx file.

minTime

UNIX timestamp of the oldest event time in the index.

partialServiceMetaPeriod

Related to serviceMetaPeriod. By default it is turned off (zero).

If set, it enables metadata sync every <integer> seconds, but only for records where the sync can be done efficiently in-place, without requiring a full re-write of the metadata file. Records that require full re-write are be sync'ed at serviceMetaPeriod.

partialServiceMetaPeriod specifies, in seconds, how frequently it should sync. Zero means that this feature is turned off and serviceMetaPeriod is the only time when metadata sync happens.

If the value of partialServiceMetaPeriod is greater than serviceMetaPeriod, this setting has no effect.

processTrackerServiceInterval

How often, in seconds, the indexer checks the status of the child OS processes it launched to see if it can launch new processes for queued requests.

quarantineFutureSecs

Events with timestamp of quarantineFutureSecs newer than "now" are dropped into quarantine bucket. Defaults to 2592000 (30 days).

This is a mechanism to prevent main hot buckets from being polluted with fringe events.

quarantinePastSecs

Events with timestamp of quarantinePastSecs older than "now" are dropped into quarantine bucket. Defaults to 77760000 (900 days).

This is a mechanism to prevent the main hot buckets from being polluted with fringe events.

rawChunkSizeBytes

Target uncompressed size in bytes for individual raw slice in the rawdata journal of the index. Defaults to 131072 (128KB). 0 is not a valid value. If 0 is specified, rawChunkSizeBytes is set to the default value.

Note: rawChunkSizeBytes only specifies a target chunk size. The actual chunk size may be slightly larger by an amount proportional to an individual event size.

Warning: This is an advanced parameter. Only change it if instructed to do so by Splunk Support.

If there are any buckets that need to be moved out hot and cold DBs, due to size constraints.

serviceMetaPeriod

Defines how frequently metadata is synced to disk, in seconds. Defaults to 25 (seconds).

You may want to set this to a higher value if the sum of your metadata file sizes is larger than many tens of megabytes, to avoid the hit on I/O in the indexing fast path.

suppressBannerList

List of indexes for which we suppress "index missing" warning banner messages.

This is a global setting, not a per index setting.

sync

Specifies the number of events that trigger the indexer to sync events.

This is a global setting, not a per index setting.

syncMeta

When true, a sync operation is called before file descriptor is closed on metadata file updates. This functionality improves integrity of metadata files, especially in regards to operating system crashes/machine failures.

Note: Do not change this parameter without the input of Splunk Support.

data/indexes-extended

Description

Provide Splunk Enterprise index bucket-level information. There are three bucket super-directories per index:

home

cold

thawed

The default update period is 10 minutes, as defined by the collectionPeriodInSecs attribute in the $SPLUNK_HOME/etc/apps/introspection_generator_addon/default/server.conf file.

Note: At least one observation period must pass after Splunk Enterprise startup for valid endpoint data to be available. The observation period is defined in the $SPLUNK_HOME/etc/system/default/server.conf file:

[introspection:generator:disk_objects]
collectionPeriodInSecs = 600

The default period is 10 seconds, but 10 minutes (600 seconds) on a Universal Forwarder.

data/indexes-extended/{name}

Description

Provide bucket-level information for the specified index. There are three bucket super-directories per index:

home

cold

thawed

The default update period is 10 minutes, as defined by the collectionPeriodInSecs attribute in the $SPLUNK_HOME/etc/apps/introspection_generator_addon/default/server.conf file.

Note: At least one observation period must pass after Splunk Enterprise startup for valid endpoint data to be available. The observation period is defined in the $SPLUNK_HOME/etc/system/default/server.conf file:

[introspection:generator:disk_objects]
collectionPeriodInSecs = 600

The default period is 10 seconds, but 10 minutes (600 seconds) on a Universal Forwarder.

timedOut - Total number of cursors that have timed out since the server process started.

open: - Information about open cursors.

noTimeout - Number of open cursors with option set to prevent timeout after a period of inactivity.

pinned - Number of pinned open cursors.

total - Number of cursors maintained for clients, typically less than zero.

document: Information about document access and modification patterns and data use. Compare these values to opcounters data, which track total number of operations.

deleted - Total number of deleted documents.

inserted - Total number of inserted documents.

returned - Total number of documents returned by queries.

updated - Total number of updated documents.

getLastError: Information about getLastError use.

wtime: getLastError operation counts with a specified write concern that wait for one or more members of a replica set to acknowledge the write operation.

num - getLastError operation counts with a specified write concern that wait for one or more members of a replica set to acknowledge the write operation.

totalMillis - Amount of time spent performing getLastError operations with write concern that wait for one or more members of a replica set to acknowledge the write operation (msec).

wtimeouts - Number of times write concern operations timed out as a result of the wtimeout threshold to getLastError.

operation: Counters for several types of update and query operations handled using special operation types.

fastmod - Number of update operations that neither cause documents to grow nor require updates to the index.

idhack - Number of queries that contain the _key field.

scanAndOrder - Number of queries that return sorted numbers that cannot perform the sort operation using an index.

queryExecutor: Data from the query execution system.

scanned - Number of index items scanned during queries and query-plan evaluation.

scannedObjects - Total number of documents scanned during the query.

record: Data related to record allocation in the on-disk memory files.

moves - Number of times documents move within the on-disk representation of the data set. Documents move as a result of operations that increase the size of the document beyond their allocated record size.

repl: Metrics related to the ordered history of logical writes.

apply: - Information about the application of ordered history of logical writes.

batches: Information on the ordered history of logical writes application process on secondaries members of replica sets.

Method summary

GET server/status method detail

Request parameters

Response data keys

server/status/dispatch-artifacts

https://<host>:<mPort>/services/server/status/dispatch-artifacts

Description

Access search job information.

Note: At least one observation period must pass after Splunk Enterprise startup for valid endpoint data to be available. The observation period is defined in the $SPLUNK_HOME/etc/system/default/server.conf file:

[introspection:generator:disk_objects]
collectionPeriodInSecs = 600

The default period is 10 seconds, but 10 minutes (600 seconds) on a Universal Forwarder.

server/status/fishbucket

https://<host>:<mPort>/services/server/status/fishbucket

Description

Access information about the private BTree database.

Note: At least one observation period must pass after Splunk Enterprise startup for valid endpoint data to be available. The observation period is defined in the $SPLUNK_HOME/etc/system/default/server.conf file:

[introspection:generator:disk_objects]
collectionPeriodInSecs = 600

The default period is 10 seconds, but 10 minutes (600 seconds) on a Universal Forwarder.

server/status/partitions-space

Description

Access disk utilization information for filesystems that have Splunk Enterprise disk objects, such as indexes, volumes, and logs. A filesystem can span multiple physical disk partitions.

Note: At least one observation period must pass after Splunk Enterprise startup for valid endpoint data to be available. The observation period is defined in the $SPLUNK_HOME/etc/system/default/server.conf file:

[introspection:generator:disk_objects]
collectionPeriodInSecs = 600

The default period is 10 seconds, but 10 minutes (600 seconds) on a Universal Forwarder.

server/status/resource-usage

Description

Learn the current levels of resource (CPU, RAM, VM, I/O, file handle) utilization for entire host, and per Splunk-related processes.

Note: At least one observation period must pass after Splunk Enterprise startup for valid endpoint data to be available. The observation period is defined in the $SPLUNK_HOME/etc/system/default/server.conf file:

[introspection:generator:resource_usage]
collectionPeriodInSecs = 600

The default period is 10 seconds, but 10 minutes (600 seconds) on a Universal Forwarder.

GET server/status/resource-usage method detail

Request parameters

Response data keys

server/status/resource-usage/hostwide

https://<host>:<mPort>/services/server/status/resource-usage/hostwide

Description

Access host-level, dynamic CPU utilization and paging information.

Note: At least one observation period must pass after Splunk Enterprise startup for valid endpoint data to be available. The observation period is defined in the $SPLUNK_HOME/etc/system/default/server.conf file:

[introspection:generator:resource_usage]
collectionPeriodInSecs = 600

The default period is 10 seconds, but 10 minutes (600 seconds) on a Universal Forwarder.

GET server/status/resource-usage/hostwide method detail

Request parameters

Response data keys

Percentage of time CPU is idle. Value reported as 100.0 on Windows except for Vista+ and XP/Win2003 English-only OSes.

cpu_system_pct

Percentage of time CPU is running in system mode. Missing from Windows except for Vista+ and XP/Win2003 English-only OSes.

cpu_user_pct

Percentage of time CPU is running in user mode. Missing from Windows except for Vista+ and XP/Win2003 English-only OSes.

forks

Cumulative number of forked processes since OS startup. Not available on Windows.

mem

Total physical memory installed (MB).

mem_used

Total physical memory used (MB). The amount of actual physical memory minus the amount of physical memory currently available. This is the amount of physical memory that can be immediately reused without having to first write its contents to disk.On Unix, mem_used = total_phys_ram - (free_mem + buffer_mem + cached_mem)On Windows, mem_used = (memoryStatus.ullTotalPhys - memoryStatus.ullAvailPhys); See GlobalMemoryStatusEx function

normalized_load_avg_1min

Normalized load average of runnable_process_count across all cores (cumulative_load_avg / number_of_cores). This value is not reliable for a VM guest.

pg_paged_out

Cumulative VM page count paged since OS startup. Not available on Windows.

pg_swapped_out

Cumulative pages swapped out since OS startup. Not available on Windows.

runnable_process_count

Number of process running or in the runnable queue. Value reported as 1 on Windows except for Vista+ and XP/Win2003 English-only OSes.

server/status/resource-usage/splunk-processes

Description

Access operating system resource utilization information.

Note: At least one observation period must pass after Splunk Enterprise startup for valid endpoint data to be available. The observation period is defined in the $SPLUNK_HOME/etc/system/default/server.conf file:

[introspection:generator:resource_usage]
collectionPeriodInSecs = 600

The default period is 10 seconds, but 10 minutes (600 seconds) on a Universal Forwarder.

Current amount of resident physical memory used (MB). (Usually far less deceiving than virtual memory because operating systems can be liberal with virtual memory size but never with resident memory size.)On Windows, mem_used is obtained by reading the WorkingSetSize property returned by the GetProcessMemoryInfo() function (see GetProcessMemoryInfo function and PROCESS_MEMORY_COUNTERS structure).

normalized_pct_cpu

Percentage of CPU usage across all cores. 100% is equivalent to all CPU resources on the machine.

page_faults

Number of major page faults. Extra field.

pct_cpu

Percentage of CPU usage, relative to one core. 100% is equivalent to 1 core.

Status from the OS scheduler. Can be R (runnable or running), W (waiting), stopped, Z (zombie), or O (other). W includes voluntary sleep or blocking on I/O. O means status is knowable but does not fit into one of those categories. Not available on Windows.

Enter your email address, and someone from the documentation team will respond to you:

Send me a copy of this feedback

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »