Caution: While processing indexes, the indexer might occasionally exceed the configured maximums for short periods of time. When setting limits, be sure to factor in some buffer space. Also, note that certain systems, such as most Unix systems, maintain a configurable reserve space on their partitions. You must take that reserve space, if any, into account when determining how large your indexes can grow.

Configure index size for each index

To set the maximum index size on a per-index basis, use the maxTotalDataSizeMB attribute. When this limit is reached, buckets begin rolling to frozen.

Configure index size according to bucket type

To set the maximum size for homePath (hot/warm bucket storage) or coldPath (cold bucket storage), use the maxDataSizeMB attributes:

The maxDataSizeMB attributes can be set globally or for each index. An index-level setting will override a global setting. To control bucket storage across groups of indexes, use the maxVolumeDataSizeMB attribute, described below.

Configure index size with volumes

You can manage disk usage across multiple indexes by creating volumes and specifying maximum data size for them. A volume represents a directory on the file system where indexed data resides.

Volumes can store data from multiple indexes. You would typically use separate volumes for hot/warm and cold buckets. For instance, you can set up one volume to contain the hot/warm buckets for all your indexes, and another volume to contain the cold buckets.

You can use volumes to define homePath and coldPath. You cannot use them to define thawedPath.

In addition, you must use volumes if you explicitly define bloomHomePath. For information on bloomHomePath, see the topic "Configure bloom filters" in this manual.

Configure a volume

To set up a volume, use this syntax:

[volume:<volume_name>]
path = <pathname_for_volume>

You can also optionally include a maxVolumeDataSizeMB attribute, which specfies the maximum size for the volume.

For example:

[volume:hot1]
path = /mnt/fast_disk
maxVolumeDataSizeMB = 100000

The example defines a volume called "hot1", located at /mnt/fast_disk, with a maximum size of 100,000MB.

Similarly, this stanza defines a volume called "cold1" that uses a maximum of 150,000MB:

[volume:cold1]
path = /mnt/big_disk
maxVolumeDataSizeMB = 150000

Use a volume

Once you configure volumes, you can use them to define an index's homePath and coldPath. For example, using the volumes configured above, you can define two indexes:

You can use volumes to manage index storage space in any way that makes sense to you. Usually, however, volumes correlate to hot/warm and cold buckets, because of the different storage requirements typical when dealing with different bucket types. So, you will probably use some volumes exclusively for designating homePath (hot/warm buckets) and others for coldPath (cold buckets).

When a volume containing warm buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to cold. When a volume containing cold buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to frozen. If a volume contains both warm and cold buckets (which will happen if an index's homePath and coldPath are both set to the same volume), the oldest bucket will be rolled to frozen.

Put it all together

This example shows how to use the per-index homePath.maxDataSizeMB and coldPath.maxDataSizeMB attributes in combination with volumes to maintain fine-grained control over index storage. In particular, it shows how to use those attributes to prevent bursts of data into one index from triggering massive bucket moves from other indexes. You can use these per-index settings to ensure that no index will ever occupy more than a specified size, thereby alleviating that concern.

Enter your email address, and someone from the documentation team will respond to you:

Send me a copy of this feedback

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »