Friday, 28 December 2018

When you have a busy Splunk environment with multiple apps, ES and custom correlation searches you need to make sure to optimize your configuration to best use your kit. Scheduling your searches and prioritizing them appropriately is usually step 1.

When you create a correlation search its important to configure the following parameters:

Cron Schedule

You can randomize the run times yourself here 2,22,42 * * * * (for a 20m search)

Scheduling

Continuous is less intensive than Real-time

Schedule Window

auto is my preferred option here

Schedule Priority

Usually preferred option is Higher (which makes it fifth overall in the priority order which you can see here)

The reason field in the majority of those searches showed:
"The maximum number of concurrent auto-summarization searches on this instance has been reached" or "The maximum number of concurrent historical scheduled searches on this instance has been reached"

The above will produce the following, which highlights that the Acceleration searches performed in the background for the highlighted data models are very inefficient. The searches in question have been successful 3373 times in 24h but have been skipped 1086913 times in the same time.

To fix the issue there are two options:

Disable acceleration to the data models that you are not using (keep in mind that dashboards based on those data models will stop working!)

Restrict data models to particular indexes.

Under each data model's configuration a macro is used to identify the indexes to be queried for that data model's relevant data.

In order to identify the relevant indexes for this case you can run the following query for the past 7 days (in fast mode) or longer if you want to be 100% certain you have all the data:

tag=change | stats values(index) as index

The result will be a small list of indexes you can add to the cim_Change_Analysis_indexes macro found under Settings -> Advanced search -> Search marcos (search with app context all and owner any to be sure). The result should look something like this:

When you have completed the above process for all the data models you will eventually see the following result in skipped searches: