Expected performance and known limitations of real-time searches and reports

Real-time search matches events that have arrived at the port but have not been persisted to disk. The rate of arrival of events and number of matches can determine how much memory is consumed and the impact on the indexing rate.

Indexing throughput

Splunk software performance is expected to be acceptable as long as the indexers are not currently heavily loaded and do not have more than a few concurrent real-time searches.

Real-time searches will have a significant impact on performance in high volume environments and network load when you have many concurrent real-time searches.

When planning your real-time searches, you should consider how it will affect the performance of both:

The search head that must process the aggregated stream of live events.

The more work that is done on the search peer, the less that is required on the search head, and vice versa. The search peer is important to the overall system function, so you do not want to burden it with too much filtering of live events. However, if the search peer does not filter at all, the processing power and bandwidth required to send all the live events to the search head may prove costly, especially when you have multiple real-time searches running concurrently.

In cases where the search head cannot keep up with the search peer, the queue on the index processor will cease to flag events for the search. However, the events will have a sequence number that you can use to tell when and how many events were omitted from search consideration.

Concurrent real-time and historical searches

You can run real-time and historical searches concurrently, within the limits of your hardware. There are no restrictions on separate searches for the same or different users.

Indexed real-time searches

The number of concurrent real-time searches can greatly affect indexing performance. To lessen the impact on the indexer, you can enable indexed real-time search. This will basically run the search like a historical search, but will also continually update it with new events as they appear on disk.

Real-time search windows

Windowed real-time searches are more expensive than non-windowed. The operations required to manage and preview the window contents can result in a windowed real time search not keeping up with a high rate of indexing. If your windowed search does not display the expected number of events, try a non-windowed search. If you are interested only in event counts, try using "timechart count" in your search.

Comments

Parkyongsu
Thank you for the followup question. None of the queues mentioned in the wiki pages that you posted in your comment are related to the realtime queue mentioned in the documentation. I sent you an email with the details.

Lstewart splunk, Splunker

November 18, 2016

Lstewart splunk, Splunker

Thank you very much for your document update.
I have one more question, please.
Could you teach me what is actually "the queue on the index processor"?
For example, what stage/queue on <http://wiki.splunk.com/Community:HowIndexingWorks> and <https://wiki.splunk.com/Community:HowDistSearchWorks>?

Parkyongsu

November 7, 2016

Tkomatsubara and Parkyonsu

The choice of wording is unfortunate. By "drop events" we do not mean the events are removed. What we mean is that some events will not be considered for the search. I have corrected the wording above to this:
In cases where the search head can't keep up with the search peer, the queue on the index processor will cease to flag events for the search. However, the events will have a sequence number that you can use to tell when and how many events were omitted from search consideration.

Lstewart splunk, Splunker

October 31, 2016

Could you answer or update documents to the below questions from "Tkomatsubara splunk, Splunker"?

Parkyongsu

October 30, 2016

>In cases where the search head can't keep up with the search peer, the queue on the index processor will drop events.

Drop events? What does this mean? Which stage/queue are those events dropped?
Can you make it clear since this is very important.

>Indexed real-time searches

If this "Indexed real-time searches" used, can we avoid this data "drop" issue?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »