Welcome to Splunk Answers, a Q&A forum for users to find answers to questions about deploying, managing, and using Splunk products. Contributors of all backgrounds and levels of expertise come here to find solutions to their issues, and to help other users in the Splunk community with their own questions.

This quick tutorial will help you get started with key features to help you find the answers you need. You will receive 10 karma points upon successful completion!

Refine your search:

ANNOUNCEMENT: Answers is being migrated to a brand new platform! answers.splunk.com will be read-only from 5:00pm PDT June 4th - 9:00am PDT June 9th. Please read this Answers thread for all details about the migration.

Welcome to Splunk Answers! Not what you were looking for? Refine your search.

How to get a report on latency between Heavy Forwarder and Indexer?

In our environment, we have syslog servers that send data to regional Heavy forwarders. The data in HFs eventually gets indexed and is searchable on Search Heads.

The issue now is, we are able to see data(logs) on HFs. But we are not able to see them on Search Heads.

Eg : The last log present on HF for a particular host is on 30th May. But the last log we can see on our Search Head for the same host will be of 27th or 28th May's. We will be able to see 30th logs, somewhere around June 1st or 2nd.

It is obvious there is some latency between HF and Indexer. It is mostly because of the bandwidth issues (confirmed).

But I would like to get a report from Splunk that gives us the time difference between the moment a log got into HF and the moment it got indexed. Is there any SPL for getting this report?

People who like this

2 Answers

Assuming that there is little-to-no latency in the arrival of the event at the HF (e.g. the timestamp in the event is very close to the time that it arrives at the HV), then you can chart _indextime - _time. So you can do something like this:

Here we are going with an assumption that there is little or no latency in the arrival of event at HF. Is there a way we can get that latency too??

So in a picture format it will be..

Endpoint (event generated) Time T1, Heavy Forwarder (the same event reached HF) Time T2, Indexer (when that same event was indexed) Time T3.

So what we need is T2 – T1 = time taken to reach HFT3 – T2 = time taken to get the event indexed T3 – T1 = total time taken for the event to be usable.

When we get the above information for each endpoint (only sample) we will be able to get to the bottom of the problem.

Then we have to go and dig deeper to find out if where the problem is:1. HF is retransmitting or2. indexer queues are full or3. we are running out CPU or 4. we are wasting time on reading and writing from the disks on the HF

At the very bottom - the key is to reduce cardinality of _time and only look for the worst case per bucket so to quickly get a general overview of your indexing delay, consider something tstatsy like this:

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website. Learn more (including how to update your settings) here. Closing this box indicates that you accept our Cookie Policy.