Troubleshooting Splunk

Have been fiddling around with Splunk lately. Splunk’s a really good tool to use for log collection and analysis (and that’s oversimplifying it, I believe it can even do event correlation…), which really made my love for data mining go crazy of late:P Best part is that it has a perpetual free license, nice!

One of the things I encountered when using Splunk was that it didn’t seem to be indexing all the log files that it was set to monitor. After some reading up and experimenting the reason became clear: Splunk will not work properly if you set it to monitor too many files.

How many is too many? For example, setting it to monitor a logfile directory which only has one active log and 100+++ rotated logs, is too many. What should be done instead is to set it to monitor the active logfile only, and use oneshot adding of the other logfiles to the index you want.

Gonna do some more sharing/writeups about this crazily great tool. There’s really a lot that this thing can do man.