This got me thinking why am I just now starting to see this alert after running SO for a couple months now? Looking into the events I started noticing how they were all stemming from our kid’s Android tablets. And of course since the alerts were over port 443 I could not see anything in the PCAPs. So I figured the next best thing would be to start looking through Elsa to see if I could find out when this started and if there was any pattern to these events. I proceeded to find out that Elsa had stopped archiving events a LONG time ago (about a month and a half ago). The first place I started to troubleshoot thingswas with the command ‘SOSTAT.’ That showed me a high amount of files sitting in the Elsa buffers section of the report (about 280+).

Googling around trying to find out why Elsa stopped working (actually archiving events) I stumbled across this thread. I started to go through the node.log file looking for errors and found errors such as these

Now since I am running SecurityOnion as a VM on a laptop with a bad battery, there had been some “unexpected” shutdowns. So just on the off chance that I borked the database, I ran the command ‘sudo mysqlcheck -A’. Nothing came back corrupted. YIPPEE! I tried to restart just the syslog_ng service to see if that would kick start things… Nope no go there either.

So going back to the node.log file for Elsa, I noticed the following line:

Total mem used: 770549356 of 3082194944, which is greater than 25 allowed percent

in the log file. Based on the post from above, I tried restarting the syslog_ng service and watching what was being written to the log file. This time I saw it in real-time. It was here that it dawned on me that this was relating back to a line that I saw in the /etc/elsa_node.conf file:

# Check to be sure that temporary indexes (which use a lot of RAM) never go above this amount of RAM, or they will be prematurely consolidated.
“allowed_mem_percent”: 25,

I made a quick change to the file; changing the ’25’ to ’35’ and then restarted the syslog_ng service. Once I did that, the files in the buffer started to go down again and Elsa started processing things again.