Turning Splunk into a Systems Management Tool

Despite its great power, Splunk is relatively static with regards to the data it processes. You cannot instruct it to simply run a script on all endpoints and index the results. The app HK Systems Management changes that. It turns Splunk into a kind of PsExec on steroids.

Background

Splunk is a fabulous tool for analysing all kinds of data, and there are many different ways of getting data into Splunk. But none of those are even close to being interactive, at least not if you want to collect data from thousands of machines.

Also, being a data analytics tools, Splunk data flow is one way only: from the endpoint to Splunk. There is no way back; at least, normally there is not.

Remote Code Execution With Captured Output

It turns out that morphing Splunk from a tool that collects data on endpoints into a tool that manages endpoints and collects data is not that difficult. Only a few simple configuration changes are required for Splunk to:

Execute arbitrary command jobs on remote systems (this part is similar to PsExec)

Capture each job’s output

Centrally index the output from all jobs

Provide easy access to each job’s results

Display job status and output on a dashboard

How It Works

We can send files to endpoints with Splunk’s Deployment Server. The only requirement is that Splunk’s Universal Forwarder is installed on the endpoints. Since Universal Forwarder typically runs as Local System (or similar, depending on the platform) and Deployment Server is able to push down new or changed apps to all forwarders, we can set up a centrally managed app that executes any scripts it gets as a payload with admin privileges on the endpoints.

With a little logic in the app we can even provide a simple way to configure whether scripts run regularly or only once, e.g. by looking for the suffix -runonce in the script name.

Implementation

The Splunk app HK Systems Management implements these concepts. It should be seen as a proof of concept and is in no way comparable to established systems management products. But it proves the point that systems management is totally doable with Splunk.

HK Systems Management consists of three Splunk apps that serve different purposes:

HK_SystemsManagement_Endpoint: runs on the endpoints you want to manage; this is the main component

HK_SystemsManagement_Indexer: indexer app, creates the index for storing the data

Endpoint App

The endpoint app’s main script hksm.cmd is run by Universal Forwarder as a scripted input every 30 minutes. It does the following:

It runs all scripts in the UserScripts subdirectory

If a user script’s name ends with -runonce successful execution is stored in the system profile’s %LocalAppData% directory and the script is not run again

All output is sent to the index hksm with source hksm

The script’s output is sent to the local splunkd.log and also to Splunk with sourcetype hksm:main

The output of user scripts is sent to Splunk with the name of the user script as sourcetype

Multiline output from user scripts is captured as a single Splunk event

Deployment

Indexer and Search Head

Deploy HK_SystemsManagement to your search head(s) using Splunk’s UI or any other method of your choosing.

The helper apps (TA and SA in Splunk terminology; I prefer to call them indexer and endpoint apps) are located in the install subdirectory of the main app. Locate $SPLUNK_HOME\etc\apps\HK_SystemsManagement\install\HK_SystemsManagement_Indexer and deploy it to your indexer(s).

Endpoints

Locate $SPLUNK_HOME\etc\apps\HK_SystemsManagement\install\HK_SystemsManagement_Endpoint and deploy it to the endpoints via Splunk’s Deployment Server. You can do so with the following configuration:

Edit $SPLUNK_HOME\etc\system\local\serverclass.conf so that it includes the following lines: