Log Management with Splunk

You can pipe system and application logs from a DC/OS cluster to your existing Splunk server. This document describes how to configure a Splunk universal forwarder to send output from each node to a Splunk installation. This document does not explain how to set up and configure a Splunk server.

These instructions are based on CoreOS and might differ substantially from other Linux distributions.

Prerequisites

An existing Splunk installation that can ingest data for indexing.

All DC/OS nodes must be able to connect to your Splunk indexer via HTTP or HTTPS.

The ulimit of open files must be set to unlimited for your user with root access.

Known issue

The agent node Splunk forwarder configuration expects tasks to write logs to stdout and stderr. Some DC/OS services, including Cassandra and Kafka, do not write logs to stdout and stderr. If you want to log these services, you must customize your agent node Splunk forwarder configuration.