Transferring Conserver Logs to Elasticsearch

If your organization manages Linux, AIX, HP-UX or Solaris servers
in-house, chances are your system administrators at least
occasionally need low-level access to those devices. Typically,
administrators use some kind of serial console—for example, traditional
serial port, Serial-over-LAN or Intelligent Platform Management Interface
(IPMI). Managing and auditing console access is not trivial, so
many organizations rely on the Conserver application to create session logs
when accessing these servers via the serial console. These logs can be
useful for various reasons—for example, maintenance or troubleshooting (to
review why something crashed), security (to find out who did what—connecting user names to actual users) or compliance (to provide detailed
session logs).

This article covers the following:

How to parse and process serial console logs using syslog-ng Open Source
Edition (Balabit).

How to send the logs to Elasticsearch (Elastic), so you get a complete, searchable
audit trail of the console access.

How to integrate the console logs into a real-time monitoring system using
Riemann.

Conserver

Conserver is a wonderful piece of software that lets you manage your
infrastructure's serial
consoles, whether they be old-style hardware serial
ports or state-of-the-art Serial-over-LAN (SOL) baseboards. Its distributed
design permits a decentralized user experience using a secure,
TLS-encrypted protocol. The straightforward workflow consists of the user
connecting to any conserver master node, which then forwards the traffic to
the node that manages the console you want to access. As all masters share
the same configuration file, it is very straightforward to
redistribute consoles among servers automatically (provided they are virtual SOL devices,
like IPMI) using configuration management (for example, using the Puppet
module we developed at CC-IN2P3).

So where is the catch? As far as we are concerned at the Computing Centre
of the National Institute of Nuclear Physics and Particle Physics
(CC-IN2P3), the logging mechanism could be greatly improved, because
conserver's design is quite ancient now. For example, it does not support
logging to syslog. From a user perspective, logging is awesome, as you can
use a keystroke to access the logs of the console, and the console logs
contain the complete session. From an architecture perspective though,
things are not so great, as every conserver master stores the logs of the
consoles it manages locally in a file.

syslog-ng

This is where syslog-ng Open Source Edition comes into play. The idea is to
transport the logfiles of the conserver masters to our favorite event store
back ends, which are Riemann and Elasticsearch. They provide powerful
real-time stream processing and long-term indexed storage capabilities,
respectively. In addition, with syslog-ng, you simply can send the logs to
Riemann and Elasticsearch directly; there is no need for any additional
agents (like Logstash). To see how this system works before going into
configuration details, watch this video.

The video shows what the user does in the console (in the
top-right section of the screen), its effect on the real-time Riemann-dash
dashboard (bottom-right) and the near-real-time Elasticsearch front end
(Kibana|Elastic, on the left).

As you can see, the user activities and events of the session are
transported to the back ends, including useful metadata like
conserver.is_attached: true. This tells you whether
or not someone
was attached to the console (which is obviously the case in this
example).

Requirements

To create the system shown in the demo video, you need the following
software:

syslog-ng Open Source Edition 3.7.2 or newer.

conserver (tested with 8.2.1).

Riemann (tested with 0.2.10).

Elasticsearch (tested with 1.6.0).

Note that this article does not cover how to install, configure (in
general) and get the above software working. You can find plenty of related
tutorials on-line. If you need help with these tasks, check the documentation,
mailing lists or on-line forums for the software you need help with. The
following sections of this article explain how to configure the components of
this infrastructure for the specific needs of our scenario as an example.

Configuring Conserver

The conserver configuration in our setup is nothing special. It creates a
unified logfile, which will serve as the glue between conserver and
syslog-ng. You can activate the unified logfile using either of the
following methods:

1) Run the conserver executable with the -U
/var/log/console.log flag.

2) Use the following configuration block in conserver.cf:

<config * { unifiedlog * /var/log/console.log; }

You also can set the server's general logfile (where conserver stores
the global messages that are unrelated to individual consoles)—for
example, to /var/log/conserver.log.

Both /var/log/conserver.log and
/var/log/console.log will be inputs for syslog-ng.
You might want to take special care of the log rotation of these files. As
you are sending them to Elasticsearch, there is no need to keep them for
too long.