Looking for an Elastic Stack ("ELK" tutorial) that shows how to set up the Elastic Stack? In
this tutorial, you learn how to get up and running quickly. First you install
the core open source products:

Then you learn how to implement a system monitoring solution that uses
Metricbeat to collect server metrics and ship the data to Elasticsearch, where you can
search and visualize the data by using Kibana. After you get the basic setup
working, you add Logstash for additional parsing.

To get started, you can install the Elastic Stack on a single VM or even on your
laptop.

Implementing security is a critical step in setting up the Elastic Stack.
To get up and running quickly with a sample installation, you skip those steps
right now. Before sending sensitive data across the network, make sure you
secure the Elastic Stack and enable
encrypted communications.

Elasticsearch is a real-time,
distributed storage, search, and analytics engine. It can be used for many
purposes, but one context where it excels is indexing streams of semi-structured
data, such as logs or decoded network packets.

Kibana is an open source analytics and
visualization platform designed to work with Elasticsearch. You use Kibana to search,
view, and interact with data stored in Elasticsearch indices. You can easily perform
advanced data analysis and visualize your data in a variety of charts, tables,
and maps.

We recommend that you install Kibana on the same server as Elasticsearch,
but it is not required. If you install the products on different servers, you’ll
need to change the URL (IP:PORT) of the Elasticsearch server in the Kibana configuration
file, kibana.yml, before starting Kibana.

To download and install Kibana, open a terminal window and use the commands that
work with your system:

The Beats are open source data shippers that you install as agents on your
servers to send operational data to Elasticsearch. Beats can send data directly to Elasticsearch
or via Logstash, where you can further process and enhance the data.

Each Beat is a separately installable product. In this tutorial, you learn how
to install and run Metricbeat with the system module enabled to collect system
metrics.

To learn more about installing and configuring other Beats, see the Getting
Started documentation:

If script execution is disabled on your system, set the execution policy
for the current session to allow the script to run. For example: PowerShell.exe
-ExecutionPolicy UnRestricted -File .\install-service-metricbeat.ps1.

Metricbeat provides pre-built modules that you can use to rapidly implement
and deploy a system monitoring solution, complete with sample dashboards and
data visualizations, in about 5 minutes.

In this section, you learn how to run the system module to collect metrics
from the operating system and services running on your server. The system module
collects system-level metrics, such as CPU usage, memory, file system, disk IO,
and network IO statistics, as well as top-like statistics for every process
running on your system.

Before you begin: Verify that Elasticsearch and Kibana are running and that Elasticsearch is
ready to receive data from Metricbeat.

If you don’t see data in Kibana, try changing the date range to a larger
range. By default, Kibana shows the last 15 minutes. If you see errors, make
sure Metricbeat is running, then refresh the page.

Click Host Overview to see detailed metrics about the selected host.

Now that you know how to set up the simplest architecture for the Elastic Stack—one
or more Beats sending event data directly to an Elasticsearch instance running on the
same server—let’s add Logstash.

Extract the contents of the zip file to a directory on your computer, for
example, C:\Program Files. Use a short path (fewer than 30 characters) to
avoid running into file path length limitations on Windows.

Logstash provides input plugins for reading from
a variety of inputs. In this tutorial, you create a Logstash pipeline configuration
that listens for Beats input and sends the received events to the Elasticsearch output.

To configure Logstash:

Create a new Logstash pipeline configuration file called demo-metrics-pipeline.conf.
If you installed Logstash as a deb or rpm package, create the file in the Logstash
config directory. The file must contain:

An input stage that configures Logstash to listen on port 5044 for incoming Beats
connections.

An output stage that indexes events into Elasticsearch. The output stage also
configures Logstash to write to the Metricbeat index.

When you start Logstash with this pipeline configuration, Beats events are routed
through Logstash, where you have full access to Logstash capabilities for collecting,
enriching, and transforming data.

Metricbeat sends events to Elasticsearch by default. To send events to Logstash, modify the
Metricbeat configuration file, metricbeat.yml. You’ll find this file under
the Metricbeat install directory, or /etc/metricbeat for rpm and deb.

Disable the output.elasticsearch section by commenting it out, then enable
the output.logstash section by uncommenting it:

Save the file, then restart Metricbeat to apply the configuration changes.

Logstash reads from the Beats input and indexes events into Elasticsearch. You haven’t
defined a filter section yet, so Logstash simply forwards events to Elasticsearch without
additional processing. Next, you learn how to define the filter stage.

Rather than sending the whole command-line argument to Elasticsearch, you might want to
send just the command’s path. One way to do that is by using a Grok filter.
Learning Grok is beyond the scope of this tutorial, but if you want to learn
more, see the Grok filter plugin
documentation.

To extract the path, add the following Grok filter between the input and output
sections in the Logstash config file that you created earlier:

Congratulations! You’ve successfully set up the Elastic Stack. You learned how to
stream system metrics to Elasticsearch and visualize the data in Kibana. You also learned
how to use Logstash to filter events collected by Metricbeat.

Next, you’ll want to set up the Elastic Stack security features and activate your
trial license so you can unlock the full capabilities of the Elastic Stack. To learn
how, read: