Install ELK stack: Manage logs with Elasticsearch, Logstash & Kibana

ELK stack is a popular, open source log management platform. It is used as a centralized management for storing , analysing & viewing of logs. Centralized management makes it easier to study the logs & identify issues if any for any number of servers.

Basically ELK stack is a combination of three open source tools,

Elasticsearch is a NoSQL database that is used for storing the logs,

Logstash is a tool that acts as a pipeline that accepts the inputs from various sources i.e. it collects, parses & stores logs for future use,

& lastly we have Kibana which is a web interface that acts as a visualization layer, it is used to search & view the logs that have been indexed by logstash.

Also we will be using Filebeat, it will be installed on all the clients & will send the logs to logstash.

In this tutorial, we will learn to install ELK stack on RHEL/CentOS based machines. So let’s start with pre-requisites.

Pre-requisite

The main dependency for installing the ELK stack is Java. Make sure that you have java 8 installed on the machine that will host ELK stack. Check the installed java version by executing the following command from terminal,

Install ELK stack

We will now start the installation of ELK stack by installing Elasticsearch first. For doing that we will add the official Elasticsearch repository on our server. Create a new repo by the name ‘elasticsearhc.repo’ in the folder ‘/etc/yum.repos.d’,

$ sudo vi /etc/yum.repos.d/elasticsearch.repo

& add the following content to the file,

[elasticsearch]

name=Elasticsearch repository

baseurl=http://packages.elastic.co/elasticsearch/2.x/centos

gpgcheck=1

gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch

enabled=1

Once the repo has been added, next we will add the gpg key for the elasticsearch repo. Execute the following command to install the key,

$ rpm –import https://packages.elastic.co/GPG-KEY-elasticsearch

We now have successfully setup the elasticsearch repo & can now install it using the following command,

$ sudo yum install elasticsearch

Next start the elasticsearch service & enable it for boot with the following commands,

$ systemctl start elasticsearch

$ systemctl enable elasticsearch

Now run the following command from the terminal to check if the elasticsearch is working properly,

$ curl -X GET http://localhost:9200

if your elasticsearch is working properly, you should get the following reply,

Next we will now install Logstash. Like we did with elasticsearch, we will first add the repository for logstash ,

$ sudo vi /etc/yum.repos.d/logstash.repo

[logstash]

name=Logstash

baseurl=http://packages.elasticsearch.org/logstash/2.2/centos

gpgcheck=1

gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch

enabled=1

We don’t need to add the gpg-key for logstash as it uses the same key as elasticsearch. Now install logstash using yum,

$ sudo yum install logstash

Now is the turn to install Kibana on the machine. Start by creating a repo for kibana,

$ sudo vi /etc/yum.repos.d/kibana.repo

[kibana]

name=Kibana repository

baseurl=http://packages.elastic.co/kibana/4.5/centos

gpgcheck=1

gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch

enabled=1

It also uses the same gpg-key as elasticsearch. Now install kibana using yum,

$ sudo yum install kibana

After installation, start service & enable it at boot time

$ systemctl start kibana

$ systemctl enable kibana

Kibana is now installed & working on our system. To check the web-page, open the web browser & go to the URL mentioned below (use the IP address for your ELK host)

http://IP-Address:5601/

We have successfully install ELK stack, we will now configure it so that it can analyse the logs.

Configure ELK stack

First thing after the installation, we need to create an SSL certificate. This certificate will be used for securing communication between logstash & filebeat clients. Before creating a SSL certificate, we will make an entry of our server IP address in openssl.cnf,

$ vi /etc/ssl/openssl.cnf

and look for section with ‘subjectAltName’ & add your server IP to it,