Tuesday, 23 August 2016

Setup a simple ELK on AWS in under 10 minutes!

So here I am... back at it again after a good long break! For this tutorial, Im going to show you how easy and simple it is to setup a basic working ELK on AWS!

ELK? Not heard of it? well if you haven't heard of it or tried it out yet, I suggest you do! ELK is actually an acronym for ElasticSearch, Logstash and Kibana.. fancy names but trust me.. together they form a really powerful log analysis tool.

ElasticSearch: Built on top of Apache Lucene, ElasticSearch is the work engine behind ELK that performs real-time data extractions and analysis on structured as well as unstructured data. To know more read HERE

Logstash: Logstash is a tool that can ingest logs, process them and forward them to another system such as ElasticSearch. Logstash comes with a huge supply of inputs, filters, codecs and outputs that can be used to consume virtually any type of logs from web servers, to syslogs, to error lgos etc. To know more read HERE

Kibana: Kibana is a visualization tool that can be used to visualize trends, patterns, read and interpret your log data. To know more read HERE

In this tutorial, I'll be walking you simple steps to get started with a single node ELK setup on AWS infrastructure.

Source: https://www.cartoonstock.com/cartoonview.asp?catref=jhen301

To get started, you will need an AWS account. You can signup for AWS and get one year of services for free* T&C apply of course!

Finally, copy Kibana's service file and enable it to start on boot using the following command:

# sudo mv kibana4 /etc/init.d/

# sudo chmod +x /etc/init.d/kibana4

# sudo update-rc.d kibana4 defaults 96 9

# sudo service kibana4 start

Kibana will start on port 5601. To test the install, point your ELK instance's public IP address and port 5601 to a browser. You should see Kibana's initial configuration screen as shown below:

Here, you need to configure the default index pattern for ElasticSearch to run indexes and analysis against. Select the "Index contains time-based events" checkbox as shown. Also, type in "logstash-*" in the Index name or pattern field and set the Time-field name value as "@timestamp"

Click the "Create" button to complete the setup.

You should see logs from your ELK instance popping up in Kibana's "Discover" tab as shown below:

There you have it! a simple ELK on a single EC2 instance!

Coming up next, Configuring ELK clients and how to setup ELK on a production scale as well, so stay tuned for much more!!

2 comments
:

Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts, have a nice weekend!

This article is very great, its content is the most unique and its website design is also very good. The speed of the website is also fast. I very much liked to share the blog very much thanks.henrystickmangames.comFriv.Pro