Graylog

Carrying on with my Graylog posts, the following will give you an introduction to creating a basic but functional dashboard.

Why do you need a dashboard? Well you can view a number a widgets very quickly which can all be configured with separate search queries. What’s really convenient is that it’s your dashboard, not one that the vendor thinks you will need. Every environment has different requirements. Perhaps you are tracking iSCSI disconnects and want to see a count of the number of iSCSI errors you are getting prior to the event, maybe want to track how many logs vpxa generates, or list the amount of port scans you are getting on your external firewall. It’s really defined by you.

Browse to your Graylog server and login.

Click on Dashboard on the Menu across the top.

Once the Dashboards tab has opened click Create Dashboard.

Give your new Dashboard a name and description.

You now have a brand new empty dashboard. lets get some widgets generated and add them in.

Click on Search in the menu bar and in the search field type something that you’d like to keep an eye on. For this purpose I’ve chosen the vpxa. It’s noisy and will be a good example.

This will return, a histogram and a bunch of messages.

Lets add the histogram to the dashboard. In the histogram pane click on Add to dashboard and select the dashboard you want to add the histogram to.

And lets add one more. In the Fields pane, expand Messages and click Quick Values. Then in the Quick values for message Click Add to dashboard and select your dashboard.

Go through the various fields and widgets and add what you think will be useful.

Back to our dashboard. Click on Dashboard in the Menu across the top and then click on the name of the dashboard you just created.

You’ll now see the widgets you’ve added to your dashboard. You’ll also see three buttons, Update inbackground, fullscreen and unlock/edit. To rearrange your widgets click unlock/edit and move them around as needed. Update in background keeps the widgets live and fullscreen puts the screen into a display mode which could be useful to display on screens around the IT department.

Once you are done move things around click lock to take it out of editing mode.

The above does go through creating a very basic dashboard by once again this demonstrates how useful Graylog really is. If you are looking for log monitoring you will be in a safe place with Graylog. The flexibility and scalability, absolutely compete with, and often exceed, the larger paid for rivals.

As a follow up to my previous post, I’ll go through deploying and configuring the Graylog OVA. It’s really, really easy. if face the whole process should only take about 20 minutes before you have a set-up ready to receive logs.

1 – OVA Deployment.

Log in the vSphere web client using an account that has permission to configure the environment.

Select Home and Hosts and Clusters.

Right click the cliuster you want to deploy Graylog into and select Deploy OVF template.

Select Browse and select the Graylog OVA.

Select Next.

Give you Graylog OVA a name and select a folder for it to go into. Select Next.

Select a Virtual Disk Format. Choose a Storage Policy and a datastore to deploy the OVA into and click Next.NOTE: If this is going into production and you anticipate a large amount of logs to come in then you should set your disk format to be Eager Zero Thick.

Choose a network.

Review your setting and click Finish.

vSphere will go off and deploy your OVA. The above process will take about 5 minutes.

2 – OS network configuration.

The Graylog OVA is based around Ubuntu and is configured with DHCP straight out of the box. If that doesn’t bother you skip this step.

Open a console to the Graylog VM. Login using the username ubuntu and the password is ubuntu.

Delete iface eth inet dhcp and replace with the following (but customising to your network requirements). exit when done (:wq!)

Next we’ll tidy up the hosts file. (sudo vi /etc/hosts).

I’ve chosen to keep my hostname as Graylog so all I needed to do was change 127.0.1.1 to127.0.0.1.

You’ll need to edit resolv.conf.

Set the nameserver entries to match the DNS servers in your environment. One for each DNS server you want to use. In addition set domain and search to match your domain.

Once you’ve done all of that run sudo graylog-ctl reconfigure. This will catch any change you have made that Graylog might rely on.

Its imporant to note here that the graylog-ctl script is quite versatile and allows you to make chages to Graylog, such as change your timezone and admin password, which should be done if you want to push this into a production environment,. Note: If you do make any changes make sure you run sudo graylog-ctl reconfigure.

OK so to be fair the above took me about 10 minutes to do, however if you are not familiar with Linux it’ll take longer but the Ubuntu community is very active and can help.

3 – Input Configuration.

So now we have our Graylog server ready to go, well almost. The amount of inputs that Graylog can receive is quite vast. In addition to the preconfigured inputs you can make your own. We’ll look at configuring the most common. the syslog input for both UDP and TCP.

Browse to your Graylog server and, if it’s running you’ll be greeted with the login prompt.

In the menu bar across the top select System and Inputs.

From the drop down menu under Inputs in Cluster select Syslog TCP and click Launch new input. In the setting box all you need to do is give your new input a name (e.g. Syslog_TCP).

Setup the same for Syslog UDP.

That’s really as difficult as it gets. Now you have the basic features set-up and configured all you need to do is point the infrastructure you want to log at it.

So the previous two posts only really scratch the surface of what is a really powerful tool. Being an opensource project,the code is readily available for anybody to look at. API;s are exposed and documented, dashboards and alerts can be configured, and custom inputs can be setup, to name a few.

Monitoring systems usually tell you when there’s a problem and what the problem is, but logs can tell you about the problem, what happened before, and what happened after. In other words logs provide a critical source of information when anything happens in your environment, from the seemingly mundane (NTP update) to the more terrifying (all paths down).

It’s always a good idea to collect logs is some form or another, being able to look through historical logs or requests from support people allows you to start looking for the cause, or a pattern. Usually a log entry gets sent from a system to a syslog box and gets added to the log file created for that system. Not only is it a good idea to have external logging for your ESXi servers but you should also log the VCSA/PSC and any supporting infrastructure, eg “first hop” switches, storage, etc… It still surprises me that many companies don’t actively do this.

Capturing all of that creates a huge amount of data which can be very labour intensive to sift through to get what you need. Apart from something to capture the logs, you should also look at a log organiser, something like vRealize Log Insight, which I really like and have marked to blog about at a later date, or Splunk. Unfortunately both solutions come with a price tag.

And this is where Graylog comes in. “Graylog is a fully integrated open source log management platform for collecting, indexing, and analyzing both structured and unstructured data from almost any source.”

All graylog deployments will have the same basic features: Graylog server, Web interface, Elasticsearch, MongoDB.

For a first look I would strongly recommend the Graylog appliance that’s distributed as an OVA. The beauty of the OVA is that is can be deployed as an all-in-on solution for smaller deployments, or configured for a single component via the graylog-ctl script for larger workloads. For those of you asking it does have beta support for Docker.

The interface is similar to other loggers, which is not a bad move in my opinion, as its what works best.

Searches are snappy and respond quickly. The query syntax is simple and doesn’t require you to have a degree in programming. Type esx01 and it will return everything with esx01. Type esx01 esx02 and it with return all entries with esx01 or esx02. But place the two in quotations, “esx01 esx02” and it will look for the exact phrase.

Dashboards are highly customisable and very easy to setup the one below was based on vsan for a rolling 5 minute window and took a couple of minutes to setup.

Support is done through the community but can be purchased at three levels, with different SLA’s for response and different ways to contact the company. I’m not sure of the cost but I guess much of that would be around the size of your deployment.

If you don’t want to use the OVA, graylog also has official deb and rpm package repositories for Ubuntu, Centos and Debian, which make it easy to install with two or three commands. I tested both the OVA, and the package install on CentOS. Both methods were really simple to deploy.

Graylog should be a serious consideration for any company, big or small, and is very good example of an enterprise opensource project.