The Goal

Have all syslogs from all servers shipped to a central server so that we can query them in one spot, and review old logs in the event of a compromise using only free software.

After looking at a number of options, I settled on rsyslog for the server, standard syslog on the clients (for now), LogAnalyzer as the UI for the web, both mysql backend for rsyslog and file based backend. The configuration for rsyslog will be a little different then most of the tutorials out there. I wanted to be able to query across all servers at one time, and LogAnalyzer only allows you to configure specific endpoints. With most configuration examples you will find on the web, they show you how to either dump it all into a database (which will no doubt get huge if you don’t clean it up) or dump each server to a single file which rotates daily, which is not ideal for a LogAnalyzer end point because your config has to change daily. This solution will dump all events into a database for all servers which will be configured as one realtime endpoint. rsyslog will also be configured to dump to a file, and this file will be rotated monthly using the usual logrotate scripts. Several archival (non-compressed) files will also be configured in LogAnalyzer for historical purposes.

Install rsyslog

I’m not going to go into this in detail. I’m going to assume you can already do this. But here are some useful notes:

My solution was built on RHEL5 using the packages available from yum. You will need the rsyslog package, as well as apache, php, and mysql for the database. I used link 2 below for this, although I didn’t remove any of the previous syslog packages. I just turned them off. Additionally, you only need to install rsyslog on your central server for this solution, not on each client.

I used this template line at the end of my rsyslog config: $template DailyPerHostLogs,”/var/log/LOGHOSTS/%HOSTNAME%/%HOSTNAME%.log”

Configure DB

I’m going to assume you have already installed the DB, but if not, here is a cheat sheet:

yum install mysql-server

Run this command (or create the database in another way):

mysql -uroot -p < /usr/share/doc/rsyslog-mysql-3.22.1/createDB.sql

Then create a user and password that can access the database from the host the server is on.

Configure rsyslog to log to both

Now this is where the magic happens. Provided you took some part of a sample config from somewhere, you should have something like this is your rsyslog:

I added the above code to the very end of the file. Then I restarted rsyslog.

Then I configured a new entry in my config file for LogAnaylzer. If you haven’t already installed it yet, you can use the install wizard to set this up. But if you have installed it, you will need to use the config.php file to make changes. The sample included in the comments seemed a little off, so this is what I used:

EDIT: My log rotate script didn’t work as expected. Try this instead. Another option could be to reverse the order of the files above, handle .3 first, then .2…etc It seemed to work ok for the first month, then started added extra .# extensions and messed things up a bit.

To purge the database and keep your logs clean, just set up this script to run as a cronjob:

Odds and Ends

If you got this all to work, you may notice that your database based sources show more information than your file based sources. If you want facility and severity to display for text based entries, you will need to setup rsyslog to include this information in the file.

First, I’m using version 3.22.1 of rsyslog from the RHEL5 repos. Which is very old. I used this template to change what is logged to the file: