Install a Centralized Log Server with Rsyslog in Debian 9

On this page

In Linux, the log files are files that contain messages about the system functions which are used by system administrators to identify eventual issues on the machines. The logs help administrators to visualize the events that happened in the system over time periods. Usually, all log files are kept under /var/log directory in Linux. In this location, there are several types of log files for storing various messages, such as a log file for recording system events, a log file for security related messages, other log files dedicated for kernel, users or cron jobs. The main purpose of log files is to troubleshoot system problems. Most log files in Linux are controlled by rsyslogd service. On newer releases of Linux distributions, the log files are also controlled and managed by the journald system service, which is a part of systemd initialization program. The logs stored by journal daemon are written in a binary format and are mainly volatile, stored in RAM and in a ring-buffer in /run/log/journal/. However, the journal service can be also configured to permanently store the Syslog messages.

In Linux, the rsyslog server can be configured to run a central log manager, in a service-client model fashion, and send log messages over the network via TCP or UDP transport protocols or receive logs from network devices, servers, routers, switches or other systems or embedded devices that generate logs.

Rsyslog daemon can be setup to run as a client and server at the same time. Configured to run as a server, Rsyslog will listen on default port 514 TCP and UDP and will start to collect log messages that are sent over the network by remote systems. As client, Rsyslog will send over the network the internal log messages to a remote Ryslog server via the same TCP or UDP ports.

Rsyslog will filter syslog messages according to selected properties and actions. The rsyslog filters are as follows:

Facility or Priority filers

Property-based filters

Expression-based filters

The facility filter is represented by the Linux internal subsystem that produces the logs. They are categorized as presented below:

The priority or severity levels are assigned to a keyword and a number as described below.

emerg = Emergency – 0

alert = Alerts – 1

err = Errors – 3

warn = Warnings – 4

notice = Notification – 5

info = Information – 6

debug = Debugging – 7 highest level

There are also some special Rsyslog keywords available such as the asterisk (*) sign to define allfacilities or priorities, the none keyword which specify no priorities, the equal sign (=) which selects only that priority and the exclamation sign (!) which negates a priority.

The action part of the syslog is represented by the destination statement. The destination of a log message can be a file stored in the file system, a file in /var/log/ system path, another local process input via a named pipe or FIFO. The log messages can be also directed to users, discarded to a black hole (/dev/null) or sent to stdout or to a remote syslog server via TCP/UDP protocol. The log messages can be also stored in a database, such as MySQL or PostgreSQL.

Configure Rsyslog as a Server

The Rsyslog daemon is automatically installed in most Linux distributions. However, if Rsyslog is not installed on your system you can issue one of the below commands in order to install the service> you will require root privileges to run the commands.

In Debian based distros:

sudo apt-get install rsyslog

In RHEL based distros like CentOS:

sudo yum install rsyslog

In order to verify if Rsyslog daemon is started on a system execute the below commands, depending on your distribution version.

On newer Linux distros with systemd:

systemctl status rsyslog.service

On older Linux versions with init:

service rsyslog status

/etc/init.d/rsyslog status

In order to start the rsyslog daemon issue the following command.

On older Linux versions with init:

service rsyslog start

/etc/init.d/rsyslog start

On latest Linux distros:

systemctl start rsyslog.service

To setup an rsyslog program to run in server mode, edit the main configuration file in /etc/rsyslog.conf. In this file make the following changes as shown in the below sample.

sudo vi /etc/rsyslog.conf

Locate and uncomment by removing the hashtag (#) the following lines in order to allow UDP log message reception on 514 port. By default, the UDP port is used by syslog to send-receive messages.

$ModLoad imudp $UDPServerRun 514

Because the UDP protocol is not reliable to exchange data over a network, you can setup Rsyslog to output log messages to a remote server via TCP protocol. To enable TCP reception protocol, open /etc/rsyslog.conf file and uncomment the following lines as shown below. This will allow the rsyslog daemon to bind and listen on a TCP socket on port 514.

$ModLoad imtcp $InputTCPServerRun 514

Both protocols can be enabled in rsyslog to run at the same time.

If you want to specify to which senders you permit access to rsyslog daemon, add the following line after the enabled protocol lines:

$AllowedSender TCP, 127.0.0.1, 10.110.50.0/24, *.yourdomain.com

You will also need to create a new template that will be parsed by rsyslog daemon before receiving the incoming logs. The template should instruct the local Rsyslog server where to store the incoming log messages. Define the template right after the $AllowedSender line as shown in the below sample.

To log only the messages generated by kern facility use the below syntax.

kern.* ?Incoming-logs

The received logs are parsed by the above template and will be stored in the local file system in /var/log/ directory, in files named after the client hostname client facility that produced the messages: %HOSTNAME% and %PROGRAMNAME% variables.

The below & ~ redirect rule configures the Rsyslog daemon to save the incoming log messages only to the above files specified by the variables names. Otherwise, the received logs will be further processed and also stored in the content of local logs, such as /var/log/syslog file.

To add a rule to discard all related log messages to mail, you can use the following statement.

In order to apply any changes made to rsyslog configuration file, you must restart the daemon to load the new configuration.

sudo service rsyslog restart

sudo systemctl restart rsyslog

To check which rsyslog sockets in listening state are opened on a Debian Linux system, you can execute the netstat command with root privileges. Pass the results via a filter utility, such as grep.

sudo netstat –tulpn | grep rsyslog

Be aware that you must also open Rsyslog ports in firewall in order to allow incoming connections to be established.

In RHEL based distros with Firewalld activated issue the below commands:

firewall-cmd --permanent --add-port=514/tcp

firewall-cmd --permanent --add-port=514/tcp

firewall-cmd –reload

In Debian based distros with UFW firewall active issue the below commands:

ufw allow 514/tcp

ufw allow 514/udp

Iptables firewall rules:

iptables -A INPUT -p tcp -m tcp --dport 514 -j ACCEPT

iptables -A INPUT -p udp --dport 514 -j ACCEPT

Configure Rsyslog as a Client

To enable rsyslog daemon to run in client mode and output local log messages to a remote Rsyslog server, edit /etc/rsyslog.conf file and add one of the following lines:

*. * @IP_REMOTE_RSYSLOG_SERVER:514

*. * @FQDN_RSYSLOG_SERVER:514

This line enables the Rsyslog service to output all internal logs to a distant Rsyslog server on UDP port 514.

To send the logs over TCP protocol use the following template:

*. * @@IP_reomte_syslog_server:514

To output only cron related logs with all priorities to a rsyslog server, use the below template:

cron.* @ IP_reomte_syslog_server:514

In cases when the Rsyslog server is not reachable via network, append the below lines to /etc/rsyslog.conf file on the client side in order temporarily store the logs in a disk buffered file, until the server comes online.