Overview

syslog-ng takes incoming log messages from defined 'sources' and forwards them to the appropriate destinations, based on powerful filter directives. In a typical simple set-up, syslog-ng will read messages from three sources:

the default /dev/log device, where most logs are sent

syslog-ng "internal" log messages

/proc/kmsg kernel messages

Sources are defined using the "source" directive. These incoming messages are then filtered according to defined filters ("filter" keyword), i.e. according to originating program or log level, and sent to the appropriate "destination". Destinations include log files (e.g. /var/log/messages.log), printing messages on a console and remote servers. The pivotal function is log. This function defines which filters should be applied to a certain source, and where the resulting messages should be sent to.

Installation

systemd/journald integration

syslog-ng pulls in the messages from the systemd journal by default. Keeping ForwardToSyslog=no in /etc/systemd/journald.conf is recommended in order to avoid the overhead associated with the socket and to avoid needless error messages in the log. If on the other hand you do not want to store your logs twice and turn journald's Storage=none, you will need ForwardToSyslog=yes, as syslog-ng tries to follow the 'journald' journal file.

Sources

You can look at the identifiers and source-drivers in the official manuals.
This will follow the manual to explain the configuration file above. The unix-stream() source-driver opens the given AF_UNIX
socket and starts listening on it for messages.
The internal() source-driver gets messages generated by syslog-ng.

Therefore, the following means: src gets messages from the /dev/log socket and syslog-ng.

source src { unix-stream("/dev/log"); internal(); };

The kernel sends log messages to /proc/kmsg and the file() driver reads log messages from files. Therefore, the following means:
kernsrc gets messages from file /proc/kmsg

source kernsrc { file("/proc/kmsg"); };

In the default configuration file after emerging syslog-ng, the source is defined as:

Reading messages by pipe("/proc/kmsg") gives a better performance but because it opens its argument in read-write mode can be a security
hazard as the syslog-ng admin guide states:

"Pipe is very similar to the file() driver, but there are a few differences, for example pipe() opens its argument in read-write mode, therefore it is not recommended to be used on special files like /proc/kmsg." (You can follow this discussion in this post.)

To open a port to read data from a remote server a source must be defined with this syntax:

source s_net { udp(); };

for UDP or

source s_net { tcp(); };

to receive log messages via TCP. Both listen on port 514.

syslog-ng and systemd journal

Starting with syslog-ng version 3.6.1 the default system() source on Linux systems using systemd uses journald as its standard system() source.

If you wish to use both the journald and syslog-ng files, ensure the following settings are in effect. For systemd-journald, in the /etc/systemd/journald.conf file, Storage= either set to auto or unset (which defaults to auto) and ForwardToSyslog= set to no or unset (defaults to no). For /etc/syslog-ng/syslog-ng.conf, you need the following source stanza:

source src {
system();
internal();
};

If, on the other hand, you wish not to retain the journald logs, but only syslog-ng's text logs, set Storage=volatile and ForwardToSyslog=yes in /etc/systemd/journald.conf. This will store journald in ram. As of syslog-ng 3.6.3, syslog-ng is using journald as the system(); source so if you set Storage=none, the systemd journal will drop all messages and not forward them to syslog-ng.

After the change restart the systemd-journald.service and syslog-ng@default.service daemons.

Destinations

In syslog-ng, log messages are sent to files. The syntax is very similar to sources:

You will be normally logging to a file, but you could log to a different destination-driver: pipe, Unix socket, TCP-UDP ports,
terminals or to specific programs. Therefore, this means sending authlog messages to /var/log/auth.log:

destination authlog { file("/var/log/auth.log"); };

If the user is logged in, usertty() sends messages to the terminal of the specified user. If you want to send console messages
to root's terminal if it is logged in:

destination console { usertty("root"); };

Messages can be sent to a pipe with pipe(). The following sends xconsole messages to the pipe /dev/xconsole.
This needs some more configuration, so you could look at the sub-section xconsole below.

destination xconsole { pipe("/dev/xconsole"); };

To send messages on the network, use udp(). The following will send your log data out to another server.

destination remote_server { udp("10.0.0.2" port(514)); };

Creating Filters for Messages

The syntax for the filter statement is:

filter <identifier> { expression; };

Functions can be used in the expression, such as the function facility() which selects messages based on the facility codes.
The Linux kernel has a few facilities you can use for logging. Each facility has a log-level; where debug is the most verbose,
and panic only shows serious errors. You can find the facilities, log levels and priority names in /usr/include/sys/syslog.h.
To filter those messages coming from authorization, like
May 11 23:42:31 mimosinnet su(pam_unix)[18569]: session opened for user root by (uid=1000), use the following:

filter f_auth { facility(auth); };

The facility expression can use the boolean operators and, or, and not, so the following filter
selects those messages not coming from authorization, network news or mail:

filter f_debug { not facility(auth, authpriv, news, mail); };

The function level() selects messages based on its priority level, so if you want to select informational levels:

filter f_info { level(info); };

Functions and boolean operators can be combined in more complex expressions. The following line filters messages with a priority level from
informational to warning not coming from auth, authpriv, mail and news facilities:

Tips and Tricks

After understanding the logic behind syslog-ng, many possible and complex configuration are possible. Here there are some examples.

Have syslog-ng reload the configuration file

You can make syslog-ng re-evaluate the configuration file. You can do so manually by sending a SIGHUP to the process, or call the reload function with systemctl:

# systemctl reload syslog-ng

Failover Logging to Remote Host

This setup shows how to send the default unencrypted syslog packets across both TCP and UDP protocols, using the standard port (514) and an alternate port. This is sending the same output to the same machine 4 different ways to try and make sure packets make it. Mostly useful if you are debugging a remote server that fails to reboot. The different ports and protocols are to make it past any firewall filters or other network problems. Also useful for port-forwarding and using tunnels. Something like this setup is ideal to tunnel across an ssh connection that the prone-to-failover host initiates through a reverse connection.

Configuring as a loghost

Configuring your system to be a loghost is quite simple. Drop the following into your configuration, and create the needed directory.
With this simple configuration, log filenames will be based on the FQDN of the remote host,
and located in /var/log/remote/. After creating the remote directory, reload your syslog-ng configuration.

The same message from the cron facility will end up in both the cron.log and messages files. To change this behavior we can use the final flag,
ending up further processing with the message. Therefore, in this example, if we want messages from the cron facility not ending up in the
messages file, we should change the cron's log sentence by:

PostgreSQL Destination

This section will use two roles: syslog and logwriter. syslog will be the administrator of the database syslog and logwriter will only be able to add records to the logs table.

No longer needed to create table for logs. syslog-ng will create automatically.

psql -U postgres

postgres=# CREATE ROLE syslog WITH LOGIN;
postgres=# \password syslog # Using the \password function is secure because
postgres=# CREATE ROLE logwriter WITH LOGIN;
postgres=# \password logwriter # the password is not saved in history.
postgres=# CREATE DATABASE syslog OWNER syslog;
postgres=# \q # You are done here for the moment

Edit pg_hba.conf to allow syslog and logwriter to establish a connection to PostgreSQL.

#logger Now THAT is a timestamp!
#tail -n 2 /var/log/messages.log
Feb 18 14:25:01 hostname logger: These timestamps are not optimal.
2010-02-18T20:23:58-05:00 electron logger: Now THAT is a timestamp!
#

RFC 3339 timestamps

Same as above, except use rfc3339 instead of iso for ts_format

Log Levels

Log levels are defined separately for each logged facility in syslog-ng config. Available log levels are listed in /usr/include/sys/syslog.h :