What is Syslog: Daemons, Message Formats and Protocols

Pretty much everyone’s heard about syslog: with its roots in the 80s, it’s still used for a lot of the logging done today. Mostly because of its long history, syslog is quite a vague concept, referring to many things. Which is why you’ve probably heard:

Check syslog, maybe it says something about the problem – referring to /var/log/messages.

In this post, we’ll explain the different facets by being specific: instead of saying “syslog”, you’ll read about syslog daemons, about syslog message formats and about syslog protocols.

Note the plurals: there are multiple options for each. We’ll show the important ones here to shed some light over the vague (and surprisingly rich) concept. Along the way, we’ll debunk some of the myths surrounding syslog. For example, you can choose to limit messages to 1K and you can choose to send them via UDP, but you don’t have to – it’s not even a default in modern syslog daemons.

Syslog daemons

can write them to a file. Traditionally /var/log/messages or /var/log/syslog will receive everything, while some categories of messages go to specific files, like /var/log/mail.

can forward them to the network or other destinations. Traditionally, via UDP. Usually, the daemon also implements equivalent network listeners (UDP in this case).

This is where syslog is often referring to syslogd or sysklogd, the original BSD syslog daemon. Development for it stopped for Linux since 2007, but continued for BSDs and OSX. There are alternatives, most notably:
* rsyslog. Originally a fork of syslogd, it still can be used as a drop in replacement for it. Over the years, it evolved into a performance-oriented, multipurpose logging tool, that can read data from multiple sources, parse and enrich logs in various ways, and ship to various destinations
* syslog-ng. Unlike rsyslog, it used a different configuration format from the start (rsyslog eventually got to the same conclusion, but still supports the BSD syslog config syntax as well – which can be confusing at times). You’d see a similar feature set to rsyslog, like parsing unstructured data and shipping it to Elasticsearch or Kafka. It’s still fast and light, and while it may not have the ultimate performance of rsyslog (depends on the use-case, see the comments section), it has better documentation and it’s more portable
* nxlog. Yet another syslog daemon which evolved into a multi-purpose log shipper, it sets itself apart by working well on Windows

Myths about syslog daemons

The one we come across most often is that syslog daemons are no good if you log to files or if you want to parse unstructured data. This used to be true years ago, but then so was Y2K. Things changed in the meantime. In the myth’s defense, some distributions ship with old versions of rsyslog and syslog-ng. Plus, the default configuration often only listens for /dev/log and kernel messages (it doesn’t need more), so it’s easy to generalize.

Syslog message formats

RFC3164 a.k.a. “the old format”

Although RFC suggests it’s a standard, RFC3164 was more of a collection of what was found in the wild at the time (2001), rather than a spec that implementations will adhere to. As a result, you’ll find slight variations of it. That said, most messages will look like the RFC3164 example:

This is how the application should log to /dev/log, and you can see some structure:

<34> is a priority number. It represents the facility number multiplied by 8, to which severity is added. In this case, facility=4 (Auth) and severity=2 (Critical).

Oct 11 22:14:15 is commonly known as syslog timestamp. It misses the year, the time-zone and doesn’t have sub-second information. For those reasons, rsyslog also parses RFC3164-formatted messages with an ISO-8601 timestamp instead

mymachine is a host name where the message was written.

su: is a tag. Typically this is the process name – sometimes having a PID (like su[1234]:). The tag typically ends in a colon, but it may end up just with the square brackets or with a space.

the message (MSG) is everything after the tag. In this example, since we have the colon to separate the tag and the message, the message actually starts with a space. This tiny detail often gives a lot of headache when parsing.

In /var/log/messages, you’ll often see something like this:

Oct 11 22:14:15 su: 'su root' failed for lonvick on /dev/pts/8

This isn’t a syslog message format, it’s just how most syslog deamons write messages to files by default. Usually, you can choose how the output data looks like, for example rsyslog has templates.

RFC5424 a.k.a. “the new format”

RFC5424 came up in 2009 to deal with the problems of RFC3164. First of all, it’s an actual standard, that daemons and libraries chose to implement. Here’s an example message:

Now we get an ISO-8601 timestamp, amongst other improvements. We also get more structure: the dashes you can see there are places for PID, message ID and other structured data you may have. That said, RFC5424 structured data never really took off, as people preferred to put JSON in the syslog message (whether it’s the old or the new format). Finally, the new format supports UTF8 and other encodings, not only ASCII, and it’s easier to extend because it has a version number (in this example, the 1 after the priority number).

Myths around the syslog message formats

The ones we see more often are:

you can’t send syslog messages over 1K. It is true that RFC3164 stated that messages shouldn’t go over 1K, because it was around the maximum size of a UDP packet (now it can be higher with jumbo packets). Modern daemons don’t respect that constraint – the message limit is configurable in both rsyslog and syslog-ng. RFC5424 made this official, not only because of UDP changes but also because many other protocols are supported (see below).

Besides writing to files and communicating to each other, modern syslog daemons can also write to other destinations. For example, datastores like MySQL or Elasticsearch or queue systems such as Kafka and RabbitMQ. Each such destination often comes with its own protocol and message format. For example, Elasticsearch uses JSON over HTTP (though you can also secure it and send syslog messages over HTTPS).

Myths around syslog protocols

The ones we hear most come from the assumption that UDP is the only option, implying there’s no reliability, authorization or encryption.

The other frequent one is that you can’t send multiline messages, like stack traces. This is only true for TCP syslog, if newlines are used for delimiting. Then, a stacktrace will end up as multiple messages at the destination – unless its newlines are escaped at the source and reverted at the destination. With UDP, multiline logs work out of the box, because you have one message per datagram. Other protocols (TLS, RELP and octet-delimited TCP) also handle multiline logs well, by framing messages.

What’s next?

Hopefully this post helped clear the fog around syslog. If you’re looking for tips on how to configure your syslog daemon, you can find a lot of them on this blog. We especially love the topic of centralizing logs with Elasticsearch. That’s because we run Logsene, our logging SaaS that exposes the Elasticsearch API which supports all the syslog we discussed here in terms of message formats (including JSON over syslog) and protocols (UDP, TCP, TLS, RELP).

You can sign up and get a free Logsene trial here. You’ll find configuration samples for all major log shippers (including the syslog daemons discussed in this post).

syslog-ng is also optimized for performance. It just scales slightly differently. Rsyslog scales better for UDP connections or single TCP connections. Here syslog-ng is limited to a single core for incoming messages and in these use cases rsyslog is definitely faster on a multi-core machine. On the other hand, when there are multiple TCP connections, syslog-ng scaled better in my measurements. Also, using complex configurations with many rules, filters, etc., syslog-ng did not slow down as much as rsyslog.
I did not check it this year, but the above results were consistent ever since threading was introduced to syslog-ng with version 3.3.
Disclaimer: I know syslog-ng much better than rsyslog 🙂

Services

About

Contact

Apache Lucene, Apache Solr and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S.
and in other countries. Sematext Group, Inc. is not affiliated with Elasticsearch BV.