Proposed new daemon would enhance and replace current Linux logging

ITworld|November 22, 2011

In an effort to foil crackers attempts to cover their tracks by altering text-based syslogs, as well as improve the syslog process as a whole, two Red Hat developers are proposing a new binary-based tool called The Journal that could replace the syslog daemon in as early as the Fedora 17 release.

And believe you me, some people are less than enthused by the proposed solution.

Developers Lennart Poettering and Kay Sievers are proposing that the current 30-year-old syslog system is inefficient and too easy to misread and hack to properly perform even its most basic function: store a log of system events on a given Linux box.

This is largely due to the free-form nature of the syslog, which basically accepts text strings in whatever form the application or daemon on the Linux system chooses to send. So, one daemon may send information about an event in one way, and another daemon in a completely different way, leaving it up to the human reader to parse the information in a useful manner. Automated log analyzer tools can help with this, but in a detailed description of The Journal, Poettering and Sievers wrote:

"The data logged is very free-form. Automated log-analyzers need to parse human language strings to a) identify message types, and b) parse parameters from them. This results in regex horrors, and a steady need to play catch-up with upstream developers who might tweak the human language log strings in new versions of their software. Effectively, in a away, in order not to break user-applied regular expressions all log messages become ABI of the software generating them, which is usually not intended by the developer."

That's just one of 14 points the two developers have highlighted as problems with the current syslog system. Others include:

And so on. Poettering and Sievers highlighted one very topical problem with the syslog system to drive their points about a needed change to syslog home:

"For example, the recent, much discussed kernel.org intrusion involved log file manipulation which was only detected by chance."

With these factors in mind, Sievers and Poettering have come up with The Journal daemon, which will store data from system events in binary--not text--form as a list of key-value pairs that includes hashing for additional security.

With this binary implementation, The Journal daemon can enable the addition of metadata to each system event, such as the process ID and name of the sender, user and group IDs, and other key system data.

"Inspired by udev events, journal entries resemble environment blocks. A number of key/value fields, separated by line breaks, with uppercase variable names. In comparison to udev device events and real environment blocks there's one major difference: while the focus is definitely on ASCII formatted strings, binary blobs as values are also supported--something which may be used to attach binary meta data such as ATA SMART health data, SCSI sense data, coredumps or firmware dumps. The code generating a journal entry can attach as many fields to an entry as he likes, which can be well-known ones, or service/subsystem/driver specific ones."

If all of this seems a bit familiar to developers, see if this rings a bell: a lot of the effort here by Poettering and Sievers was inspired by the key/value, hash, and metadata provided to developers who use the git version control system.

Not only will implementing The Journal make a Linux system more secure (as unauthorized log entries or unexpected data field entries will immediately be flagged by the journal daemon), its inventors hope to actually reduce the footprint of the logging system on Linux by unifying all log systems on a Linux machine and efficiently restructuring the data.

"It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the new additions. The fields, an entry consists off, are stored as individual objects in the journal file, which are then referenced by all entries, which need them. This saves substantial disk space since journal entries are usually highly repetitive (think: every local message will include the same _HOSTNAME= and _MACHINE_ID= field). Data fields are compressed in order to save disk space. The net effect is that even though substantially more meta data is logged by the journal than by classic syslog the disk footprint does not immediately reflect that."

But not everyone is thrilled with the proposal. Poettering and Sievers anticipated that many developers and system admins would be unhappy with The Journal's use of UUIDs to identify messages--as evidenced by their tongue-in-cheek attention to the issue in the FAQ section of their proposal document.

But many of the objections voiced on Linux Weekly News, where the proposal was first highlighted, lament the replacement of a simple text-based system with a binary data format that will rely on one tool--The Journal--which in turn will only be available with the systemd daemon.

Several commenters picked up on this entry in The Journal proposal FAQ:

"Will the journal file format be standardized? Where can I find an explanation of the on-disk data structures?

"At this point we have no intention to standardize the format and we take the liberty to alter it as we see fit. We might document the on-disk format eventually, but at this point we don't want any other software to read, write or manipulate our journal files directly. The access is granted by a shared library and a command line tool. (But then again, it's Free Software, so you can always read the source code!)"

That entry, more than any other in the proposal document, generated a lot of controversy, as many LWN commenters objected to the idea of using a non-standard format for The Journal's data. Backwards compatibility was also a big point of concern.

"It's a shame that we will lose the simplicity of the plain-text syslog format. But syslogs are usually compressed using gzip anyway. So essentially for me, all this means is that I use <magic-lennart-tool> instead of gzcat as the first part of my shell command, wrote commenter C. McCabe. "The big issue that I see is that a lot of system administrators will treat this as magic security dust, and not realize that they need to periodically save those hashes to a remote (and secure!) system in order to get any security benefit.

"I also hope Lennart and co. realize the absolute necessity of backwards compatibility for the on-disk format," McCabe added. "It would really embitter a lot of system administrators if their old logs became unreadable after upgrading to the shiniest new version. But assuming this is managed well, I don't see any reason why this couldn't be a good idea."

How this plays out in the broader Linux community will be interesting, to be sure. I personally find it notable that Fedora (and its commercial parent Red Hat) now seems to be the project where many internal infrastructure changes to the Linux operating system are getting implemented, even as distros like Ubuntu focus on the interface and user space.

This is not a judgmental statement, but rather an observation. Linux is clearly undergoing some significant evolutionary changes and shedding some of its UNIX legacy. What remains to be seen is how these changes will affect Linux as it moves ahead.