Rsyslog vs Systemd Journal Presentation

Do rsyslog and the journal cooperate? If so, how? This is the presentation from the LinuxTag 2013 conference. It details the rsyslog team's current position on the journal, how it affected rsyslog, what is being done for integration and some notes about how to configure rsyslog to do things that the journal announcement claimed to be impossible.

Rsyslog vs Systemd Journal Presentation

1.
rsyslog vs journal?Rainer Gerhards

2.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyMe & the Talk• Rainer Gerhards▫ Data center guy▫ Involved 15+ years in logging▫ Founded rsyslog in 2003• The talk▫ Will rsyslog fight the journal?▫ Some history on journal-like system▫ Ways of integration▫ How to do things the journal announcementclaimed as impossible

3.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyDoes journal replace syslog?• The initial announcement sounded a bit in thatway, or was at least interpreted by most(including me) in that direction.• Looking at how things have evolved▫ There of course is overlap between both systems▫ But there are also (large) regions that do notoverlap• This is not a new situation, there is some historylesson...

6.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanySo what does history tell us?• If such a system can totally replace syslog, thereshould be no syslog on Windows at all – andnever have been.• Well... there are ample of applications▫ WinSyslog (initial version by me, 1996)▫ Kiwi Syslog (Solarwinds)▫ EventReporter (first ever Windows-to-syslog tool,1997)▫ Snare▫ and many more!

7.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyObviously, there must be someneed to syslog technology...• Face it: syslog is the lingua franca of networkevent logging.▫ If you want to process messages from differentsources, chances are high you will need it.▫ Even if not syslog (protocol) is used, you usuallyneed some common denominator e.g. Linux does not understand native WindowsEventLog Windows neither does understand native journal

8.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyA key problem solved by syslog• You want to integrate all of your systems into aconsolidated log• This either means▫ A common protocol▫ A system that is capable of processing multipleprotocols and somehow “normalize” them• Syslog is ubiquitous – because a basic client isdumb easy to implement!

10.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyWindows as a receiver...• Windows acts as syslog server• Messages are written to▫ Local files▫ Windows Event Log (!)▫ Some other processing (like alerting)• Typical deployment scenario for SOHO• But some large Windows-only shops also use itfor integration of non-Windows sources

11.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyWhy I am talking so much aboutWindows?• As I said, I see strong similarities betweenjournal and Windows Event Log• Except that journal has much more quicklygotten some network functionality• So my best guess is that deployments and end-user needs will evolve into mostly the samedirections

12.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyJournal vs. Syslog:low end systems• Usually users of these machines are not at allinterested in logging• Journal is very convenient as a troubleshootingtool• Works perfect on personal desktop & notebook• Rsyslog will be needed by some users tointegrate e.g. their DSL routers messages intothe journal

13.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyJournal vs. Rsyslog:enterprise systems• Impossible to manage without any syslog• Journal integrated as another event source▫ Journal-centric As much as possible is done with journal Integration happens at central head server(s)▫ Syslog-centric Journal is used only as much as unavoidable Each machine runs rsyslog and forwards events▫ Mode depends on end-users philosophy

14.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyHow did the journal affect thersyslog project?• Obviously, we expect less presence on low-endsystems• So we re-focussed the project▫ Previously low-end and enterprise needs wereequal peers▫ Now strong focus on enterprise• The logging world at large got benefit assuddenly everyone was interested in logging –which also helps rsyslog!

15.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyWhat have we done to integratewith the journal?• Module omjournal▫ Provides ability to store messages into the journal▫ Traditional syslog, text files, ...▫ Caters for the low-end use case• Module imjournal▫ Provides ability to pull messages off the journal,just as another event source▫ Contributed by Red Hat▫ Caters for the enterprise use case

19.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyWhy is it simple to integrate thejournal?• Rsyslog is actually▫ A message router▫ With dynamically loadable inputs and outputs▫ Highly configurable• So, journal support is as easy as adding somenew inputs and outputs!• The rest of the plumbing is already there.

22.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyNow lets look at some“impossible” things• The original journal paper claimed that syslog is▫ Seriously broken▫ Cannot provide some important features• Ill show how to do these “impossible” things▫ Based on 2011 technology▫ And on current one (v7.4)

23.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyLog File Manipulation Protection• The traditional approach is to ship logs off themachine, to a central and highly secured system• Keeping them on a system that is “easilycompromised” is asking for trouble.• Problem is that local secrets can always becompromised• In rsyslog 7.4, we address these problems via logsignatures and encryption...

34.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyMessage authentication andMetadata availablility• Rsyslog uses the same SCM_CREDENTIALSfacility that journal does• And in both cases it can be faked – as journaldemonstrates when it actually fakes it on thesystem log socket ;)• The volume of metadata available has beenincreased starting in 2012• Total authenticity requires signatures atthe original originator level (each app),what currently is impossible in the *nixframework.

35.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyFree-Formedness of Log Records• Traditional syslog messages are much like free-form text• Today, we see the same for typical journalmessages• There are a couple of standardization effortsunderway to provided structured logging• Project lumberjack (lead by Red Hat) providesJSON-based structured logs

36.
Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, GermanyUnstructured Text Log Duality• If a log format does not support freeform-text, itis not used (at least not more than one can avoidto...)• If it supports freeform-text (among others), thatfreeform-text will be abused• → unstructured logs wont go away!• Weve seen this in Windows Event Log and looksmuch the same for journal.