Friday, 10 November 2017

Re: Should Ubuntu systemd journal logs be persistent by default?

I fully understand the need to minimize redundancy and performance degradation that this incurs, which you don't want on every desktop or embedded system. But for production or test servers, this could prove invaluable (I work in a high volume system validation lab where system crashes are the norm).

My suggestion (at least for existing releases) is to document the process for enabling persistent logging from the commandline, as this will allow sysadmins to easily implement it for post reboot reviews on a selective basis. Having it documented in a central location that is easily searched in google helps new sysadmins that would otherwise have to wade through tons of documentation and google searches of irrelevant information (sorry, not volunteering for this - too busy).

For 18.04, maybe a separate package or a package install prompt during installation (similar to unattended-updates) could be implemented.

Some time around the 15.04 release, a policy change was made to quit making some logging persistent by default.

A number of users did not realize there was a policy change until they went to debug something that happened on a previous boot, only to find the logs were missing. Some of these user responses are captured in the bug report below [1], which prompted this policy discussion.

The change was made because of the introduction of systemd and the introduction of the `systemd journal` in addition to the existing `rsyslog`.

The concern about making the systemd journal persistent by default is that some logs could end up duplicated between the systemd journald and rsyslog, along with disk space and performance concerns of the additional logging.

On the other hand, having systemd journal logs persist seems to be a safer option: It a malicious app could cause or entice a reboot, it could erase logs of it's earlier activity. Also, deleting key logs at shutdown breaks a decades-log precedent of system logs persisting through reboots.

The compromise option seems to be make the systemd journal persistent by default, but minimize the amount of logging that is sent to both rsyslog and systemd to mitigate resource considerations of duplicate logging.

So the policy question here is: should the systemd journal be persistent by default?