logrotate

The story

The disk space for /var/log/audit/audit.log tends to get filled up. The audit daemon has an ability to rotate its own logs. See the man page for auditd.conf.

max_log_file = 100
max_log_file_action = rotate

That’s swell and all, until you realize that auditd cannot compress its rotated logs. On a small /var/log/audit mount point, you’ll fill it up with uncompressed logs.

/dev/mapper/os-var_log_audit 2490M 2136M 355M 86% /var/log/audit

So on a RHEL7 system with logrotate, you can adjust logrotate to handle the audit.log file. Now, logrotate is a finicky application. It has caused me many hours of grief in the past.
You would want to set auditd.conf a certain way:

After a few days, I learned that my logs were getting filled up so fast, the weekly rotation wasn’t good enough. So I had to place it in my cron.hourly.
And then I learned that it wasn’t running every hour. I spent a few days investigating, and eventually learned that some systems use a specific status file for logrotate. I remember in the past logrotate needs an execution with a -f flag to force the rotation the first time and add a new file to the status file. So if a new file was never force-rotated, it won’t be added to the status file.
My manual logrotate -f command was indeed adding my audit.log log file to the status file, but to the wrong one!
Some of my systems use -s /var/lib/logrotate/logrotate.status but the default is /var/lib/logrotate.status.
So I had to reflect that in my ansible playbook. Actually, I had to write some logic to find the one used by the cronjob and then use that status file.

So I got the correct logrotate status file set up in the ansible playbook. I spent the next week figuring out that logrotate simply couldn’t rotate the file when called from cron. I piped the utility to tee, and also included the -v flag on logrotate. I saw a permission denied.
With the permission issue, I had no choices left by selinux. I had to use the audit.log file to determine that the audit.log file is not readable by logrotate when called by cron.
I finally set captured all the actions performed by logrotate by setting the selinux process context to be permissive:

semanage permissive -a logrotate_t

I let it run, and then had to collect all the actions it performed, and saw what had happened.

Now, I wrote a master ansible playbook that performs this whole operation, from loading the .te file and compiling it and installing it, to setting logrotate to watch the audit file, and telling auditd to ignore rotating it.Note: It is outside the scope of this task to ensure that the selinux tools are in place on each server. My environment already ensures package libselinux-python is present on each system, which should bring in all the dependencies of this ansible playbook.