As requested in this new PR, this needs a lot more explaining. To start with, the ticket description includes a proposed solution, which is always a bad sign, because the explanation of why and how this problem occurs had been glossed over.

Please explain exactly what happens and why this is a problem. So, there's a prefix added to log lines. What didn't like that? A regex in our syslog config? Psql? How come it only affects relays and not the root server?

Once we've covered the "why", let's start looking at solutions. It looks to me like this could potentially be fixed in any one of three places (on the node that adds the prefix, on the relay or on the root server). Please explore these options and explain if/if not possible and why the solution you propose is best.

Basically, what's happens then is that rsyslog on the relay tries to parse the message to get the program name, and understands that the program name is "forwarded". We only want "rudder", so rsyslog skips the send to remote syslog step, and worse, stores the message locally instead (like any other message by default).

There are basically three solutions:

Stop the AIX syslog from doing this (-n argument)

We cannot just ask people to change their syslog default arguments, or worse, do it ourselves sneakily

Use a rsyslog module that does it automatically

There is the pmaixforwardedfrom.so module, but it is far from compiled everywhere, especially on RHEL ...

Be a little more generic about which kind of messages we want to forward

This is the safest and most efficient method: try to match more messages

What I do here is say to rsyslog "forward any message that has the program name "rudder" OR contains "rudder" in the message, thus enabling rsyslog to forward them without issue.

Thank you for clarifying this Matthieu. This makes this bug easy to understand, and following your analysis is simple. Now I can help.

I understand and confirm your analysis of the problem. However, there seems to be one thing missing: if "it works" on the root server, why don't we make the relay config for rsyslog more like the root server config for rsyslog?

In particular, I note that the root server config doesn't check the programname at all, it just uses a (admittedly complex) regex. Shouldn't we be doing the same on the relay, for consistency? And if we need to change the programname check on the relay, I also note that the root server config ends with a check on the programname (to discard logs from being logged elsewhere) - surely that should be changed also.

Actually, I used a mix of my idea and your suggestion, to make sure every remaining Rudder log (not matching the report formar) sent to a relay OR a root server get dropped, since even with the report filtering, regular CFEngine warnings would still pass through (when started using -KI for example ...)