EFK follow-up of log system: monitor alarm monitoring

Previous articles EFK Follow-up of Logging System: fluent-bit Service Independence It completes fluent-bit acquisition, fluentd forwarding to kafka, and then to elastic search. Later, it is mentioned that the server logs should be synchronized to the fluent-bit environment. This can be accomplished by incremental synchronization through rsync, without recording in detail. Now we mainly record the alarm log monitoring and sending messages in kafka. The process of notification.

copy: copy each event to multiple outputs, store is equivalent to match

rewrite_tag_filter: Rewrite tag according to rule rule rule for matching event, send message with new tag, and re-process from top to bottom, so pay attention to rewrite tag does not match the current match, otherwise it will fall into a dead cycle.

Here matches the message tag fb.dapeng:

Messages matching level as ERROR rewrite tag as error.fb.dapeng,

Messages are sent directly to the topic of kafka: efk

For the message rewritten as error.fb.dapeng in tag 1, send the message to the topic of kafka: efk_error

In this way, elastic search only consumes kafka's topic: efk, which contains all levels of log information. For alarm monitoring monitor, only the topic of consuming kafka: efk_error, which is ERROR-level log, is used.

Note: The rewrite_tag_filter plug-in needs to be installed and the Dockerfile of fluentd needs to be modified to rebuild the image