Configure Filebeat

Beware, the YAML syntax is very strict. For example tab characters are not allowed, your text editor may automatically use them for indentations but you will hardly notice it. Sometimes Filebeat reports the error, sometimes not. Verifying the configuration file with a YAML syntax checker might help.

Input

Propectors

Prospectors are used to locate and process log files. Each prospector item begins with a dash (-) and specifies prospector-specific configuration options.

Parameter

Default

Mandatory

Description

paths

/var/log/*.log

Yes

Specify the list of paths that are scanned to locate log files.

Each path item begins with a dash (-).

Example

paths:
# Parse all files with the extension .log directly under folder /home/ubuntu/somelogs
- /home/ubuntu/somelogs/*.log
# Parse all files with the extension .log under the first level of subdirectories of folder /home/ubuntu/somelogs
- /home/ubuntu/somelogs/*/*.log
# Parse all files with an extension prefixed by .log located directly under folder /home/ubuntu/somelogs
- /home/ubuntu/somelogs/*.log*

Recursively matching all files in a directory and its subdirectories is currently not supported, but you can use a wildcard (*) for directory names. This means that you will have to declare a paths entry for each level of subdirectories you want to monitor.

Multiple prospectors

When using more than one prospector, you must ensure that each log file is monitored in the paths of only one prospector. Sharing the same file between multiple prospectors can lead to unexpected behaviour.

multiline

Not set

No

By default, Filebeat will treat each line in a log file as a separate log message.

When monitoring log messages that span multiple lines, you can use the multiline to group all lines of a message together following a pattern.

Recommendations

force_close_files for Filebeat v1.2 or close_removed for Filebeat v5.2.2.

To ensure that no line remain unprocessed upon file renaming, the new file name must be monitored in the prospector paths.

Logs scan rhythm

In the default configuration, a prospector will detect a new file within 10 seconds after it is actually created. The prospector will also detect a new line added to a known file within the next second after it is actually added. These settings can be adjusted with the scan_frequency and backoff parameters.

Path definition

Absolute and relative paths can be used, but absolute paths are more straightforward: relative paths correspond to the working directory. When restarting, changing the working directory will mean that the previous registry file won't be used and log parsing will be restarted.

Stop monitoring files based on their last modification time

The more files you are monitoring, the bigger your registry_file will become. To optimize its use, you should adjust the values of close_older (Filebeat v1.2) orclose_inactive (Filebeat v5.2.2) and ignore_older to suit the lifetime of your log files. In the default configuration, files will never be ignored and handlers on those that haven't been updated in more than 1h will be closed.

Compression

By default, Filebeat will compress all outgoing messages. In a situation where network bandwitdh is not a bottleneck and a minimum impact on the CPU usage of the monitored host is required, compression can be deactivated by setting compression_level to 0.