We are having some difficulty configuring our Filebeat to forward a large number of log files to our System.

Since we are deploying on another tenant's machine we are required to keep the resource consumption to a reasonable level. However with this many files it has become difficult for us to create a configuration that meets that requirement and keeps up with the updates.

The logging files:

Data Type 1:

Data Type 2:

Number of files: 323+

File creation: 1/day

Update rate: 1 row/minute

Total size per file: 55 KB

Data Type 3:

Number of files: 323+

File creation: 1/day

Update rate: ~6 row/10 minutes

Total size per file: 2 KB

Note: This file has strange logging behavior when it updates the file, It is deleted to and then recreated when updating new records. Optionally, to avoid this mess we are able to consume historical logs which are rolled over to at midnight and not updated after that.