Hi, I'm David

Watch a log file and send new lines to an HTTP endpoint – with log2http

1 minute read

Recently, I wanted to watch a couple of log files for new entries and have them sent to an http endpoint for collection and later analysis. I did a quick research on what tools exist, but eventually decided to create a small Python app myself which doesn’t require a complicated setup.

I thought of something along the lines of:

pip install <the module>

Define which log files to watch and where to send the contents to

Run it from the terminal.

And so I built it.

log2http

You can find the Python module log2http on GitHub, and here’s how you can use it:

Install

Install the module in a Python 3.6+ environment:

pip install log2http

Create a config file

Create a new YAML config file (anywhere you wish) and insert the following key:values for each file you want to watch:

-logfile:/Users/demo/example.logendpoint:http://endpoint1.example.com/tag/samplemin_lines:5# only send to endpoint when at least 5 new lines were collected-logfile:/var/log/system.logendpoint:http://endpoint2.example.com/tag/syslogmin_lines:150

The endpoint is the address to which new lines are being POSTed to. You could use a logging service like Loggly (endpoint would be something like this: http://logs-01.loggly.com/bulk/token/tag/example/) or build a small service yourself.

Run it

Now you can run the module from your terminal with:

log2http --config=/absolute/path/to/the/config.yml

To see how it looks like when events are being collected, generate a few sample entries for the file you are watching, e.g. in bash: