In order to support all of the kinds of globs, pass
GlobSafeConfigParser
into the Configuration object so that it parses section headers
correctly.

Update dependency on conf_d

Fix line spacing + trigger travis

Remove chdir in test

This should fix the unit test to run properly when run from the main
directory.

When shipping logs, use millisecond-precision timestamps. [Ryan Park]

Logstash 1.3.2 has a problem with microsecond
precision timestamps in the
@timestamp field, which is the default behavior of Python’s .isoformat
method. Logstash uses the JodaTime library to parse timestamps, and
Joda
doesn’t support nanosecond timestamp resolution. As a result,
Logstash
1.3.2 throws an exception on every log item shipped from Beaver.

My change allows that, by making delimiter not string
escaped until
after the config file is parsed. I’m naive about python, so there is a
strong possibility I’ve gone about it horribly wrong. This would also
easily allow splitting on nulls, tabs, unicode characters and other
things that ConfigParser may not find kosher.

By doing this sort of multiline parsing with beaver, it allows one to
run logstash without the multiline filter, which due to its lack of
thread
safety, forces you to run logstash with only one worker thread.

Each worker has a self._file_map attribute which is a mapping of
file ids to file data. When retrieving lines or checking on the status
of the file, we use iteritems() which gives us a generator as
opposed to a copy such as with items(). This generator allows us to
iterate over the files without having issues where the file handle may
open several times or other random Python issues.

Using a generator also means that the set that we are iterating over
should not change mid
iteration, which it does if a file is unwatched. To circumvent this,
we should use a separate list to keep track of files we need to
unwatch or rewatch, and do it out of band.

We should also take care to catch RuntimeError which may arise when
closing the Worker out of band
such as in the cleanup step of the worker dispatcher
but nowhere else.

This should fix issues where logrotate suddenly causes files to
disappear for a time and beaver tries to tail the file at the exact
time it is being recreated.

Typo in SQS docs. [Jonathan Quail]

Remove ujson requirement. [Jose Diaz-Gonzalez]

This allows users that do not have a compiler in their deployment area
to install beaver.

Please note that the primary beaver stanza MUST be located in the
file specified by the `

configfile` argument. Any other such beaver stanzas will be ignored.

This change also unifies the BeaverConfig and FileConfig classes,
and simplifies the api for retrieving global vs file
specific data.

Please note that this commit BREAKS custom transport classes, as the
interface for creating a transport class has changed. If you are
referencing a file_config.get(field, filename) anywhere, please omit
this and refer to beaver_config.get_field(field, filename).

16 (2012-12-27)

Previously there were issues where files that were updated frequently
such as varnish or server logs
would overwhelm the naive implementation of file.readlines() within
Beaver. This would cause Beaver to slowly read larger and larger
portions of a file before processing any of the lines, eventually
causing Beaver to take forever to process log lines.

This patch adds the ability to use an internal work queue for log
lines. Whenever file.readlines() is called, the lines are placed in
the queue, which is shared with a child process. The child process
creates its own transport, allowing us to potentially create a Process
Pool in the future to handle a larger queue size.

Note that the limitation of file.readlines() reading in too many lines
is still in existence, and may continue to cause issues for certain
log files.

This argument was only used when no globs were specified in a config
file.
Since it is not configurable, there is no sense leaving around the
extra logic.

Remove extra callback invocation on readlines. [Jose Diaz-Gonzalez]

Remove extra file_config module. [Jose Diaz-Gonzalez]

General code reorganization. [Jose Diaz-Gonzalez]

Move both BeaverConfig and FileConfig into a single class

Consolidated run_worker code with code in beaver binary file. This
will create a clearer path for Exception handling, as it is now the
responsibility of the calling class, allowing us to remove duplicative
exception handling code.

Added docstrings to many fuctions and methods

Moved extra configuration and setup code to beaver.utils module. In
many cases, code was added hastily before.

Made many logger calls debug as opposed to info. The info level should
be generally reserved for instances where files are watched,
unwatched, or some change in the file state has occurred.

Remove duplicative and old beaver instructions from binary. [Jose
Diaz-Gonzalez]

13 (2012-12-17)

This code should allow us to create an ssh tunnel between two distinct
servers for the purposes of sending and receiving data.

This is useful in certain cases where you would otherwise need to
whitelist in your Firewall or iptables setup, such as when running in
two different regions on AWS.

Allow for initial connection lag. Helpful when waiting for an SSH
proxy to connect. [Jose Diaz-Gonzalez]

Fix issue where certain config defaults were of an improper value.
[Jose Diaz-Gonzalez]

Allow specifying host via flag. Closes #70. [Jose Diaz-Gonzalez]

12 (2012-12-17)

Reload tailed files on non-linux platforms. [Jose Diaz-Gonzalez]

Python has an issue on OS X were the underlying C implementation of
file.read() caches the EOF, therefore causing readlines() to only
work once. This happens to also fail miserably when you are seeking to
the end before calling readlines.

This fix solves the issue by constantly re
reading the files changed.

Note that this also causes debug mode to be very noisy on OS X. We all
have to make sacrifices…

Deprecate all environment variables. [Jose Diaz-Gonzalez]

This shifts configuration management into the BeaverConfig class.
Note that we currently throw a warning if you are using environment
variables.

11 (2012-12-16)

For my setup I need to have the fqdn used at all times since my
hostnames are the same but the environment (among other things) is
found in the rest of the FQDN.

Since just changing socket.gethostname to socket.getfqdn has lots of
potential for breakage, and socket.gethostname doesn’t always return
an
FQDN, it’s now an option to explicitly always use the fqdn.

Fixes #68

Check for log file truncation fixes #55. [Jeremy Kitchen]

This adds a simple check for log file truncation and resets the watch
when detected.

There do exist 2 race conditions here:
1. Any log data written prior to truncation which beaver has not yet
read and processed is lost. Nothing we can do about that.
2. Should the file be truncated, rewritten, and end up being larger
than
the original file during the sleep interval, beaver won’t detect
this. After some experimentation, this behavior also exists in GNU
tail, so I’m going to call this a “don’t do that then” bug :)

Additionally, the files beaver will most likely be called upon to
watch which may be truncated are generally going to be large enough
and slow
filling enough that this won’t crop up in the wild.

Timestamp in ISO 8601 format with the “Z” sufix to express UTC.
[Xabier de Zuazo]

Adding udp support. [Morgan Delagrange]

Lpush changed to rpush on redis transport. This is required to always
read the events in the correct order on the logstash side. See: https:
//github.com/logstash/logstash/blob/6f745110671b5d9d66bf082fbfed99d145
af4620/lib/logstash/outputs/redis.rb#L4. [Xabier de Zuazo]