Description

To solve certain known issues and limitations, Flume requires a refactoring of some core classes and systems. This bug is a parent issue to track the development of a "Flume NG" - a poorly named, but necessary refactoring. Subtasks should be added to track individual systems and components.

The following known issues are specifically to be addressed:

Code complexity; Flume has evolved over the last few years and has a fair amount of extraneous code.

Core component lifecycle standardization and control code (e.g. anything that can be start()ed or stop()ed, sources, sinks).

(Static) Configuration access throughout the code base.

Drastic simplification of common data paths (e.g. durability as an element of the source rather than a disconnected sink).

Heartbeat and master rearchitecture.

Renaming packages to org.apache.flume.

This is a large and far reaching set of tasks. The intent is to perform this work in a branch as to not disrupt immediate releases or short term forthcoming releases while still allowing open development in the community.