Update Tail to get rid of races and truncation problems.

Details

Description

The first tail implementation used buffered readers and file readers. This caused problems because the read call was blocking and couldn't shutdown properly.

The second tail implementation was mroe closely based on gnu tail's C implementation but relies on RandomAccessFile. This version had problems with races (restarting from beginning FLUME-218) and truncation (new test in FLUME-218 patch by Eric Sammer).

The new approach will likely use NIO and nonblocking IO to act cleanly, or possibly use a JNI based approach to get to unix system calls to get and follow file descriptors or inode numbers.

This version significantly reduces the chances of data duplication
encountered due to the FLUME-252 (races in tail) bug. However, it
does not completely fix the problem. A workaround is to use
'exec("tail -F <file>")' instead of the tail source Linux/Unix
systems. A new issue has been filed as FLUME-320 to continue tracking
this problem.

Jonathan Hsieh
added a comment - 04/Nov/10 21:02 This version significantly reduces the chances of data duplication
encountered due to the FLUME-252 (races in tail) bug. However, it
does not completely fix the problem. A workaround is to use
'exec("tail -F <file>")' instead of the tail source Linux/Unix
systems. A new issue has been filed as FLUME-320 to continue tracking
this problem.

Scratch FLUME-205. I tested with some chinese characters (我是中國人) and data did not get through properly. My guess is that reads in this tail are correct on output there is an endian or output encoding issue. Will leave FLUME-205 open.

Jonathan Hsieh
added a comment - 06/Oct/10 00:32 Scratch FLUME-205 . I tested with some chinese characters (我是中國人) and data did not get through properly. My guess is that reads in this tail are correct on output there is an endian or output encoding issue. Will leave FLUME-205 open.

Jonathan Hsieh
added a comment - 05/Oct/10 22:46 - edited The solution I have address these problems with the following mechanisms:
FLUME-205 : by using NIO api and only using byte[] (never doing charater encoding translations)
FLUME-248 : added a method to cursor that is called on close and read file rotate, as well as a test that fails if not done properly
FLUME-218 : Passes the python script found in that test.
This does not address FLUME-148 .