This StoredFieldsFormat compresses blocks of 16KB of documents in
order to improve the compression ratio compared to document-level
compression. It uses the LZ4
compression algorithm, which is fast to compress and very fast to decompress
data. Although the compression method that is used focuses more on speed
than on compression ratio, it should provide interesting compression ratios
for redundant inputs (such as log files, HTML or plain text).

File formats

Stored fields are represented by two files:

A fields data file (extension .fdt). This file stores a compact
representation of documents in compressed blocks of 16KB or more. When
writing a segment, documents are appended to an in-memory byte[]
buffer. When its size reaches 16KB or more, some metadata about the documents
is flushed to disk, immediately followed by a compressed representation of
the buffer using the
LZ4compression format.