This StoredFieldsFormat compresses blocks of 16KB of documents in
order to improve the compression ratio compared to document-level
compression. It uses the LZ4
compression algorithm, which is fast to compress and very fast to decompress
data. Although the compression method that is used focuses more on speed
than on compression ratio, it should provide interesting compression ratios
for redundant inputs (such as log files, HTML or plain text).

File formats

Stored fields are represented by two files:

A fields data file (extension .fdt). This file stores a compact
representation of documents in compressed blocks of 16KB or more. When
writing a segment, documents are appended to an in-memory byte[]
buffer. When its size reaches 16KB or more, some metadata about the documents
is flushed to disk, immediately followed by a compressed representation of
the buffer using the
LZ4compression format.

If documents are larger than 16KB then chunks will likely contain only
one document. However, documents can never spread across several chunks (all
fields of a single document are in the same chunk).

When at least one document in a chunk is large enough so that the chunk
is larger than 32KB, the chunk will actually be compressed in several LZ4
blocks of 16KB. This allows StoredFieldVisitors which are only
interested in the first fields of a document to not have to decompress 10MB
of data if the document is 10MB, but only 16KB.

Given that the original lengths are written in the metadata of the chunk,
the decompressor can leverage this information to stop decoding as soon as
enough data has been decompressed.

In case documents are incompressible, CompressedDocs will be less than
0.5% larger than Docs.