zs 0.0.0-dev

ZS is a simple, read-only, binary file format designed for
distributing, querying, and archiving arbitrarily large
record-oriented datasets (up to tens of terabytes and beyond). It
allows the data to be stored in compressed form, while still
supporting very fast queries for either specific entries, or for all
entries in a specified range of values (e.g., prefix searches), and
allows highly-CPU-parallel decompression. It also places an emphasis
on data integrity – all data is protected by 64-bit CRC checksums –
and on discoverability – every ZS file includes arbitrarily detailed
structured metadata stored directly inside it.

Basically you can think of ZS as a turbo-charged replacement for
storing data in line-based text file formats. It was originally
developed to provide a better way to work with the massive Google N-grams,
but is potentially useful for data sets of any size.