zstd – A Fast Data Compression Algorithm Used By Facebook

Zstandard (also known as zstd) is a free open source, fast real-time data compression program with better compression ratios, developed by Facebook. It is a lossless compression algorithm written in C (there is a...

Zstandard (also known as zstd) is a free open source, fast real-time data compression program with better compression ratios, developed by Facebook. It is a lossless compression algorithm written in C (there is a re-implementation in Java) – its thus a native Linux program.

Read Also: 10 7zip (Data Comperssion) Command Examples in Linux

When required, it can trade compression speed for stronger compression ratios (compression speed vs compression ratio trade-off can be configured by small increments), vice versa. It has a special mode for small data compression, known as dictionary compression, and can build dictionaries from any sample set provided. It comes with a command line utility for creating and decoding .zst, .gz, .xz and .lz4 files.

It is actively used to compress large volumes of data in multiple formats and use cases in Facebook; services such as Amazon Redshift data warehousing; databases such as Hadoop and Redis; the Tor network and many other applications including games.

The following results are obtained by doing several fast compression algorithms tests on a server running Linux Debian using lzbench, an open-source in-memory benchmark tool.

Zstandard Compression Testing

How to Install Zstandard Compression Tool in Linux

To install Zstandard on a Linux distribution, you need to compile it from sources, but before that first you need to install the necessary development tools on your system using your distribution package manager as shown.

4. To set a compression level, zstd has a number of operation modifiers, for instance you can specify a compression level as -6(a number 1-19, default is 3) as shown.

$ zstd -6 --rm etcher-1.3.1-x86_64.AppImage

5. To set a compression speed, zstd has a compression speed ratio 1-10, the default compression speed is 1. You can trade compression ratio for compression speed with the --fast option, the higher the number the faster the compression speed.

$ zstd --fast=10 etcher-1.3.1-x86_64.AppImage

6. To display information about a compressed file, use the -l flag, which is used to display information about a compressed file, for example.

$ zstd -l etcher-1.3.1-x86_64.AppImage.zst

7. To test the integrity of a compressed files, use the -t flag as shown.

$ zstd -t etcher-1.3.1-x86_64.AppImage.zst

8. To enable verbose mode, use the -v option.

$ zstd -v -5 etcher-1.3.1-x86_64.AppImage

9. To use other file compression or decompression formats such as gzip, xz, lzma, and lz4, using the --format=FORMAT as shown.

10. To set a zstd process priority to real-time, you can use the option –priority=rt as shown.

$zstd --priority=rt etcher-1.3.1-x86_64.AppImage

The -r flag instructs zstd to operate recursively on dictionaries. You can find lots of useful and advanced options, how to read or create dictionaries by consulting the zstd man page.

$ man zstd

Zstandard Github Repository: https://github.com/facebook/zstd

Zstandard is a fast real-time, lossless data compression algorithm and compression tool which offers high compression ratios. Try it out and share your thoughts about it or ask questions via the feedback form below.