2/27/2018

We're pleased to announce the release of Oodle 2.6.0 featuring the new Leviathan codec. Leviathan
achieves high compression ratios (comparable to 7z/LZMA) with unparalleled fast decompression.

Oodle is an SDK for high performance lossless data compression.
For more about Oodle, or licensing inquiries,
visit the RAD Game Tools web site.
This is my personal blog where I post supplemental material about Oodle.

Almost two years ago, we released Oodle Kraken. Kraken roared onto the scene with high compression ratios
and crazy fast decompression (over 1 GB/s on a Core i7 3770).
The performance of Kraken immediately made lots of codecs obsolete. We could also see that something was wrong
in the higher compression domain.

Kraken gets high compression (much more that zlib; usually more than things like RAR, ZStd and Brotli), but
it gets a little less than 7z/LZMA. But to get that small step up in compression ratio, you had to accept a
20X decode speed penalty.

For example, on the "seven" testset (a private test set with a variety of data types) :

LZMA gets 3% more compression than Kraken, but decodes almost 20X slower.
That's not the right price to pay for that compression gain. There had to be a better way to
take Kraken's great space-speed performance and extend it into higher compression without giving up
so much speed.

It's easy to see that there was a big gap in a plot of decode speed vs compression ratio :

(this is a log-log plot on the seven test set, on a Core i7 3770. We're not looking at encode speed
here at all, we're running the compressors in their max compression mode and we care about the tradeoff
of size vs decode speed.)

We've spent the last year searching in that mystery zone and we have found the great beast that
lives there and fills the gap : Leviathan.

Leviathan gets to high compression ratios with the correct amount of speed decrease from Kraken (about 33%).
This means that even with better than LZMA ratios on the seven test set, it is still over 2X faster to
decode than zlib :

Leviathan doesn't always beat LZMA's compression ratio (they have slightly different strengths) but
they are comparable. Leviathan is 7-20X faster to decompress than LZMA (usually around 10X).

Leviathan is ideal for distribution use cases, in which you will compress once and decompress many times.
Leviathan allows you to serve highly compressed data to clients without slow decompression.
We do try to keep Leviathan encode's time reasonable, but it is not a realtime encoder and not the right
answer where fast encoding is needed.

Leviathan is a game changer. It makes high ratio decompression possible where it wasn't before. It
can be used on video game consoles and mobile devices where other decoders took far too long. It can be
used for in-game loading where CPU use needs to be minimized. It can be used to keep data compressed
on disk with no install, because its decompression is fast enough to do on every load.

Leviathan now makes up a part of the Oodle Kraken-Mermaid-Selkie family. These codecs now provide
excellent solutions over a wider range of compression needs.

Read more about Leviathan and Oodle 2.6.0 in these other posts on my blog :

Oodle is an SDK for high performance lossless data compression.
For more about Oodle, or licensing inquiries,
visit the RAD Game Tools web site.
This is my personal blog where I post supplemental material about Oodle.

A quick run down of all the exciting new stuff in Oodle 2.6.0 :

1. Leviathan!
2. The Kraken, Mermaid & Selkie fast-level encoders are now much faster.
3. Kraken & Mermaid's optimal level encoders now get more compression.
4. Kraken & Mermaid have new bit stream options which allow them to reach even higher compression.
5. Kraken and Mermaid are now more tuneable to different compression ratios and decode speeds.

The speed of the fastest encoder (level 1 = "SuperFast") is up by about 60% in Kraken & Mermaid.

Kraken's encode speed vs ratio is now competitive with ZStd, which has long been the best codec for
encode speed tradeoff.
For example, matching Kraken1 to the closest comparable ZStd levels on the same machine :

Compression ratio improvements around 1% might not sound like much, but when you're already on the Pareto frontier,
finding another 1% without sacrificing any decode speed or changing the bit stream is quite significant.

4. Kraken & Mermaid have new bit stream options which allow them to reach even higher compression.

Kraken in Oodle 2.6.0 now gets Silesia to 50,006,565 bytes at the default space-speed tradeoff target.
Kraken in max-compression space-speed setting gets Silesia to 49,571,429 bytes (and is still far faster
to decode than anything close).

If we look back at where Kraken started in
April of 2016 , it was
getting 4.05 to 1 on Silesia , now 4.24 to 1.

Kraken now usually gets more compression than anything remotely close to its decode speed.

Looking back at the old
Performance of Oodle Kraken ,
Kraken only got 2.70:1 on win81. On some files, Kraken has always out-compressed the competition, but win81 was one
where it lagged. It does better now :

At the time of Kraken's release, it was a huge decode speed win vs comparable compressors, but it sometimes
lagged a bit in compression ratio. No longer.

NOTE : Oodle 2.6.0 by default makes bit streams that are decodable by version >= 2.6.0 only. If you need
bit streams that can be read by earlier versions, you must set the backward compatible version number that
you need. See the Oodle FAQ on backward compatibility.

5. Kraken and Mermaid are now more tuneable to different compression ratios and decode speeds.

The new v6 bit stream has more options, which allows them to smoothly trade off compression ratio for decode speed.
The user can set this goal with a space-speed tradeoff parameter.

All the Oodle codecs have a compression level setting (similar to the familiar zip 1-9 level) that trades encode time for decode speed.
Unlike many other codecs, Oodle's compressors do not lose *decode* speed at higher encode effort levels. We are not finding more compact
encodings by making the decoder slower. Instead you can dial decode speed vs ratio with a separate parameter that changes how the encoder
scores decisions.

Oodle is an SDK for high performance lossless data compression.
For more about Oodle, or licensing inquiries,
visit the RAD Game Tools web site.
This is my personal blog where I post supplemental material about Oodle.

Oodle's speed on the Sony PS4 (and Microsoft Xbox One) and Nintendo Switch is superb.
With the slower processors in these consoles (compared to a modern PC), the speed advantage of Oodle
makes a big difference in total load time or CPU use.

These are run on the private test file "lzt99". I'm mainly looking at the speed numbers here, not
the compression ratio (compression wise, we do so well on lzt99 that it's a bit silly, and also not
entirely fair to the competition).

The Microsoft XBox One has similar performance to the PS4. Mermaid & Selkie can decode faster than the
hardware DMA compression engine in the PS4 and Xbox One, and usually compress more if they aren't limited
to small chunks like the hardware DMA engine needs.

Note that the PS4 non-Oodle reference data is from my earlier runs back in 2016 :
Oodle Mermaid and Selkie on PS4 and
PS4 Battle : MiniZ vs Zlib-NG vs ZStd vs Brotli vs Oodle .
They should be considered only rough reference points; I imagine some of those codecs are slightly different now, but does even a 10 or 20 or 50%
improvement really make much difference? (also note that there's no true zlib reference in that PS4 set; miniz is close but a
little different, and zlib-ng is faster than standard zlib).

Leviathan is in a different compression class than any of the other options, and is still 2-3X faster than zlib.

Something I spotted while gathering the old numbers that I think is worth talking about:

Well no, it hasn't. But this is a good example of how looking at just space or speed on their own can be
misleading.

Oodle's encoders are always optimizing for a space-speed goal. There are a range of solutions to that problem
which have nearly the same space-speed score, but have different sizes or speeds.

So part of what's happened here is that Oodle 2.6.0 is just hitting a slightly different spot in the space-speed
solution space than Oodle 2.3.0 is. It's finding a bit stream that is smaller, and trades off some decode speed
for that. With its space-speed cost model, it measures that tradeoff as being a good value. (the user can set the
relative value of time & bytes that Oodle uses in its scoring via the spaceSpeedTradeoffBytes parameter).

But something else has also happened - Oodle 2.6.0 has just gotten much better. It hasn't just stepped along
the Pareto curve to a different but equally good solution - it has stepped perpendicularly to the old Pareto curve
and is finding better solutions.

At RAD we measure that using the "correct" Weissman score
which provides a way of combining a space-speed point into a single number that can be used to tell whether you have
made a real Pareto improvement or just a tangential step.

The easiest way to see that you have definitely made an improvement is to run Oodle 2.6.0 with a different
spaceSpeedTradeoffBytes price so that it provides a simpler relationship :

Now we have higher compression and higher speed, so there's no question of whether we lost anything.

In general the Oodle 2.6.0 Kraken & Mermaid encoders are making decisions that slightly bias for higher
compression (and slower decode; though often the decode speed is very close) than before 2.6.0.
If you find you've lost a little decode speed and want it back, increase spaceSpeedTradeoffBytes (try 400).

Read more about Leviathan and Oodle 2.6.0 in these other posts on my blog :

Oodle is an SDK for high performance lossless data compression.
For more about Oodle, or licensing inquiries,
visit the RAD Game Tools web site.
This is my personal blog where I post supplemental material about Oodle.

All the compressors in this test are run on in their slowest encoding level.
We're looking for high compression and fast decompression, we're not looking for fast compression
here. See @@ for a look at encode times.

Tests are on groups of files. The results show the total compressed size & total decode time over
the group of files, but each file is encoded independently. The maximum window size is used for all compressors.

LZMA and zlib here were built with MSVC; I also checked there speed in a GCC build and confirmed it is nearly
identical.

Two settings for LZMA are tested to try to give it the best chance of competing with Leviathan. LZMA's default settings
tend to be good on text but not great on binary (often even Kraken can beat "lzma_def" on binary structured data).
I've also increased LZMA's fast bytes to 128 in the non-default options, the default value of 32 is a bit of a detriment to
compression ratio, and most of the competition (ZStd, Brotli, LZHAM) use a value more like 128. I want to give LZMA
a great chance to compete; we don't need to play any games with selective testing to make Leviathan look good.

2/12/2018

Starting in Oodle 2.6.0 the pre-Kraken codecs will be slowly phased out.

In 2.6.0 they will still be available for both encoding & decoding. However, they will be marked "deprecated" to discourage their use.
I'm doing this two ways.

One : if you encode with them, they will log a warning through the new Oodle "UsageWarning" system.
Decoding with them will not log a warning. This warning can be disabled by calling Oodle_SetUsageWarnings(false).
(Of course in shipping you might also have set the Oodle log function to NULL via OodlePlugins_SetPrintf, which will also disable the
warning log). (Also note that in MSVC builds of Oodle the default log function only goes to OutputDebugString so you will not see
anything unless you either have a debugger attached, change the log function, or use OodleX to install the OodleX logger).

Two : the enum for the old compressor names will be hidden in oodle2.h by default. You must define OODLE_ALLOW_DEPRECATED_COMPRESSORS
before including oodle2.h to enable their definition.

As noted, decoding with the old codecs will not log a usage warning, and can be done without setting OODLE_ALLOW_DEPRECATED_COMPRESSORS.
That is, Oodle 2.6.0 requires no modifications to decode old codecs and will not log a warning. You only need to consider these steps
if you want to *encode* with the old codecs.

In some future version the encoders for the old codecs will be removed completely. All decoders will continue to be shipped for the time
being.

I'm intentionally make it difficult to encode with the old codecs so that you transition to one of the new codecs.

The new codecs simply obsolete the old codecs and they should not be used any more. The old codecs are also
a mix of some fuzz safe, some not. The new codecs are all fuzz safe and we want to remove support for
non-fuzz-safe decoding so that it's not possible to do by accident.

In pursuit of that last point, another change in 2.6.0 is removing the default argument value for OodleLZ_FuzzSafe
in the OodleLZ_Decompress call.

Previously that argument had a default value of OodleLZ_FuzzSafe_No , so that it would allow all the codecs to
decompress and was backwards compatible when it was introduced.

If you are not explicitly passing something there, you will get a compiler error and need to pass something.

If possible, you should pass OodleLZ_FuzzSafe_Yes. This will ensure the decode is fuzz safe (ie. won't crash
if given invalid data to decode).

The only reason that you would not pass FuzzSafe_Yes is if you need to decode some of the older non-fuzz-safe
codecs. We recommend moving away from those codecs to the new post-Kraken codecs (which are all fuzz safe).

If you need to decode one of the non-fuzz-safe codecs, you must pass FuzzSafe_No. If you pass FuzzSafe_Yes, and
the decoder encounters data made by that codec, it will return failure.

Fuzz safety in Oodle means that unexpected data will not crash the decoder. It will not necessarilly be
detected; the decode might still return success. For full safety, your systems that consume the data post
decompression must all be fuzz safe.

Another semi-deprecation coming in Oodle is removing high-performance optimization for 32-bit builds (where 64-bit is available).

What I mean is, we will continue to support & work in 32-bit, but it will no longer be an optimization priority.
We may allow it to get slower than it could be. (for example in some cases we just run the 64-bit code that requires
two registers per qword; not the ideal way to write the 32-bit version, but it saves us from making yet more code paths to
optimize and test).

We're not aware of any games that are still shipping in 32-bit. We have decided to focus our time budget on 64-bit
performance. We recommend evaluating Oodle and always running your tools in 64-bit.

If you're a user who believes that 32-bit performance is important, please let us know by emailing your contact at RAD.