Trixter wrote:EDIT: I just thought of another optimization: Don't use FF for the code; instead, use the value that shows up the least. Scan the entire file before compressing to determine which value shows up the least, then use that as the code. Otherwise, data that has a lot of FFs in it (like most 8-bit programming!) won't compress very well...

I don't think I'll get a lot by switching to rarest byte - usually a lot of FFs will be successfully eaten by LZ77, but I can at least estimate using outputs from my current design to see possible impact of that...

P.S. So I took results of compression of JRCARTS7.IMG and count number of literals FF that was coded in:

so by switching to rarest byte I'll get extra 1.2KB (-1.13%) so 112,715 will turn into 111,444 - from one hand it's better, from other it's not too much yet to convince me to get rid of FF at the name of my compression utility (it's SHAFF because of FF : ) - it's still a little far from ZX7 and RNC method 2...

Actually I found some sample of data that gave me 4% improvement by using rarest byte instead of FF, so I think I can support it as an option...

Very cool. But looking at LZ4 size, I don't know why you just don't use that. My LZ4 decoder only decodes up to a single segment, so you'd have to compress and decompress each ROM by itself if you wanted to use my code, but even still it seems like a win...

Trixter wrote:Very cool. But looking at LZ4 size, I don't know why you just don't use that. My LZ4 decoder only decodes up to a single segment, so you'd have to compress and decompress each ROM by itself if you wanted to use my code, but even still it seems like a win...

ZX7 always compresses smaller than LZ4 because ZX7 has an entropy encoder (range coding) on the back-end, whereas LZ4 does not. Where you pay for it is on the decompression side: LZ4 is extremely fast, ranging from 0.8x to 3.2x memcpy() speed, whereas ZX7 hovers around 9x memcpy() speed. Of course, decompression speed isn't a concern for a ROM cartridge -- nobody will care if it takes 0.5 seconds vs. 1 second to start the game...