After some comments by David, I've decided to revise my question. The original question can be found below as well as the newly revised question. I'm leaving the original question simply to have a history as to why this question was started.

Original Question (Setting LZMA properties for jslzma)

I've got some large json files I need to transfer with ajax. I'm currently using jQuery and $.getJSON(). I'd like to use the jslzma library to decompress the files upon receiving them. Currently, I'm using django with the pylzma library to compress the files.

The only problem is that there's a lack of documentation for the jslzma library. There is some, but not enough. So I have two questions about how to use the library.

It gives this as an example:

LZMA.decompress(properties, inStream, outStream, outSize);

I know how to set the inStream and outStream variables, but not the properties or the outSize. So can anyone give an example(s) on how to set the properties variable (ie. what's expected) and how to calculate the outSize...

Thanks.

Edit #1 (Revised Question)

I'm looking for a compression scheme that lends itself to highly repeatable data using python (django) and javascript.

The data being transferred contains elevation measurements. Each file has 1200x1200 data points, which equates to about 2.75MB in it's raw binary form uncompressed. JSON balloons it to between 5-6MB. I've also looked into base64 (just to cover all the bases), which would reduce the size but I haven't had any success reading it in js. I think the data lends itself to easy compression just because of the highly repeatable data values. For example, one file only has 83 unique elevation values to describe 1440000 data points.

I just haven't had much luck, mainly because I'm just starting to learn JavaScript.

So can anyone suggest a compression scheme for this type of data? The goal is to minimize the transfer time by reducing the size for the data.

2 Answers
2

For what it's worth LZMA is typically very slow to compress as well as decompress; and thus it is more common to use bit faster compression schemes. Standard GZIP (deflate) has reasonably good balance: its compression ratio is acceptable, and its compression speed is MUCH better than that of LZMA or bzip2.
Also: most web servers and clients support automatic handling of gzip compression, which makes it even more convenient to use.

+1 Assuming the client is a web browser, just use the builtin content compression.
–
BNLNov 14 '11 at 22:04

Thanks StaxMan. I've heard that somewhere else too. Is it as simple as changing the MIME type and compressing it server side? And is it automatically unzipped on the client side? Basically, I just want to know the steps involved. A link would be great if you have one.
–
maxNov 14 '11 at 22:04

1

The main thing needed is to ensure client sends header "Accept-Encoding"; and server can then compress and return HTTP header "Content-Encoding" with compression type ("gzip"). One link google gave was [betterexplained.com/articles/… but it should be easy to find more.
–
StaxManNov 14 '11 at 22:07

Thanks, StaxMan. Now I know what to search for.
–
maxNov 14 '11 at 22:17

Decompression on the client side with Javacscript can take a significant longer time and highly depends on the available bandwidth of the client's box. Why not just implement a lesser but faster and easier to write decompression like rle, delta or golomb code? Or maybe you want to look into compressed Jsons?

Thanks. I'll look into those. I'd like to use binary instead of json, but javascript isn't the easiest language to work with binary. I've tried some of the modules for binary data, I'm just not an expert in working with binary in the first place. Thanks for those suggestions I'll see if I can find a simpler compression algorithm to use. I was mainly looking into LZMA because of high compression ratios.
–
maxNov 14 '11 at 4:45

Compression depends on the data you want to compress. There isn't a best compression scheme. If you can check your data and ask again about a good compression scheme. Then you can decide better.
–
PhpdevpadNov 14 '11 at 8:01

Thanks again David. I've revised/edited the question to go along with the conversation so far. So going back to your comment, can you suggest a compression method that suits this type of data?
–
maxNov 14 '11 at 21:54

I'm not an expert either and I can second to use builtin gzip or bzip2 server and browser compression but maybe you can try a delta-encoding first and then let the browser compress it. You may get better compression result with little effort.
–
PhpdevpadNov 14 '11 at 23:49