Well, I've gotten a lot done today. I just applied code to SVN for variable block size. You can test it out using "-v 1" on the commandline. It has much less of an adverse effect on speed than I had anticipated, which is great. Hopefully in the future I'll be able to enhance the algorithm for the block splitting decision. This works pretty well for a first attempt though.

One down-side is that the resulting files are not within the FLAC Subset format. They do, however, decode just fine with the libFLAC and FFmpeg decoders. Hardware support may be sporatic though.

I'm not going to add anything big for a few days and just focus on bug fixes and small changes. Then I'll release a new version.

Well, I've gotten a lot done today. I just applied code to SVN for variable block size. You can test it out using "-v 1" on the commandline. It has much less of an adverse effect on speed than I had anticipated, which is great. ...I'm not going to add anything big for a few days and just focus on bug fixes and small changes. Then I'll release a new version.

Cool, sounds interesting. Once I have wisodev's build I'll set some scripts running.

OK, I've managed to compile flake v42 by myself and to do a quick test with variable block sized flac on my iAudio U3 - and I'm simply astonished because it just WORKS! There is small glitch though, the player doesn't display remaining play time of the track anymore but that's not a problem for me.

How bloody annoying! Only time they may have been of use and they weren't right! I have recompiled, checked the -v switch, and uploaded, just in case. I have no idea what happend this morning, I'm sure some files had updated...

Thanks for highlighting the error.

Edit: OK, I've just done some (after the horse has bolted) testing with -v 1 at -0, -5, -8 and -12 and in all cases the -v versions are bigger than the fixed block size versions. Am I missing something?

Edit: OK, I've just done some (after the horse has bolted) testing with -v 1 at -0, -5, -8 and -12 and in all cases the -v versions are bigger than the fixed block size versions. Am I missing something?

Hmmm. It worked for my samples. I'll do more testing tonight. I may temporarily add a variable frame-split-threshold until I can figure things out.

Justin, my post was a bit premature. I was just a little surprised at the (minor) increase in size over various compression levels, so posted without really thinking.

What I totaly forgot was that this was just one file.

To rectify I have just tested -5 on my "FLAC corpus" (28 files) and the results are a lot better.

Out of the 28 files 12 files suffer with the -v switch, but on the whole the compression improves from 59.843% to 59.754%, or 1,221,384 bytes in 1,371,283,088 (a 0.001% improvement).

As you can see from the figures, those that suffer from variable blocks (-v 1) are the files at the outer edges - generally those that compress the best (e.g.: Charlotte Church, Andrea Bocelli), but also two Motorhead tracks (-274 and -5,698). This variation reminds me of the window testing I was involved in for FLAC 1.1.2_CVS (1.1.3), with the two bounds acting differently than the core files... not sure if that's relevant.

When wisodev is kind enough to provide a compile of Rev. 42 I'll soon provide some more figures, using my Yalac corpus, including timings and using various "key" compression levels.

I would caution against encoding variable-blocksize files in native FLAC unless familiar with the implications. there is a design flaw in the native FLAC container that can make variable blocksize files difficult to handle in some ways (I may fix this later but it's a format change). this is not a problem when in other containers e.g. Ogg FLAC.

definitely I would say that a fraction of a percent size decrease is not worth the useability tradeoff.

Ratio is the compression ratio in percent. No speed data this time, because i am not sure, if both flake versions have been bulit with the same compiler optimizations. Subjectively Flake sv42 with -v 1 seems not to be significantly slower than V0.10.

"songs" contains CD-Ripps of the beginning of some songs. This set contains many files that are benefiting from higher predictor orders. It may be less representative than set rw, but i keep it because it has been my primary set for many years.

Some remarks

Up to 0.23 percent better compression for sample set "rw", only 0.01 better compression for "songs".

Not too surprising for me: "rw" contains many samples with very fast changes of the signal characteristics, where variable block sizes should help most.

Possibly Justin could try to use some of the "rw" files for further optimizations of the block length switching. I found them very useful for this.

"songs" contains files, which can quite well be encoded with constant frame sizes of up to 200 ms.

Yes, they are right. This file is one of the very rare cases, where a 16 bit file (or a part of it) contains a constant least significant bit, which can be removed by Flac's "Look for wasted bits" option. Probably in Flake this option is not implemented.

N.B. What does that exactly mean? That the 16th bit is always the same through the whole file? In other words 15 bit recording pad to 16 bit?

I have not performed a deeper analysis of this file. Possibly one channel has a constant least significant bit (LSB), because compression can be improved by about 3 percent by the wasted bits option. If both channels were affected, it would be about 6 percent.

"That the 16th bit is always the same through the whole file?" Through the whole file or through a whole frame (my notation) or block (Flac's notation). Usually the analysis is beeing performed on a frame basis (implementaion dependend). And theoretically more than 1 bit can be constant.

"In other words 15 bit recording pad to 16 bit?" If someone multiplies each sample with 2 (or shifts it by one place), the least significant bit will become zero:

I would caution against encoding variable-blocksize files in native FLAC unless familiar with the implications. there is a design flaw in the native FLAC container that can make variable blocksize files difficult to handle in some ways (I may fix this later but it's a format change). this is not a problem when in other containers e.g. Ogg FLAC.

definitely I would say that a fraction of a percent size decrease is not worth the useability tradeoff.

Josh

I agree. The variable block size headers are a bit tricky. I don't have any issue with using it for my personal files because I know the players I use support it. If I was sharing the files with other people I would definitely not use that option though. Perhaps I should change the warning to something less technical like "Warning! The chosen options are not part of the FLAC Subset format and therefore may not be compatible with some FLAC players."

N.B. What does that exactly mean? That the 16th bit is always the same through the whole file? In other words 15 bit recording pad to 16 bit?

I have not performed a deeper analysis of this file. Possibly one channel has a constant least significant bit (LSB), because compression can be improved by about 3 percent by the wasted bits option. If both channels were affected, it would be about 6 percent.

"That the 16th bit is always the same through the whole file?" Through the whole file or through a whole frame (my notation) or block (Flac's notation). Usually the analysis is beeing performed on a frame basis (implementaion dependend). And theoretically more than 1 bit can be constant.

"In other words 15 bit recording pad to 16 bit?" If someone multiplies each sample with 2 (or shifts it by one place), the least significant bit will become zero:

I think I may have removed that feature because I could not find any files which had any wasted bits...I can't really remember. I at least know I had implemented it at one point, but it's definitely no longer there. It would be easy enough to add.

As far as the compression gain for variable block size, I have a feeling I can do better. This was a first attempt and pretty much a shot-in-the-dark. What I think I'll do next is a really slow brute-force approach so that I will have something to compare results against.

Thanks for all the great results. It would be interesting to see some speed comparisons. I know that the current SVN version is quite a bit faster on my machine than 0.10, but my machine is certainly not typical (Linux, AMD K6-2).

Thanks for all the great results. It would be interesting to see some speed comparisons. I know that the current SVN version is quite a bit faster on my machine than 0.10, but my machine is certainly not typical (Linux, AMD K6-2).

Hi,Flake r46 has a new variable block size algorithm, which is selected by using "-v 2". It gives a lower-bound for even 3-level splitting (splitting frames exactly in half up to 3 times). Encoding takes about 4x as long. Hopefully I will be able to use this to help improve -v 1 by comparing the resulting files. Future developments might also include uneven splitting, which could better isolate changes within a frame with less overhead.-Justin