I'm not at ease with vorbis at this bitrate during a blind test : it sounds too particular (hiss, desquilibrated tonal range : more treble, poor low-medium, and limited stereo), and it's easy for me to detect the encoder. I'm rating vorbis, and not an unknow encoder. So it isn't blind anymore.

I agree 100%. I often knew right away which one was vorbis, and I struggled to not let that influence my ratings. Am I rating it too high because I'm an OSS true-believer? Am I rating it too low because I'm overcompensating?

I also agree that its performance was spotty. A few samples had very serious problems (for me, those included Illinois, Polonaise, gone). I hope 1.0.1 fixes these issues and some of the problems exposed in the 128k test.

I am *expanding!* It is so much *squishy* to *smell* you! *Campers* are the best! I have *anticipation* and then what? Better parties in *the middle* for sure.http://www.phong.org/

Gosh! I only listened to one sample (New York City)... but I gave Vorbis a 1.3 Even the muffled lo-fi sounds of QT and WMA sounded nicer to me than the lavish sweeps of distortion and generous, unrestrained servings of noise that were dished out by Vorbis. I really wasn't expecting that at all. Still, I suppose I shouldn't judge it on one sample alone...

And Lame really is very good, (even if twice the bit-rate)... it's nice to know that.

Great test! However, I wonder who is doing 64kbps encoding. I believe that most people use at least 128kbps or 96kbps at the very minimum...flash card prices are getting lower and lower...anyway, that is why I'm very glad that a 128kbps test already took place...I hope that other test will follow...how about a 160kbps test

By the way: What is HE-ACC?

Thanks

--alt-presets are there for a reason! These other switches DO NOT work better than it, trust me on this.LAME + Joint Stereo doesn't destroy 'Stereo'

HE AAC is one of the profiles of MPEG-4 AAC. It uses SBR (Spectral Band Replication) in order to achieve a high efficiency at low bitrates. Currently there's only one publicly available HE AAC encoder implementation - Nero AAC/AAC-HE encoder.Check here for more info about HE AAC.

Actually, I'd like to see it go even lower... a 32kbps test (with samples containing some music, but mainly speech) would be a fairly good reflection on low-bitrate streaming -- it's around the rate used by the BBC's RealAudio streams, anyhow. Many people found it hard to detect artifacts even at 64kbps, so as we increase the bitrate the likelihood of getting decent statistically valid results crashes through the floor.

Thinking about this test has revived an old idea of mine, which would be to test the samples without the original present -- the users would then mark which sample sounds better, rather than which sounds closest to the original. I've not yet been able to figure out a decent way to analyse the results, though. Not having a scale opens the possibility of a non-transitive chain: i.e. a set of samples X_1,..., X_k where X_1 is preferred to X_2, X_2 to X_3, ..., *and X_k to X_1*. I'd love to see something like that happen in practise.

According to the averege the best my preferred encoder seems to be HE AAC even if i should say that i’m quite surprised for the result of mp3PRO (or maybe i’ve overestimated HE AAC).

I’ve done a sort of ranking for the best, real antagonists at 64 Kbps. The first position is colored in green, the second in yellow and the third in red. With this direct comparison Mp3PRO is (according to my preferences) often better than HE AAC.

Mp3Pro has shown a detectable lowpass (16Khz) but this is not the real problem for a 64 Kbps, artifacts are more annoying. There is an interesting thing i’ve perceived: with HE AAC the high frequencies seem unnatural, attenuated, as if it was lowpassed. While doing the blind test i imputed this to the lowpass, but later i’ve discovered that the HE AAC files are lowpassed at about 20 kHz (surely inaudible for me). Does FAAD use dithering when decoding ? If not, i think that the fact could be explained with the SBR “problem” of which guruboolez give us an excellent description.

While the two codecs above scored a very close quality level, i can’t say the same for Vorbis that is often behind the others two. I surely agree with guruboolez: Vorbis in this bitrate range is easy detectable because of noise and exaggerated highs (with sharp attacks the result is quite annoying).We all are waiting for the 1.1 version that should give better result with this type of artifacts.

At the end, i sincerely want to thank Roberto for his effort organizing this useful test. The number of participants is increased and this is a clear indication of good organization.

At first, let me thank Roberto for his efforts: thank you! It was nice to see that the participation level was high.

For me HE AAC and MP3 Pro came out on top, while Ogg was just mediocre. Forget the rest. The below is only true for my personal ratings and usually the anchors are disregarded, unless mentioned otherwise.

The most surprising result was with sample 06, Illinios. Here HE AAC totally sucked, while Ogg shone (and the others did well too). This is contrary to the average public ranking.

On sample 9, Polonaise, MP3 Pro was worst. MP3 Pro also has the highest standard deviation of all encoders (including anchors).

Sample 07 was also interesting. While all other encoders dipped real low in quality, HE AAC was doing well.

Also worth mentioning is that I didn't rate Lame 128k as best on 6 samples. In 5 of these cases, HE AAC was rated higher than Lame.

WMA std was rated worst 9 out of 12 times. WMA std was also most consistent in quality. It consistently sucked.

Many people found it hard to detect artifacts even at 64kbps, so as we increase the bitrate the likelihood of getting decent statistically valid results crashes through the floor.

You got that right!

Quote

Thinking about this test has revived an old idea of mine, which would be to test the samples without the original present -- the users would then mark which sample sounds better, rather than which sounds closest to the original.

Menno came with an idea: Replacing "Lame 128" and "FhG MP3" with "High/Low Anchor" on the plots.

As to completely avoid confusion, since several people are thinking Lame won, and it wasn't there to win or lose to start with.

Any comments? Suggestions on alternatives?

Yes, it's a good thing. A different color maybe for each anchor, in order to avoid subconscious confusion.Is it possible to re-assignate a different place for the different plot ? I mean : from left to right, the winner (HE-AAC) to loser (Real?). This may be useful, to see on which sample winner(s) fail(ed). Random position aren't useful in my opinion.