The much awaited results of the Public, Multiformat Listening Test @ 64 kbps are ready - partially. So far, I only uploaded an overall plot along with a zoomed version. The details will be available tomorrow. You can also download the encryption key on the results page that is located here:

Wow, thanks a lot for posting so fast these results.WMAPro is competitive against HE-AAC at 64 kbps... great result for this new format. What were Microsoft listening tests on this subject (I forgot it)?

Compare to the last 48kbit/s listening test, 64kbits will only bring slightly better results.

I-tunes at 96kbits ist transparent for most users on both tests.

WMA is not interesting for me.

Nero-AAC HE score was 3,64 points at 48kbits, now we can see 3,74 points at 64kbits.This is not very impressive for me. I thought Nero will performe better at 64kbit/s.Of course, it is still usable for e.g. portable devices or good quality webradio.

Vorbis is also better at 64kbits/ (3,16 to 3,32 points )

So i can go with itunes at 96kbit for high quality use (maybe nero performing better at this bitrate?), and 48-64kbits for medium quality use.

maybe 80kbits/s will hit a 4.xx score?

i think the next test should be a 96-112kbit multi-format test, also including Lame.

They're actually both correct, but now I agree that the first format which I posted doesn't make sense anymore since the listening tests have their own page. That htaccess redirection was good for the time where the tests were in subfolders of the MaresWEB site.

Compared to the whole group of testers my global evaluation for all competitors is clearly more harsh (-1.03 points on average), especially with the high anchor (-1.3 points) and HE-AAC (biggest deviation with -1.38 points). It confirms the lake of sympathy I feel for the SBR trick (there's several complains in my log files against the "SBR texture/noise"). I'm more disappointed by the high anchor which doesn't sound great to my ears. I expected more from LC-AAC two years after my previous test at 96 kbps.

WMAPro is a weird case. I'm not familar at all with this format (I never tested since its last metamorphosis in WMP11) and the new kind of distortion it produces. I disliked it on the beginning but I was much more enthousiastic after some times. Indeed, the second half of tested samples was better marked than the first one while it was at best the same for all other competitors. In other words my notation was more harsh during the second half but WMAPro's one has drastically grown in this severe period WMAPro artefacts were close to HE-AAC ones; it has a stronger smearing (cf kraftwerk, eig...) and share the same kind of SBRish issue (noise packets altering tonal sound, cymbals...) but often with less annoyance. It also has a kind of "noise sharpening" (for people knowing this foobar2000's plug-in) which tends to add some energy to high frequencies. Sound is often a bit brighter than reference to my ears. It's unexpected, not necessary a good thing but I find it rather pleasant in some situations, and certainly more enjoying than stereo reduction, pre-echo, lowpass or noise filtering. I simply fear that this kind of enhancement would quickly appear as tiresome (like noise sharpening IMO). That's why I wonder if I would still consider WMAPro so kindly with additionnal experience with this encoder and its own texture...

I was never fond of Vorbis at <80 kbps so I'm not surprised to see it inferior to HE-AAC with a confidence >95%. It often sound coarse, fat, with serious stereo issues (and a bit lowpassed too, but a smaller one would maybe increase the ringing...). I'm simply disappointed that for my taste no other format could currently outdistance this format.

As a consequence I'm disappointed. I maybe expected a miracle too soon after reading other people's comments. I will see in a future test if 80 or 96 kbps are more enjoyable for my taste.

Compare to the last 48kbit/s listening test, 64kbits will only bring slightly better results.

I-tunes at 96kbits ist transparent for most users on both tests.

WMA is not interesting for me.

Nero-AAC HE score was 3,64 points at 48kbits, now we can see 3,74 points at 64kbits.This is not very impressive for me. I thought Nero will performe better at 64kbit/s.Of course, it is still usable for e.g. portable devices or good quality webradio.

Vorbis is also better at 64kbits/ (3,16 to 3,32 points )

So i can go with itunes at 96kbit for high quality use (maybe nero performing better at this bitrate?), and 48-64kbits for medium quality use.

maybe 80kbits/s will hit a 4.xx score?

i think the next test should be a 96-112kbit multi-format test, also including Lame.

It's technically not valid to compare results between tests, although the ratings differences do seem to make some sense.

It's technically not valid to compare results between tests, although the ratings differences do seem to make some sense.

I think it's not completely pointless to note that both high and low anchor (which haven't change in the meantime - iTunes's version excepted) are now slightly worse than previously (samples are harder and/or listeners a bit more sensitive on average). A direct comparison between 48 kbps and 64 kbps performance should take this difference into account. It increases a bit the difference between 48 and 64 kbps encodings.

high is better than Nero, Vorbis, WMApro, lowNero is better than Vorbis, WMApro, low

Kudos to Nero! A clear winner according to me. Probably I must like SBR sort of trickery. Ranked it "annoying" only twice.(And I guess Nero needs some work on the classical orchestra sample "macabre")

WMA pro is disappointing. I'm not impressed. All narrow stereo problems turned out to be WMA.

Vorbis is not worse than WMA but it sounds to me that it didn't really improve very much (at this bitrate) for the last couple of years.

Both WMA and Vorbis tend to distort lower frequencies, which is very easy for me to notice on natural acoustic instruments (guitars, violin, trumpet, also voice). Too distorted sometimes, even worse than low anchor.

(I am not so sensitive to high frequency artifacts. At least typically I don't find it annoying.)

High anchor is very good. Almost transparent. However, I didn't really concentrate very much on the high anchor. Otherwise I could have given it a few more "4"s. But very impressive anyways.

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

I think you have to just stick with your description and refer to the graph. Otherwise the explanation becomes unwieldy. A=B and B=C because if you repeated the test, there's a fair chance (more than 1 in 20) that A would score higher than B, or that C would score higher than B. But we say A>C because there's less than a 1 in 20 chance that a repeat test would show the opposite.

BTW, these results do seem to contradict the NSTL results, but they can actually both be consistent because neither yielded a clear winner between nero he-aac and wma pro 10.

Guru, my taste mirrors yours on Vorbis...anything below 80 kbps and the codec is displeasing with the artifacts. At 80 kbps, without a reference, my tin ears (a place where our similarities vanish) simply couldn't be happier. *This* is the reason that I request that we stick with the original plan and do an 80 kbps multiformat test next.

I used the key and decrypted results through java abc/hr menu Tools/Process and got 18 text files. Some resulting text files don't include all 5 ratings in text file (I rated all 5 tracks s of all 18 samples). Is this some kind of bug?

Wow, much more results than what I expected!Thank you Mares for organizing the test!Thanks to all participants for doing the test!

QUOTE (Alexxander @ Aug 16 2007, 10:46)

I used the key and decrypted results through java abc/hr menu Tools/Process and got 18 text files. Some resulting text files don't include all 5 ratings in text file (I rated all 5 tracks s of all 18 samples). Is this some kind of bug?

I also have suspicion that java abc/hr has some bugs in processing encrypted results. Just never had time to check it.

QUOTE (kennedyb4 @ Aug 16 2007, 03:58)

It seems that Itunes at 96 VBR has outscored Itunes 128 CBR from the previous multi-format test.That's a substantial improvement unless the difficulty of the samples is not comparable.

Different samples, different participants. Just look at how personal results posted here differ from the average.Results from different listening tests are just not easily comparable.

QUOTE (kwanbis @ Aug 16 2007, 02:58)

QUOTE (Sebastian Mares @ Aug 15 2007, 23:00)

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

not an expert, but at leas mathematically if A=B and B=C, A=C.

Operator = and < have in this case different meaning. If average score of A is greater than average score of B, then B=A means that there is chance greater than threshold x that in another test B could have higher average score. B<A means that the chance that in another test B is in average better than A is less than x (x is predefined by procedure used for ranking). This is roughly speaking, correct definitions would be more complicated.

I too am a bit disappointed. I would have expected a few pleasant surprises where the new codecs would have reached almost transparent listening experience. For me, only the high anchor would be usable, even though it is far from transparency.

Out of curiosity, I played some of the samples through my big & good Hi-Fi speakers. I did know that only headphones can reveal codec problems properly, but I was still surprised about how much better the encoded samples sounded through a standard stereo speaker system in a casual listening situation. I suppose that the normal room echoes get mixed with pre-echo and other codec faults and the listener's brain "calculates" subconsciously a new "combined acoustic space", which does not sound completely wrong.

WMA Pro behavior is interesting. It clearly produces more distortion than the other encoders (I mean constant distortion like an analog amp produces when it is played too loud) and behaves rather oddly with some samples. Despite these problems it was occasionally the best contender.

When the WMA Pro samples are inspected with an audio analyzer it looks like the MS developers are very optimistic about how high frequencies their codec can successfully fit in 64 kbps files. WMA Pro uses a lowpass filter at around 20 kHz. However, I suspect that the highest frequency range is more like an artificial byproduct of the MS version of "HE" than a real attempt to represent the original sound faithfully. The WMA Pro samples seem to produce quite altered waterfall displays at about 15-20 kHz when compared with the reference.

I used the key and decrypted results through java abc/hr menu Tools/Process and got 18 text files. Some resulting text files don't include all 5 ratings in text file (I rated all 5 tracks s of all 18 samples). Is this some kind of bug?

Now this is weird!

OK, I uploaded all user comments - you can either browse here or download everything as signed, solid and locked RAR. Notice that those were the comments used for evaluating. Please check if you find all five codecs rated in my decrypted result files.