Autumn 2006 Listening Test

So, for the encoder decision and settings, do you think some pre-tests are appropriate? Obviously, not all encoders can be included to have a comparison between iTunes CBR and iTunes VBR, FhG CBR and FhG VBR, etc. What I know is that iTunes VBR performed very bad in Roberto's test. That's why I thought CBR should be used now. As for FhG, I always thought their CBR engine would also be better than their VBR one, but as you guys said, this has to be checked. However, this has to be checked in some pre-tests because there are already enough contenders.

Autumn 2006 Listening Test

Did I miss the listening test(s) proving that? Do you have a link?The problem with iTunes (but correct me if I'm wrong) MP3 VBR is the following one:1/ iTunes doesn't offer a scalable VBR engine (like HELIX or Fastenc does); you can only use "VBR" as option for fixed bitrate settings (like 96 kbps VBR, 112 kbps VBR, 128 kbps VBR etc... with nothing in the middle). It's a bit like current LAME VBR scale.2/ But iTunes also uses the "target" bitrate as the minimal one per frame. In clear, if you select "192 kbps VBR", the bitrate will vary from... 192 kbps up to 320 kbps. "Low complexity" samples won't compensate for the bloating bitrate of "complex" samples. As a consequence the average bitrate of this setting could only be higher than 192 kbps.

In our case, "128 kbps + VBR" iTunes setting will for sure correspond to an average bitrate superior to 128 kbps. How greater? I can't say. But it will be superior.An alternative choice would be "112 kbps + VBR" which would correspond to a bitrate higher than 112 kbps and --who knows-- maybe as close to 128 kbps than the previous setting. But as Roberto explained it this choice would handicap iTunes encoder (as it handicap it in 2004).So assuming that iTunes 112@VBR is lower quality than 128@CBR, we still don't know how iTunes 128@VBR would perform against CBR@128. As I said, VBR@128 won't go below 128 kbps. Therefore it's hard to imaging this setting performing poorer than 128@CBR! So what we really need to know is the average bitrate of iTunes VBR at "128 kbps". If the average bitrate isn't too far from 128 kbps then iTunes VBR would easily be include in the test. If the bitrate is a bit too high (138...140 kbps) then we could easily increase a bit the setting for HELIX, FHG (and also GOGO). Then we would perform a listening test at ~135...140 kbps instead of a ~128...132 kbps one. Not a big issue in my opinion.

Quote

As for FhG, MMJB could be an option if VBR is needed. Aren't both WMP's and MMJB's MP3 encoders based on FastEnc?

Autumn 2006 Listening Test

I find it hard to believe that 6 kbps difference between iTunes and LAME can make such a difference in quality. And if iTunes VBR implementation is so good, why didn't it allocate more bits for hard parts like LAME did?

Anyways, I will let iTunes encode my music collection (electronic, rock, instrumental, pop, could add some classical music too) and see what it comes up with.

Autumn 2006 Listening Test

I think we are nitpicking regarding the title. I used the term "@ 128 kbps" because of the tradition. I can also call it "@ ca. 43 * Pi kbps".

Again, I happen to agree with guruboolez. I don't believe we're picking nits. I think that 128 kbps implies CBR and is therefore misleading. 130, OTOH, is clearly VBR. IMO, tradition is usually never a good reason to do anything and I have taken issue with this idea of passing apples off as oranges with previous tests but I don't believe I've said anything about it until now. I apologize for voicing my opion too strongly.

As far as genres and killer samples go, bitrates can vary. I don't think you'd want to change the quality setting just to achieve a certain bitrate because of a certain genre. This also reinforces the point of not putting CBR up against VBR. Perhaps using samples of metal can demonstrate this, especially where the use of VBR will result in bitrate increases of greater than 10%?

Is 24-bit/192kHz good enough for your lo-fi vinyl, or do you need 32/384?

Autumn 2006 Listening Test

So, for the encoder decision and settings, do you think some pre-tests are appropriate?

Pre-test would enlight us a lot if they're done properly (i.e. several samples, several people). A big pre-test performed by a single person has a well-known limitation, and a small-one performed by several people but on a small selection of samples may also lead to false conclusion.And what we need here is more than one pre-test. Suggestions for pre-tests:- iTunes CBR vs iTunes VBR- Helix VBR vs Helix CBR- Helix default setting vs Helix 'tuned' setting- Fhg VBR vs Fhg CBR- Fhg from {one software} vs Fhg from {another software}- LAME -V5 vs LAME -V5 --vbr-new- LAME 3.97 vs LAME 3.98

If someone start a pre-test to "extract" the best setting for one encoder (let's say iTunes as example) then it would be unfair to refuse the same treatment for all other competitors (excepted LAME maybe which is the one we're more familiar with). iTunes, Helix, Fhg... our lack of knowledge is the same for all these encoders and I wouldn't really see as fair to privilege one over the other just before the test begins. On the other side, I can't imagine how great the enthousiasm would be to the idea of performing three pre-tests as a prelude to another one dedicated to MP3 encoders.

Not to mention than even if pre-tests are done some people would still attack your choice, especially if a fight against CBR vs VBR will occur :/

Autumn 2006 Listening Test

Jesus, that's a lot of pre-tests. I am thinking of another possibility... Only thing that comes into my mind would be running two tests: one with all encoders using CBR and one with all encoders using VBR. However, this is rather complicated. We would have to use the same samples, but then the factor tester could have an impact on the results (different testers or even moods = different results).

Autumn 2006 Listening Test

I find it hard to believe that 6 kbps difference between iTunes and LAME can make such a difference in quality.

Don't forget that the bitrate table is based on short samples (selected ones, who all tend to use more bits than average). It implies that iTunes 112@VBR average bitrate would logically be lower and therefore that the difference with LAME ABR bitrate is therefore higher than 6 kbps.

Quote

And if iTunes VBR implementation is so good, why didn't it allocate more bits for hard parts like LAME did?

It's indeed a good question. Someone said once as hypothesis that the limited bitrate's variations of this encoder is linked to the (old) iPod CPU clock issue. I tend to believe that this limitation simply correspond to the will of keeping the bitrate as close to the mentioned one (128@VBR with iTunes is indeed close to 128 kbps whatever the situation and it's probably what most users are expecting from this setting).

N.B. what is interesting us at the moment is not to see if iTunes VBR is "so good" but rather if it isn't "poor" (in order to see if CBR would be more suitable than VBR). One advantage of iTunes trick (i.e. using the target bitrate as the minimal floor per frame) is at least to prevent the encoder of big encoding mistakes. I expect from iTunes CBR and VBR to be very close each others (quality-wise) simply because both modes are always leading to the same bitrate (approximately). With LAME on the contrary you can expect from VBR to sound much better on critical material (some samples could reach 200 kbps with -V5) but also to sound poorer in some cases (it was the case with quiet samples - problem partially fixed with --athaa-sensitivity 1 switch and now close to be totally fixed with 3.98 alpha 6).

Jesus, that's a lot of pre-tests. I am thinking of another possibility... Only thing that comes into my mind would be running two tests: one with all encoders using CBR and one with all encoders using VBR. However, this is rather complicated. We would have to use the same samples, but then the factor tester could have an impact on the results (different testers or even moods = different results).

I have something more easy: let people having choice of testing either CBR-only settings or VBR-only ones. A poll could be an answer.

If a majority of people are rather interested by 128 kbps CBR => test everything with CBR. Then no bitrate table fight. Only problem: people interested to optimize the quality for their DivX/Portable player... at moderate bitrate won't learn anything by a test focusing on... unoptimized settings (CBR instead of VBR).

If a majority of people are rather interested by 128 kbps VBR => test everything with VBR and discard all encoder which doesn't offer either VBR coding or simply a ~128 kbps VBR setting.

Autumn 2006 Listening Test

Thanks for accepting l3enc. Very long ago i had a few mp3's with a short burst of noise at the start, now i know why.

As for the other contenders, since we are testing encoders i believe that each encoder should contend at the most optimal settings for the target bitrate, and it doesn't matter if that's cbr or vbr, just whatever is best.

Indeed a pre-test may be a good way to determine the most optimal settings, but that really means we're doing the developers' homework.

Autumn 2006 Listening Test

Ahhh, am I too late?? I wanted to suggest slipping a standard iTunes AAC @ 128 kbps as "pseudo-high" anchor. Might be interesting to see if you could say AAC really is "out of MP3's league" or if MP3 in general (...ok, Lame ) is a good contender. Ok, shut me up...

Autumn 2006 Listening Test

Ahhh, am I too late?? I wanted to suggest slipping a standard iTunes AAC @ 128 kbps as "pseudo-high" anchor. Might be interesting to see if you could say AAC really is "out of MP3's league" or if MP3 in general (...ok, Lame ) is a good contender. Ok, shut me up...

Autumn 2006 Listening Test

As I mentioned above, the FHG Fastenc encoder forces simple stereo in VBR mode. So I suggest testing it in CBR (even if all other contenders are in VBR), because I'm sure it will perform better with CBR.

Autumn 2006 Listening Test

Guru, a poll would ease up everything, but in the end, we won't know if CBR or VBR was better for certain encoders. For this purpose, we really have the two options only: lots of pre-tests or two big listening tests.

Autumn 2006 Listening Test

Well, for small groups or individual developers (LAME, AoTuV, etc.) this is an option, but who is responsible for iTunes' MP3 encoder? Who is responsible for WMPs encoder? And I already made the experience with writing mails to big companies and the time it takes to receive a reply.

Autumn 2006 Listening Test

I don't understand the point in testing CBR when a VBR option is available. As far as I know the only advantage of VBR should be its superior quality at the same average bitrate of CBR. If, for a particular codec, VBR has a lower quality than CBR, then it's a developers fault, in that case they should have provided only the CBR mode. So I don't see the point in testing CBR when a VBR option is available and that VBR option is not tagged by the developers as "experimental" or "you should use CBR, VBR option is provided here only to be useful for that particular case....".

Autumn 2006 Listening Test

WMP doesn't come with a VBR option in first place. Maybe MS tuned the CBR encoder only and then focused on WMA - no idea. The problem with FhG encoders - as far as I can tell - is that each company might have done their own tunings unless this was prohibited from FhG's side. Therefore, you cannot say that FhG performed like this or that.

Autumn 2006 Listening Test

As I mentioned above, the FHG Fastenc encoder forces simple stereo in VBR mode. So I suggest testing it in CBR (even if all other contenders are in VBR), because I'm sure it will perform better with CBR.

It's maybe for a good reason, who knows... The feature, or this bug, is present for years. Even Audition 2.0 (Trial version) embedd a MP3 encoder forcing joint-stereo. On the other side, didn't Fastenc joint-stereo suffer from another kind of bug (stereo collapse or something like that)?

Quote

we won't know if CBR or VBR was better for certain encoders.

Quote

If you view the test as a study that should have some authorative value over which encoder is best and how they compare, a poll isn't very scientific.

It doesn't have a scientific purpose. And it's not intended to tell us which kind of coding is supposed to be better for each encoder. It's just a way for us to make a decision and to avoid inextricable problems. And yes, final decision would appear as an arbitrary one If people are interested to see how various MP3 encoders perform compared to LAME, I'd tend to say that it wouldn't be a bad idea to test them all with similar settings: VBR against VBR, CBR against CBR. VBR is per default supposed to offer a better quality (or efficiency) than CBR, especially on hard-to-encode samples (precisely what we're used to include in listening tests). HELIX, Apple, Fraunhofer... have all put energy and money to develop a VBR mode; this mode is offered to the public; there is no contraindication nor warning against VBR in the manual telling to users that VBR isn't tuned enough. And as far as I can remember, we didn't mail to Apple developers neither the Microsoft ones to see if we had to prefer CBR over VBR with iTunes AAC or WMAPro (last 130 kbps multiformat listening test).

I'm not against the idea of questionning developers themselves, but as you said, answer may take a long time. I'm not against making a pre-test, but it looks like filling ourselves a hole in the encoder/software's manual (and I'm very sceptical about the success of such pre-tests -- which are requiring as energy as the final test: we can't count on a pre-test to help us and spoil it by a lack of samples/testers). The idea of having two distinct tests (one for CBR, another for VBR) is also a good one, but again I'm not sure that many people are interested to spend free time to test twice what is often considered as an outdated format (MP3), including what is often perceived as outdated encoders (Fhg, iTunes), with one test dedicated to what is considered with right as a wrong coding method (CBR).

Autumn 2006 Listening Test

And as far as I can remember, we didn't mail to Apple developers neither the Microsoft ones to see if we had to prefer CBR over VBR with iTunes AAC or WMAPro (last 130 kbps multiformat listening test).

WMAPro is developed entirely by Microsoft and WMP / WME offer VBR encoding. WMP's MP3 encoder on the other hand is CBR only. There is no VBR option provided. Using MMJB or AA is an option, but just like for WMP and its CBR encoder, the encoders are not only FhG, but FhG + company tunings.