The signal analysis in the ITS report suggests ATRAC3 being far superior to mp3 and wma, whereas the subjective listening tests in test 2 suggest it is the worst.

Can signal analysis be relied on, when it comes to subjective perception of audio signals ?Is it possible, that a bad codec colours the sound file in such a way, that it actually sounds better but not authentic ? Perhaps a good codec should not sound good, but reproduce in a transparent way. What do you think ?

Can signal analysis be relied on, when it comes to subjective perception of audio signals ?

No, at least not without taking into account certain aspects of the faculties which color its perception. We aren't testing for signal level accuracy, we are testing for how it appears to sound. There is not a linear correlation between the two.

QUOTE

Is it possible, that a bad codec colours the sound file in such a way, that it actually sounds better but not authentic ?

Sure.

QUOTE

Perhaps a good codec should not sound good, but reproduce in a transparent way. What do you think ?

Well then all you're doing is redefining the term "good codec." You can define it to mean whatever you want, whether that means objective accuracy, or subjective accuracy. What the judgement hinges upon then is what is important to you. Most people on this board will probably choose sound quality over pretty looking frequency sweeps, but then your mileage may very.

Note though that in your case "good codec" has nothing to do with perceived sound quality and is simply a rather arbitrary designation.

In my opinion, what is important is that it "sounds good" and "sounds transprent". If these conditions are met then whether or not the encoded signal deviates more statistically from the original signal in comparison to another so-called "good codec" is not important (in fact, to favor that approach would be to miss the entire idea behind psychoacoustic lossy audio compression).

We can notice also, that their subjective tests both show ATRAC3 132 kbps superior to MP3 128 kbps, with proper blind listening methods, and with confidence intervals of 95 %. If I'm not mistaken, MusicMatch Jukebox uses Fraunhofer encoder, like CoolEdit. The MP3 files they got being slightly inferior to ATRAC, it shows the overwhelming superiority of the Lame VBR mode used in Roberto's test over the CBR fraunhofer version they used.

Oops, I missed a serious flaw in the two tests linked above : one of them was run with 4 musical samples given by Sony, and only 1 chosen by the lab, and in the other, 7 samples were chosen by Sony, while 3 were chosen by the lab.It is likely, Sony having very big interests in the ATRAC success, that they gave musical samples especially chosen so as to advantage ATRAC over other codecs. Thus there might be a strong bias for ATRAC in both tests.

Oops, I missed a serious flaw in the two tests linked above : one of them was run with 4 musical samples given by Sony, and only 1 chosen by the lab, and in the other, 7 samples were chosen by Sony, while 3 were chosen by the lab.It is likely, Sony having very big interests in the ATRAC success, that they gave musical samples especially chosen so as to advantage ATRAC over other codecs. Thus there might be a strong bias for ATRAC in both tests.

heehee. How clever of them.

Besides, I don't believe in paid-for tests mentioned at the company's site. They could well have asked 20 labs to test their codec, and only choose to announce the results that hailed it as winner.

--------------------

Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:http://www.rarewares.org

I've been lurking for awhile, but I think it's time to speak up a bit.

I think there has been a somewhat undeservedly negative view taken towards ATRAC3 AND MiniDisc in general, without a real understanding of why ATRAC3 is the way it is and what it is targeted to.

I felt that the multiformat128 test result was probably accurate in its results, but I don't feel that the test was particularly fair for ATRAC3, and I also feel that no real conclusions can be made directly regarding the performance of ATRAC3 (MDLP) on an actual MD player.

With regards to an actual MD player, assuming that the DSP on the minidisc player is in use, I'd like to note that:

1. ATRAC Type S, used in the newer lines of MD recorders is actually a different encoder (and decoder?) for MDLP/ATRAC3 compared to the one used originally when the Sony R900 series came out, and it purportedly improved quality. In addition, though it is compatible, I don't think it is the same as the software codec that is used by SonicStage on the PC, which more than likely uses something similar to the original algorithm used in the first generation of MDLP recorders. Unfortunately, this makes it difficult to truly compare performance when it comes to MD players versus MP3 players.

2. On a personal note, I've always found 132kbps ATRAC3/LP2 mode to be superior to CBR 128kbps MP3 (assume Frauenhofer). Perhaps, more equivalent to 160kbps. I don't have the ABX test to show that personally onhand, but I find that the main difference is that ATRAC3 does not suffer from upper-frequency artefacting nearly as much. It might be that ATRAC3 changes the sound from the original, but it certainly less grating to my ears compared to CBR MP3 at less than 160kbps.

3. Why is it NOT so fair to compare Minidisc ATRAC3 to VBR MP3?The whole reason of ATRAC3's CBR has to do with backwards compatibility to a technology that is over 10 years old! I believe that because of CBR and its less strenuous requirements on hardware design and flexibility, Sony and other manufacturers have been able to achieve much better battery life and editing flexibility with MiniDisc players compared to competitors, along with much a cheaper storage medium (at this point) on which to store music. ATRAC3 had to conform to a preexisting standard and is aimed at a certain specific use that the other codecs don't really do.

Note that as it is, today, for most music you still find 128kbps CBR MP3 as the standard, so I wouldn't rap on the MD users for saying that they find it to be superior to MP3, because in general, for most of the music they've heard, it probably IS better. Yup, maybe they're a little less informed than the crew here.

I won't argue that on today's terms, that if you were going to choose the best-quality codec with which to store your music on a computer or hard disk at ~128kbps for some reason, that you should choose the ATRAC3 software encoder. Certainly, the other VBR codecs will do a better job on a computer. However, in use for a portable device, I think it's still quite competitive for what it can do.

Instead of it, I tell you that even Sony is pushing Sonic Stage 2 as the main Atrac encoder nowadays. First of all, I higly doubt that record companies use hardware encoders to encode their files for Connect instead of that they propably have some software solution which is based on the same software encoder as Sonic Stage. And! The new Hi-MD devices are all about transferring music to device from CDs using Sonic Stage. And the all mighty and definite answer: This thing was discussed about in the threads of the listening test! You should read them before posting.

Instead of it, I tell you that even Sony is pushing Sonic Stage 2 as the main Atrac encoder nowadays. First of all, I higly doubt that record companies use hardware encoders to encode their files for Connect instead of that they propably have some software solution which is based on the same software encoder as Sonic Stage. And! The new Hi-MD devices are all about transferring music to device from CDs using Sonic Stage. And the all mighty and definite answer: This thing was discussed about in the threads of the listening test! You should read them before posting.

Whoops... bad spelling mistake... corrected... :-)

I did read the threads of the listening test (though admittedly not in the past 2-3 weeks) but I felt that people were making comments that weren't particularly in context or fair regarding ATRAC and MiniDisc in particular.

When it comes down to Hi-MD we can't make a conclusion either as none of us in our right minds would be using ATRAC3 MDLP2 modes anymore. I don't particularly care about what Sony tells me to use - otherwise I'd be encoding everything at 64kbps in LP4 mode, believing that I've got 320 minutes of listenable music on my MD. We haven't done any testing on ATRAC3Plus at 256kbit modes so I don't think it's easy to conclude.

1. Creating an high quality audio codec for hardware is much more difficult than for software, due to constraints as complexity, battery life etc. as you pointed out yourself under item 3. Why on earth would Sony cripple their software to be worse than this? Wouldn't they wanna showcase their codec at their best? It would be the worst marketing ever to actually reduce quality.

2. Well, claims without proof usually isn't accepted around here. Why not just perform an ABX test yourself quickly, and you will get taken much more seriously here. And personally I can't understand howcome a codec that (maybe) colors the sound (as you speculate yourself) can be better. To me the only thing important is staying as close to the original.

3. For me this test is as fair as it gets. Its about audio quality and nothing else. Maybe the MiniDisc players do have excellent battery life and cheap storage, but how is that relevant to audio quality? If we were to take such things into account in these kinds of tests, any ranking would be impossible. And in addition to portable use, the test was also aimed at finding out what online music stores uses the highest quality audio format, of which ATRAC is what is used in Sonys music store (don't remember the name).

3. Why is it NOT so fair to compare Minidisc ATRAC3 to VBR MP3?The whole reason of ATRAC3's CBR has to do with backwards compatibility to a technology that is over 10 years old! I believe that because of CBR and its less strenuous requirements on hardware design and flexibility, Sony and other manufacturers have been able to achieve much better battery life and editing flexibility with MiniDisc players compared to competitors, along with much a cheaper storage medium (at this point) on which to store music. ATRAC3 had to conform to a preexisting standard and is aimed at a certain specific use that the other codecs don't really do.

Note that as it is, today, for most music you still find 128kbps CBR MP3 as the standard, so I wouldn't rap on the MD users for saying that they find it to be superior to MP3, because in general, for most of the music they've heard, it probably IS better. Yup, maybe they're a little less informed than the crew here.

And MP3 is not an ancient spec Last I checked it was also more then ten years old and filled with stupid limitations like the sfb21 problem. How does this make it unfair to compare to ATRAC3? If anything i think that ATARC looseing to such an outdated codec just make's ATRAC look less appealing, certainly not more as you seem to imply.

QUOTE

On a personal note, I've always found 132kbps ATRAC3/LP2 mode to be superior to CBR 128kbps MP3 (assume Frauenhofer). Perhaps, more equivalent to 160kbps. I don't have the ABX test to show that personally onhand, but I find that the main difference is that ATRAC3 does not suffer from upper-frequency artefacting nearly as much. It might be that ATRAC3 changes the sound from the original, but it certainly less grating to my ears compared to CBR MP3 at less than 160kbps.

Check out rule #8 from the TOS:

QUOTE

8. Any statement about sound quality must be supported by the author responsible for such statements by a double blind listening test demonstrating that he can hear a difference, together with a test sample.

Perhaps you could post results? Afterall we already have a lot of double blind test results saying ATRAC is a lot worse then LAME. Some fhg tests on the samples you're refering to would be interesting to look at.

1. Creating an high quality audio codec for hardware is much more difficult than for software, due to constraints as complexity, battery life etc. as you pointed out yourself under item 3. Why on earth would Sony cripple their software to be worse than this? Wouldn't they wanna showcase their codec at their best? It would be the worst marketing ever to actually reduce quality.

2. Well, claims without proof usually isn't accepted around here. Why not just perform an ABX test yourself quickly, and you will get taken much more seriously here. And personally I can't understand howcome a codec that (maybe) colors the sound (as you speculate yourself) can be better. To me the only thing important is staying as close to the original.

3. For me this test is as fair as it gets. Its about audio quality and nothing else. Maybe the MiniDisc players do have excellent battery life and cheap storage, but how is that relevant to audio quality? If we were to take such things into account in these kinds of tests, any ranking would be impossible. And in addition to portable use, the test was also aimed at finding out what online music stores uses the highest quality audio format, of which ATRAC is what is used in Sonys music store (don't remember the name).

1. Yes, creating a hardware codec is certainly difficult... but Sony has two sides which have constantly been at each other's throat these days: one is the technology, and the other is the music industry, so really, you could argue that Sony's music / software side has no interest in improving the codec, as they don't want people just distributing their ATRAC files everywhere...

Obviously, on the other hand, they'd like to show their best hand (which hand?), but I think making it exclusive to the hardware can actually make it more attractive to purchase it.

Regardless of this speculation, the fact remains that they changed their ATRAC3 codec from the original hardware and we do not know whether the software version was updated at the same time.

2. An ABX test is something I will certainly try and set up when I have time. In regards to colouring sound, I mean that it is coloured in a way that is much less obvious/distracting compared to a Fraunhofer CBR codec at 128kbps, at least to me. Obviously, I can't make any real claims at this point according to the forum rules.

3. If the whole point is to compare quality (regardless of application), I would accept the results without reservation. On a quality basis, I do accept the results reached. The issue is that really, I don't think ATRAC3's design goals really fit the same profile as any of the others. You're right, this makes it very difficult if not impossible to compare, especially if it comes down to hardware. For portable use, I really do think other factors come into play and that quality must be balanced at times against them. Otherwise, we'd all be walking around with portable tube amps and battery backpacks.

On the side, if one of the points here was to compare online music store codecs, then I don't think that was achieved by including VBR LAME MP3 (never mind Vorbis or MPC even), as it is not being used by any of them (aside from allofmp3.com, possibly, though the legality of that service is in question).

As a result of this test, everyone has taken a liking to criticizing users of MDs (and of course, I'm one of those) because the sound quality is obviously inferior to some available solutions... but comparably speaking, I believe that most of these MD users are talking from a 128kbps Fraunhofer MP3 CBR (which is what the majority of MP3s have been encoded in) vs. ATRAC3 132kbps situation.

To Mike Giacomelli:

I see nothing wrong with comparing ATRAC3 to MP3... but I do think the settings and codec used are important. MP3 is old too, but it wasn't designed with particular hardware specifications in mind, and certainly it didn't really show up until much later than MiniDisc did. Note that LAME doesn't count in this case since it is hardly the age that MD is on those terms. Also note that MiniDisc took off in Japan/Asia in general when it was introduced, so that's why I'm saying that it had a presence compared to MP3, which wouldn't have been useable at the time at all because computers that were powerful enough to decode MP3 weren't readily available. Also, as mentioned in the listening results thread, ATRAC3 can't simply just be constantly developing, as it is tied to hardware, so to try to compare something that hasn't had the chance to develop...

Again, I'm NOT comparing ATRAC3 vs LAME... I'm saying that a more useful comparison is ATRAC3 vs. Fraunhofer, if we want to understand why people have argued that MiniDisc is superior, as opposed to just putting them down as unreasonable. LAME is better in many ways, but I'm trying to compare things from a more real-world applied point of view. I guess I should do LAME CBR as well as Fraunhofer. Something to put on the list.

Generally, I think most of this will be pretty moot as standards reach higher bitrates, 256kbps and above, as they all seem to be heading. The only reason NOT to go higher is so that the music industry can continue to think they are not just giving away all their money if they use higher bitrate songs, in which case this is their justification for limiting the quality of their songs distributed online.

I also use MiniDisc on a fairly regular basis, although I have a Sharp unit. This hasn't even been taken into account, as early on it was pretty much accepted, as far as I know, that the Sharp DSP (especially version 6 and higher) were better than the Sony DSP's available at the same time. I haven't seen any proof of this however, just heresay, and as for today's DSP's, I have no idea which is better.

That set aside, I would be interested in some test sample of ATRAC3+ (the codec used in the new Hi-MD units) at 256kbit (perhaps someone with a Hi-MD can record it via optical and post a sample?).

Also I think it should be mentioned that no one seems to use ATRAC, ATRAC3 or ATRAC3+ as a codec for storing music on their computer (in general...), while MP3 is widely used. I for one use my MiniDisc unit (and therefore use ATRAC) for running or other active outside uses, which introduces a lot of outside noise, which brings me to my final point....

I think it would be interesting to run some sort of ABX test in a noise outdoor environment to find out which codec really performs the best in this situation -- an active environment with a reasonable amount of ambient noise/background noise... maybe like on a train or bus (as I would imagine this is where most users use their portables...)

Maybe then we could see which is the best portable codec, or if they all end up sounding generally the same, due to the extraneous noise.

I think it would be interesting to run some sort of ABX test in a noise outdoor environment to find out which codec really performs the best in this situation -- an active environment with a reasonable amount of ambient noise/background noise... maybe like on a train or bus (as I would imagine this is where most users use their portables...)

All other conditions being relatively equal (bitrate, etc.), I think it should pretty much always be the case that the codec that performs best in a quiet environment is also going to sound the best in a noisy environment, unless something really strange is going on.

The noise might diminish ones ability to discern deviations in quality compared to the original sample, but I don't think it should fundamentally change the sound in a way which makes the kind of difference you're talking about.

it's conceivable that in a particular noise environment the spectral nature of the noise might mask some corresponding deficiencies in the codec.

for example the harmonic whine of a jet engine could well make ATRAC listenable...

btw, does anyone have some tech details for ATRAC/ATRAC3? in particular where the scalefactor bands lie, and things like that. from looking at spectrograms (please don't hit me), ATRAC3 seems derivative of mp3 in many respects, whereas ATRAC (SP mode) is a completely different beast.

of course that says nothing of the sound quality - i've found ATRAC3 to sound better than ATRAC in most cases, but i've only really dealt with human speech with MD sources.

*ATRAC3 is NOT backward compatible with ATRAC*MP3 standard was finalized in 1992*ATRAC3 was introduced several years after

It is quite right in the sense of a CODEC, but not in the sense I was talking about. I was addressing the fact that ATRAC3 had to be useable on a hardware platform 10 years old, i.e. MiniDisc (which just happens to use ATRAC).

Even though ATRAC3 is new(er), it is restricted on a fundamental basis by its tie-in to the MiniDisc platform, which is relatively quite old.

One of the more obvious results of this restriction was that even though ATRAC3 should have been half the bitrate of MiniDisc (292kbps), it ended up being at 132 instead of 146, with 14kbps wasted as blank space in the name of compatibility with older MD players (so they wouldn't play noise).

Oops, I missed a serious flaw in the two tests linked above : one of them was run with 4 musical samples given by Sony, and only 1 chosen by the lab, and in the other, 7 samples were chosen by Sony, while 3 were chosen by the lab.It is likely, Sony having very big interests in the ATRAC success, that they gave musical samples especially chosen so as to advantage ATRAC over other codecs. Thus there might be a strong bias for ATRAC in both tests.

You've maybe missed this other important flaw:

CODE

1.3 - Listening test material

All Atrac3 and Atrac3plus files were encoded and converted by Sony Corporation. The Laboratory takes on trust that these files were encoded with good intention and at the correct bitrate.

Highly scientific approach Samples came from Sony, encoded material from Sony, headphones from Sony. I wonder if listeners are not also from Sony.

Anyway, they didn't precise the mp3 settings used. Are files in full-stereo like Atrac3 at the same bitrate? Could be possible.

Other important thing to precise: the frequency graphs at the end of the report (APPENDIX III) are showing a strange cut-off behaviour of atrac3plus:- 20 Khz lowpass with atrac3plus@48 kbps- 21 Khz lowpass with atrac3plus@64 kbps- 22 Khz lowpass with atrac3@132 kbpsIf you look on the lowpass used in commercial atrac3/atrac3plus encoders, these values are totally different- ~18 Khz for atrac3@132- not sure for the other, but it must be something like 16 KHz for 64kbps and 14 KHz for 48 kbps.

These graphs are very important: they are proving either that encoders used for this test are totally different from encoders available for MD users via SonicStage or they are proving that Sony didn't respected the bitrate asked for this test.

It is quite right in the sense of a CODEC, but not in the sense I was talking about. I was addressing the fact that ATRAC3 had to be useable on a hardware platform 10 years old, i.e. MiniDisc

Wrong.ATRAC3 is not usable on older minidisc players.

I apologize for my use of the word 'useable' in a different manner, as ATRAC3 is indeed not possible to LISTEN to on older hardware, but if you read my argument, what I am trying to say is that ATRAC3 still has to fit the MiniDisc standard mold.

So even on an old player, an ATRAC3 track *WILL PLAY*, though it will be mute, as opposed to simply being dysfunctional and causing errors or noise.

These graphs are very important: they are proving either that encoders used for this test are totally different from encoders available for MD users via SonicStage or they are proving that Sony didn't respected the bitrate asked for this test.

Wow ! I would never have noticed this ! Are you sure that ATRAC3 lowpasses everything at 18 kHz ? Couldn't it be an apparent lowpass caused by the ATH, like the apparent 16 kHz lowpass of MP3 APS -Y ?

Could only talk about music encoding: but I never seen any sample, any bit, any information beyond 18...19 KHz. I seriously doubt that dedicated signal tones could force the same encoder to reach higher values.Even with APS -Y, it's easy to find short parts going with maximum lowpass on regular music.

Could only talk about music encoding: but I never seen any sample, any bit, any information beyond 18...19 KHz. I seriously doubt that dedicated signal tones could force the same encoder to reach higher values.

Have you checked any hardware implementations? We can't say that they are spreading disinformation before we have checked all the possibilities.