ABX testing vs. unconscious perception/decision, enjoyment, and so on

2012-03-16 11:31:57

SOME THOUGHTS

All this talk about skin “hearing” music, “feeling” the vibrations of the music etc raises for me a separate debate.

Whilst I completely agree that ABX is an excellent tool for spotting consciously detectable differences (ie respondents must be wilfully aware of the difference to make their choice) what about our unconscious decisioning? Just the other day their was a BBC documentary “out of control” which they were explaining how your unconscious mind is often calling the shots and influencing the decisions you make.

So we have 3 different types of tests:1) Signal waveform analysis (as was used to check to see if JPlay was doing anything or FLAC v WAV using Audio DiffMaker2) ABX audio testing (man in the middle) for checking for conscious discrimination of different samples3) Enjoyment factor (or some other form of what some philosophers call qualia)

I am not saying to know that our skin reacts to sound or anything like that however I can believe that there is more to our enjoyment of music than just what our ears hear, for example I was at a classical concert yesterday and we slipped from the back to some seats in the front where we could feel the power of the music more in our bellies. Is this just due to the increased volume and would our ears have provided our brains with sufficient information if I had desensitised my stomach – I am not so sure. Now clearly in this example I was conscious of this effect, but there may be others I was not and yet still added to my overall enjoyment.

So even assuming that enjoyment is a wholly conscious act there could be unconscious inputs that the brain turns into conscious states. However these states could be impacted by the ABX test in 2 ways:A. Not enough time is allowed for in playing the samples for these unconscious inputs to take effectB. The act of enjoying a concert vs the act of trying to spot differences mean the brain is trying to do different things and thus doing the second might not require the same inputs as used in the first (I assume we are happy that the outputs of those 2 exercises are likely to be different and in a different form

Now bear in mind that for this thread I am in the 16/44 is sufficient for playback camp – I am still saying I am not fully satisfied that tests (1) and (2) are sufficient to provide complete proof. It is a bit like evolution (as some other poster mentioned). There is some proof for the theory, there are many wholes (for example in the fossil line) however there is no proof for any other theory...

ABX testing vs. unconscious perception/decision, enjoyment, and so on

I was at a classical concert yesterday and we slipped from the back to some seats in the front where we could feel the power of the music more in our bellies. Is this just due to the increased volume[?]

It is a bit like evolution (as some other poster mentioned). There is some proof for the theory, there are many wholes (for example in the fossil line) however there is no proof for any other theory...

Implying that audio testing is just as vague as evolution does a gross disservice to both fields.

ABX testing vs. unconscious perception/decision, enjoyment, and so on

Don't assume any limitations to how ABX testing is applied. The only real limitation is that it needs to be applied double-blind, i.e. there can not be any means other than the effect being tested to differentiate A from B from X.

Your reference to conscious decisions isn't even a limitation. There is no reason that the subject even be aware that he is taking an ABX test if the observer can evaluate his reactions without them being verbalized. Similarly there is no requirement that the testing be within some time period. Testing could take place over months or years and still be just as valid.

Finally, I will point out what has been said many many times before. ABX testing cannot prove that there is no difference between A and B. It can only provide reasonable certainty that there is.

ABX testing vs. unconscious perception/decision, enjoyment, and so on

I was at a classical concert yesterday and we slipped from the back to some seats in the front where we could feel the power of the music more in our bellies. Is this just due to the increased volume[?]

Yes.

but that does not answer how exactly that increased volume is captured by the brain

Your reference to conscious decisions isn't even a limitation. (1) There is no reason that the subject even be aware that he is taking an ABX test if the observer can evaluate his reactions without them being verbalized. (2) Similarly there is no requirement that the testing be within some time period. Testing could take place over months or years and still be just as valid.

I really like both of these and both of these were in my head when I wrote the first post. However what I could not decide is how I capture the results from the tests and how and when I ask for their answers.

However I disagree with your statement that I have now put in bold. My reasoning is as per my first post. What you do with your suggestions goes someway to address the problem, but we need some other way of capturing responses for a comparison

ABX testing vs. unconscious perception/decision, enjoyment, and so on

There are many ways that one could envision determining a subject's response without asking him/her. Just off the top of my head, someone may subconsciously avoid coughing if one is enjoying what one is listening to. Count how many times the subject coughs in a certain time period and use that as the basis of the test.

ABX testing vs. unconscious perception/decision, enjoyment, and so on

pdq - I agree that there should be a way to measure it, I am just not clear on the specifics that will work for our cases.As I said in the now OP, I am comfortable assuming we are interested (at least at this stage) is conscious stages of mind, even if these are driven from unconscious inputs. So I "know" that I am happy, even if I don’t know what exactly I am experiencing to make me happy.

ABX is looking at a different conscious stage, rather than "I am happy" it is looking at "I am spotting a difference". We can agree that it is possible for 2 different samples to trigger the second in the specific for of "I am spotting a difference in these 2 samples". Whereas you seem to be wanting to use it to also answer the first. I suggest that this is really the second state in a different form "I am spotting a difference in the happiness state created by these 2 samples". This is slightly different from what we wanted to measure.

That is the act of us comparing our own happiness states actually changes our state of mind which could be stopping us conducting the measurement we actually wish to measure.

Your example of coughs is an example of a world we could explore to get around this, but as I started off this reply by saying, I am not clear on the approach we should take.

ABX testing vs. unconscious perception/decision, enjoyment, and so on

I was at a classical concert yesterday and we slipped from the back to some seats in the front where we could feel the power of the music more in our bellies. Is this just due to the increased volume[?]

Yes.

but that does not answer how exactly that increased volume is captured by the brain

There is this book named "This is Your Brain on Music". Answers your question.

ABX testing vs. unconscious perception/decision, enjoyment, and so on

I agree that being engrosed in a listening experience and trying to spot the difference between two sources is a different application of the intellect.

That said, faulting bias controlled listening tests on the grounds that there is a difference between them and a listening experience at a live venue (obviously true!) is an example of a false comparison.

Nobody brings a pair of speakers to a venue, and compares them to the live performance, except maybe guys like me who record them. Then we mostly use headphones for pretty obvious reasons.

If you want to compare ABX to something relevant, compare it to a listening comparison done at a dealer's or a friend's.

ABX testing vs. unconscious perception/decision, enjoyment, and so on

Don't assume any limitations to how ABX testing is applied. The only real limitation is that it needs to be applied double-blind,

The ABX test method and the double-blindness are distinct concepts. The former is a type of experiment, the latter is a method to address and mitigate a certain type of bias. You can have double-blind experiments that are not of the ABX type, and you can have ABX with or without double-blinding. Question is rather, which one(s) of these four combinations (ABX or not, double-blind or not) it makes sense to use.

If we are facing the issue of use of A/B tests which are prone to placebo-type bias, then a very modest request for improvement would be: Suggest the smallest possible blinded (i.e. 'blindable'!) modification to sighted A/B. The mitigation of placebo is an end, the double-blinding is a well-established means to that, and ABX is ... well, basically iterated comparison, so it passes as a small modification to AB. Problem solved.

(In addition to 'ABX in place of A/B' and 'double-blind', this forum also would expect a standard to statistical procedure to evaluate the scores before drawing inference from them. That is a third distinct concept!)

i.e. there can not be any means other than the effect being tested to differentiate A from B from X. [...]Testing could take place over months or years and still be just as valid.

Ah, well, in principle. In practice, you could easily violate the ceteris paribus desideratum, and it would be hard to get sensible results. There is an entire branch of the statistical science dedicated to the design of experiment.

ABX testing vs. unconscious perception/decision, enjoyment, and so on

I only mention testing over longer periods to counter the usual argument against ABX that it doesn't take into account long-term effects. If the length of the test is too short to take into account long-term effects then that is a limitation on how the test was applied, not in the methodology itself. Clearly it would be difficult to avoid other variables if the testing period were as long as years although random variation of the stimulus would help to alleviate that.

ABX testing vs. unconscious perception/decision, enjoyment, and so on

SOME THOUGHTSSo even assuming that enjoyment is a wholly conscious act there could be unconscious inputs that the brain turns into conscious states.

One prime example of a proven and often unconscious input is expectation bias.

So the big problem is if you suppose that there are things that are perceptible only unconsciously and that can't be determined through ABX/DBT, how do you test for them with any reliability?

And if you can't get any sort of reliable evidence, does it make sense to suppose that such things really exist? Based on what?

You can certainly theorize that something might possibly exist, but should also ask, "How likely is it that something is only unconsciously perceptible in ways that specifically and effectively evade detection in controlled testing?".

ABX testing vs. unconscious perception/decision, enjoyment, and so on

@drewfx I think you are missing what I am saying. I am not saying that "there is a flying spegetti monter or a tea pot orbiting the moon it is just we cannot see them". I am simply saying that there is a chance that ABX/DBT where the listener is having to make a choise is not the right way to verify the unconscious elements.

As KMD says, listening for enjoyment and listening for a test are 2 different exercises. We (I) would like to know a method that can measure the difference between 2 samples when the listener is trying to listen for enjoyment. Maybe a EEG, maybe something else, but I would like to know.

ABX testing vs. unconscious perception/decision, enjoyment, and so on

I am simply saying that there is a chance that ABX/DBT where the listener is having to make a choise is not the right way to verify the unconscious elements.

But you are supposing here that there are (or might be) otherwise undetectable unconscious elements. My point was, based on what?

It seems to me you are trying to improve the testing procedure to catch test cases that might not exist. Now that's not necessarily a bad thing, as ideally testing would catch everything conceivably possible.

But in the case of audio, it plays into the belief many people out there have (and I definitely don't mean to imply you're one of them) that "ABX/DBT is flawed in just such a way that it can't detect all this stuff that I know I can hear!".

ABX testing vs. unconscious perception/decision, enjoyment, and so on

Don't assume any limitations to how ABX testing is applied. The only real limitation is that it needs to be applied double-blind,

The ABX test method and the double-blindness are distinct concepts. The former is a type of experiment, the latter is a method to address and mitigate a certain type of bias. You can have double-blind experiments that are not of the ABX type, and you can have ABX with or without double-blinding. Question is rather, which one(s) of these four combinations (ABX or not, double-blind or not) it makes sense to use.

If we are facing the issue of use of A/B tests which are prone to placebo-type bias, then a very modest request for improvement would be: Suggest the smallest possible blinded (i.e. 'blindable'!) modification to sighted A/B. The mitigation of placebo is an end, the double-blinding is a well-established means to that, and ABX is ... well, basically iterated comparison, so it passes as a small modification to AB. Problem solved.

The above paragraph pretty much describes our state of mind when we invented ABX.

Our first attempt involved creating a hidden list of As and Bs that were assigned randomly before the test and maintained throughout the test.

We'd play the alternatives in accordance with the hidden list and ask people to respond by writing down on a personal piece of paper first which one they liked the most, and later on whether they thought that List Item N was A or B. We made this change of direction because people complained that they thought they could hear a difference, but the alternatives were too much alike to create a preference.

This was found to be too hard because it put too much burden on our memory for the subtle details of what things sounded like.

Then we provided sources known to be A and B as references that the listener could compare to the unknown (which we started calling X) as many times as desired. That was ABX and it always seemed to be wonderfully easy compared to what went before it.

ABX testing vs. unconscious perception/decision, enjoyment, and so on

The above paragraph pretty much describes our state of mind when we invented ABX.

I tried to be very careful not to make any claims on precisely what was actually intended or thought when it originally came about (or when it found its way into this forum's terms).

(And I actually wrote, but deleted before submitting, that I think ABXing is very modest indeed, as the typical mode of the unscientific layman (and the ingorant professional) would be to 'compare A to B and report the difference'. I was totally unaware that you actually attempted a higher level of ambition first. But this goes into a pet principle of mine as a ... well, professional geek: if you try to explain differences, don't forget to check whether there is one. Otherwise, be aware that you are basically doing what-if analyses (which is perfectly fine, for example for precautionary purposes, as long as you are aware of it).)

ABX testing vs. unconscious perception/decision, enjoyment, and so on

If your primary listening situation is sighted, and not blind, and if seeing the music coming from a nicely furbished, expensive amp & HD source alters your perception of the sound, go for it! That alteration is real and modern science would not disagree. You will very likely not be able to tell it apart from a regular CD player and amp, when blindfolded, but you are usually not going to be listening while being blindfolded. So what?

ABX testing vs. unconscious perception/decision, enjoyment, and so on

As KMD says, listening for enjoyment and listening for a test are 2 different exercises. We (I) would like to know a method that can measure the difference between 2 samples when the listener is trying to listen for enjoyment. Maybe a EEG, maybe something else, but I would like to know.

Unless you believe that you're somehow more sensitive to artifacts when listening for enjoyment then actively looking for artifacts, I don't see that it matters. If I concentrate and have a reference and still hear nothing, then I am quite confident that under my normal listening conditions I will also hear nothing. I think its universally true that differences become more apparent under an A/B comparison, not less.

ABX testing vs. unconscious perception/decision, enjoyment, and so on

If your primary listening situation is sighted, and not blind, and if seeing the music coming from a nicely furbished, expensive amp & HD source alters your perception of the sound, go for it! That alteration is real and modern science would not disagree. You will very likely not be able to tell it apart from a regular CD player and amp, when blindfolded, but you are usually not going to be listening while being blindfolded. So what?

I have a better suggestion:I am fairly certain that the FIFO buffer makes my DAC immune to any jitter from my $chickens**tmoney digital output. Knowing that the mumbojumbo segment of self-proclaimed audiophiles will have spent $$elephantsize on some oversensitive ill-constructed DAC, just because it 'reveals all the differences' between digital outputs, and then spent $$mammoth on a dejitter/reclock box which basically has a RAM the size of a 90's telephone, that gives me the pleasure of 'Gawd, what a bargain knowledge can buy' any time I hook up to my fb2k, and if I want to listen to Mahler or funeral doom, I can switch to 'heck, what has mankind come to be?'.

I know I shouldn't pull a 'my placebo is better than your placebo' without supplying the ABX logs to prove it, so let me stick to 'your placebo is not better than mine, and mine is much cheaper'.

ABX testing vs. unconscious perception/decision, enjoyment, and so on

I am fairly certain that the FIFO buffer makes my DAC immune to any jitter from my $chickens**tmoney digital output.

It's nice, that a simple FIFO component is able to boost your musical enjoyment and assurance. It certainly is placebo, though. A FIFO can buffer samples, but the clock rate must still be reconstructed from the S/PDIF input signal, and is thus prone to jitter. S/PDIF doesn't transmit timing information as a discrete value but implicitly and eventually analog.

ABX testing vs. unconscious perception/decision, enjoyment, and so on

... the clock rate must still be reconstructed from the S/PDIF input signal, and is thus prone to jitter. ...

Er... not directly. The output clock is derived from the state of the FIFO buffer. It does vary slowly, but short-order intersample variations on the clock derived from the input S/PDIF signal are effectively suppressed. Consider the FIFO as a low-pass "jitter filter" with a cutoff frequency set by the size of the FIFO.