I've noticed an increase in forum debate about the validity of transferring the credibility of ABX from the physical domain to perception testing. I'm wondering if anyone has found a way past this issue?

The purpose of blind testing is to subtract subjectivity from the effect of - for instance - a drug trial: to assess a medication's impact on a subject's physiology with interference from their psychology. But what about when the purpose of a test is subjective perception? How do we then subtract the effect of the method to arrive at a meaningful outcome?

While we would like to remove expectation bias from the equation, if the conditions under which this is done also change the perceptive state of the listener, the test is invalidated as surely as they would be by tissue sample contamination.

Recent large scale public experiments by Lotto Labs (http://www.lottolab.org/) demonstrated that perception acuity is dramatically altered by test conditions: for instance, that time contraction/dilation effects are experienced when exposed to colour fields. In one experiment, two groups were asked to perform an identical fine-grained visual acuity test. One group was pre-emptively 'manipulated' by filling in a questionnaire designed to lower their self-esteem. This 'less confident' group consistently performed worse on the test that the unmanipulated one: their acuity was significantly impaired by a subtle psychological 'tweak' that wasn't even in effect during the test.

It seems undeniable that the much grosser differences between the mental states of sighted and 'blind' listening - considered generously - cast serious doubt on the results thus obtained.

The harder line is that blind perception tests are a fundamental misappropriation of methodology. In psychology it's axiomatic that for many experiments the subject must be unaware of the nature of the test (see Milgram). If a normalised state is not cunningly contrived, results are at best only indicative of what a subject thinks they should do; at worst, entirely invalid.

Probing hearing, the point is that a test must not change the mental state of the listener.

The contrast between outcomes of sighted and listening tests is as stark as those demonstrating suggestibility (see McGurk), but giving too much credence to such an intrinsically unsound experimental approach (not spotting this difficulty) does no favours to our credibility at all.

The only way past the dilemma seems to be direct mechanical examination of the mind during 'normal' listening to explore why the experiences of sighted and unsighted listening differ. This seems to be an interesting question.

In the meantime, the idea that - despite the method problem - results from blind ABX are valid is at least supported by the majority of data derived from home testing, Audio DiffMaker et al, so we needn't get hung up on it.

While the OP's reasoning and claimed inference from his cited studies are certainly flawed, he touches a valid point: Double-blind testing of isolated senses puts a subject into an artificial mode of perception, that is different from our usual perception of the world, which is always a multisensory blend. It should also be out of question, that the human brain makes excessive use of inter sensoric correlation while forming a consolidated mental representation of the outside world.

An ABX test can tell you, what an attentive mind, with artificially blocked non-auditory senses, can differentiate at best through the remaining, isolated auditory channel. I do not question, that this is probably as objective as it can get, when the auditory channel alone is all you want to map. I do question how much can be inferred regarding to the experience of actual listening situations in peoples' homes, where not only other senses (seeing your carefully composed system), but also a history of attached memories, associations and whatnot are constantly part of your perception of the world. I'm not claiming that aiming to be objective in a subjective environment is senseless, but it certainly might also make sense, that well situated, older men write magazines for each other, reporting about their experiences of trying to transform their surplus of dollars back to some sense of meaning. Subjective prose, that isn't castrated by some blinded, pain-in-the ass protocol as ABX, might be an actually more "objective" guide to identify an perfectly matching audio system for a member of any common enough group of individuals.

Long term HA usage might turn your mind into something, that has become unable to extract joy from owning expensive audio gear. A history of personal ABX comparisons can attach enough associations, so that this road is just closed. Not everyone might like that and I have come to a point, where I think that both is fine. I do also have come to the belief, that a man convinced that his gold cables sound better in a sighted test, even when he is unable to verify the same results blindly, is probably not lying to us.

PS & BTW Could it be shown already that results from double-blind listening tests correlate with the results of sighted tests over a large enough pool of listeners and different setups?

While the OP's reasoning and claimed inference from his cited studies are certainly flawed, he touches a valid point: Double-blind testing of isolated senses puts a subject into an artificial mode of perception, that is different from our usual perception of the world, which is always a multisensory blend.

Why do sighted test regularly lead to different results, then? Just calling it "bias, that should be eliminated" doesn't change the fact.

Imagine the following test setup: A test subject is presented music supposedly sourced from either a Sansa Clip or his favorite Burmester rack. You present an expensive looking switch to him, that's basically a dummy and that only inserts a small pause, but connects to the Clip at all times. Now imagine, you'd get a statistically significant result, that the subject rates the sound quality consistently higher, when he believes it to be coming from his Burmester rack / not coming from the Sansa Clip.

Now do a second test, this time double blind with both sources actually connected. Imagine the subject now fails to identify a difference.

What can we draw from this, especially when the subject was a honest type, sincerely motivated to rate the quality exactly as he perceived it in the first setup, without trying to prove or defying anything?

First, HA habit, the subject should stop claiming, that his Burmester setup sounds better than a Sansa Clip, as proven by the DBT. HA usually stops here.

But maybe one shouldn't. The belief, that sound was coming from a impressively crafted sound system, was able to significantly alter the subjects perception. In addition, the subjects usual mode of listening is reflected much better in the first setup than in the second (DBT).

Why do sighted test regularly lead to different results, then? Just calling it "bias, that should be eliminated" doesn't change the fact.

You wrote: Double-blind testing of isolated senses...

DBT does not necessarily 'isolate' any senses. Your can see, hear, taste, touch smell. All that has changed is what you *know*.

QUOTE

Imagine the following test setup: A test subject is presented music supposedly sourced from either a Sansa Clip or his favorite Burmester rack. You present an expensive looking switch to him, that's basically a dummy and that only inserts a small pause, but connects to the Clip at all times. Now imagine, you'd get a statistically significant result, that the subject rates the sound quality consistently higher, when he believes it to be coming from his Burmester rack / not coming from the Sansa Clip.

Now do a second test, this time double blind with both sources actually connected. Imagine the subject now fails to identify a difference.

What can we draw from this, especially when the subject was a honest type, sincerely motivated to rate the quality exactly as he perceived it in the first setup, without trying to prove or defying anything?

The first time he failed to identify that there was in fact no difference, and we can reasonably attribute that to sighted bias. The second time he may well have successfully identified that there was no difference, or may have failed to identify a real, but small, difference.

QUOTE

First, HA habit, the subject should stop claiming, that his Burmester setup sounds better than a Sansa Clip, as proven by the DBT. HA usually stops here.

But maybe one shouldn't. The belief, that sound was coming from a impressively crafted sound system, was able to significantly alter the subjects perception. In addition, the subjects usual mode of listening is reflected much better in the first setup than in the second (DBT).

This is no different from putting the same cheap wine in differently-priced bottles. Subjects often think the pricier wine tastes better. So, what does that tell us about the *wine*? What does your listeners *beliefs* about a piece of gear, tell us about the *gear*? What claims can reasonably be made about the relative performance of A and B?