Case Study 2: Loudspeakers
During a visit to Canada's National Research Council many years ago, I noticed, stuck to the wall of the prototype IEC listening room, a page of results from one of Floyd Toole's seminal papers on the blind testing of loudspeakers. The scoring system was the one that Floyd developed and that I subsequently used for blind tests at Stereophile: "0" represents the worst sound that could possibly exist, "10" the perfection of live sound. On this scale, a telephone, for example, rates a "2." The speakers in Floyd's test pretty much covered the range of possible performance, yet their normalized scoring spread, from the worst to the best, was just 1.9 points. Other than some pathological designs, the audiophile speakers I test for Stereophile probably cover a range of 0.5 point on Floyd's scale; but, as I pointed out in my 1997 AES convention paper, our reviewers' ratings generally follow Floyd's findings: that people tended to prefer speakers with flat response and controlled dispersion.

But not always.

There are vocal advocates of high-sensitivity horn speakers. There are equally vocal advocates of low-sensitivity, large-panel speakers. The horn advocates, among whom was Stereophile's founder, the late J. Gordon Holt, enthuse about these speakers' "jump factor." I used to ask Gordon what he meant by that: was it dynamic range? No, it wasn't. Was it low distortion? No. His reply was always that there was something about the presentation of these speakers that more closely resembled what he experienced from live classical music.

What about panel speakers? Being dipoles, their interaction with the room is very different, so maybe that's why their advocates love them, not even considering the fact that, with a large panel, the listener is almost never far enough away from the speaker to be in the farfield, the result being more basswhich everyone likes, at least in two-channel audio. Of course, there are no cabinet resonances because there is no cabinet, and perhaps people's preference is due just to the fact that he very large radiating area inherently ensures more linear behavior when moving the same mass of air as a much smaller cone driver. But panel speakers' impulse responses look terrible. Well, I say "impulse response"; what I actually publish and use for my reviews is not the response of a loudspeaker to an impulse, but the impulse response calculated by cross-correlating an MLS signal or a chirp signal, based on the loudspeaker being a perfectly linear system. (John Vanderkooy has investigated the shortcomings of this assumption.)

Perhaps, then, the problem with measuring panel speakers is that the calculated impulse response does not accurately reflect the behavior of a large panel in the same manner as it does a conventional speaker's. At the 1989 AES Convention in New York, mathematician Manfred Schroeder discussed Chaotic systems in audio and mentioned, in passing, that a Chaotic system tended to produce subharmonics. And indeed, when I have measured panel speakers I have found that they produce subharmonics. You feed the panel speaker a 1kHz tone and in its output you find a 500Hz tone as well as the usual distortion harmonics at 2, 3, 4kHz, etc.

So in a large panel speaker, while the average position of the diaphragm follows the driving signal, the motions of the individual elements of that diaphragm are Chaotic. The speaker sounds much better than its measured behavior suggests, as all that randomness is integrated by human hearing's latency.

Live vs recorded; Steinway D vs Vivid GiyaG1 speakers

Or do people like the sound of panel speakers because the intensity of the sound they producethe sound power per unit area of the radiating surfaceis closer to what people hear live? This question was triggered by a live vs recorded demo in which I took part in 2009, in Maryland. Two pianists performed for an hour on a well-prepared Steinway D grand piano, which I recorded at 24/96. Then, after a break, the audience heard the recital again, this time played back through a system. (This meant miking the piano far more closely than I would do for a conventional recording, to avoid a double dose of the room acoustic when the recording was played back.)

The system was full-range, with low coloration and sufficient dynamic-range capability that I was able to match sound-pressure levels. But even though the SPL and the sound of the piano matched closely, listeners felt that the reproduced sound just didn't have the "bigness" of the real thing. I wasn't sure what they meant by "bigness." Yes, the piano's dispersion would excite the room very differently than would a pair of speakers, but perhaps these listeners were suggesting that they were aware that the intensity of the sound was not correct, even when the SPL and response were correct.

So when people say they like panel speakers that measure poorly, perhaps they're responding to those speakers' more accurate intensity, at least when it comes to large sources.

If Stereophile has the principle of not doing DBT and for you doing so is a matter of principle why oh earth would you even care to read Stereophile? Your statement above suggests you have very extensive knowledge of the content of Stereophile's articles. So you keep reading stereophile's reviews knowing they are irremediably flawed?? How perverse is this?

The real question lies on your double blind tested hatred.

Whatever your answer is I will have to assume that your hatred is a broken record and it will just go on.

......on your error filled post above so why should I comment on your assumptions? If you wish to worship Fearless Leader then by all means bow down and do so. I myself prefer to ask questions and be skeptical about his "methods" and the reason "why" he doesn't test ALL items under review.

It's a pity that some proponents of DBT as the only valid methodology have used the comments thread here to launch personal attacks against JA.

Indeed. I am not sure why I have become a lightning rod for these audio skeptics. Perhaps it is because it is a matter of religious belief on their part - having had extensive experience of blind testing, I rejected it and thus became a heretic in their eyes. Or perhaps it is just our courtesy in allowing them the space on this website to express themselves.

Andreasmaaan wrote:

JA does measure every piece of gear his reviewers review - a practice which ensures that the opinions of the reviewers are grounded in objective data, or otherwise as the case may be.

As my reviewers do not see the measurements until after they have written the review, I take my hat off to them.

Andreasmaaan wrote:

To cut what is risking becoming a lengthy expression of indignation short: thank-you JA for a wise and thought-provoking read.

You're welcome and my thanks to everyone else who appreciated the preprint. It was an honor to have been invited by the Audio Engineering Society to give this lecture - it's not often that you get the opportunity to look back over a 4-decade career!

You ignore the errors and just plow along blissfully gobbling up the unwarranted praise.

"JA does measure every piece of gear his reviewers review"

As I pointed out above, along with other errors on the part of Andreasmaaan,you do NOT test every piece of gear your reviewers review, but then you pretended that you do in your response. Only certain items are tested by you. Please get your own facts straight if that's even possible for you to do so.

Unlike yourself Raghead, I will be DESIGNING and BUILDING my own speakers and learning. You do know that word "learning"? Hmmmm maybe you stopped after age 10. You on the other hand will be on here *yawning* and acting the fool and spending your money on Stereophile reviewed crapola.

Johnny never offered to show you anything other than his backside which I think is approriate considering the childish actions of yourself.What does it matter what our audio components are? Anything we mention would be ridiculed by yourself no doubt. Sorry but I'm not playing your silly game. Good God grow up and act like a man for once will you?

As I pointed out above . . . you do NOT test every piece of gear your reviewers review, but then you pretended that you do in your response. Only certain items are tested by you. Please get your own facts straight if that's even possible for you to do so.

Out of morbid curiosity, I looked at the past 15 issues of Stereophile - we published 98 full reviews and follow-up reviews. Of those 98 reviews, 83 were accompanied with a full set of measurements, or 84.5%. So even if not every review includes measurements, the vast majority do, which supports Andreasmaaan's point.

I have explained to you before, in other comment threads, why we do not publish measurements of analog playback components, which is due to lack of resources. I have also explained to you why we do not publish measurements of tweaks, which is that it it is difficult to determine exactly what to measure. I don't see any reason to reopen those discussions, but that you continue to be fixated on these two issues and continue to raise them is sad.

And, as I have warned you, JohnnyR, I am continuing to delete comments from you, and others, that are nothing more than flames aimed at other readers.

"Reviews" means just that, any item reviewed by your staff. If you think "tweaks" don't qualify then they should be labled as "Opinions" not a review. All of those full reviews you speak of were what speakers, amps, preamps, cd transposrts and the like?

Yeah I know why you don't test tweaks, that would involve you using those horrible SBT or DBT. That's a really nice system you have set up. "We don't believe in DBTs or do them or can't afford to do them"......"We don't know what to test on the tweaks"..........add the two up and you get "We don't have to show that tweaks actually do anything other than what our "reviewer" subjectivly alludes to in their sighted biased listening using their golden ears"......EXCUSES Mr Atkinson plain and simple.

You seem to think DIY can't do anything right in your previous post in answer to Raghead. Insulting intelligent people that actually build better speakers than the status quo can produce is beyond being pompus in your case.THAT is what's SAD.

Don't you have some real work to be doing instead of being on the forums? I thought that was Ariel's job. Let him earn his pay.

I looked at the past 15 issues of Stereophile - we published 98 full reviews and follow-up reviews. Of those 98 reviews, 83 were accompanied with a full set of measurements, or 84.5%. So even if not every review includes measurements, the vast majority do, which supports Andreasmaaan's point.

"Reviews" means just that, any item reviewed by your staff. If you think "tweaks" don't qualify then they should be labled as "Opinions" not a review.

I am not sure what you mean. You were objecting to Andreasmaan's statement that Stereophile accompanies its reviews with measurements as not being true. I offered the analysis above to show that he was correct.

You need to consult with your fellow traveler GeorgeHolland, who criticized me about linking to other sites :-)

JohnnyR wrote:

You seem to think DIY can't do anything right in your previous post in answer to [Regadude.] Insulting intelligent people that actually build better speakers than the status quo can produce is beyond being pompus in your case.THAT is what's SAD.

With respect, you are arguing with the voices in your head here. I haven't written anything about DIY speaker designers not being able to do "anything right." What I have said is that a DIY designer, like yourself, doesn't subject his loudpeakers to the scrutiny of a disinterested marketplace, something professional designers do as a matter of course. So the question whether or not DIY designs are superior to commercial designs isn't put under independent scrutiny. It remains unsupported conjecture, as far as I am concerned.

Mr Atkinson, you could compare DIY against manufactured speakers but of course you won't. Might upset those that can't make good speakers and I'm not talking about the DIY

Johnny plainly stated that "tweaks" are reviewed in your magazine and online yet they aren't tested, hence you don't test every component under review. You have a reading comprehension problem? How many components reviewed by Jason Serinus have been tested? uh huh, right.

So now you wuss out about commenting on links? That proves you had nothing to say in the first place about it so you shouldn't have opened your mouth.Cop out time for Mr Atkinson.

The only unsupported conjecture on here are your abilities to print honest reviews. Your ability to avoid testing alot of audio components is well known by now. Very impressive spinning you do on here.

Mr Atkinson, you could compare DIY against manufactured speakers but of course you won't. Might upset those that can't make good speakers and I'm not talking about the DIY

Back in the early 1990s, Stereophile's Corey Greenberg was one of the judges for a DIY loudpeaker competition in San Francisco. There were 2 winning designs and as part of the prize, those 2 speakers were subject to a full set of measurements in Stereophile, published in the March 1992 issue. The speakers' measured performance was good but not great and certainly didn't embarass professional designers.

GeorgeHolland wrote:

Johnny plainly stated that "tweaks" are reviewed in your magazine and online yet they aren't tested, hence you don't test every component under review. You have a reading comprehension problem?

I don't believe so. JohnnyR was wrong, in that we very rarely publish full reviews of tweak products. Of the 98 reviews published in the magazine January 2012 through March 2013, there was just one review of a "tweak," Robert Deutsch on the HiFi Tuning Supreme Fuses in May 2012. ( I don't count the AudioQuest power strip reviewed by Kal Rubinson's review in December 2012 as Kal didn't make any comment on its sound quality, only on its utllity.)

GeorgeHolland wrote:

How many components reviewed by Jason Serinus have been tested? uh huh, right.

The correct answer is none. But that doesn't support JohnnyR's case either, as Jason's reviews are published on the website Secrets of Home Theater and High Fidelity, which has no connection with Stereophile. I hardly believe I am obliged to support reviews of products in competing publications with measurements in Stereophile.

"Back in the early 1990s, Stereophile's Corey Greenberg was one of the judges for a DIY loudpeaker competition in San Francisco. There were 2 winning designs and as part of the prize, those 2 speakers were subject to a full set of measurements in Stereophile, published in the March 1992 issue. The speakers' measured performance was good but not great and certainly didn't embarass professional designers."

Golly "only" 20 years ago and as we all know, nothing new has happened since then to make speaker design any better (Insert BIG sarcasm face here) Talk about a lame example Atkinson.

Yeah yeah, "full reviews" whatever, Plenty of "little revews" though all the freakin time and all they are , are OPINIONS but you know what? You let your reviewers get away with murdering the intergrity of the audio world with their stupid banter about "blacker backgrounds,lifted veils and HUGE improvements in the sound" without them having to justify their claims, Pretty slick there pal.

Oh and Jason Serinus "reviews" as posted about the recent RMAF isn't connected to Stereophile? Give us all a break, Your EXCUSES are even lamer this time than usual.

Yep typical Fearless Leader goobley gook. Did you major in Universisty in the double speak of Orwell's 1984? If not then you are gulity of plagiarizing the concept.

Golly "only" 20 years ago and as we all know, nothing new has happened since then to make speaker design any better...

Again you are arguing with the voices in your head, JohnnyR. Of course a lot has happened in the past 20 years, especially the advent of low-cost measuring equipment. But you're missing my point, which is, as I wrote earlier today, that a DIY designer doesn't subject his loudpeakers to the scrutiny of a disinterested marketplace, something professional designers do as a matter of course. You have repeatedy claimed that your speakers are as good as if not better than commercial designs. However, unlike engineers like Kevin Voecks, Paul Barton, Richard Vandersteen, Jeff Joseph, etc, your ability to pay your mortgage and feed your family doesn't depend on your skill as a speaker designer. Their's does, and it makes a difference.

Look, if you seriously believe your speaker designs are fully competitve with commercial designs, email me your measurements. I'll get back to you with my interpretation of what they mean, assuming they are as comprehensive as what I publish with every Stereophile speaker review. Put up or shut up.

JohnnyR wrote:

Yeah yeah, "full reviews" whatever, Plenty of "little revews" though all the freakin time and all they are , are OPINIONS but you know what? You let your reviewers get away with murdering the intergrity of the audio world with their stupid banter about "blacker backgrounds,lifted veils and HUGE improvements in the sound" without them having to justify their claim...

You're not making sense. "Andreasmaaan" was discussing reviews in the magazine being accompanied by measurements. I was discussing reviews in the magazine being accompanied by measurements. You seem to be discussing something that exists only in your mind. And unlike Professor Dumbledore's quote, that doesn't mean it's real :-)

JohnnyR wrote:

Oh and Jason Serinus "reviews" as posted about the recent RMAF isn't connected to Stereophile?

That was a show report. No-one other than yourself equates a magazine's coverage of a show with its formal equipment reviews. Are you seriously suggesting that magazines shouldn't publish show reports without including a full set of measurement data for each room they report on? Really? There isn't an audio magazine or website that does that.

Look, if you seriously believe your speaker designs are fully competitve
with commercial designs, email me your measurements. I'll get back to
you with my interpretation of what they mean, assuming they are as
comprehensive as what I publish with every Stereophile speaker review.

Me let YOU interpet my frequency response? Don't make me LAUGH.

You make my point for me, JohnnyR. It was a serious offer on my part. Professional loudspeaker designers send out their products for review and for the public scrutiny of possible customers. Amateur speaker designers, such as yourself, are often afraid to let anyone else judge their work.

What should have been an enlightening discussion of your great lecture has been contaminated by ill-conceived comments. Some individuals have posted comments that have created an environment of nastiness.

I would seriously urge you to start deleting ill-conceived comments from now on, whether they use flamable language or not. Some people are just turned on by the flames, but others aren't. And those others would much rather focus on the joy of engaging in the music and exchanges related to it.

You do not run a state-owned site but a private one that is likely to become a lot less lucrative (in all senses) if such nasty comments are to become visible.

I am not even pointing out names and we all know who I am talking about.

I am just giving you feedback that comes from an avid reader of Stereophile online.

Finally, the correction of my use of English was mentioned by one of these individuals. Please allow me to say that English is my second language and it is oxidized. Sorry about that.