Header

Objective Reviews & Commentary - An Engineer's Perspective

March 17, 2011

DAC Listening Challenge Results

A BIG THANK YOU! I appreciate all the responses to both listening tests. I know listening for subtle differences requires some effort and I want to thank everyone for taking the time and sharing their results.

NOTE TO HEAD-FI MEMBERS (revised 3/28): Unfortunately, before I posted this article, the administrators at Head-Fi.org started censoring links to this blog. So if you had some trouble finding the results here, I’m sorry. The links were apparently not a problem when I was posting at Head-Fi on several other products and topics. The censorship only started after a paying Head-Fi sponsors (NuForce) publically complained about my review of their product. Head-Fi has gone so far as even deleting posts from other Head-Fi members that reference this blog.

BACKGROUND: NuForce responded to their uDAC-2 measuring poorly by saying it was designed to sound good even if several measurements are notably bad. So I came up with the best way I knew how to judge the NuForce purely on sound quality. I also thought it would be interesting to compare the $29 Behringer UCA202 with a high-end product like the Benchmark DAC1 Pre. The DACs were recorded playing real music under as realistic and similar conditions as possible. Anyone could download the recordings and compare them without knowing which was which—sort of like a “brown bag” wine tasting. Would people like the $3 wine better than the $30 wine? I thought it would be fun to find out!

TEST METHODS: Not having conducted a public web-based listening test before, this was something of a learning experience for me. For the first round, to make the comparison as fair as possible, I used the line outputs of all three DACs . The second round of tests used the headphone outputs driving real headphones and included a modified version of the Behringer UCA202. The modifications were made using a few dollars worth of parts and improve the headphone output of the Behringer. The files were given names of US Presidents in the first test and common trees in the second. The original listening tests, and all the details, can be found here:

A NOTE ABOUT “MASKING” (revised 3/17): Some have questioned the validity of the tests because, as NuForce put it, the output of the DACs have been “re-digitized”. And some also argue the equipment used to play back the files may not be high enough quality. The concern is these could mask the differences between the DACs. In reality this should not be much of an issue using playback hardware with reasonable fidelity. Here’s why:

These trials are only about the differences between the files, not the absolute accuracy. It’s like shopping online for a shirt. You might look at 3 different blue shirts at the same online retailer. Even if your computer display can’t convey every color and detail perfectly, you can still easily tell most of the differences between the shirts if they’re good pictures, all taken the same way, and your have at least a reasonably decent display. For example if your computer distorts the exact shade of blue, you should still be able to tell which of the 3 shirts is the darkest blue. The same is true in comparing the sound files. Even if your headphones might exaggerate the bass, you can still tell differences between the bass in the various files. This “difference effect” has been well documented in research.

As to “re-digitized”, the Benchmark ADC1 used to convert the output of the DACs to a CD quality file is already better than most studio gear used to record the music we all listen to. Studio A/D converters typically cost about $100 – $400 per channel of conversion. The ADC1 is a reference-grade A/D with a $900 per channel price tag. It generally has better specifications than nearly all studio gear. If the less expensive studio gear is good to enough to capture the subtle difference between say a high-end Steinway and a high-end Yamaha grand piano, the ADC1 should also capture audible differences between DACs.

Most recordings have already been through many more steps and kinds of digital processing yet are still very revealing of subtle differences. So adding one more relatively “pure” step isn’t going to make much difference. The recordings used in these tests are unusually pure and transparent.

PARTICIPANTS: This was a very informal survey. A total of 20 listeners picked their favorite (and sometimes least favorite) tracks. Many ranked them top to bottom. Some only participated in one of the tests and/or one of the songs. Some used ABX and some listened conventionally. More participated in the first (line out) trial than the second (headphone out) trial.

SCORING: Not everyone participated in every trial, some devices were offered in more trials than others, and different listeners provided different sorts of “votes”. Someone who’s better with statistical analysis than I am is reviewing the raw data. But, for now, to summarize the results here’s my attempt at a rough analysis:

Top choices scored 2 points

Second (“Runner Up”) choices scored 1 point

Least favorite (worst) scored –1 point (several indicated only their least favorite and not a favorite)

RESULTS (corrected 2 minor errors 3/16): Here are the total points scored for each downloaded file ranked from most favored (highest score) to least favored using the above scoring method:

Taft - Benchmark Line Out Brick House

9

Harrison - NuForce Line Out Brick House

8

Juniper - Benchmark CX300 Brick House

7

Wilson - Behringer Line Out Brick House

6

Jefferson - Reference CD Just Dance

4

Lincoln - Benchmark Line Out Just Dance

3

Oak - Behringer CX300 Brick House

2

Spruce - Mod UCA202 CX300 Brick House

2

Acorn - Benchmark CX300 Tis of Thee

2

Maple - Mod UCA202 CX300 Tis of Thee

3

Monroe - Reference CD Brick House (!)

1

Hawthorn - NuForce CX300 Tis of Thee

1

Fig - Behringer CX300 Tis of Thee

1

Jackson - NuForce Line Out Just Dance

0

Cypress - NuForce CX300 Brick House

0

Adams - Behringer Line Out Just Dance

-2

Pine - NuForce UE SF5 Tis of Thee

-4

Total Benchmark Line Out

12

Total Benchmark Headphone Out

9

Total NuForce Line Out

8

Total Behringer Line Out

4

Total Modified Behringer Headphone Out

5

Total Behringer Headphone Out

3

Total NuForce Headphone Out CX300

1

Total NuForce Headphone Out Ultimate Ears SF5

-4

COMMENTS ON RESULTS (edited 3/16): Even with the small sample size a few things were fairly clear:

The NuForce headphone output with the Ultimate Ears SuperFi 5 Pro headphones was an obvious “fail”. Several expressed a clear dislike of “Pine” and nobody favored it. I think this is due to the large frequency response variations caused by the relatively high output impedance of the NuForce uDAC-2 with these headphones. Several commented the high frequencies were rolled off or “dull”. This consistent with what the measurements would predict.

The reference CD tracks didn’t score as well as expected. For “Monroe” it’s likely because the reference track stood out as being different. Early in the forums it was labeled “by far the worst”. And that public comment likely tainted others into also hearing it as the “worst”. Six people rated it as the worst of that group of tracks. I personally think this is a good example of subjective bias--the original piece of music was deemed the “worst”! The Just Dance reference faired much better with a positive score of 4 likely because it wasn’t labeled publically as being bad. And much fewer even bothered to vote on Just Dance so the score of 4 was close to “perfect”.

In terms of total votes, the Benchmark stood out as a clear favorite. It was included mainly as a “reference” and not necessarily as fair competition to the other two, much cheaper, DACs. But, some might have expected all the DACs to sound roughly similar via the line outputs based on the measurements. The could be many reasons why this didn’t happen. The most likely is the same “peer bias” that caused a reference track to be rated poorly (see above). But it’s also possible the cheaper DACs do have audible problems.

The NuForce did much better via the line outs than the headphone outputs. Given the output impedance problem and higher distortion of the headphone output, this is what the measurements would predict.

Any channel balance error was corrected in all tests. So this removed the audible imbalance of the NuForce helping it score better.

NuForce’s claim of “better sound” doesn’t seem to be true when using the headphones. It faired poorly even with the more common and “impedance friendly” Sennheiser CX300’s.

Trying to draw other conclusions is a bit more difficult, but some things worth noting:

Nearly half the people couldn’t hear any differences at all.

The Just Dance track had a lot of inherent distortion that made hearing differences difficult for most. So those results are likely less valid (and there are fewer of them).

A number of votes were somewhat randomly distributed between all the middle scoring test files. This may suggest more guessing rather than clear preferences.

The line outputs of the $29 Behringer didn’t do as well as its measurements would suggest. The NuForce and Benchmark were preferred. This could be “peer bias”.

The differences among the individual files were more obvious via the headphone outputs rather than the line outputs. This is to be expected due to the impedance interaction.

For those who argue tests like this mask differences, it’s interesting to note the test may not have masked any subtle advantages of the high-end Benchmark. This is despite the fact that few listened to the files on similarly high-end gear.

BOTTOM LINE: I think the most obvious thing is roughly half the participants couldn’t tell any difference and another group of the “middle scoring” results is almost random with no clear preferences. Comparing the headphone outputs, however, the differences seemed more obvious. And they were very obvious when using balanced armature headphones (the SuperFi’s) on the NuForce. It’s clear the frequency response variations created by the NuForce’s relatively high output impedance creates audible problems.

It’s apparent subjective opinions of differences are easily swayed by public comments. The reference track from the CD of Brick House was strongly disliked by six listeners. They heard something different and were easily swayed into thinking different was worse based on previous public comments. It just took one person to say something negative and several others followed. This is exactly the sort of subjective bias that affects the majority of online subjective reviews in forums. As further proof, others using ABX (which is blind) voted the same track their favorite.

There’s also significant evidence the NuForce uDAC-2 does not sound better—at least using 2 different types of headphones—as NuForce claims it does. But it seemed to do a respectable job via the line outputs—at least with the channel balance error removed.

Overall I think this has been an interesting experiment. I learned a lot about how to run (and not run!) a listening test. And there were some fairly clear results—some expected and some not. If there’s sufficient interest, I may try to build on what I’ve learned here and conduct future listening tests?

COMMENTS WELCOME: Please feel free to add comments to the end of this article on the results—especially if you participated. I’d also like to know how many are interested in future listening tests? They’re a fair amount of work to put together and only really valid if you get a reasonable number of votes. So it’s something I only want to do if there’s enough interest. Please feel free to make suggestions, etc?

TECH SECTION:

IMPROVING THE RESULTS: I realize this wasn’t the best run study. It was more an informal experiment than anything. I think a larger scale listening test, that had more uniformity, would be needed to verify some of the closer results. If I do this again, I’ll research better methods, and I also welcome input from others with experience in this area?

DIFFERENT LOADS: It’s been suggested it would be also useful to include high impedance headphones and, except for the complexity that adds to the mix, I agree that would be interesting. I chose low impedance headphones as they’re, by far, the most popular—especially for use with a portable entry level DAC.

PASSWORD: An encrypted 7-Zip file was included with the file descriptions to prevent me from cheating or changing anything after people’s votes. The password is:

CY&YUMN5cZ9x2X8BhNj2t

PINE EXPLAINED: The graph below, in blue, shows the frequency response of the NuForce uDAC-2 using the Ultimate Ears headphones used in the trial:

The 4+ dB of response variation seen above is caused by the relatively high output impedance of the uDAC-2 interacting with the SuperFi headphones (typical of balanced armature designs from many manufactures). For more on this see Headphone & Amp Impedance.

OTHER DETAILS: The other details of how the test was run can be found in the original articles:

36
comments:

I wanted to thank you for the posts on your blog so far. I was Googling reviews of the Udac2, because it was on the top of my list for my next purchase, when I found your original review. It offered me a refreshing and informative approach to considering audio products; and through participating in the listening test I have learned much about my own methods of forming quality judgements. I sincerely hope that the ill feeling Head-Fi currently possess disappears to allow more people the opportunity to easily discover and access the blog. Thanks again!

Thanks for your comments Keith, and to Geek, I wasn't aware Google Docs even had a public survey option. I'll certainly look into that but I assume only those with a Google account could vote? I didn't want to use a single forum survey for the same reason. But it might be possible to use a few different options and still keep the scoring confidential until the end.

If there's enough interest, I'd certainly like to do this sort of thing again with a better run trial and perhaps subject matter that has a broader appeal.

I feel really sorry for the way that head-fi is treating you. You are one of the very few headfiers that back the subjective impressions with hard data and the way head-fi pays back is censoring your posts because one device measured badly. That kind of behaviour is what i expect from big corporations or Cuba like countries not from a sound forum. It seems that headfi is headed to he big corp management.I'm with you man. Keep it up.

wierdly, no. you can link it to anyone, anywhere , and anyone can use it i believe. There's nothing to stop someone from filling it out more than once, and that's about the only issue, and that comes from not needing a logon.

Thanks Ben. I managed to get a screenshot of at least one of them and have several more of other posts they've modified or deleted related to this listening test. You really have to wonder what Head-Fi is so afraid of for them to behave this way? And you also have to wonder how many other posts/threads are "silently deleted" that nobody even knows about?

As I've said to you repeatedly, you can post your information on Head-Fi. Your posts about the sponsor in question are still there, unedited, except to remove your rather constant linking to this site. If your posts are intact, and you've been told you can post your further findings there, then, as I've said to you before, you're making a straw man argument about sponsor protection--*none* of your findings posted there have been removed. Every criticism you've made of that sponsor and/or its product(s) remains.

Again, almost all of your posts (more than 95% of them) were made on Head-Fi after you started your blog here, with many of those posts linking here. When a person comes to Head-Fi, and it seems one of their primary purposes is to promote his site (be it personal or professional), they're treated just the same--and it has been that way there for nearly 10 years. The exceptions you see are for sponsors (who are allowed signature links as a component of sponsorship) and members who have long since proven that they're at Head-Fi more to be members of the community than promote their associated sites.

When it seems to me that redirecting people here from Head-Fi is not one of your primary reasons for being at Head-Fi, then, again, I'll reconsider my position on this with you.

So let's recount what has happened so far: What has been edited? Your posted findings about the sponsor in question? Nope. Your criticisms of the sponsor or its products? Nope. Your constant linking to this blog? Yup. Have you been told you can post your findings here over on Head-Fi? Yup.

The more emphatically you scream about not being able to post links to your site--the more vehemently you campaign for that--the more you convince me that *that* is one of your primary reasons for being at Head-Fi.

On most days, between 2500 and 3000 posts are made on Head-Fi, and many of those posts criticize sponsors and/or their products. One need not look too hard to prove that statement true.

You've made many complaints of censorship, and yet you've censored me here repeatedly. You might want to put that high horse back in the stable.

@HeadFi (Jude) I tried to put this to rest. I even extended the olive branch yesterday after I deleting some things here but you never responded although Currawong (another Head-Fi Admin) was kind enough to.

I've deleted exactly ONE comment of yours containing erroneous and misleading information--among other things you claimed to have not deleted any posts when in fact you had. So I don't agree with "repeatedly". Rather than get into an argument it seemed better to just remove it and move on. You did try to post the SAME comment 4 times in succession yesterday, but Blogspot automatically flagged them as SPAM.

You completely left out you're not just deleting links and posts of mine, but other posts as well--some with no links at all. Why?

What's so threatening here if it's not the negative NuForce review? What justifies all the censorship of not just me, but several others at Head-Fi? What justifies you blocking the very people from Head-Fi who participated in this listening test from getting the results? And the 2 threads for this test are even tucked away in the infrequently visited "back of the bus" Sound Science forum because you forbid blind testing discussions anywhere else on your site.

I count about 41 threads I've participated in on Head-Fi. Of my 145 total posts I would be surprised if more than a few dozen have any links to this blog in them. So your "95%" number is misleading. Anyone is welcome to check the 145 posts unless you're going to delete all of those too?

What do you mean "constant linking"? You have deleted exactly one link of mine, and removed my signature with its link. I got the message loud and clear after trying to reason with you and made no more attempts to post links after you threatened me if I even tried. All the posts since of mine you deleted, and at least some posted by others, have contained NO links. Again, I have screenshots.

I've never claimed you have deleted my posts that are critical of NuForce. The issue is all the other rampant censorship and what's behind it.

I can't help but feel there's a double standard at Head-Fi. I was on the "right" side of that double standard when I started very similar threads about the Sansa Clip+, Cowon i9, and Behringer UCA202. These all had links to this blog and a link in my signature:

I never received a single message from any Head-Fi moderator or admin over any of the threads above dating back to February. Nor did I get any message when I first added my signature or from dozens of other posts. If your policy is really as uniform as you suggest, why not? Why did I suddenly fall on the other side of the "double standard" after the NuForce drama unfolded?

My primary reason at Head-Fi was to contribute and provide objective information. I get nothing from "re-directing people" to this blog except it saves duplicating the same info in multiple places.

Unlike your sponsor-fest of a website, more "hits" to this blog don't get me anything. I have no advertisers. I don't get a free prize if I get 20,000 hits.

So I really don't understand your argument it's somehow harmful for Head-Fi members to be directed here to learn about jitter, or headphone impedance, or lots of other things.

As for "scream about not being able to post links" I've already gave up on that on that when you freaked out yesterday. I've only been trying to help Head-Fi members but I give up if this is what I get in return.

Again, you can post your information on Head-Fi if you choose. If you choose not to, that's your choice.

That "sponsor-fest of a website" happens to contain one of the closest-knit communities of its size that I've so far seen, and I'm proud of it.

Once again, when it seems to me you're there more to be a part of that community than to self-promote, I'll reconsider my position.

We do not curate the 2500 to 3000 posts per day to weed out posts critical of sponsors or their products. And, again, one needn't look hard to prove that true.

I don't see how anyone could argue this sponsor has been protected when (a) your posts and measurements criticizing them/their product(s) remain, and (b) you've been told you can continue to post your findings there. I'd be absolutely shocked if that sponsor felt protected at all.

I think we've exchanged enough messages on this now. I've explained my position, and I respect and accept that you and some others don't like it.

If you decide to post your information on Head-Fi now or in the future, I'll see you there.

Whenever I see a volley of posts between two people where they break it down into rebuttals to each other's words, my eyes glaze over and I skip those posts. This happens a lot on head-fi. No offense to you guys; I understand you have to make your positions clear, but to the rest of us, it reads as noise that doesn't add to what we're here for: to learn more about our passion for audio and the equipment that reproduces it.

Head-Fi IS a great community, and it's not so much the site itself as the people who contribute to it that make it that way. I haven't found any other place that's as useful or current a resource on headphones, amps, DACs, and the like. I first heard about this blog on head-fi, and for that I'm grateful.

NwAvGuy, as I've said before, this blog is a rare and informative resource. It is a bit of a bummer if you can't post links to here, or at the very least have a sig link without being a sponsor, but if those are head-fi's rules (and they do have some specific rules) I guess you have to play by them. I can only imagine they were implemented with good intentions...because we all know how vehement people in this hobby can be expressing their opinions.

Whenever you have a post that you think will be of interest to head-fiers, copy it there, verbatim if you want--that doesn't seem to be a problem--and people who want to hear what you have to say will find your blog and become regular readers. Try not to let the tension between you guys keep you away from head-fi. You won't hurt the admins one bit if you don't post there, but you also won't be sharing your findings with a community that wants to hear them.

Thanks Nate. I was trying to avoid the whole "eyes glaze over" thing yesterday when I deleted the first post which was even longer. In hindsight, I probably should have done the same again. But I wanted to be fair and give Jude his say which is more than he's done for me these last few days on Head-Fi.

I don't have a problem with rules. I have a problem with rules that are applied SELECTIVELY as they have been here. I'm not out to hurt the admins. And, and as I understand it, some are unpaid volunteers just doing as they're told.

For me it's much more about contributing to sites I can respect and believe in. Head-Fi, due to Jude's actions, what's gone on behind the scenes, and more, has lost A LOT of my respect since the weekend and I know several others who feel the same way.

As for copying content, how many times do you see someone copy entire Wikipedia pages into a forum thread? They don't. They just post a link. That's what the web is supposed to be all about. And it's even OK on Head-Fi until Jude decides you crossed some invisible line.

NwAvGuy, don't worry about the Head-Fi business. I wouldn't be surprised if Sound Science subforum gets deleted there at some point. Too much bad vibe for advertisers and sponspors. Your blog on the other hand is awsome and it's a wonderful example about the trouble brewing beneath the warm and fuzzy marketing claims. To me it seems there is at least a potential to a paradigm shift. More and more people are getting to know about the science and objective measurements behind audio products. At some point the industry has to aknowledge this and start selling their products based on objective information, otherwise they will go out of business. At least I hope so. Much of the industry is still literally covering their ears and babbling on about how all cables sound different and so on. I'm pretty sure, that if you keep coming up with interesting stuff in this blog, the number of readers is going to increase significantly.There are already a lot of people who really appreciate your work.

Thanks Cer. I'll do my best to keep coming up with interesting stuff including the E-mu review. This blog isn't for me, it's for everyone interested in this stuff. So I also welcome input on what sorts of reviews and articles others would like to see here?

I guess it would be pretty nice to have an article about why digital IS better than analog (at least on most real world scenarios), and how this can be proved by stuff like the Nyquist Theorem.Though it'll probably end up in flame wars.

The thing that's difficult for many people to understand is that Head-fi is a commercial private site. It appears to look like an open BBS or public forum, but it's not. People who post there should know that they are guests. The rules are spelled out - you must accept the terms during registration.

As far as applying rules selectively, the Head-fi admins have a lot of posts to go through. Also, the admins have every right to do whatever they want: selectively delete posts, highlight certain products, impose rules, etc. because they are the hosts. Kind of like how you would have the right to do anything that you wanted to for your own house-party, including putting selective restrictions on your guests.

Head-fi doesn't exactly make this in super clear terms so sometimes, folks like us are left wondering. The heavily commercial nature of Head-fi is difficult to distinguish because it doesn't look like one of those official review sites with discussion forums on the side - where one would implicitly know that he is not in a public playground.

On the other hand, if Head-fi did make it's "private club, we make the rules" much more explicit, some people might be less inclined to post there.

If you don't like it, make you own site. And that's you've done here - a good job too.

Vittau, by "real world" do you mean vinyl vs CD? If so, that topic has already been well covered from lots of angles. Sort of like many political issues, the preferences pretty much fall along "party lines"--the vinyl lovers vs everyone else.

The most amusing (and telling) thing of all to me is you can insert an A/D and D/A into an extremely high-end ($10,000+) all analog system unknown to said system's owner. And they won't even notice it's there. Playing their vinyl puts just as big of smile on their face even when it's being converted to digital and back again as long as they don't know that's happening. I'd even wager I could do such a thing to Michael Fremer's personal home system and he'd probably never notice. That should tell everyone something.

What's interesting to me is vinyl fans have traditionally been people who grew up with vinyl and have a nostalgic connection to it. But they're increasingly finding themselves among the hearing aid wearing AARP crowd. And one might think the era of vinyl would fade away with them.

But, there seems to be a significant new wave of vinyl lovers that are often in their 20's. I think in part because of the vast amount of music that's only available on vinyl. It's also "retro" which draws another group in. And some just want to be different.

There are some technical advantages to certain old vinyl releases versus certain CD releases of the same music. That's a topic I hope to cover in a future blog article. But in terms of comparing the raw formats for accuracy it's no contest and that's already been well documented.

"There are some technical advantages to certain old vinyl releases versus certain CD releases of the same music. That's a topic I hope to cover in a future blog article." -- NwAvGuy

---------------------------------------

This is something I've observed for several years in comparing my own vinyl LP rips [using a high quality analog playback front end and Apogee Duet A/D/A converter] with the CD versions of the same recordings -- in many cases the vinyl rip sounds much more 'musical.'

I believe this is so because the vinyl pressings were usually manufactured relatively soon after the analog studio tapes were recorded and hence the signal strength on these tapes was high relative to tape noise. However, the old analog tapes from which many reissue CD's are derived are at least 20+ years old and by that time the effects of tape aging has taken its toll. My own collection of old commercially recorded open reel tapes bears this out.

In addition, many CD's were manufactured in the '80's and early '90's using A/D converters which were not of the same quality as are available today.

The difference between a hi-res vinyl LP rip [at 96/24 or 192/24 bit] and later CD reissues clearly illustrates the ability of well-cared-for quality vinyl recordings to preserve the original full beauty of older recorded music. It seems that vinyl is less prone to aging than magnetic tape in my experience -- sort of analogous to the ability of amber to preserve fossils for millions of years!

Vinylripper you are 100% correct and that's part of what I was hinting at above and going to cover in a future article.

Keith Johnson, a long time well respected audio engineer, gave a talk I attended at an Audio Engineering Society conference. He worked at Ampex back in the day when Ampex was making many of the studio tape machines. One of his key points was the degradation of studio master tapes used in CD mastering.

Magnetic tape deteriorates with time--especially the high frequencies. So later CD's issued from old master tapes are at a significant disadvantage. And to make things worse, especially in the early days of CDs, a lot of the engineers used high frequency EQ to try and "restore" the lost high frequencies due to the age of the tape. That HF EQ boost, combined with the tape's distorted HF content, was a big part of the original "harsh digital" sound that audiophiles branded the CD format with.

And you are also correct about the early digital hardware. The equipment used varied a lot in quality and not everyone even knew how to get the most out of what they had.

For years early digital recordings used magnetic tape and were "edited" using a razor blade to crudely cut the tape. In 1983 when the CD was introduced to the public, the most powerful computers available, like the CRAY, had only 4 MB of memory. That's not enough for even 1 minute of CD quality audio so digital editing, mixing, etc. wasn't a cost effective option.

As a result, early multi-track digital recordings required a lot of additional steps in the analog domain compared to today's digital workstations. Today everything is converted to digital only once and 32+ bit software processing is used for all the mixing, editing, etc.

But it's worth noting vinyl also ages and is prone to all sorts of problems--especially if it's mistreated. Things like warping, brittleness, groove wear, mis-tracking damage, and surface damage cannot be removed by any sort of cleaning. But, on the whole, well cared for vinyl can certainly age more gracefully than magnetic tape.

Too bad you let the psychotic bunch at headfi draw you in. If you had done some searches at other headphone and audio forums first, you would have found lots of similar stories.

Headfi drives most of the smart ones away leaving only a faint signal against a massive backdrop of deluded noise. Go figure but they seem to like it that way. I have no idea why its as popular as it is.

Cool comparo but I have to wonder if you had tighter control and a more solid test if the line outputs would have all scored about the same? It's easy to see how the headphone tests could sound different tho. Sorry I missed the test but I just found your blog.

And nuforce is kinda like headfi. If you just do the research you will learn they have lots of crappy products with obvious problems. Like check out the much hated icon mobile. They obviously dont learn from their mistakes and reading some of their responses to you I now get why.

@RockinCans, I admit I should have done more homework on NuForce before ordering the uDAC-2. I found some good reviews, assumed they had fixed all the problems because it was a 2nd generation product, and they seemed like a decent company. Yeah, I was wrong.

And thanks for the feedback on the listening test. I agree it's not the most rigorous of trials. I did send the raw data to a guy who knows statistical analysis a lot better than I do. So it will be interesting to see what his conclusions are and I hope to do some more trials in the future.

As for Head-Fi, it just seemed like the obvious place to hang out. But, like NuForce, things weren't so impressive under the well-marketed surface. There's no shortage of members raving about Qables that cost 4 times as much as the iPod they're plugged into but there are at least some rational members who appreciate the objective side. So I was willing to wade through all the "noise".

As Itsallaboutthemoney said above, Head-Fi is more like a private club and one person apparently dictates how it's run. And if he wants to censor me, and censor those who like what I'm doing, that's his choice but it sends a clear message regardless of his words above. So yeah, I learned the hard way with Head-Fi too.

Awesome - this is what we needed for a very long time !! keep it up maan !! we want more data !! awesome.I'm tired of reading posts with paragraphs after paragraphs of the "air between the strings" and the airiness of xyz.. on various forums.Even dedicated blogs by people reviewing expensive dacs, amps and cans and not a single technical explanation..going on and on about "synergy" between a portable amp and a can, silly 150 ohm iems requiring powerful amping, 250 ohms being tough to drive etc. Waiting for some more high quality technical posts..

I used to be a big headfi regular. I hung out in the DIY section and contributed quite a bit (design, build, advice, etc).

I was forced to be 'MOT for a day' even though I'm not a company and don't accept their terms for 'playing in their sandbox'. after having my photos and posts deleted and fighting one war after another with the DIY mods, I just simply LEFT HEADFI. I am now on another couple of diy forums and have no such problems anymore.

Goodbye headfi. You are dead to me, now. Some good people are still there, but the admins favor vendors no matter how honest your tech reviews are.

NwAvGuy...forget Head-Fi. Many of the threads have become nothing more than a kindergarten full of inexperienced users posing as experts. Not all, mind you, but many. And the fanboyism and ridiculous statements made as fact pretty much render much of the place useless. I too used to be a Head-Fi regular but lost interest for the reasons above. But there is no doubt a double standard at Head-Fi, selective enforcement of its sacred rules is the rule, not the exception. Some are warned and banned, others not, for much of the same behaviors. And while the .org suffix still applies by web standards (it used to be reserved for non-profit type orgs, but not more), it is much more of .com site. I mean, when I sign on, ads pop up specifically devoted to my web activity, no doubt Head-Fi is making a ton of money, or should I say Jude is making serious money, from this so-called hobbyist site.

I agree with the above poster. There are still some decent people on Head-Fi, of course. But the sponsors/advertiser bias is so obvious. And all the free samples and subsequent "reviews" by the well-know shills operating on Head-Fi is so obvious. As for linking, there are a few members whose sole purpose is to post on Head-Fi to drive traffic to their sites, which are also loaded with advertising, unlike you very nice blog (and I don't even agree on the FiiO E5, either).

Anyway, if the term "Jumped the Shark" ever applied to a forum, Head-Fi is it. It's a mess, a collection of more and more new members who state opinion as fact, and who deliver pretty much worthless information.

If you want to know why Jude at head-fi is such a psychotic autocrat just read the March 2011 Wired article "The Overlords" by Matt Schwartz. He explains how guys like Jude exploit vast numbers of unpaid workers for the "pleasure of the kingly founder whose authority is absolute". Ding! We have a match!

He goes on to say "by keeping their organizations in a state of minimalist adolescence, the founders are able to keep the control they crave and use their sites--and the massive user base that powers them--as tools". Sounds EXACTLY LIKE HEAD-FI don't you think?

I stopped feeding Jude's messed up empire a long time ago. I predict it will collapse under its own misguided weight if he doesn't stop driving away all the best contributors who try to keep it real. People won't put up with it forever. Just look at all the Arab tyranny. Jude should get a clue.

I was being repeatedly berated by another member about liking grado rs1's.He made several personally insulting attacks on me saying that "people like yourself with deep pockets could continue pretending to believe in your imaginary "quality"". When I responded that I was if fact near poverty and used every cent I had(to the point of living in my friend's barn and not eating regularly) I was censored as being "offensive".

thanks for exposing nuforce for their shoddy work, i find it disgusting how admins @ head-fi have treated you, instead of dropping nuforce as their sponsors they ban you for letting everyone know about a crappy dac...

they must be desperate for the dollar...

and as for mysteriously disappearing threads i've seen the same thing happen with Burson HA-160 & Burson HA-160D threads, both of them were deleted, when someone made a thread asking about it that thread got swiftly deleted it as well...

Hiplease forgive my lack of technical knowlege.A couple questions - -regarding the benchmark - what is the difference between its line out and headphone out?-what signifigance does a DAC's output impedance have if any?for example I own a schiit bifrost.its specs are as follows.

as a side note, yet another O2 thread was locked in head-fi today, no explanation, and the thread seemed quite civil to me too. Such a shame. I have a feeling i'm going to be spending a lot more time here, and a lot less time at head-fi.

The output impedance of a pure DAC is usually not a problem unless it uses a tube output stage or has a very "esoteric" design. But if you're talking about headphone DACs, then the headphone output impedance is very important and should be under 2 ohms or under your headphone impedance divided by 8.

Unless you're running Rockbox firmware, and increasing the internal gain setting above 0 dB, the Clip+ will not distort into 40 ohms even at full volume. With the Sansa firmware it won't distort no matter what you do. It's also odd your Onkyo distorts. So if you're hearing distortion with all those sources there might be something wrong with your Ultrasones or perhaps the music you listen to just sounds bad.

This is off topic for this article. See the O2 Summary article, the FiiO E10 article, the ODA/ODAC article, and my newest updates for comments about DACs.

Hi, NwAvGuy, I stumbled upon your DAC blind listening tests while doing a google search for "DAC blind listening." It's nice to see that there are people actually interested in controlled equipment comparisons rather than methods subject to bias (like so-called "reviews.")

I spent some time looking at your methodology. The idea to ADC the outputs of various DACs and headphone amps under load, then post the captured level-matched audio files for anyone to compare was brilliant. This allows trusted audio community members to participate, so that readers cannot argue that the listeners were unqualified. It also allowed you to recruit a larger group of listeners than would otherwise be practical.

However, I do have some concerns regarding your analysis and the conclusions drawn from the experiment. The primary issue is that there is no statistical analysis presented. It is impossible for the reader to determine if the "ranked" order was due to random chance or if there were differences among the tested devices that truly existed. I can tell you based on experience that with 20 listeners, each selecting either several "favorites" or providing a single rank list, you probably will need a much larger sample size and series of repetitions in order to achieve results that meet statistical significance (i.e. your "findings" are likely to be reflecting actual differences rather than due to random chance). I know you stated that this was an "informal survey," but if it is, there are no meaningful conclusions that can be drawn. That is, even if the Benchmark DAC1 had the most "votes," it doesn't necessarily mean, based on your study, that it was sonically superior to the uDac (I "expect" that it would be, but your study doesn't necessarily demonstrate his hypothesis.)

I'm very interested in how your ODAC work will turn out. It would be fabulous if a DAC that is sonically indistinguishable from the Benchmark DAC1 can be purchased for a fraction of the price. You mentioned that you would be conducting blind listening tests in a similar fashion, and I'm wondering if you have access to a statistician who can help you design your test methodology and the data analysis?

(cont)If I can make a friendly suggestion: one elegant way to demonstrate that two DAC's are similar (which I think is what you'll want to show), is individually testing each audio clip and output pair separately. i.e.Test#1: ABX of ClipA played on lineout of DacA vs. DacB - The clips from each DAC are randomized, and the listener selects same/different. Test is repeated X number of times, and statistical analysis is done.Test #2: ABX of ClipA played on headphone-out of DacA vs. DacB - again, same/different test done X number of times, analysis done on this test separately.Test #3: ABX of ClipB played on lineout of DacA vs. DACB - and so on....

At the end of the day, you'll be able to obtain analysis results like: (examples)- 5 out of 20 listeners were able to differentiate the lineout of DacA from DacB using clipA more reliably than would be expected by chance- 8 out of 20 listeners were able to differentiate the lineout of DacA from DacB using clipB- 18 out of 20 listeners were able to differentiate the headphone-out of DacA from DacB using clipA....and so on.OR NONE of the 20 listeners was able to differentiate the lineout of DacA from DacB using ClipB more often than would be expected by chance.

Statistics would be very simple/straightforward doing it this way. But fair warning, if you have negative results, people are going to argue at least one of the following:- the listener's equipment (listener's DAC, amp, and headphones) had insufficient resolution to distinguish actual differences between DacA and DacB. To borrow your example, if you are trying to see if 3 blue shirts displayed on a monitor are different shades of blue, and you're using an old-school monitor that only has 16 colors, all 3 blue shirts will look the same, even though they are actually different shades. - your listeners, either due to poor hearing or inexperience or whatever, cannot actually hear small differences in sound quality even though they exist- the added ADC+DAC process actually introduced artifacts or affected SQ in such a way that actual differences in DacA vs. DacB were "masked."

Thanks for all the comments and suggestions! The blind test was something I just threw together in response to all the subjectivists on Head-Fi irrationally defending NuForce's poorly measuring uDAC-2. As I said in the articles, I know the methodology leaves a lot to be desired.

Unfortunately when you throw out the "bad data" from those who didn't follow the rules, etc., the sample size was really marginal. I gave the raw data to a statistics guru and that was his overall conclusion.

If I try something similar again, I will make a better effort to both capture better data, and get the help of a statistics expert to properly interpret it and perhaps help design the methodology.

You're correct about the likely arguments. In my experience the only way to really silence (or at least shame) critics of blind testing is to offer the same test to them live on as many of their terms as possible. Faced with that challenge, everyone I've known makes some weak excuse why they can't or won't participate. That usually takes a lot of credibility out of their argument.

A problem with doing live ABX comparisons with DACs is the time sync. Unlike doing analog ABX trials, DACs have some time delay (latency) and even fed with the same bitstream may have a noticeable time shift. It's also not trivial to feed two DACs with the same bitstream. There are workarounds like restarting the clip when a switch is made or adding a long enough time delay to mask the time shift. But those lower the sensitivity of the test somewhat. Although it's still vastly more sensitive than the typical home listening scenario of listing to DAC A for a while, disconnecting it and replacing it with DAC B, etc.

PLEASE NOTE! If you see "LOAD MORE..." OR "LOADING..." just above this paragraph, you need to click that link to load the newest comments. It may take several seconds. When all the comments are displayed the Load More/Loading link disappears.

COMMENT APPROVAL: Comments may take anywhere from a few minutes to a few days to be approved. You might want to save a copy (highlight Ctrl-C) of your comments before submitting in case something goes wrong. The approval process is necessary to prevent obscenity, etc. Please keep comments on topic. Comments promoting commercial interests will not be approved and may be reported as SPAM.