Category Archives: Harris

So, I dipped by toe back in to web research this morning. Since I generally have decent experiences with Harris polls, I decided to give them a shot. As usual with them, it was a fairly painless experience; nothing really glaringly wrong or obnoxious — so let’s not dwell too much on either of these things.

First, from a pure design standpoint, I don’t understand the point of these massively over-wide columns. If you’re going to answer true for some and false for some, it’s really a lot of left-right mouse or trackpad motion — enough that it created a minor annoyance for me. In a 3-question true/false setup like this, it’s really not terrible — but in a longer series of questions, it might drive me to drink:

Wouldn’t this shopped version be easier to use?

Like I said, pretty minor. Which brings us to my second and final observation on this poll:

Still fighting the good fight, but haven’t had time to write about (or even look at) much research lately. I did catch this grid a couple days ago, and I think it’s worth throwing up and looking at, not because it’s a particularly terrible example (it’s sadly just typical), but because I can imagine so many better ways to measure this:

Can’t you picture something with different carmaker logos (or, maybe even better, images of their most popular models) that you can drag up and down or left and right to indicate exactly how likely you would be to consider each of them? And that’s just my very first thought on this one.

Flash makes pretty much anything possible, but we’re still using virtual #2 pencils to fill in virtual scantron bubbles, aren’t we? What do you think?

It’s always possible this was a Firefox-only problem, but I (a) doubt it and (b) don’t think that makes this OK either, actually.

It’s not eggregious, of course, just … off, and as such, distracting. Don’t distract your respondents; don’t make them take time out from thinking about their answers to think about why your boxes won’t fit on your page.

Of the online polls I see regularly, Harris Interactive seems to be the best of the lot. They actually did something I thought was particularly good in a piece of research I saw today, but of course I’m going to post on the thing they did today that I didn’t like as much.

First, I was asked this:

Nothing horrible there; it’s a matrix, but it’s not too huge, and the answer choices are fine. But then they ask this as the very next screen:

I check my bank statements all the time, so I started clicking the right-most column, just like I’d done with the Facebook questions — and then I realized that the right-most column wasn’t “very likely” anymore, but had mutated into “not applicable.” I suppose you can make an argument that the column is necessary on this page — the Facebook questions were asked only of Facebook users, and it’s possible some people answering this screen might not have a 401(k), or might not have any credit cards — but I think it would have been better to either let the “not at all likely” column take care of those folks or to have added the “not applicable” onto the Facebook questions as well.

The more you can keep answer choices identical, the more your respondents can glide effortlessly through the research. Note that what Harris did here is really quite minor, and not at all as aggravating as what I’ve seen done elsewhere from time to time, where a painfully over-sized matrix runs across multiple pages, but with the answer choices randomly shifting to abuse the respondents make sure the respondents are paying attention. What I’m talking about here is nitpicking compared to that sort of thing — but the common element is that your respondent is going to click the wrong thing because they expect their answer choice to be somewhere else.

Harris often asks respondents to click a radio button at the end of the survey, asking something along the lines of “How much do you agree with this statement: I noticed mistakes in this survey,” with answer choices ranging from strongly agree to strongly disagree. (I keep neglecting to get a screencap of that, but it’s along those general lines.)

Here’s NPD’s similar approach:

Polling Point has asked, at various times, something along the same basic lines; I remember because there was something they kept getting wrong, and I kept complaining about it, to no avail. (It was something minor related to the answer choices they were giving about which news program you watched most often, but I can’t remember the details.)

All of these approaches are, on the one hand, commendable: respondents should be able to provide feedback, and asking directly in the research instrument itself lets the feedback go directly to the survey designers, as opposed to making those with issues go through the front office of the giant research company to try to make a complaint that will never be properly routed.

But here’s the other side of the coin:

Wouldn’t it be better to extensively test our research before it goes into the field?

Offer special incentives to especially active panelists who have proven themselves in past feedback to be critical of errors in exchange for them “previewing” certain surveys. You can probably get them to do a decent job of quality assurance testing/user acceptance testing/proofreading/whatever you want to call it for nothing other than an additional entry in the sweepstakes or an extra $2 in their prize money account. (Yes, itoccurs to me they could be doing this right now, just without the special incentives, and I’m doing their work for free.)

The thing is, finding errors and other usability problems in complicated market research instruments really isn’t the sort of thing you want to be doing in real-time on a live survey after people work their way through it. Whether you’re exposing the whole panel or just the selected subset of early reviewers to the unedited project, you’re still trying to do triage on something that should have been right before it launched.