Saturday, February 28, 2015

What people say, what people do, and what they say they do are entirely different things. (Margaret Mead)

I think the perfect example of this is when, at the end of a test, I ask the user how things went. Invariably, they’ll say it went “pretty good.” And they’ll say this no matter whether they aced it, flunked every task I gave them, or – as most commonly happens – fell somewhere in the middle.

I’ll typically also have observers who will hone in on that particular part of the test, citing it as evidence that the users “liked it,” the test went well, and nothing really needs to be changed. This usually only happens for brand-new observers however. I’ve invoked this particular quote so often that I’m pretty sure that anyone who’s sat in on more than a few of my tests has already committed it to memory.

I do also explain, though, that what’s going on has a lot to do with social niceties and – to a lesser degree – the limitations of human memory. I then try to focus them on what actually happened, reiterating that usability tests are task-based.

I also make the point that a task-based test is a real reflection of how people actually use websites. So, yes, Mr. Art Director, when the user says that they liked the colors? They’re really just being nice (or are not sure what else to say).

I’ve also used Dr. Mead’s quote when I need to emphasize the limitations of surveys – including self-reporting, self-selection, and retrospection. And, though I run focus groups myself, I also use the quote to make sure that my team realizes what focus groups are – and are not – good for. Finally, I also use it to make a plug for ethnography, a field that owes a lot to Margaret Mead.

Unfortunately, I cannot use that quote when it comes to web analytics. In fact, I once had a senior exec take a shot at usability testing by firing that quote right back at me. After all, web analytics really are what people do.

After fumbling around a good bit, I was able to make the point that this is one time where the “say” part of the equation is pretty darn important. In a usability test, what the user says while he completes his task lets us know what’s going on in his brain – he’s looking for x when we call it y, he’s looking at the top of the page when we put the button at the bottom, he thinks he’s finished but there’s one more step he needs to take. And analytics might not tell you any of that. And that's why this particular quote is still one of the best ones in my armory.