Hello -
I'd like to thank the people who responded to my questions about
questionnaire design. My original email and the responses are included
below.
W. Jeffrey Rankin
Lead Web Application Developer
O'NEIL & ASSOCIATES, INC. http://www.oneil.com
495 Byers Rd.
Miamisburg, Ohio 45342-3662
Phone: (937) 865-0846 ext. 3504
Fax: (937) 865-5858
E-mail: [log in to unmask]
### Original Posting ###
#
From: Jeff Rankin
Sent: Friday, January 23, 2004 7:57 AM
To: [log in to unmask]
Subject: C&S: Questionnaire Design
Hello:
To help evaluate one of our more complex web applications, we want to
develop a questionnaire that users can fill out online. We're interested
in their general level of satisfaction with the application, areas where
it works well, areas where it's hard to use, etc. Are there guidelines
available for this kind of questionnaire design? Are there particular
ways in which the questions should be asked? Is the information gathered
from questionnaires reliable? What other considerations exist?
Please send replies to me and I will summarize for the list.
### Responses ###
### Response 1 ###
#
From: Marc Hassenzahl [[log in to unmask]]
Sent: Monday, January 26, 2004 3:17 AM
To: Jeff Rankin
Subject: AW: Questionnaire Design
hi jeff,
i think you should use an established instrument such as the WAMMI or
SUMI. if you really want to develop your own questionnaire, which must
be reliable and valid, be prepared for a lot of work.
i developped a semantic differntial to measure appeal (satisfaction) and
its determinants, which are pragmatic quality (usability) and hedonic
quality (stimulation, identification). it is very short and universally
applicable. there is also an online version of it: www.attrakdiff.de
complete with automatic analysis and reporting. if you interested in the
underlying ideas refer to:
Hassenzahl, M., Platz, A., Burmester, M., & Lehner, K. (2000). Hedonic
and Ergonomic Quality Aspects Determine a Software's Appeal. In T.Turner
& G. Szwillus (Eds.), Proceedings of the CHI 2000 Conference on Human
Factors in Computing (pp. 201-208). New York: ACM, Addison-Wesley.
Hassenzahl, M. (2002). The effect of perceived hedonic quality on
product appealingness. International Journal of Human-Computer
Interaction, 13, 479-497.
Hassenzahl, M. (2003). The thing and I: understanding the relationship
between user and product. In M.Blythe, C. Overbeeke, A. F. Monk, & P. C.
Wright (Eds.), Funology: From Usability to Enjoyment (pp. 31-42).
Dordrecht: Kluwer.
the major drawback: it is all in german. however, there is an untested
english translation. if you are interested, we may find ways of working
together.
all the best marc
### Response 2 ###
#
From: David Unsworth [[log in to unmask]]
Sent: Monday, January 26, 2004 3:49 AM
To: Jeff Rankin
Subject: Re: C&S: Questionnaire Design (long)
Jeff,
The extremist in me says don't do it. Questionnaires offer such
unreliable feedback they can send you off in the wrong direction.
The reasons they offer bad feedback are:- 1. People cannot answer
questions about why they did things or how easy or hard things were, or
indeed about any cognitive processes, because we have no introspective
access to those processes. If you don't believe me, listen to yourself
speaking. The only way to tell how easy something is for people is to
watch them use it. If you ask people any questions about cognitive
processes they just speculate. 2. They are self-selecting, so you only
get those inclined to answer. Why do they answer? You can't get a sample
that is random or representetive. A great example of this is those
"vote" questionnaires on the news sites. You know, "Should Senator X
resign?" etc. Unfortunately, the sample that selects itself has a vested
interest in the outcome. 3. The people who reply want to impress you.
Whenever we do questionnaires with our usability tests, we always get
diversion in results.
When we have asked people which was the quickest site to get a Motor
Insurance Quote they rarely pick the one that was genuinely the
quickest. They picked the one with the fewest screens even though it
took them the longest time between start and finish.
When we asked people which site they found easiest to use, they picked a
site with beautiful icons where they had to ask the observers what to do
next.
One of my favourites was when we asked what qualities they liked best in
a website and all agreed on simplicity, uncluttered design, not too many
links etc, yet when we followed on to ask what sites they used the most
they said www.ft.com and www.bbc.co.uk. If you take a look at these two
you'll see why I like this example. They are both pretty busy designs,
to put it mildly.
If, however, you absolutely insist on using questionnaires, ask them
questions they have a chance of being accurate with:- How oft6en do you
use (say) news sites, How often do you visit this one, On average, how
long do you spend on news sites when you visit, What are the areas of
the news site that interest you most, What features of a news site do
you find most interesting....
If you intend to use the data internally, to convince people of action,
you could include some demographic questions, so you can emphasise
certain issues are stronger with your target demographic.
### Response 3 ###
#
From: [log in to unmask]
Sent: Wednesday, January 28, 2004 1:31 AM
To: Jeff Rankin
Subject: Re: C&S: Questionnaire Design
Jeff,
For some short answers to your questions: 1. Are there guidelines
available for this kind of questionnaire design? Yes, but don't take any
as gospel. Look for research on Human Factors, Social Psychology, and
Cognitive Psychology research methods for more info.
2. Are there particular ways in which the questions should be asked?
Most definately. Way too many to mention here though. The main gist is -
know what kinds of questions to ask, and remove as much bias as
possible.
Is the information gathered from questionnaires reliable? No. Research
shows that data gathered from questionnaires often tends to the
extremes.
What other considerations exist? Too many to mention.
Questionnaire design is quite an in-depth area (see the
'Self-assessment' section of my last posted proceedings -
http://www.humanfactors.com.au/). Good questionnaire design is an art
form that takes a solid knowledge of cognitive psychology and decades of
practise to fine tune.
Probably the best person to ask would be the Father of some of the most
famous subjective satisfaction questionnaires in the field of HCI, Jurek
Kirakowski.
I hope this helps.
Best regards,
Ash Donaldson User Experience Designer
### End Summary ###
Confidentiality Notice
The information contained in this e-mail is confidential and intended for use only by the person(s) or organization listed in the address. If you have received this communication in error, please contact the sender at O'Neil & Associates, Inc., immediately. Any copying, dissemination, or distribution of this communication, other than by the intended recipient, is strictly prohibited.
--------------------------------------------------------------
Tip of the Day: Email mailto:[log in to unmask]
with any comments, questions or problems
CHI-WEB: www.sigchi.org/web POSTINGS: mailto:[log in to unmask]
MODERATORS: mailto:[log in to unmask]
SUBSCRIPTION CHANGES & FAQ: www.sigchi.org/web/faq.html
--------------------------------------------------------------