One of the best tools I've found for card sort experiments seems to be websort, mainly because you can just send out a link for participants to go through the exercise at their own convenience.

However, at the end of the day this method just returns limited quantitative data, and prevents the researcher from exploring another entirely separate dimension of information. I've never actually performed a card sort, so I don't know if performing the experiment in person is really worth the time. But at the end of other types of research sessions, I like to get inside the participant's head.

Have you found that gathering qualitative data from card sort experiments is worth the time? Based on the nature of card sorting, I expect this would generally need to be more involved than a few follow-up questions that could be included in a non-proctored experiment.

Edit: For the bounty, I'm looking for a why-or-why-not response. If you feel that qualitative data is important, bonus points for specifying what type of information should be gathered and/or which questions should be asked.

5 Answers
5

Yes, qualitative data from card sorts can be valuable as a knowledge elicitation technique, especially when you're working in domains outside your expertise. The main benefit is that in addition to the categories, you get to know a bit more about the criteria that your participants are using and their thinking process. (Although if you've never done a card sort before, you might be surprised how much you can learn from a standard computer based card sort.)

If working with one participant at a time, use normal think-aloud instructions. Do a warm-up exercise if necessary. Alternatively, you could pair up your participants and have them talk to each other, or use a larger group.

Encourage to make a pile for 'other'
or 'don't know this item'. That way
the sort will be focused on items
that the participants have something
to say about. You can always revisit
the rest pile at a later stage.

Ask why questions (that is, if they
don't talk enough on their own). Why
does this item belong in pile A? With
really strong participants such as
true domain experts you can even ask
'why not' without influencing the
results (for example when asking a
medical doctor to categorize
symptoms).

Often, during the sort people will
change their mind. Add a new
category, change a label, or move an
item. This is where it gets really
interesting. What was wrong, why is
this better?

Have some blank cards ready, in case
a participant can't choose between
two or more piles because he feels
the item can belong to all of them.
In that case you can duplicate the
card.

When the sort is complete, take a
look at the results. Does your
participant feel that any items were
missing? If that's the case, it's
easy to create a few extra cards and
at them to the relevant piles.

After the initial sort, you can move
up or down into the hierarchy. Ask to
split up large categories into
subcategories. And ask to group
multiple categories together into
supercategories. You can strongly encourage splitting up by taking 3 random items, and asking which 2 are most similar (the other one will go into a new category, taking all items that belong there too with it).

Some members of a category are more
central than others. You can ask
which items are the best, most
representative, example(s) for each
category.

Ask questions about the chosen labels. What makes them a good description? Alternative wordings?

Ask questions about the participant. How familiar are they with the items? Do they consider themselves an expert in the domain? If so, did that influence their sorting?

With this type of card sorting the final groupings are less important than the thoughts during the process. Your questions may influence the participant's behavior, so I wouldn't mix the results from such sessions with those where participants work on their own.

Thanks for your feedback. Are you discouraging the use of all card sort programs, including those for administering the test in person (and using a more traditional method like note cards instead)?
–
BaaApr 25 '11 at 16:22

2

+1 Could work remote as well though, e.g. with Skype or something similar used in parallel (see also ‘Remote Research’ by Nate Bolt & Tony Tulathimutte).
–
Sascha BrossmannApr 25 '11 at 22:26

I'm looking for more specific information as to why qualitative data is valuable for card sorts. I feel that both open card sorts and close card sorts have a time and place to be used. I would not really consider an open sort (the ability to create new items) to be qualitative data, per se.
–
BaaMay 1 '11 at 17:09

I'm curious, does this program tell you how long a person took to sort a particular card? or if they placed it in one group first and then the other? Does it let them keep any cards unsorted?

I'm also a novice to card sorting so I would like to add a sub question:
Do people physically arrange cards or card groups in particular ways while or after sorting the cards? Would you as the person conducting the card sort care about the distances between the card groups or their location on the table?

Hi Viraj, for websort all of the cards are available at once. The user can sort them in any order. I think the paid version gives information like how long the user took to complete the sort, and perhaps how many times each card was moved. As for your second question, there are many different ways to perform a card sort. Sometimes there are no groups to start out with, and the user creates their own groups. My understanding is that physical distance does not matter as much, and these experiments probably take a lot of space when a lot of cards are being used.
–
BaaMay 1 '11 at 17:26

In my opinion, especially performing an open card sort where participants can define their own categories can yield very helpful qualitative results.

The reason: often we have a lot of items that have to be displayed on our website/in our application and we may have a mental model already which suggests a sensible categorization scheme, but the categories our users come up with frequently have very little in common with our own.

Since we are not our users (and neither are our bosses), it's very important to figure out which categorization scheme is likely to work best for our target users, and card sorting can be a very valuable tool for this.

We have dosins of dosins of methods you can use in your usability engineering, but everything you do must have a purpose. You don't just spend a few hours (and worse, a few users) if you don't need that information.

Obiously you can learn a lot when you work with users, and that is the main reason for gathering qualitative data during card sorting. Insight. But if the main purpose is to confirm an information hierarky that you already have worked a lot with, you don't need the qualitative information. You can do with some quantitative statistics.

Thanks, Jorn. Good point - I guess should just use online card sorts when I want to perform a closed sort to validate an existing architecture, but I should consider in-person sessions for open card sorts that pertain to newer projects.
–
BaaMay 3 '11 at 0:08