What kinds of questions do you ask playtesters?

I have a Google docs questionnaire that I made that asks kind of broad things like "How long did the game take?" "Who won?" "Were any of the rules too obscure or confusing?" etc, but I was wondering what kinds of questions you all ask of your playtesters?

Apologies for the wall of text. I tend to over-answer and over-explain myself (though with the best of intentions). I don't have a lot of experience getting feedback from play testers, but I have had several classes that have dealt with surveys and what makes them effective, as well as personal experience designing surveys and interpreting results, so here are my thoughts. Hopefully they're of some use - take them for what they're worth:

I think the key to a good survey is focus. More than anything, I'd suggest making a list of information you want to find out from the play testers and prioritize them. I'm sure you've got a catch all field for additional comments at the end so use that to your advantage. Don't be anxious about having overlooked something - responders will use that catch-all field to let you know if you have. In fact, I find I'm actually more likely to write something in the catch-all field at the end if the survey is made up of specific, detailed questions than ones that are too broad or open ended.

Getting a good indication of people's true gut reactions is crucial. I'd ask testers something like, "On a scale of 1 to 5, 5 being best, how would you rate the quality of your experience with this game overall?" before I asked them any thing else, because you want that initial gut reaction. If you ask people to analyse/justify their initial reaction, they often over-analyse things and actually give less accurate responses. If you're doing an electronic survey, I'd consider going as far as to asking this on a separate page so that people can't second guess themselves and go back to change it. (You can find an interesting study on this phenomenon here, if you're interested - if you can't access the full write up, the abstract has all you really need: http://psycnet.apa.org/index.cfm?fa=search.displayRecord&uid=1991-17498-001 )

How easy a survey is to respond to will dictate how many (and how complete of) responses you get more than anything else. Targeted, specific questions are actually easier for people to answer than broad, general ones. Having people rate things on a scale from 1 to 5 (with 5 being best) is a great way to do things because people are more likely to give you an accurate reflection of their first impression. It's much easier to rate the speed of setup, how smooth the turn progression went, etc, than it is to answer a question like "Can you think of anything that slowed down the game or phases of the game that could be more streamlined?" For questions that don't lend themselves to a rating scale, people should ideally be able to answer with one or two words.

Also, don't fall into the trap of trying to make the responders do you job for you. While you want to ask for suggestions (in the catch all and perhaps about specific elements), trying to identify potential problems should take priority. Questions like, "How could this game be made better?," are hard to answer, so most people will leave them blank. And while it's worth asking just in case someone does come up with a great suggestion, most of the ones that you do get aren't that useful. You're the expert on game design (or at the very least, the expert on your game), and your play testers usually aren't. Play testers are experts in an area you aren't: their experiences playing your game. If you construct a survey that centers on your play testers' expertise, you'll get a more accurate picture of they're experiences and you'll be better able apply your expertise (tweaking your game so that other people have a better experience playing it). Like I said, there's no harm in asking for suggestions - you'll just get better results realizing your (and your play testers') areas of expertise and playing to them as much as possible.

Here are some questions I thought of that might yield useful responses (some are open ended, some lend themselves to a rating scale, some could go both ways, depending on how they're phrased). They just came off the top of my head, so this list is by no means definitive (or even balanced), but hopefully there's something in here that will be of some help:

About how long did it take to set up the game?

How difficult was the setup? (scale: "It went smoothly" to "It was complicated and/or confusing")

How does this game compare to other games you enjoy? How is it different? How would those similarities and differences affect your likelihood, either positively or negatively of buying this game? (Could be open ended, could have possible choices along the lines of: Positively, because this shares elements of games I already enjoy; Negatively, because it is too similar to games I already own; Positively, because I do not own many games like this; Negatively, because this game does not have much in common with games I already enjoy; Neutral)

Rate the clarity of the instructions (could even get more specific and have them rate clarity of instructions for setup, player turns, global events, etc). Might also phrase as, "How clear were the explanations for the rules of the game?" and "Were you able to navigate the rule book easily/Was the rule book well organized?" (since I've played many games where the rules themselves were easy to understand, but were presented in a confusing manner)

About how many turns did it take for you to "get into" the flow of the game?

Were there any phases of the game or elements of play which stood out as engaging? Were there any which stood out as boring or which took too long? (Similar to obligatory "What part of the game did you like best/least?," but might elicit different response from more experienced gamers, especially for something like a tabletop wargame)

Is this a game you see yourself playing frequently? If not, why? (possible response options: It's not my type of game, it took longer to play than I usually have time for, needed too much space to set up, etc)

Would you pay [projected list price of $x] for this game and purchase it either for yourself or as a gift for someone else? If not, why? (perhaps with options: Not a game I would normally buy; I would be interested in that game, but would not consider buying it unless it was on sale; I like this type of game but there are similar games available for a better price; etc)

Were players able to devise a "best" strategy and use it repeatedly? What was that strategy? How might the game be better balanced so that other strategies were more viable?

How quickly were players able to make decisions? Did the game progress smoothly from turn to turn (or phase to phase), or where there portions which routinely acted as speed bumps?

Did you feel confined in the number of choices you had when making decisions? Were you overwhelmed by too many choices?

Did you end up adopting and "house rules"? If so, why? (i.e. lack of balance, unclear rules, desire for more/fewer options, etc)

Did the game feel like it was missing a crucial element? Were there elements that felt extraneous?

How many times did you play the game? (If more than once) Was the game more enjoyable, less enjoyable or equally enjoyable during subsequent plays?

Hey, no problem on the wall of text. A lot of it was pretty helpful. Some of the things you've suggest (asking about how long it took to play or how many times they played) I have included in my own questionnaire, but I think the advice on asking more specific questions and allowing the catch all at the end to do more work was really helpful. I'll definitely be revising my survey before the next time I have people play.

I like to bluntly ask 'what didn't you like about this game', or 'what didn't feel right'.

Go into the conversation with the attitude that something must be wrong, and you want them to tell you what it was - that way they're more likely to think of something than they would be if you ask whether anything was wrong.

I've just been giving my testers a piece of paper labeled "Notes and Observations" and letting them put what they feel. This has worked well for finding the hitches in rules and such, but after seeing this i'm so excited to give a much more directed feedback form.

Ask is the key here; Questionnaires can ask about things you already know are a problem or might be a problem. Problems tend to crop up in unseen ways. I prefer to interview the players so that my questioning can be guided by the answers I get.

Are we talking about blind testers, or about in-person testers? I never give questionnaires to in-person playtesters. What's important to me is watching what they actually do during the game. In that respect I follow Jakob Nielsen, who says that what people say they do or would do, isn't what they actually do. If you want to ask questions in-person try some variation of Edard de Bono's Six Hats method. It's simple and fairly specific, but gets at several different angles.

One more comment on the price question... I actually find that this is a good way to get more game feedback than you might initially think. If playtesters respond with a lower price point that you were going for, push them on why. I've had folks tell me that the game was too simple and/or not engaging enough and thus they put a lower price tag on it.

In one way, it makes no logical sense... you could make a tic-tac-toe set with diamond 'x's and pure gold 'o's. It won't be a cheap game, but it is simple. But you've got to look past that and engage with their actual reasoning as to the price point.

Guys, I just want to say a thanks to all of you. What an incredible resource this thread is. I have a design that is pretty awesome and I'm chewing through my playtesting (one lined up tomorrow). I'm really good at brainstorming, but the ideas in this thread are so great and cut my time figuring out what I wanted to ask by probably 80-90%. AMAZING! Thanks so much!

If you get responses like Bored, Confused, Tired, Disheartened, or things that lean towards a feeling of isolation, then the game needs serious fixing.

But if you get responses like Envy, Tension, Excitement, or things that lean towards a feeling of communication or interaction, then you're on the right track!

I had a game once where every tester basically said "I get the mechanics, but I just don't feel the incentive." The game worked and was a sound system, but there wasn't much in the way of emotional involvement.

I'd much rather have a game that gets me emotionally invested only to have someone else destroy me than to be bored the whole time and win.

I often like to ask if there are parts that players find to predictable.

"Which parts do you find predictable or boring?"

These parts often are the first boring parts of a game, indicating a decline in the fun that players have.
A second question following this one would be if they think it was because of their opponent(s) or the game itself. After all, it might be a player that immidiately goes "systematic".

"Was this predictability/boring part due to the game? Or was it your opponent(s)?"

Then I also know, if the opponent went automatic, if the game is obvious balanced or not.

To conclude, I can ask the opponent(s) why they did the same thing over and over.

Well, I have many more. But they are not general questions. More in dept questions.

In the interest of sharing, here is what my survey became. I used it for the first time tonight in a three-player test with great feedback. Discussion after the game went for 30 minutes and the game only took about twenty. hah

This is all formatted in a two-to-a-page MS Word document that I print and cut. "*****" is the stand-in for the game name.

***** Playtest Survey
Date __________________ Version 5.0 Experience 1st 2nd 3rd 4th+
Player Count 2 3 4 5 6 Player # 1 2 3 4 5 6
Your Final Score ___________ Your Rating (1-10, 10 best) ___________
What is the most would you pay for a full-art, high quality version of this game for yourself or to give as a gift?
$0 $9 $15 $19 $25 $29 $39 $49
How many turns before you got into the “flow” of the game? __________
How much time did it feel like you were playing? ____________________
What game have you played that is most similar to *****? ___________
What are a couple of your favorite games? _________________________
Could you play the game again without reading the rules? YES NO
What was your strategy? ________________________________________
Why did the winner succeed? ____________________________________
How much control did you have over the outcome? __________________
How quickly were players able to make decisions? ____________________
What aspect stood out as more engaging than the rest? _______________
What element was “missing” from the game? _______________________
THANK YOU VERY MUCH FOR YOUR TIME AND FEEDBACK!