Designing great surveys in any medium

A survey that doesn’t work well on paper or in-person is not going to work well over the phone – Good mobile-phone survey design must begin with good survey design.

Fortunately, much research in other fields has already been done on what makes a survey accurate, easy-to-answer and useful. Here is a quick primer on survey design, based on industry best practices in all mediums.

Survey focus and intent

Know why you are sending out a survey and exactly what information you need. If you do not know why you are asking a question, don’t ask it.

Are you trying to identify what works in a program? Are you looking for input on a particular decision? Do you need to understand the context for future work?

Stating your purpose clearly will help you focus and avoid the cost and time of including irrelevant questions. Communicate your purpose to participants to help them understand why the survey is relevant to them, and what effect participation will have on them.

Viamo team members are happy to work with you to move from your objectives and key decisions into building the appropriate set of survey questions.

Ask clear, unbiased questions

The way questions and surveys are phrased can influence the results. Always aim to let participants pick responses freely – uninfluenced by the structure of the survey.

Include Comprehensive and Exclusive options.

One goal in question design is to have exactly one response that applies for each participant – this avoids requiring participants to pick a response they do not agree with in order to move forward with the survey. Consider including a “none of the above” option, or using conditions to make sure people are not asked questions to which they cannot respond.

Instead of: Where do your children go to school?

Use: Are there currently children under the age of 18 living in your household? If yes, Do the children go to school, and where?

Avoid combining multiple statements in one question, such that a respondent cannot agree or disagree with one portion of the question without agreeing or disagreeing with the whole.

Instead of: Do you think that the Electricity Company of Ghana should increase rates and construct a new power plant?

Use: Should the Electricity Company of Ghana increase rates, decrease rates, or keep them the same?

Or, Would you support an increase in electricity tariffs, if the additional funds funds are directly earmarked for power plant construction?

Avoid common biases in question phrasing.

There are broadly-identified patterns in how respondents answer particular types of questions. Avoiding these phrasings can help participants respond only to the content of the question, without a pre-established preference for one answer.

Agreement Bias:

Respondents are more likely to choose yes and confirm the question asker’s statements than to disagree with them.

Instead of: Do you agree that the Ministry of Education needs to do more to support teacher training?

Use: Should the Ministry of Education increase its level of support for teacher training, or not?

Primacy Bias and Recency Bias:

Respondents are more likely to pick the first and last options in a list. Try re-ordering the response options to account for this.

“Loaded Question” Bias:

Use neutral, balanced language in question phrasing.

Instead of: How serious a problem is the theft of public funds by fat-cat corrupt politicians in your country?

Use: Do politicians in your country misuse public funds? Frequently, sometimes or never?”

Or, Is theft by public officials a common problem, or not?

Use clear and specific language

You want to be sure that each respondent interprets the question in the same way you do. When possible, use specific time frames and objective indicators (unless your purpose focuses on perceptions).

Instead of: How politically involved are you? A lot, a little, not at all

Use: In the last year, how many times have you attended a political protest, rally, community meeting or town hall? Never, once or twice, 3-10 times, or more.

Is your target audience familiar with any technical terms, abbreviations or jargon in your questions?

If a translator struggles to find an appropriate word, it may be a tip-off that the content will be difficult to understand. If you are not sure, simply find a few target respondents and ask them.

Instead of: Do you think the implementation of MDG policies have affected GDP in Ghana?

Use: Do you think that the current primary schools in your community help the local economy, or harm the local economy?

Survey flow and framing

Completing a survey should be a smooth, intuitive process for your users. Begin with an introduction that lets them know who is asking, why and what will happen with the answers. During the survey, group similar questions together in a logical path. When switching topics, include a brief transition to tie the sections together.

This doesn’t need to be long. Consider something like: “Thank you. We have a few more questions to help us understand your situation. Do you live in a rural area or village, or an urban area or city?”

It is a good idea to begin with easier questions to give the respondent time to become comfortable with the survey. More difficult or sensitive questions should come at the end, after building more trust with the respondent.

Starting with a great survey design will make converting it to mobile format much easier. In addition to these resources, we also recommend testing your survey in person with a few respondents. Ask them if any questions are unclear or difficult to answer, or if they think the surveyor wants to hear particular responses.

This post originally appeared on votomobile.org. The content has been updated for clarity, and to reflect Viamo as the solution provider.

External Resources

Look here for more information about good survey design in any medium: