Survey Response Bias: 3 Most Harmful Biases (& How To Avoid Them)

16 May 2019 /product

Vrinda Singh

Completely removing response bias from your surveys is next to impossible, but taking the right steps to reduce it can be the point of difference between gaining accurate insights and completely wasting your resources.

Everything from the manner in which you phrase your survey questions to the delivery method of your survey can significantly affect the quality of your results. Here’s how to set yourself up for success from the get go.

How Survey Bias Affects Your Data

Source: Seek

When your survey responses are affected by bias, the data you receive is of a lower quality, often rendering the results of your survey useless.

One thing to keep in mind is that response bias isn't really the fault of your respondents. It's your survey that leads them to answer in the way that they do. You might not be able to do anything about any preconceived bias that the respondent walks in with, but you can encourage them to give more honest and accurate answers by wording and formatting your surveys the right way.

Types of Survey Bias

Here are the different types of bias to cater and watch out for when crafting your survey.

Acquiescence Bias

People tend to want to be agreeable when they're answering questions. Survey respondents are no different - they can often give the answers that they think they should give (as opposed to giving answers they really mean).

This means that if you have questions in your survey that require respondents to agree or disagree with a statement, they are more likely to agree.

Source: Paperform

How to reduce acquiescence bias: Use contradictory statements to test the accuracy of your respondents’ answers. Present them with an original statement (for eg. “I prefer to approach people first”), and later present them with a contradictory statement (eg. “I prefer people who strike up conversations with me first”).

If the respondent agrees with both statements, it’s generally a good indication that their responses are not entirely accurate as they’re agreeing with two contradictory statements.

If they get any indication of what information you’re to extract with your survey, they’ll likely alter their behaviour to match the expectations of the study. This can be detrimental to the accuracy of your results, as it reduces the authenticity of the respondent’s answers.

How to reduce demand effects: Reducing human contact is a good way to avoid demand effects. Conducting a survey face-to-face can often lead the surveyor to unintentionally reveal certain things about the study through body language, reactions or emotions to responses. Using an online survey can help eliminate the human factor and subsequently avoid demand effects.

Extreme (or Neutral) Responses

This occurs when respondents provide extremely positive or negative answers for questions. Surveys that include questions that require respondents to rate something along a scale (For eg. from 1-5 or ‘very satisfied’ to ‘very unsatisfied’) are most prone to this kind of bias.

Culture, indifference towards the survey or boredom are generally the most common reasons for such answers. Regardless of motivation, extreme responses can inaccurately skew your data and misrepresent how the respondent truly feels about the question.

In other cases, respondents might choose to select predominantly neutral answers to avoid having to contemplate the statement too much.

How to reduce extreme or neutral responses: Try to make your survey as short as possible to avoid respondent fatigue and indifference. Also try to cut down on the amount of available responses for scale questions, to avoid skewed results.

Most importantly, try to use interactive elements throughout your survey to ensure that respondents remain engaged. By using things like button animations, you can make your survey more fun for respondents to interact with. Play around with the Paperform editor to see all the cool ways you can make your survey more interactive.

Desirability Bias

People might also respond to a survey with answers that they feel are desirable, and avoid giving undesirable answers, regardless of how they actually feel. Like other types of bias, this occurs because people want to be seen as someone who thinks, says and does the right thing.

How to reduce desirability bias: Validate the respondents’ answers with multiple questions. As an extreme example, if a respondent declares that they are 15 years old, but claim that they have 2 children and have an income of $100K, you can likely rule their responses out as being inaccurate.

Response bias often stems from people wanting to give the answer that they think you want to hear. When you create surveys, you need to make sure that you're not directing them toward any particular answers. You also need to help them feel as if there are no “right” answers, allowing them to be as honest as possible.

How to Avoid Response Bias in Your Surveys

There are some more steps you can take to keep your survey data bias-free.

Word your questions carefully

Choosing your words carefully is one of the most important things when it comes to reducing response bias. Writing questions that sway your respondents in a certain direction are guaranteed to elicit inaccurate responses.

Do: How would you rate your flying experience on a scale from 1 - 5?

Don’t: How tiring was your flight?

While the first question doesn’t assume any emotions, the second basically encourages respondents to respond negatively.

By choosing the wrong words, you can lead your respondents to believe that you are looking for a particular answer from them.

Another assumption-based mistake to avoid is assuming your respondents’ traits. For example, if you ask your respondents “Which gym do you train at?”, you’ll most likely alienate a portion of your audience that doesn’t go to the gym at all.

Instead, it would be better to use question logic (as shown above) to ensure that you’re asking questions that are targeted to better suit your respondent. By using question logic, you can first “qualify” respondents by asking them “Do you train at a gym?” to make sure that your “Which gym to you train at?” question is only displayed to respondents who answered your previous question with a “Yes.” This is incredibly easy to set up Paperform.

Think about your answer options

If your survey questions give your respondents a range of predefined answers to choose from, you’ll need to be careful about which options you give them. One thing to keep in mind is that sometimes you need to provide more options than you might think.

For example, the option to select "I don't know" or to add their own answer can be helpful for many people. It means that they won't be forced to select an answer that they don't feel is consistent with their opinion but is closest to what they really want to say. Make sure that your answer options are clear and simple, while offering enough choices for your respondents to choose from.

Consider the order of your questions and answers

Even the order that you place your questions in can make a big difference in how people respond to them. For example, one thing to consider is how personal the questions you ask are, or how much they rely on personal opinion. These can often be best at the end of the survey because people are less likely to drop out if they are not asked these types of questions right away.

The order of your answer options can also make a difference. Often, people choose the first option automatically. Changing up the order of the answers for different questions can help to reduce this effect and weed out inaccurate respondents. It also bears repeating: tailoring your survey so that it's more relevant to each participant is also a good idea.

Make your survey anonymous

Making your survey anonymous can also help you to collect more accurate data. If your participants don't have to provide any personal information, they can feel more comfortable providing honest answers. For many surveys, it's not necessary to ask for personal details or contact details.

However, you can still ask questions relating to demographics, even if you don't ask for names. Just make sure that it’s very clear to the respondent that your survey is completely anonymous. Assure your respondents of this fact multiple times if necessary. If respondents feel safe in knowing that their identity won’t be available to anyone (include you), they’ll be more likely to respond without feeling judged for their answers.

It can also be helpful to remove any branding or indication of the person or organisation that’s conducting the survey, to make the purpose of the survey less clear.

While respondents generally walk in with some element of bias regardless of the survey in question, reducing any likelihood of bias on your end can make a world of difference to the accuracy of your results. Ultimately, if your survey is engaging and makes the respondent feel safe enough to be candid, you’ll be sure to collect some incredibly insightful data to power better decisions.