Recent OFCOM Research highlighted that 71% of the UK receive 9 nuisance calls a month, and that telephone research is the #4 culprit (1). So has telephone research had its day? At the same time online grows apace. We’ve looked closely at the merits of telephone, online and face-to-face (ftf). So if you wish to commission a quantitative research survey, this article summarises some insights and ideas to help you make the most of your research investment.

Quantitative research survey costs

Quantitative research costs are sensitive to sample size, ease of reaching an audience or ‘incidence’, the length of survey, mode and complexity of fieldwork and analysis. Compared with online (index =100) fieldwork costs are typically higher for face-to-face (index 350-450) than telephone (index 250-300) due to the greater human time involved. Other costs such as coding for online research, computer aided telephone interviewing (CATI) and computer assisted personal interviewing (CAPI) are similar.

Coverage or reach

82% of the UK are online though many online surveys use panels which cover just 5% of the population. There are some geographic gaps and respondents are more ‘Internet experienced’. Thus some sample bias is possible. Nearly all homes have access to at least one phone though telephone databases cover just 60% UK homes (though even fewer will have agreed to take part in research!). Fixed line telephone reaches 82% homes (and proportionately more of the elderly) while mobiles reach 81% (and proportionately more of the young) (1). Face-to-face can reach most places (though with extra travel costs).

Online response depends on the nature of the panel, and how responsive and interested respondents are. Expect between 5-30%. Response from links on websites or emails will similarly depend on the nature of the source. Telephone responses have fallen over the last decade and responses are now around 10-15%. Face-to-face response is also around 15-20%.

Avidy bias (Sample bias)

The self-selection nature of online panels means there is a greater risk of respondents only participating in surveys that interest them. So-called avidy bias. Typically online respondents are younger, more familiar with the online world and spend more time on it. They are also more informed, more opinionated and more politically activist. (2) Panels also contain more early technology adopters though it remains possible to discern other types on the ‘diffusion of innovation’ spectrum.

Social desirability bias

Telephone research respondents present more socially desirable responses more often than face-to-face (3,4). This is particularly the case with those with lower intellectual ability or fewer years of formal education (i.e. C2DEs). Research has also shown that respondents are more comfortable discussing sensitive subjects face-to-face as they can see, and thus have greater trust in, the interviewer. Conversely, face-to-face interviews conducted in the respondent’s home minimises anonymity, making socially desirable responses more pronounced. Overall however, interpersonal trust between the interviewer and the respondent has a greater influence resulting in more honest responses. Face-to-face shows similar results to online (where there is no interviewer effect) though some research (5) has observed higher valuation responses to some ‘willingness to pay’ questions (for example, when there is a perceived ‘civic virtue’ in being seen to add to a common good).

Satisficing

Satisficing (a combination of the words satisfy and sacrifice) involves short-cutting the response process, settling on a solution that is ‘good enough’ but could be ‘optimised’.

Telephone poses an increased cognitive burden. The increased difficulty to comprehend questions, reduces the effort to cooperate, search the memory and process information. Perceived time pressure also fatigues and demotivates. This results in questions being less considered, giving rise to higher acquiescence (answering affirmatively regardless of the question), having no opinions, choosing mid-points or only extremes in rating scales, easier to defend answers and reduced disclosure. Again this is more clear with those with lower intellectual ability. Research (3,4,5) suggests face-to-face researchers are better able to judge confusion, waning motivation, distraction (via watching a tv, eating etc.) and be able motivate and make it easier for the respondent to understand the questionnaire and improve cooperation on complex tasks. With online respondents go at their own pace.

Research Inspiration

1. Useful research starts with a clear market research brief. Decide your objectives, target market, what you need to know and any guidance. Beyond feasibility and answers to questions, what’s the relative importance of cost, speed, ‘reliability’ etc? Be as clear as you can about expected sample sizes.

2. Understand the pitfalls in conducting quantitative research. Larger samples give greater reliability, for example a sample in excess of 1000 will give more reliability than a sample of 500, i.e. if repeating a survey 100 times, in 95 instances a confidence interval i.e. variance of responses will be within +/- 1%. Prefer shorter surveys to cut the risk of satisficing.

4. There are even more pitfalls if you would like to repeat a quantitative research survey or set up a brand tracker. Take extra care to make sure the pool of respondents will deliver a sufficient and matched sample for each survey wave so findings are comparable.

5. Beware spurious analysis. Remember the Whiskas advert that famously told us that ‘8 out of 10 cats prefer Whiskas’. This was eventually changed to ‘8 out of 10 owners that expressed a preference said their cats preferred Whiskas’. What we still don’t know is how many said ‘don’t know’, how many expressed a preference, and the sample size. Whatever the survey mode, be clear about the sample size, what is statistically significant or merely directional to make the context clear. Ensuring clear and fair analysis gives more useful insights and ensures better decision-making!