Main outcome measure: The proportion of completed or partially completed questionnaires returned.

Results: The odds of response were more than doubled when a monetary incentive was used (odds ratio 2.02; 95% confidence interval 1.79 to 2.27) and almost doubled when incentives were not conditional on response (1.71; 1.29 to 2.26). Response was more likely when short questionnaires were used (1.86; 1.55 to 2.24). Personalised questionnaires and letters increased response (1.16; 1.06 to 1.28), as did the use of coloured ink (1.39; 1.16 to 1.67). The odds of response were more than doubled when the questionnaires were sent by recorded delivery (2.21; 1.51 to 3.25) and increased when stamped return envelopes were used (1.26; 1.13 to 1.41) and questionnaires were sent by first class post (1.12; 1.02 to 1.23). Contacting participants before sending questionnaires increased response (1.54; 1.24 to 1.92), as did follow up contact (1.44; 1.22 to 1.70) and providing non-respondents with a second copy of the questionnaire (1.41; 1.02 to 1.94). Questionnaires designed to be of more interest to participants were more likely to be returned (2.44; 1.99 to 3.01), but questionnaires containing questions of a sensitive nature were less likely to be returned (0.92; 0.87 to 0.98). Questionnaires originating from universities were more likely to be returned than were questionnaires from other sources, such as commercial organisations (1.31; 1.11 to 1.54).

Conclusions: Health researchers using postal questionnaires can improve the quality of their research by using the strategies shown to be effective in this systematic review.

What is already known on this topic

What is already known on this topic Postal questionnaires are widely used in the collection of data in epidemiological studies and health research

Non-response to postal questionnaires reduces the effective sample size and can introduce bias

What this study adds

What this study adds This systematic review includes more randomised controlled trials than any previously published review or meta-analysis no questionnaire response

The review has identified effective ways to increase response to postal questionnaires

The review will be updated regularly in the Cochrane Library

Editorial by Smeeth and Fletcher

Introduction

Postal questionnaires are widely used to collect data in health research and are often the only financially viable option when collecting information from large, geographically dispersed populations. Non-response to postal questionnaires reduces the effective sample size and can introduce bias.1 As non-response can affect the validity of epidemiological studies, assessment of response is important in the critical appraisal of health research. For the same reason, the identification of effective strategies to increase response to postal questionnaires could improve the quality of health research. To identify such strategies we conducted a systematic review of randomised controlled trials.

Methods

Identification of trials

We aimed to identify all randomised controlled trials of strategies to influence the response to a postal questionnaire. Eligible studies were not restricted to medical surveys and included any questionnaire topic in any population. Studies in languages other than English were included. Strategies requiring telephone contact were included, but strategies requiring home visits by investigators were excluded for reasons of cost. We searched 14 electronic bibliographical databases (table 1). Two reviewers independently screened each record for eligibility by examining titles, abstracts, and keywords. Records identified by either reviewer were retrieved. We searched the reference lists of relevant trials and reviews, and two journals in which the largest number of eligible trials had been published (Public Opinion Quarterly and American Journal of Epidemiology). We contacted authors of eligible trials and reviews to ask about unpublished trials. Reports of potentially relevant trials were obtained, and two reviewers assessed each for eligibility. We estimated the sensitivity of the combined search strategy (electronic searching and manual searches of reference lists) by comparing the trials identified by using this strategy with the trials identified by manually searching journals. We used ascertainment intersection methods to estimate the number of trials that may have been missed during screening.2

Table 1

Electronic bibliographical databases and search strategies used in systematic review of response to postal questionnaires

Data extraction and outcome measures

Two reviewers independently extracted data from eligible reports by using a standard form. Disagreements were resolved by a third reviewer. We extracted data on the type of intervention evaluated, the number of participants randomised to intervention or control groups, the quality of the concealment of participants' allocation, and the types of participants, materials, and follow up methods used. Two outcomes were used to estimate the effect of each intervention on response: the proportion of completed or partially completed questionnaires returned after the first mailing and the proportion returned after all follow up contacts had been made. We wrote to the authors of reports when these data were missing or the methods used to allocate participants were unclear (for example, where reports said only that participants were “divided” into groups).

Interventions were classified and analysed within distinct strategies to increase response. In trials with factorial designs, interventions were classified under two or more strategies. When interventions were evaluated at more than two levels (for example, highly, moderately, and slightly personalised questionnaires), we combined the upper levels to create a dichotomy. To assess the influence of a personalised questionnaire on response, for example, we compared response to the least personalised questionnaire with the combined response for the moderately and highly personalised questionnaires.

Data analysis and statistical methods

We used Stata statistical software to analyse our data. For each strategy, we estimated pooled odds ratios in a random effects model. We calculated 95% confidence intervals and two sided P values for each outcome. Selection bias was assessed by using Egger's weighted regression method and Begg's rank correlation test and funnel plot.3 Heterogeneity among the trials' odds ratios was assessed by using a 2 test at a 5% significance level. In trials of monetary incentives, we specified a priori that the amount of the incentive might explain any heterogeneity between trial results. To investigate this, we used regression to examine the relation between response and the current value of the incentive in US dollars. When the year of the study was not known, we used the average delay between year of study and year of publication for other trials (three years). We also specified a priori that, in trials of questionnaire length, the number of pages used might explain any heterogeneity between trial results, and to investigate this, the odds of response were regressed on the number of pages.

Results

We identified 292 eligible trials including a total of 258 315 participants that evaluated 75 different strategies for increasing response to postal questionnaires. The average number of participants per trial was 1091 (range 39-10 047). The trials were published in 251 reports—80 (32%) in medical, epidemiological, or health related journals, 58 (23%) in psychological, educational, or sociological journals, 105 (42%) in marketing, business, or statistical journals, and 8 (3%) in engineering journals or dissertations, or they had not yet been published (see Appendix Appendix A).

All tests for selection bias were significant (P<0.05) in five strategies: monetary incentives, varying length of questionnaire, follow up contact with non-respondents, saying that the sponsor will benefit if participants return questionnaires, and saying that society will benefit if participants return questionnaires. Tests were not possible in 15 strategies where fewer than three trials were included. The method of randomisation was not known in most of the eligible trials. Where information was available, the quality of the concealment of participants' allocation was poor in 30 trials and good in 12 trials. The figure shows the pooled odds ratios and 95% confidence intervals for the 40 different strategies in which the combined trials included more than 1000 participants.

Effects on questionnaire response of 40 strategies where combined trials included over 1000 participants

Table 2 may be used to translate odds ratios into response rates from different baseline rates. At least one strategy in each category was found to influence response. For example, when incentives were used the odds of response were more than doubled when money was the incentive (odds ratio 2.02; 95% confidence interval 1.79 to 2.27) and were almost doubled when incentives were not conditional on response (1.71; 1.29 to 2.26). The length of questionnaires influenced response: short questionnaires made response more likely (1.86; 1.55 to 2.24). The use of coloured ink as opposed to blue or black ink increased response (1.39; 1.16 to 1.67) as did making questionnaires and letters more personal (1.16; 1.06 to 1.28). When recorded delivery was used the odds of response were more than doubled (2.21; 1.51 to 3.25), and they were increased when stamped return envelopes were used (1.26; 1.13 to 1.41) and questionnaires were sent by first class post (1.12; 1.02 to 1.23). Contacting participants before sending questionnaires increased response (1.54; 1.24 to 1.92), as did follow up contact (1.44; 1.22 to 1.70) and providing non-respondents with a second copy of the questionnaire (1.41; 1.02 to 1.94). Questionnaires designed to be of more interest to participants were more likely to be returned (2.44; 1.99 to 3.01), but questionnaires containing questions of a sensitive nature were less likely to be returned (0.92; 0.87 to 0.98). Questionnaires originating from universities were more likely to be returned than questionnaires from other sources, such as commercial organisations (1.31; 1.11 to 1.54).

Table 2

Conversion of odds ratios to response rates from different baseline rates

We found significant heterogeneity among trial results in 17 out of the 31 strategies that included more than one trial. For trials of monetary incentives the heterogeneity among the results was significant (P<0.0001). In regression analysis, the relation was positive between the size of the incentive and the odds of response:

Log(OR)=0.69[SE 0.05]+0.084 [SE 0.03] Log (Amount in US$)

The model predicts, for example, that the odds of response with a $1 incentive will be twice that with no incentive. The model also predicts that the marginal benefit will diminish, in terms of increasing the odds of response, for each additional $1 increase in the amount given (for example, the odds of response with a $15 incentive will be only 2.5 times that with no incentive). For trials examining the effect of questionnaire length, heterogeneity between results was apparent on inspection of the forest plot (P<0.00001). In regression analysis a relation was found between the number of pages used and the odds of response:

The model predicts, for example, that the odds of response with a single page will be twice that with three pages.

Discussion

Several reviews and meta-analyses of strategies to increase response to postal questionnaires have been published in the literature on research surveys over the past 40 years. Our review, which was based on a systematic search of published and unpublished literature in English and other languages, includes more than twice as many trials as any previously published review.4 The trials identified were not restricted to medical surveys: one third were medical, epidemiological, or health related; one quarter were psychological, educational, or sociological; and two fifths were marketing, business, or statistical.

We have identified a range of strategies that seem to increase response to postal questionnaires. The pooled effect measures for some strategies are precise because large numbers of participants were included in the combined trials. Before these results are implemented, several methodological issues must be considered.

Identification and inclusion of all relevant trials

Identifying and including all relevant trials in systematic reviews reduces random error in meta-analyses and, because ease of identification of trials is associated with the size of treatment effects, complete ascertainment may reduce bias.5 We estimate that our search strategy retrieved nearly all eligible trials (estimated sensitivity 95%; 84% to 99%) and that we missed very few relevant records during screening.2 We excluded some trials because we could not confirm that participants had been randomly allocated to intervention or control groups, and we have not examined whether the results of these trials differ systematically from the included trials. Tests for selection bias were significant in five strategies. Although these results may be due to true heterogeneity between trial results rather than bias in the selection of trials,3 we cannot rule out the possibility of selection bias having an effect on the results.

Methodological quality of trials

Inadequate allocation concealment can bias the results of clinical trials.6 In our review, information on allocation concealment was unavailable for most of the included trials. If they had inadequate concealment, this may have biased the results, which is unlikely in this context because the researchers making the allocations would find it difficult to predict propensity to respond to a questionnaire.

Heterogeneity among trial results

We found substantial heterogeneity among the results of trials in half of the strategies, and for these it may be inappropriate to combine results to produce a single estimate of effect.7 Before undertaking the analyses we developed hypotheses concerning underlying differences in the trials of monetary incentives and length of questionnaire that might explain heterogeneity. Regression analyses identified relations between response and amounts of incentive and between response and questionnaire length. These models explain some of the heterogeneity. For other strategies, variation between trial interventions and populations is likely to explain some of the heterogeneity. For example, among trials evaluating non-monetary incentives, the types of incentive used are very different, ranging from donations to charity to free key rings. Much of the heterogeneity between results may disappear when subgroups of trials are analysed. Further exploratory subgroup analyses may show important sources of variation—for example, according to methodological quality, questionnaire topic, age of the study, or type of population. In this review, our aim was to identify eligible trials systematically, critically appraise them, and present the relevant data. We did not intend to produce single effect estimates for every strategy. For many statistically heterogeneous strategies the direction of the effects is the same. For these strategies we cannot be sure about the size of the effect, but we can be reasonably confident that there was an effect on response.

Conclusions

Researchers can increase response to postal questionnaires by using the strategies shown to be effective in this systematic review. Some strategies will require additional materials or administrative time, whereas others can be implemented at little extra cost.

We have presented odds ratios for methodological reasons,7 but the practical implications of the odds ratio for a strategy may be difficult to interpret without knowing the response at baseline without the strategy. If the size of the effect that would be expected if a specific strategy were used is an important consideration for researchers, the data used in this review may be accessed through the Cochrane Library, where they will be updated regularly.8

Alutto JA. Some dynamics of questionnaire completion and return among professional and managerial personnel: the relative impacts of reception at work site or place of residence. J Appl Psychol 1970;54:430-2.

Etter J-F, Perneger TV, Laporte J-D. Unexpected effects of a prior feedback letter and a professional layout on the response rate to a mail survey in Geneva. J Epidemiol Community Health 1998;52:128-9.

Everett SA, Price JH, Bedell A, Telljohann SK. The effect of a monetary incentive in increasing the return rate of a survey to family physicians. Eval Health Professions 1997;20:207-4.

McConochie RM, Rankin CA. Effects of monetary premium variations on response/non response bias: Representation of black and non black respondents in surveys of radio listening. American Statistical Association—Proceedings of the Section on Survey Research Methods, 1985:42-5.

Miller MM. The effects of cover letter appeal and non monetary incentives on university professors' response to a mail survey. Paper presented at the annual meeting of the American Educational Research Association, 1994.

Peters TJ, Harvey IM, Bachmann MO, Eachus JI. Does requesting sensitive information on postal questionnaires have an impact on response rates? A randomised controlled trial in the south west of England. J Epidemiol Community Health 1998;52:130.

See Tai S, Nazareth I, Haines A, Jowett C. A randomized trial of the impact of telephone and recorded delivery reminders on the response rate to research questionnaires. J Public Health Med 1997;19:219-21.