Abstract

Under a grant from the National Science Foundation (NSF),
Westat conducted an evaluation of how different data collection devices affect
data quality, usability, respondent acceptance, and response behaviors for
sensitive questions. This evaluation was unique in that sampled respondents
were assigned to a device type for completing the survey. Devices included a
laptop PC, tablet, or smartphone.

Westat conducted more than 500 in-person interviews across
eight primary sampling units (PSUs) using an address-based sample (ABS).
Respondents were randomly assigned to one of the three device types within each
PSU. Westat developed the web questionnaire, which was accessible only by the
assigned device type, and trained field interviewers on the experimental design
and procedures. Westat also provided respondents with the assigned device type
and mobile web access. Interviewers had MiFi devices that could connect to the
Internet across a cellphone network.

The
web questionnaire included several experiments to test the effect of visibility
as a function of screen size and to examine the potential of mobile devices for
collecting sensitive information. The results will inform and guide the
understanding of how survey data collection is being affected by technological
changes and how respondents use these devices to access surveys. Deliverables
included a number of peer-reviewed publications.

Technical Information

Response Rates: A total of 512 interviews were completed, for an overall response rate (AAPOR RR3) of 30.4 percent. The response rates did not differ
significantly across the three device conditions (smartphone 32.9 percent,
tablet 29.6 percent, and laptop 28.8 percent). Only two cases could not be completed because
of problems with cellular coverage (one each in the smartphone and laptop
conditions).

We selected address-based samples in each
of eight counties around the United States. The counties were selected purposively, based
on the availability of experienced field staff there. Still, the set of eight sample counties encompassed
a range of different types of areas. Seven
were in metropolitan statistical areas and the other was rural; one of the seven
metropolitan areas involved a small central city. The eight counties included all four Census
regions.
Within each county, we selected a full
probability sample of addresses. We used
Census block groups as the second-stage sampling units (SSUs). Six SSUs were selected from each county with probabilities
proportional to size (in housing units). One SSU was on a military base and was replaced
with the next SSU in the county. In total,
we selected 1,824 addresses, 228 within each sample county and 38 in each
sample SSU. One hundred forty-two of the
addresses were vacant, could not be found, or turned out not to be dwellings,
leaving a final sample of 1,682 occupied dwelling units.
When they contacted a household,
interviewers tried to complete a short screener with an adult there. In households with just one adult, he or she was
selected for the main interview; in households with two adults, one of the two was
selected randomly; and, in households with three or more adults, the last
birthday method was used to select one for the main interview. Once the sample person had been designated,
he or she was asked to complete the interview on a device that had been
randomly determined ahead of time. Sample
persons were offered a $20 incentive to complete the questionnaire.