Methods of Data Collection

Census. A census is a study that obtains data
from every member of a
population.
In most studies, a census is not practical, because of the
cost and/or time required.

Sample survey. A sample survey is a study
that obtains data from a subset of a population, in order
to estimate population attributes.

Experiment. An experiment is a controlled
study in which the researcher attempts to understand
cause-and-effect relationships. The study is "controlled" in
the sense that the researcher controls (1) how subjects are
assigned to groups and (2) which treatments each group
receives.

In the analysis phase, the researcher compares group scores on
some
dependent variable.
Based on the analysis, the researcher draws a conclusion about
whether the treatment
(independent variable) had a causal effect on the
dependent variable.

Observational study. Like experiments,
observational studies attempt to understand cause-and-effect
relationships. However, unlike experiments, the researcher
is not able to control (1) how subjects are
assigned to groups and/or (2) which treatments each group
receives.

Data Collection Methods: Pros and Cons

Each method of data collection has advantages and disadvantages.

Resources. When the population is large, a
sample survey has a big resource advantage
over a census. A well-designed
sample survey can provide very precise estimates of population
parameters - quicker, cheaper, and with less
manpower than a census.

Generalizability. Generalizability refers to the
appropriateness of applying findings from a study to a larger
population. Generalizability requires random selection. If
participants in a study are randomly selected from a larger
population, it is appropriate to generalize study results to
the larger population; if not, it is not appropriate to generalize.

Observational studies do not feature random selection; so generalizing
from the results of an observational
study to a larger population can be a problem.

Causal inference. Cause-and-effect
relationships can be teased out when subjects are
randomly assigned to groups. Therefore, experiments, which allow
the researcher to control assignment of subjects to treatment
groups, are
the best method for investigating causal relationships.

Test Your Understanding

Problem

Which of the following statements are true?

I. A sample survey is a type of experiment.
II. An observational study requires fewer resources than an experiment.
III. The best method for investigating causal relationships is an
observational study.

(A) I only
(B) II only
(C) III only
(D) All of the above.
(E) None of the above.

Solution

The correct answer is (E). Unlike an experiment, a sample survey does not require the researcher
to assign treatments to survey respondents. Therefore, a sample
survey is not necessarily an experiment. A sample survey could be an observational study, rather than an experiment.
An observational study may or may not require fewer
resources (time, money, manpower) than an experiment. The best
method for investigating causal relationships is an experiment - not
an observational study - because an experiment features randomized
assignment of subjects to treatment groups.