Researchers usually cannot make direct observations of every individual
in the population they are studying.
Instead, they collect data from a subset of individuals  a sample  and
use those observations to make inferences about the entire population.

Ideally, the sample corresponds to the larger population on the characteristic(s) of interest.
In that case, the researcher's conclusions from the sample are probably
applicable to the entire population.

This type of correspondence between the sample and the larger population
is most important when a researcher wants to know what
proportion of the population has a certain characteristic  like a particular
opinion or a demographic feature.
Public opinion polls that try to describe the percentage of the population
that plans to vote for a particular candidate, for example, require a
sample that is highly representative of the population.

Probability samples and convenience samples

Two general approaches to sampling are used in social science research.
With probability sampling, all elements (e.g.,
persons, households) in the population have some opportunity of
being included in the sample, and the mathematical probability
that any one of them will be selected can be calculated.

Recruiting a probability sample is not always a priority for researchers.
A scientist can demonstrate that a particular trait occurs in a population by
documenting a single instance.
For example, the assertion that all lesbians are mentally ill can be refuted by
documenting the existence of even one lesbian who is free from psychopathology.

Another situation in which a probability sample is not necessary is when a
researcher wishes to describe a particular group in an exploratory way.
For example, interviewing 25 people with AIDS (PWAs) about their experiences with
HIV could provide valuable insights about stress and coping, even though it would
not yield data about the proportion of PWAs in the general population who share
those experiences.

Types of probability samples

Many strategies can be used to create a probability sample.
Each starts with a sampling frame, which can be thought of as a list of all
elements in the population of interest (e.g., names of individuals, telephone numbers,
house addresses, census tracts).
The sampling frame operationally defines the target population from which the sample is drawn
and to which the sample data will be generalized.

Probably the most familiar type of probability sample is
the simple random sample, for which all elements in the sampling frame
have an equal chance of selection,
and sampling is done in a single stage with each element selected independently
(rather than, for example, in clusters).

Somewhat more common than simple random samples are systematic samples, which
are drawn by starting at a randomly selected element in the sampling frame and then
taking every nth element (e.g., starting at a random location in a telephone
book and then taking every 100th name).

In yet another approach, cluster sampling, a researcher
selects the sample in stages, first selecting groups of elements, or clusters
(e.g., city blocks, census tracts, schools), and then selecting individual
elements from each cluster (e.g., randomly or by systematic sampling).

An example

Suppose some researchers want to find out which of two mayoral
candidates is favored by voters.
Obtaining a probability sample would involve defining
the target population (in this case, all eligible voters in the city)
and using one of many available procedures for selecting a
relatively small number (probably fewer than 1,000) of those
people for interviewing.
For example, the researchers might create a systematic sample by obtaining
the voter registration roster, starting at a randomly selected name, and
contacting every 500th person thereafter.
Or, in a more sophisticated procedure, the researchers might use a computer
to randomly select telephone numbers from all of those in use in the city,
and then interview a registered voter at each telephone number.
(This procedure would yield a sample that represents only those people who have a telephone.)

Several procedures would also be available for recruiting a convenience sample,
but none of them would include the entire population as potential respondents.
For example, the researchers might ascertain the voting preferences
of their own friends and acquaintances.
Or they might interview shoppers at a local mall.
Or they might publish two telephone numbers in the local newspaper and ask readers to
call either number in order to "vote" for one of the candidates.
The important feature of these methods is that they would
systematically exclude some members of the population
(respectively, eligible voters who do not know the
researchers, do not go to the shopping mall, and do not read the newspaper).
Consequently, their findings could not be generalized to the population of city voters.

Evaluating samples

Samples are evaluated primarily according to the procedures by
which they were selected rather than by their final composition or size.
In the example above, it would be impossible to know if the convenience sample consisting
of the researchers' friends or mall shoppers is representative,
even if its demographic characteristics closely resembled
those of the city electorate (e.g., the same ratios of women to
men and Blacks to Whites).
And even if several thousand people called the published telephone numbers,
the sample would be seriously biased.

Of course, results from a probability sample might not be accurate for many reasons.
Using probability sampling procedures is necessary but not sufficient for obtaining results
that can be generalized with confidence to the entire population.
One of the major concerns about a probability sample is
that its response rate is sufficiently high.

Response rates. Once a sample is selected, an attempt is made to collect data (e.g., through interviews or questionnaires) from all of its members. In practice, researchers never obtain responses from 100% of the sample. Some sample members inevitably are traveling, hospitalized, incarcerated, away at school, or in the military. Others cannot be contacted because of their work schedule, community involvement, or social life. Others simply refuse to participate in the study, even after the best efforts of the researcher to persuade them otherwise.

Each type of nonparticipation biases the final sample, usually in unknown ways. In the 1980 General Social Survey (GSS), for example, those who refused to be interviewed were later found to be more likely than others to be married, middle-income, and over 30 years of age, whereas those who were excluded from the survey because they were never at home were less likely to be married and more likely to live alone (Smith, 1983). The importance of intensive efforts at recontacting sample members who are difficult to reach (e.g., because they are rarely at home) was apparent in that GSS respondents who required multiple contact attempts before an interview was completed (the "hard-to-gets") differed significantly from other respondents in their labor force participation, socioeconomic status, age, marital status, number of children, health, and sex (Smith, 1983).

The response rate describes the extent to which the final data set includes all sample members. It is calculated as the number of people with whom interviews are completed ("completes") divided by the total number of people or households in the entire sample, including those who refused to participate and those who were not at home.

Whether data are collected through face-to-face interviews, telephone interviews, or mail-in surveys, a high response rate is extremely important when results will be generalized to a larger population. The lower the response rate, the greater the sample bias. Fowler (1984), for example, warned that data from mail-in surveys with return rates of "20 or 30 percent, which are not uncommon for mail surveys that are not followed up effectively, usually look nothing at all like the sampled populations" (Fowler, 1984, p. 49). This is because "people who have a particular interest in the subject matter or the research itself are more likely to return mail questionnaires than those who are less interested" (p. 49).

Fowler (1984) warned that: "[O]ne occasionally will see reports of mail surveys in which 5 to 20 percent of the sample responded. In such instances, the final sample has little relationship to the original sampling process. Those responding are essentially self-selected. It is very unlikely that such procedures will provide any credible statistics about the characteristics of the population as a whole" (p. 48).

Sample size and sampling error. The use of appropriate sampling methods and an adequate response rate are necessary for a representative sample, but not sufficient. In addition, the sample size must be evaluated.

All other things being equal, smaller samples (e.g., those with fewer than 1,000 respondents) have greater sampling error than larger samples. To better understand the notion of sampling error, it is helpful to recall that data from a sample provide merely an estimate of the true proportion of the population that has a particular characteristic. If 100 different samples are drawn from the same sampling frame, they could potentially result in 100 different patterns of responses to the same question. These patterns, however, would converge around the true pattern in the population.

The sampling error is a number that describes the precision of an estimate from any one of those samples. It is usually expressed as a margin of error associated with a statistical level of confidence. For example, a presidential preference poll may report that the incumbent is favored by 51% of the voters, with a margin of error of plus-or-minus 3 points at a confidence level of 95%. This means that if the same survey were conducted with 100 different samples of voters, 95 of them would be expected to show the incumbent favored by between 48% and 54% of the voters (51% ± 3%).

The margin of error due to sampling decreases as sample size increases, to a point. For most purposes, samples of between 1,000 and 2,000 respondents have a sufficiently small margin of error that larger samples are not cost-effective. However, if subgroups are to be examined, a larger sample may be necessary because the margin of error for each subgroup is determined by the number of people in it. For example, although a national survey with a probability sample of 1000 adults has a margin of error of roughly 1-3 percentage points (using a 95% confidence interval), analyses of responses from the African Americans in that sample (who would probably number about 100) would have a margin of error of roughly 4-10 points.

Other considerations

This brief discussion has focused on sampling procedures. However, many other factors also affect the quality of data from a research study. For example, it is always important to critically evaluate the specific procedures used for obtaining responses, including the questions that were asked.

In order for research data to be meaningful, the questionnaire and the procedures used to collect the data must be valid. The validity of a method (e.g., a survey questionnaire) refers to how accurately it measures what it is supposed to measure. If survey items are so complex or ambiguous that different respondents interpret them differently, for example, their validity is compromised. Validity is also threatened if respondents do not provide accurate or honest answers, either because of their inability to do so (e.g., due to memory problems) or their unwillingness to answer truthfully (e.g., because the researchers communicated their biases or expectations to the respondents).

Samples in social and behavioral research

Most behavioral and social science studies use convenience samples consisting of students, paid volunteers, patients, prisoners, or
members of friendship networks or organizations. Studies with
such samples are useful primarily for documenting that a
particular characteristic or phenomenon occurs within a given group or,
alternatively, demonstrating that not all members of that group
manifest a particular trait. Such studies are also very useful for detecting relationships among different phenomena.

Sometimes matched convenience
samples are used to compare two groups (e.g., psychological test scores of gay people and
heterosexuals). With this procedure, each individual in the first
sample has a counterpart in the second sample who is of the same gender,
race, educational background, age, or whatever other
characteristics are judged to be relevant. The purpose of
matching is to eliminate known sources of bias; however, the
problem of potential bias from hidden sources still remains.

With a hard-to-reach population (e.g., gay people or persons
who engage in homosexual behavior), a series of studies with
nonprobability samples can suggest rough estimates of the
proportion of the population manifesting various characteristics.
When similar results are obtained repeatedly with many different
nonprobability samples, the likelihood that those results apply
to the population is greater than when only a single
nonprobability sample is used. Nevertheless, inferences based on
such data must be cautious because of the possibility of hidden
systematic bias.

Strictly speaking, inferences cannot be drawn
from a nonprobability sample about the proportion of the
population manifesting (or not manifesting) a particular
characteristic. Realistically, however, funding limitations and
the methodological difficulties of sampling a relatively small
and partially hidden population have usually prohibited the use of
probability samples in research on sexual orientation.

It is extremely important, therefore, that findings obtained with convenience samples be critically evaluated. Readers should always ask the following questions:

What types of people were systematically excluded from the sample?

What types of people were over-represented in the sample?

Have the findings been replicated by different researchers using a variety of data-collection methods with different samples?