Designing, organizing, and conducting a questionnaire survey is like
establishing and running a business. Although these activities require
technical knowledge and skill, administrative ability, and specific
experience or training, the general impression is that, with a little
luck, anyone can do them well. Thus, many people assume that starting a
business requires no special skills beyond a willingness to try. The
annual failure of thousands of new small businesses demonstrates the
naivety of this assumption. Similarly, the findings from thousands of
questionnaire surveys that are unreliable or misleading prove that
conducting a viable, reliable questionnaire survey is not as easy as it
might seem.

Contrary to common assumptions, a questionnaire is more than a list
of questions. Rather, it is a scientific instrument for measuring and
collecting particular kinds of data. And, like all other scientific
instruments, a questionnaire must be designed in accordance with
particular specifications and tailored to the specific aims of the
surveyor.[2] Only by careful "soup-to-nuts" planning,
construction, and implementation can reliance be placed upon the results
of the questionnaire survey. To assist anyone planning to conduct a
questionnaire survey, this article provides a comprehensive checklist of
items to consider in questionnaire development (figure 1). These items
take the surveyor from the initial survey planning stages through
instrument design and development and on to actual survey
implementation. By following the information and suggestions provided
here, surveyors can use a systematic methodology to collect valid data
and achieve reliable results.

Planning the Survey

What question or questions are to be answered by the survey?

Before the survey can be undertaken, the surveyor must be able to
present its objective clearly and specifically. What is the problem that
the survey will address? Typically, the general nature of this problem
has been determined already. However, the surveyor must further define
the specific problem that is to be studied objectively and
scientifically in the survey.

Next, the surveyor must determine the facts needed to answer the
problem statement. What are the specific questions to be answered by the
survey? What data need to be collected? Collecting information for its
own sake rarely justifies the effort required to assemble and present
it. Therefore, the survey should not pose questions whose answers are
interesting but unessential to the main thrust of the investigation. In
making these determinations, the surveyor should keep in mind the
survey's end user - the agency or individuals who will use the
survey findings. By keeping these potential consumers in mind, the
surveyor is more likely to determine those questions that will be most
meaningful and useful.

How and by whom will the results be used?

This question is related to the previous one. In developing the
survey plan, the surveyor must keep in mind the characteristics and
information needs of the survey's eventual consumers. While it is
not possible to foresee all the misuses to which survey statistics may
be put, many abuses can be prevented if the surveyor is aware of the
consumers' needs.

How large an effort is needed?

Depending on the nature of the problem to be addressed and the amount
of information already available on that topic, the surveyor may want to
use the survey as a small preliminary or pilot study to form the basis
for recommended subsequent research.

This option, however, is rarely available: more often the problem
needs to be addressed in full immediately. In these cases, the surveyor
should have some control or input relative to planning the survey's
required funding and time. The surveyor should ensure that sufficient
funds will be available not only for the collection of the necessary
data but for their tabulation, analysis, and presentation. Similarly,
the surveyor should ensure that sufficient time is allotted for
questionnaire development, distribution, collection, followup, and for
resolving any problems that may arise. If either the survey budget or
schedule is under-estimated, the survey is not likely to be successful.

Is the desired information already available?

No survey should be undertaken until the surveyor is satisfied that
the required facts have not already been collected. Often, a careful
search will show that some of the desired information is already
published or on file. However, if the available facts are inconclusive,
outdated, or otherwise insufficient, the surveyor can profit by avoiding
the mistakes of the earlier survey or by expanding upon those aspects
that the original did not emphasize.[3]

How will information be obtained?

Once the problem to be investigated has been defined and the decision
to conduct a survey finalized, the surveyor should next decide what
survey method should be used to obtain the desired information from the
respondents. Three general survey procedures are available:

* Personal (face-to-face) interview.

* Telephone interview.

* Self-administered questionnaires.

The advantages and disadvantages of these procedures are shown in
figure 2.

What kind of sampling is appropriate?

Although it is sometimes possible to survey an entire population,
generally this technique would result in an unwieldy amount of data. To
limit the population surveyed, a sample is drawn to reflect the
characteristics of the total population. By using a carefully drawn
sample, the surveyor is assured that potential respondents have been
selected in a standard, scientific manner.

Many surveyors assume that a sample can be drawn unsystematically -
e.g., by accident or self-selection - and thereby yield a random sample.
This assumption is incorrect: unsystematic sampling may be biased at
worst or unrepresentative at best. Because many surveyors fail to decide
what population they are sampling, they have no way of knowing if they
are exposing all of the population to the sampling process. A similar
problem exists when the source or list from which the sample is chosen
does not represent the population. Therefore, the surveyor must first
define the survey population and then select a method or methods of
sampling it. Following are descriptions of the most commonly used
sampling types:

* Random sampling.

* Stratified sampling.

* Purposive sampling.

By using two or more of these sampling techniques, a survey can
capitalize on their combined advantages and counterbalance their
disadvantages. This method is called double or mixed sampling.

Random Sampling. Random sampling assures each individual or element
in the population the same chance of being chosen for inclusion in the
survey. If the sample is chosen at random and is sufficiently large, it
will represent all the groups in the population in approximately the
correct proportions. One advantage of random sampling is no advance
knowledge of the characteristics of the population is necessary. A
disadvantage of the technique is that a low percentage of returns could
result in an accidental sample.[3]

Stratified Sampling. Stratified sampling is frequently recommended as
the most efficient procedure for ensuring representativeness.[3] The
procedure calls for dividing the population into two or more strata and
then drawing a random sample from each stratum. Stratified sampling may
be proportional or disproportional depending on the number of samples
selected within each stratum. In proportional stratified sampling,
samples are drawn from each stratum in the same proportion as they occur
in the population. Consequently, the only difference between
proportional sampling and random sampling is that the former ensures
that the right proportion is obtained from each stratum.[3] In
disproportional stratified sampling, an equal number of samples is
secured from each stratum, thereby facilitating reliable comparison of
the different strata. The goal of obtaining an equal number of samples,
however, is very difficult to achieve in actual practice.

Purposive Sampling. The objective of purposive sampling is to make
the sample look like the population. There are several ways of doing
this:

* Select the sample so its averages

ages are the same as the population

averages for known characteristics

believed relevant

to the phenomenon being

investigated.

* Select the sample so its frequency

distribution of each relevant

characteristic resembles

the population's frequency distribution.

* Select the sample so it will contain

the same proportions as

the population's at the upper

and lower ends of the frequency

distribution (deciles,

quartiles, etc.).

* Select the sample so that it's

variability (e.g., standard deviation)

is the same as the population

variability.

One advantage of purposive sampling is it may be cheaper than other
sampling techniques. A major disadvantage is that much must be known
about the population before sample selection.

How big a sample is needed?

The optimum sample size ensures efficiency, representativeness, and
reliability. It should provide the most useful information with the
required level of reliability at the minimum cost. The sample should be
large enough to yield statistically representative and significant
results in all key proposed tabulations and to avoid intolerable sample
error. It should not be so large, however, as to waste funds, delay the
project, or achieve a needlessly high level of precision.

When trying to determine the optimum size of the sample, the surveyor
should follow the guidance provided below. In general, the surveyor
should note that it is sound practice to err on the side of too large a
sample rather than too small.[3]

To determine optimum sample size, the surveyor should perform the
following.

Secure preliminary information about the population. If the sample
will be used to make a percentage estimate of the population, a
preliminary estimate of the approximate size of this percentage is
needed. On the other hand, if the purpose is to predict the average of
the population, the surveyor needs an estimate of the standard
deviation. Fortunately, both these estimates can be rough and still
prove useful.

Determine the required precision of the prediction. This
determination depends upon the error permissible in the estimate and the
degree of confidence that the estimate will fall within the permissible
error.

Calculate the required sample size. For estimating percentages, use

n = pq[z.sup.2]/[t.sup.2] where p = the preliminary estimate

of the percentage q = 100 - p z = the number of standard

error units (from a normal

probability table) T = the required precision or

tolerance Example: Suppose we are permitted a [+ or -] 5-percent
error in estimating from a population the percentage of engineers with
10 years or more of experience. Assume we will be satisfied with a
90-percent degree of confidence that the estimate will fall within the
5-percent tolerance. We have reason to believe that the percentage of
engineers with 10 years or more of experience is somewhere around 75
percent. Substituting in the formula above, we have:

n = (75)(25)[(1.64).sup.2] /[(5).sup.2]

n = 202

n = (75)(25)[(1.64).sup.2]/[5.sup.2]

Thus, we need to take a random sample large enough to assure
responses from about 200 engineers if we want a 90-percent probability
that the true percentage is within [+ or -] 5-percent.

For estimating averages, use [Mathematical Expression Omitted] where

[sigma] = the estimated standard

deviation

z = the number of standard

error units

T = the required precision or

tolerance Example: Suppose we want to estimate the average
design strength of concrete pavement mixes within 100 lb per square in
(psi) (7.0 kg/[cm.sup.2]). Let us assume we will be satisfied with a
95-percent degree of confidence that our estimate will fall within the
100-psi (7.0 kg/[cm.sup.2]) tolerance.

If we use 300 psi (21.1 kg/[cm.sup.2]) as the estimated standard
deviation of concrete pavement mix design strength, we can substitute in
the formula above: [Mathematical Expression Omitted]

n = 35

Thus, we need to take a random sample large enough to assure 35
responses regarding concrete pavement mix design strength if we want a
95-percent probability of estimating the average design strength within
[+ or -] 100 psi (7.0 kg/[cm.sup.2]).

If the preliminary estimate of either the percentage or standard
deviation proves to have been too large, the computed sample size also
will have been larger than necessary and therefore conservative; the
preliminary estimate may also come out too small, with the opposite
result. To help set sample size, the surveyor should try various
estimates in the formula to see what effect each has on the computed
sample size.

Construction of the

Questionnaire

Is there a clear understanding of the topic to be addressed?

If the questionnaire is to be meaningful to others, the surveyor
needs to have a good working knowledge of the topic to be addressed. If
the surveyor does not fully understand the topic, the questions may be
vague or ambiguous, complex issues may be oversimplified, insufficient
space may be provided for responses, etc. In such cases, respondents may
show their frustration by either rushing through the questionnaire,
providing only superficial answers or not responding at all.

Are needed definitions provided?

Related to understanding the survey topic is the concept of
understanding the respondents. The surveyor must: * Understand the
respondents'

language, and use it correctly.

* Not talk down to the respondents.

* Not assume too quickly that respondents

will understand the

question in the intended frame

of reference.

* Ensure that the questions have

the same meaning for all respondents.[4]

Additionally, definitions of key terms used in the questionnaire
should be provided. There should not be too many definitions, nor should
the definitions be lengthy or complicated. The definitions may appear
together in a section above the questions themselves; if only one or two
definitions are to be given, it may be more appropriate to include them
in notes following the respective question. The objective is to get the
respondents to read and understand each definition.

How long should the questionnaire be?

The percentage of questionnaires returned decreases as the number of
pages increases from 1 to 10. (Interestingly, there does not seem to be
a significant difference in the percentage of returns as pages increase
from 10 to 35.) Although evidence suggests that with the proper
incentives and a carefully pretested form, certain groups will respond
thoroughly to a very long questionnaire. However, the best approach is
to keep the form as short as possible to obtain the needed
information.[3]

To keep the questionnaire's length down, the surveyor should
evaluate each proposed question separately. Only those questions bearing
directly on the problem should be included; answers to question that can
be secured from other sources may be deleted unless absolutely necessary
as a check.

Are the right type(s) of questions being asked?

There is no generally accepted type of question that can be
universally recommended for all questionnaires. Rather the type(s) of
questions selected for use in a given questionnaire should depend
primarily on the anticipated tabulation plan (i.e., how will the data be
grouped in the tables?). To determine the appropriate type(s) of
questions to use, the surveyor should both consider the tabulation plan
and the strengths and limitations of the various types of questions as
presented below.

Open-End Questions. Open-end questions give respondents free latitude in making their responses. An example of such a question is "What
is the principal reason for your dislike of product A?" A drawback to their use is that open-end questions may yield such a variety of
responses that the task of condensing and analyzing them is both time
consuming and difficult to handle statistically. Open-end questions are
often used in the exploratory phases of a study when the area covered by
the question is still not well understood.

Dichotomous Questions. These questions elicit either/or responses
(yes/no, true/false, good/bad, etc.) They usually also allow for a third
response such as "don't know," "neither," etc.
Surveryors, however have found that if the noncommittal third
alternative is not given, relatively few respondents will fail to make a
choice between the two options presented, thereby "forcing"
some respondents to take definite stand when actually they would not do
so otherwise. The chief advantage of the dichotomous question is its
simplicity. One of its limitations is that a slight misunderstanding of
the question's meaning may result in a complete reversal of
response. (3)

Multiple Choice Questions. Multiple choice questions allow
respondents to choose from among several possible answers. The list of
alternatives must be complete enough to cover all possible answers.
Multiple choice question results can be tabulated easily and interpreted
with greater precision than can answers to open-end questions. Multiple
choice questions may take any of several forms; arguably the most
popular of these are checklists and rating scales.

Checklists. Checklist are multiple choice questions consisting of a
statement of the problem or question followed by a list of three or more
possible answers. Checklists should be made up only after some
preliminary survey or pretest has determined the types of replies that
may be expected; this ensures that most answers will not fall in the
"other" category that is usually provided with a limited list.
The advantage of the checklist is that it serves to remind respondents
of their various options.

Rating Scales. Rating scales allow respondents to choose among
various degrees of opinion. The scale may be numerical (e.g., ranging
from 1 to 5) or verbal (e.g., ranging from "poor" to
"excellent"). Most dichotomous questions can be turned into
rating scale questions simply by introducing degrees of opinion to the
response options. After doing so, the extreme responses can be sorted
out for use in comparative analyses where a high statistical
significance level is desired. One problem with rating scale questions,
however, is there is often a tendency for the respondent to choose a
moderate rating. Ranking of Items. In ranking of items questions, the
respondent is asked to arrange a list of words, phrases, statements,
pictures, etc., in some order. The ranking adds a dimension not found in
checklists. However, a limitation of these questions is the fact that
the size of the intervals between rankings is usually not the same: the
difference between the first and second choices may be much greater than
the difference between the second and third choices. Another limitation
is the relatively small number of items that may be included. It has
been suggested that as few as three choices give the best results; the
optimum number probably varies with the subject matter. (3)

Are questions properly worded?

Asking questions to gain desired responses is more difficult than it
may seem. The greatest danger in wording questions lies in taking too
much for granted regarding language and terminology, reader
comprehension and background, etc. Surveyors must develop a critical
attitude toward their questions and should carefully analyze the wording
of each. The critique is best done during pretest of the questionnaire.
Every objection that may be raised about the phrasing should be
carefully studied and answered. (4)

Following are some suggestions for wording questions:

* Use simple words and correct punctuation

* Use underline or boldface type for emphasis.

* Make questions concise and unambiguous.

* Formulate questions to yield exactly the information desired (e.g.,
specify the units for the answer) and avoid leading questions, i.e.,
questions worded in such a way as to suggest the answers.

* Avoid words such as "always" or "any,"
"about" or "near," "possible," and
"should." When using "you" or "your," make
it clear whether the second person singular or plural is intended.

* Cover only one point per question. If there are two or more ideas
in the question, it will be confusing to determine to which thought the
answer applies.

* Allow for all possible responses, i.e., multiple choice answers
should be mutually exclusive and complementary and provide for
"don't know" and "other" answers.

* Keep asking "What am I assuming or taking for granted by this
question?"

Are questions in logical order?

To avoid confusion and misunderstanding, questions should be arranged
logically. The questionnaire should, for example, open with a question
that can be easily answered, preferably with a simple yes or no. If a
difficult question is asked at the beginning, respondents may refuse to
continue with the questionnaire.

After this important opening question, there must be an
easy-to-follow logic, an established order, to the form. Order should be
considered at three presentation levels:

* Topics covered by the questionnaire.

* Questions under each topic.

* Response alternatives to a single question.[5]

The surveyor should note that once a train of thought has been
established, breaking that sequence can lead to poor results. Thus, a
totally unrelated question, suddenly, asked, is not desirable.

Is the questionnaire attractive?

An attractive questionnaire is a strong selling point for respondent
cooperation. Attractiveness can be accomplished by the appropriate use
of colors, pictorial materials, spacing, type, arrangement, etc.
Research has shown that yellow paper - followed closely by pink - has
the highest percentage of returns; dark-colored questionnaires usually
are not effective. (3) The questionnaire should look easy to complete.
This goal can be met by requiring very little writing of the respondent
and/or arranging the various items so the questionnaire does not appear
crowded.

Sending the Questionnaire

Has the questionnaire been pretested?

Regardless of how carefully the questionnaire has been worded, it
should not be assumed that it will work well until it has been tested
under field conditions. A pretest can be used to check the wording and
sequence of questions, questionnaire length, clarity of instructions,
effectiveness of cover letter, and other related items. Pretests also
enable the surveyor to turn open-ended questions into multiple choice
ones.

A pretest is normally done by furnishing the questionnaire to typical
respondents, reviewing their responses, and interviewing them to get
their reactions, suggestions, and other comments. Based on this
feedback, the questionnaire is then revised as appropriate. Especially
difficult questionnaires may require several revisions.

Another way to pretest the questionnaire is by developing several
different drafts and simultaneously sending a different draft to each of
several small samples of respondents. Such a pretest will provide
comparative information on relative rates of return, data completeness
and appropriateness, data suitability for tabulation, etc.[3]

Is the cover letter effective?

A letter of transmittal soliciting respondent cooperation should
accompany the questionnaire. This letter should:

* Be on official letterhead.

* Explain the purpose of the survey and the uses to be made of the
data.

* Provide instructions on how to fill out the form (if such
instructions are not printed on it).

* Thank the respondent for cooperating.

Customized, personal touches to the letter can be effective in
motivating potential respondents. Examples include a hand-written
postscript or personal signature.[3] Another possible
"motivator" is including self-addressed stamped or franked
envelopes. Finally, promising to furnish survey results to respondents
can also be effective. Of course, such a promise must be fulfilled in a
timely manner.

When should the questionnaire be sent?

The surveyor must determine the most opportune time for sending out
the questionnaire. This choice may be based on several factors, since
the season of the year, the month, week, or time of the week during
which a questionnaire should arrive in order to have the best chance of
being filled out will vary with the type of respondent. (For example, a
survey of construction engineers might best be done in the winter months
when work is not at its peak.) The surveyor should also take into
account the timing of the mailing vis-a-vis periods such as holidays,
extremely hot weather, and important national events.

When and how to followup?

Followup of some kind is usually required to ensure an adequate rate
of response. (3) The first followup might take the form of a short note
or postcard reminding the recipient of the overdue questionnaire. A
tactful suggestion may be included to disregard the reminder if the
reply is on the way. The second reminder may include a copy of the
questionnaire since the respondent may have misplaced the original. This
second followup may also take the form of a telephone call.

The first followup activity should be undertaken when the daily
returns have dwindled down to a trickle. Subsequent followups, if
needed, should be spaced closer and closer together.

The acceptable percentage of responses depends on such variables as
type of respondent, size of the questionnaire, subject matter, etc.
However, it is not unrealistic to expect a 75-percent or greater rate of
response for most questionnaires.[6]

Summary

This article is intended to provide guidance for the development of
generic questionnaires. While the examples provided are engineering- or
highway-related, the discussion may be equally applicable to
questionnaires in other fields.

There is more to developing good questionnaires than just wording
good questions. Weeks of planning and exploratory work are often needed.
Thus, it is important that the surveyor schedule sufficient time.
Successful questionnaire development includes identifying the specific
problem, understanding the subject matter, and knowing the respondents.
Appropriate survey method, sample size, type of sampling, and type of
questions have to be given adequate consideration. The use of pretest is
probably the greatest help in devising the actual wording of questions.
The surveyor should also think well ahead of the data analysis phase.
Because each survey tends to present its own problems, no amount of
advice or guidance can substitute for thoughtful care, common sense, and
good organization.

[6] Office of Management and Budget, "Memorandum on Reporting
Burden Reduction Program to Heads of Executive Departments,"
February 17, 1977, Washington, DC.

PHOTO : Figure 1. - Questions development checklist.

PHOTO : Figure 2. - Advantages and disadvantages of three popular
survey procedures.[3]

Peter A. Kopac is a research highway engineer in the Pavements
Division, Office of Engineering and Highway Operations Research and
Development, Federal Highway Administration (FHWA). He is a program
manager for the Nationally Coordinated Program, "Highway
Maintenance." Since joining the FHWA in 1977, Mr. Kopac has worked
in the areas of statistical quality control, data collection and
analysis, and performance-related specifications. His responsibilities
have included management and design of numerous research studies that
relied on questionnaire surveys.

COPYRIGHT 1991 Superintendent of Documents
No portion of this article can be reproduced without the express written permission from the copyright holder.