An earlier version of this essay was presented at the 1985 Annual Meeting of the International Society
of Political Psychology, Washington, D.C. I am indebted to Jerry Gaus, Judith Gillespie, Richard
Hudelson, Don Kurtz, Beth Lau, Dana Ward, and several anonymous journal reviewers for their
encouragement and/or intelligent commentary. I remain, of course, solely responsible for any lapses of
judgment.

DEFINITIONAL VALIDITY: THE CONCEPT
AND A CASE STUDY

ABSTRACT

All researchers implicitly make three well-known validity claims for their research: of "internal
validity", of "external validity", and (implicitly) of substantive importance ("substantive validity").
However, researchers also need to examine a less well-known validity claim: that the research's
methods and measures conform to researchers' implicit claims for the substantive significance of their
work ("definitional validity"). This article sets forth the claim of definitional validity and explains its
use by application to two fields: the measurement of economic inequality and the measurement of belief
constraints. xx on inequality. The article argues that the belief regularities contemplated by correlational
measures of constraint are not consonant with the substantive importance constraint is held to have for
democracy. These critiques are not new; the contribution of this article is to recognize the claim of
definitional validity and demonstrate how it organizes and clarifies the significance of otherwise diverse
criticisms.

DEFINITIONAL VALIDITY: THE CONCEPT
AND A CASE STUDY

Social-scientific research derives its claim to significance from four successive validity claims - four
inferences whereby the immediate research methods, data and measure acquire significance as shedding
light on issues of acknowledged ethical consequence. Three of these claims are familiar: the claims of
internal validity; of external validity; and of substantive significance, which we will call substantive
validity, both for parallelism's sake and because it is indeed a validity claim. The claim of definitional
validity, however, is not a familiar one, even though theorists already implicitly evaluate research in
terms of it. The purpose of this paper is to define this claim clearly, to show its logical relationship to
the claims of internal, external and substantive validity, and to apply this concept to the case of
constraint research to show how the claim is tested and how the failure to redeem the claim can
undercut an entire research tradition.

Beyond the above discussion of the four validity claims and the associated explanation of their relation
to one another, the purpose of this article is to elaborate the unfamiliar claim of definitional validity by
evaluating the claims to it made by belief systems research, particularly that using correlational
constraint measures. The point of the article is not to develop new attacks on constraint research but
instead to show how earlier, diverse criticisms all flow coherently and straightforwardly from the
evaluation of definitional validity. Indeed, almost all of the critiques appearing in this evaluation have
been made previously, and it would be strange if we found many new ones: constraint research has been
subjected to nearly thirty years of theoretical scrutiny, and such scrutiny has always been conducted
with at least a background recognition that methods must match their research's substantive
significance. This essay simply brings this recognition to the foreground, and constraint research
happens to provide good examples.

Note that the claim to definitional validity is made, implicitly or explicitly, by all research, not just
constraint research. Constraint research happens to be an exceptionally good example of where previous
theoretical critiques can almost all be gathered under the one rubric of analyses of definitional validity.

1. Social Research's Four Validity Claims

The four inferences whereby social-scientific research sets claim to significance are shown in Figure 1
below.(1)

Figure 1: Research projects' four validity claims

Observational methods and analytic measures

|

|

| INTERNAL

| VALIDITY

|

Empirical findings & interpretations applied

solely within the domain of the study

|

|

| EXTERNAL

| VALIDITY

|

Conclusions about society in general

|

|

| DEFINITIONAL

| VALIDITY

|

Relevance to issues believed by the

researcher to be of ethical consequence

|

|

| SUBSTANTIVE

| VALIDITY

|

Relevance to issues of ethical consequence

To start with, researchers must establish that their choice of data collection and data analysis methods
supports their empirical findings and interpretations of what is happening in the immediate
experimental setting. This issue of internal validity concerns whether the outcome of the process being
studied is a consequence of the explanatory variable(s) or of some other variable or process. In other
words, are there alternative explanations for the results within the boundaries of the research study
itself?

The next question is whether the interpretations can be generalized to some intended domain of
generalization beyond the research project. This issue of external validity asks whether the experimental
results or interpretations are likely to apply outside the immediate context of the experiment. Do the
findings represent a general law, or only one highly contingent on the specific research context?

Skipping over the third validity claim, we take up the fourth claim of substantive validity: that the
results are of substantive significance. The claim here is that we want to know these results for reasons
beyond the sheer accumulation of curious facts. In other words, researchers would like to claim that the
results make some important difference to the way we view (or act within) the social world.

The claim of definitional validity mediates between the first two claims and the fourth claim. The claim
to definitional validity states that the researcher's measures are consonant with her substantive intent.
The researcher wishes to assert that her findings actually address the questions she sees as substantively
significant. Her judgment of substantive importance may be idiosyncratic, but others' agreement is not
the issue: the issue of definitional validity only concerns whether the interpretations speak to her sense
of substantive value.

There is, of course, a metaethical debate over the truth status of moral claims: whether they have
objective truth, intersubjective truth, or no truth at all beyond individual opinion. People who hold that
moral claims have no truth value at all can simply ignore the final step of substantive validity. The real
focus of this paper (the issue of definitional validity) is not affected, however.

But suppose that we accept that claims to substantive validity do have some truth status: why then
should we be concerned with definitional validity? Why attend to what the researcher feels is important
about the research instead of dealing directly with the claims of real substantive validity? The answer is
that even if claims to substantive importance have some truth status, they very often remain unsettled
for a long time. Claims to definitional validity, on the other hand, can be evaluated rather
straightforwardly (as the present essay shows), involving a dispute only over the hermeneutic
interpretation of what the researcher believes the substantive significance of her findings are. In other
words, the evaluation of definitional validity does not negate that of substantive validity, but it can be
much easier to handle.

To summarize, of these four claims, only that of substantive validity projects the methodologist into
directly moral argumentation. The first two claims depend only on logic and empirical judgment, and
while the claim of definitional validity may require a hermeneutic analysis of the text in order to infer
the researcher's substantive concerns, it does not require a review of the actual significance of those
concerns.(2)

To see concrete examples of these four validity claims, let us look at Cary Covington's (1988) recent
work on presidential coalitions. Briefly, his "study examines roll-call votes in the House of
Representatives identified by the White House as 'critical' motions during the period 1961-67 to
determine the extent to which cross-pressured but supportive Democrats supported the presidents'
positions by means other than voting for them. ... Findings demonstrate that ... cross-pressured
supportive Democrats abstain [a form of 'alternative support'] on key votes more often than consonantly
pressured supporters..." (Covington 1988:3). The study raises the four validity claims, as exemplified by
sample claims listed below:

Sample Claims of Internal Validity: Cross-pressured Representatives abstain, switch, and pair to some
extent because of Presidential pressure. To state this in another way, there is no variable that partials out
the observed relationship in this dataset between the importance Presidents attached to different bills
and the likelihood of cross-pressured Representatives providing alternative modes of voting support.

Sample Claims of External Validity: The period 1961-1967 is representative of the American national
political process generally. They apply to Republican as well as Democratic administrations.

Sample Claims of Definitional Validity: The measure of cross-pressure really measures cross-pressure;
ditto for the measures of "critical motions" and "alternative support". In particular, "alternative support"
really is support for the President's position to some degree and "critical motions" can be chosen by the
President independent of constraints of party and ideological balance.

Sample Claims of Substantive Validity: The findings show that our political system gives scope to
individual action, so that concepts of skill and will remain important for political actors beyond such
impersonal constraints as party dominance, ideological balance, and presidential popularity.(3)

If we look at standard social science methodology articles and texts, we observe that methodologists
concentrate primarily on internal validity, markedly less on external validity, and little at all on
definitional validity. Issues of substantive validity are considered to be the concern of political theory,
and are virtually unconsidered in methodological discussion.(4)

This pattern of attention is a natural consequence of the general social-scientific division between issues
of fact and value (and avoidance of the latter). Methodologists can handle most issues of internal (and,
to a lesser degree, external) validity by reference to accepted standards of logical inference -- i.e., the
canons of the scientific method. Issues of whether findings are truly of substantive significance, on the
other hand, can be resolved only by arguments involving ethics, for which we have as yet no settled
standards. Given the very different character of these two domains of discourse, and given the settled
nature of one relative to the other, it should be no surprise that most methodological attention has been
to the former domain, where relatively firm conclusions are possible and much work still remains to be
done. It is clear that issues of definitional validity are important ones, however. If research methods do
not support any sense that the findings have substantive significance, then we have no reason to pay
attention to the findings, regardless of the methods' internal and external validity.

Under what circumstances is this scrutiny of definitional validity possible? It is obviously possible
whenever there is no dispute about the substantive consequence of the research question. If everyone
agrees on the nature of our concern about a certain issue, then we methodologists are on firm ground
when we ask whether the methods and measures yield conclusions addressing that concern. But even
when the research's substantive concerns are disputed, we can still ask whether the methods yield
conclusions addressing the researcher's concern. We need not deal with the issue of substantive validity
-- that is, about whether the researcher's concern is of real moral consequence. We are always entitled to
ask whether the researcher's methods and moral intentions form a consistent whole; we need not get
trapped in debates about what has true substantive consequence.

One difficulty with the evaluation of definitional validity is that researchers frequently do not state
explicitly what claims of substantive significance they are making for their research. Working in an
existing tradition of inquiry, they customarily take their research's significance for granted. To evaluate
definitional validity, then, we will often have to infer the nature of the significance intended, and our
evaluation will be conditioned upon our having gotten the sense of significance right.

2. Belief Systems Research Asserts Substantive Significance

Mainstream analyses of democratic politics commonly view it as the politics of belief systems: a
process by which policy preferences and the beliefs embedding them are translated into political and
finally governmental action. Political debates involve the clash of beliefs over both the factual
correctness of various assertions and the virtue of adopting various policies. The outcomes of these
debates are correspondingly affected by the distribution and organization of beliefs about such
assertions and policies.

Ideally, the citizens of a working democracy would have rational belief structures: accurate in
assessment of the world (realism); containing no internal contradictions (logical consistency); and
smoothly linking means to ends (goal rationality).(5) Unfortunately, citizens (including researchers!)
have no objective way to assess belief systems' realism, logical consistency, and goal rationality. Belief
systems theorists accordingly pay most of their attention to the looser demands of what we call
ideology. Quite reasonably, researchers hold that if they can't determine rationality, they must settle for
what is feasible: the determination of ideological coherence. Belief systems are ideologically coherent
when they are "particularly elaborate, close-woven, and far-ranging" (Campbell et al. 1960:192).
Ideology "links particular actions and mundane practices with a wider set of meanings" (Apter,
1964:16).

Belief systems theorists argue the substantive importance of belief systems structure in two major
ways.(6) First, ideology facilitates democratic politics. In a society where only a limited number of
ideologies are present, political leaders can readily search for and attract support, whereas in a society
with no ideologies or with thousands, each new political issue requires the laborious construction of
completely new alliances. Further, a unidimensional variation of ideologies is especially helpful for
party groupings and electoral appeals. Early researchers like V. O. Key, Jr. (1961) and Campbell et al.
(1960) emphasize the importance of belief systems research for an understanding of our political
system.(7)

Second, structured belief systems also are seen as desirable for individual political actors, not just for
the political system as a whole. Ideology gives coherence and meaning to political issues, enabling
individuals to grasp and react to new issues more readily. Even though logic may not apply to belief
systems, ideology purports to provide the same assurances as logic: that goals are not self-contradictory
and that means are appropriate and efficient. In short, ideology is supposed to facilitate citizens' ability
to act effectively in political contexts.

The focus on individuals is especially marked in the seminal works of Campbell et al. (1960) and
Converse (1964), most explicitly in the latter. Converse lauds "the idea organization that leads to
constraint" [his term for ideological structure], because it "permits [the actor] to locate and make sense
of a wider range of information from a particular domain than he would find possible without such an
organization. ... The efficiency of such a yardstick in the evaluation of events is quite obvious" <214:
bracketed page numbers refer to Converse, 1964>. "In our estimation, the use of [ideologies] betokens a
conceptual grasp of polities that permits a wide range of more specific ideo-elements to be organized
into more tightly constrained wholes. We feel, furthermore, that there are many crucial consequences of
such organization: With it, for example, new political events have more meaning, retention of political
information from the past is far more adequate, and political behavior increasingly approximates that of
sophisticated 'rational' models, which assume relatively full information" <227>. When Converse
discusses his coding of the respondents' answers as "broadly philosophical" <222>, a footnote indicates
his preference for ideological responses: "In all candor, it should probably be mentioned that a teacher
grading papers would be unlikely to give passing marks to more than 20% of the attempted
definitions.... We made an effort, however, to be as generous as possible in our assignments" <257:
emphasis added>. In summary, Converse takes a directly evaluative stance toward the ideological
structure of belief systems: they make "adequate" the retention of political information, and the political
actor becomes more "efficient" in "making sense of" or "giving meaning to" the political environment.
Such structure is "sophisticated," "rational," and -- the academic emerging -- will receive "passing
marks".

3. Measuring Belief System Structure through Correlational
Constraint

Belief systems research has two potential standards by which to judge how ideologically structured
people's beliefs are (Barton and Parsons, 1977). The first approach, applicable to individuals considered
singly, employs standards of logic or "psycho-logic". Such standards are implicitly applied when
researchers content-analyze subjects' open-ended responses, looking for the presence of coherent
approaches to issue, party, candidate or other evaluations. Originally, this content analysis was simply a
matter of finding key words or classifiable statements. In recent years, however, researchers have come
to employ the more sophisticated methods associated with Piagetian psychology (e.g., Ward, 1982;
Rosenberg, Ward, and Chilton, 1989) and schema theory (e.g., Axelrod, 1973; Fiske and Linville, 1980;
Conover and Feldman, 1984).

This essay, however, is concerned not with this first approach to assessing ideological structure but
rather with a second general approach: the "constraint" or "model-fitting" approach. This second
approach, applicable only to people considered in groups, employs standards derived from the group as
a whole. In these methods, response patterns are collectively fit to a statistical model of what an
ideologically structured population looks like. Greater aggregate fit then implies more ideological
structure. Several models have been used:

The mean correlation measure. This measure was suggested by Campbell et al. (1960:191) and was
implemented by Converse (1964:227-231). Its model assumes that all opinion questions range along
a single common ideological scale (e.g., liberalism-conservatism), so that for an ideologically
constrained population we should observe perfect correlations between different opinion questions.

The Guttman scale measure. This measure was also suggested by Campbell et al. (1960:191-192),
but apparently has never been used. Its model assumes that all belief questions are cutting points on
an underlying ideological continuum, so that in an ideologically constrained population, the
responses will jointly show and define a Guttman scale and its response pattern.

Factor-analytic and other structural model measures. The mean correlation measure is the primitive
ancestor of a variety of increasingly elaborate measures. All of these, however, share the mean
correlation measure's assumption that the ideologically constrained population shares a common
orientation to one (or very few) underlying ideological dimensions, even though individual positions
on the dimension(s) may differ. This approach is taken by Luttbeg (1968) and Nie and Rabjohn
(1979), who use principal components analysis; by Marcus, Tabb, and Sullivan (1974), who use
individual differences scaling (INDSCAL)(8); by Achen (1975), who uses a simple form of structural
model to control for individual and question error variation; and by Sullivan et al. (1979), who use
confirmatory maximum likelihood factor analysis to assess the unidimensionality of tolerance items
while controlling for time period, target group, and act.

Applying the above methods to the U.S. population, constraint researchers have obtained such findings
as: elites are more ideologically constrained than the population at large (e.g., Converse, 1964); and the
population has become more ideologically constrained of late (e.g., Nie and Andersen, 1974).

4. Correlational Constraint Measures Have No Definitional
Validity

These conclusions have been vigorously attacked on many grounds.(9) The present essay evaluates the
constraint methods in terms of their definitional validity, not their internal, external, or substantive
validity. This evaluation has three parts. First, we interpret the constraint research literature to determine
what claims it makes for substantive significance. This was the burden of the last section, which
concluded that belief systems research claimed that its strictures on both individual beliefs and group
patterns of beliefs made democratic politics easier. Second, we must elucidate the nature of the moral
positions constituted in the methods and measures used to assess the constraint concept. Since
something of substantive significance is being measured, we may ask what "good" that measurement
implies and whether we believe that version of goodness. Third, we evaluate these moral positions
against the claims of substantive consequence. These last two steps are the burden of this section.

Because the mean correlation method is the simplest to describe, and because it implies the same
metaethical premises as more sophisticated methods, it will be the focus of the critique.

The constraint approach makes no direct judgments on the content of beliefs or ideological positions at
issue, so to a first approximation it takes no ethical position.(10) Converse (and Campbell et al. 1960)
specifically note that they accept ideological terms from any tradition (socialist, etc.). Insofar as the
constraint approach treats all ideologies even-handedly, it resists any direct attack on its ethical
premises.

The approach does, however, hold that we can evaluate belief systems in terms of whether group belief
configurations are constrained. This is a metaethical position: a criterion by which one may judge the
form or structure of ethical positions.(11) "Constraint" is not a statement of what beliefs are good but
rather a criterion for recognizing how beliefs should be related to one another: a "good" belief system
must be "constrained".

The constraint approach goes no farther than the position that constraint is only one virtue of a belief
system. There may well be other ethical or metaethical criteria that distinguish among equally
constrained belief systems, but clearly constraint is important. The possibility of such other factors has
not been a prominent feature of constraint theorists' defense of their work, however.

There are several methods of measuring constraint, but the mean correlation method has the greatest
practical simplicity, avoiding the need for complex computations, subjective coding of responses, or
follow-up surveys. So, for example, Converse uses the data from seven questions administered to a
1958 sample of American adults, computes tau-gamma correlation coefficients among the questions,
and takes as his measure of constraint the mean absolute value of these correlations.(12) The higher the
mean absolute correlation, the more constrained the belief system. A mean of 1 indicates a completely
constrained system, since all questions intercorrelate perfectly; a mean of 0 indicates no constraint,
since no question correlates with any other.(13)

The constraint criterion evaluates aggregates and thus cannot be applied directly to the belief system of
an individual: philosophically, the method interprets actors' beliefs only in the context of other actors';
mathematically, correlations cannot be computed for single individuals. How are aggregates evaluated?
They are not evaluated in terms of their adherence to a morally "correct" position. They are not
evaluated even in terms of their adherence to any of a range of correct positions (e.g., positions ranging
from liberal to conservative). Instead, the aggregates are evaluated in terms of the extent to which their
beliefs lie along a straight line in multidimensional belief-space.(14)

Individuals' beliefs are not evaluated by this method, but it is a reasonable conclusion, consistent with
the overall method, that this approach deems an individual's beliefs good to the degree that they adhere
to, rather than deviate from, the straight line in belief-space defined by this and all other actors' beliefs.
The constraint measure therefore creates two evaluations: it explicitly measures the goodness of an
aggregate's constellation of beliefs, and it implicitly measures the goodness of each actor's belief
system.

Casting the evaluation methods in these terms, we can see immediately several difficulties. For
aggregates, the constraint measure fails to recognize obvious good or evil in several circumstances:

A group of like-thinking philosopher-saints whose beliefs differed only through slight error variance
would, according to the constraint measure, possess no ideological constraint. But obviously their
identical beliefs could each be based on the same systematic, coherent, and profoundly moral
structure. Democratic government in such a society is scarcely hindered by the presence of slight
error variation.(15)

A group of democrats neatly arranged on a liberal-conservative continuum would be considered to
have highly constrained belief systems. If, however, a like number of equally constrained
anarcho-syndicalists, Marxians, monarchists, white supremacists, or libertarians were added to the
original group, the revised measure would clearly indicate that the enlarged group did not have a
highly constrained belief system. Thus the constraint measure can vary widely, depending on the
arbitrary and accidental context used.(16)

Imagine an isolated village where the people live along one main street. At the two ends of the
village live two powerful leaders. Although both are violent, evil, and completely insane, they have
entirely different insanities. The villagers, fearing both leaders, shape their beliefs in exact proportion
to their relative distance from the two. These conditions, according to the constraint criterion, would
make the village a pinnacle of ideological thought. The evil nature and logical incoherence of the
two polar positions would be of no significance.

It might be argued that, unusual as these examples appear, such unidimensional belief structures do
have the virtue of making more orderly democratic processes possible. This argument assumes,
however, that democracy requires a linear variation of beliefs. That assumption is both unproven and
highly suspect. Unidimensional variation of beliefs may simplify political coalition-building, but it is a
long step to the conclusion that democracy needs much cognitive simplicity. Constraint theorists offer
no compelling reason to weight simplicity so heavily in the evaluation of societies.(17)

But perhaps constraint is really meant to apply to individuals. After all, are we not concerned that
political actors grasp in an integrated fashion the complex issues of the day? Regardless of any
collective unidimensionality, will it not be hard to construct truly democratic politics when the demos
cannot think rationally? Let us turn, then, from constraint's measure of group virtue to its implied
measure of individual virtue. Two arguments reveal the arbitrary nature of the constraint measure
applied to an individual's beliefs:

The measure suffers from the "Alphonse-Gaston paradox": each individual's beliefs are evaluated by
the degree to which they correspond to an overall pattern, but the overall pattern is itself composed
of people being similarly evaluated. This is the Durkheimian view that morality is adhering to the
norms of one's culture, that is, attempting the impossible feat of agreeing with everyone else while
they are trying to agree with you. "After you, Alphonse." "No, no; after you, Gaston."(18)

An actor's minor deviation from the group pattern will degrade the correlation, but a large deviation
can increase the correlation. This is a well-known phenomenon for the Pearson product-moment
correlation: the addition of an outlier to a scattergram (Figure 2) degrades the correlation, but the
addition of a more distant outlier to the same scattergram (Figure 3) raises it. The effect changes
according to the number of points in the main cluster, but the principle always holds eventually:
beyond a certain distance, a single outlier can improve any non-perfect correlation. Applying this
principle to belief systems, the implication is that while minor deviation from one's group is bad,
major deviation (if major enough) is good, because it makes one's fellows' differences pale in
comparison.(19)

It is fantastic that such a seemingly natural measure as mean correlation could produce such
monstrosities. Even if researchers were happy with what correlational constraint measures, its measure
of subgroups would still have no relation to its measure of their combination. And if researchers were
not happy with correlational constraint and so used it only as one criterion out of possibly many, it
would not aid the measurement of the sort of ordered belief structure intended. In short, correlational
constraint has no significance as defined, and whatever meaning it does have in a particular context
applies only as long as the context remains fixed. Belief structure is too central a concept of political
analysis to be associated with such a flawed measure.

Our focus on definitional validity has clarified where corrections must be made. We have seen that the
problems of the constraint measure arise from its Durkheimian premises that knowledge, rationality,
and moral obligation are determined by group agreement. The examples advanced above to demonstrate
the absurd consequences of this position all draw on the contrast between the individual's beliefs and
the configuration of the group's beliefs. While social actors' beliefs are certainly affected by others'
beliefs, and while a rational actor's belief system will take cognizance of those other beliefs, it is not
reasonable to define ideological coherence solely in terms of belief configurations defined primarily by
others. Studies of belief structure must accordingly employ concepts and analytic methods based on
individuals, not groups.(20) Thus evaluating definitional validity is of immediate theoretical use.

6. The Status of Definitional Validity

The implications of definitional validity go deeper than simply the organization of critiques, however. it
is a fundamental aspect of research logic, and it carries us beyond the traditional fact-value distinction.

(1) Definitional validity is an implicit claim of every research project. Even though it is not as well
known as the claims of internal, external and substantive validity, any research project always already
asserts that its methods and operationalizations support its normative intentions. For the most part
researchers implicitly acknowledge this claim by developing measures consistent with their normative
concerns. Thus scrutiny of definitional validity is always appropriate but usually can be taken for
granted. In this sense, the validity problems of correlational constraint are unusual, but in principle there
is no difference between scrutinizing its claims and those of any other research program.

(2) Definitional validity also mediates between claims of fact and claims of value. Definitional validity
connects the purely methodological issues of internal and external validity with the purely normative
issues of substantive validity. This intermediate role of definitional validity may explain why it has not
previously been distinguished: because of the traditional split in social science between fact and value,
connections between them are ignored. If empirical researchers openly state their claims to the
substantive significance of their work, they become vulnerable to attacks on the substantive validity of
their position. If normative analysts openly state their reliance on empirical presuppositions, they
become vulnerable to empirical falsification. Neither side has an incentive to cross the middle ground.
But surely our concern for the substantive significance of our work should be the centerpiece of social
research, not locked away in its theoretical dungeons. Direct examination of the definitional validity of
our research offers us a greater meaningfulness to our work, at the cost only of opening up our positions
to attacks on their substantive validity - that is, at the cost only of admitting that social science is rooted
in moral concerns.

Bennett, W. Lance (1975). The Political Mind and the Political Environment. Lexington,
Massachusetts: Lexington Books.

Bishop, George F., Alfred J. Tuchfarber and Robert W. Oldendick (1978). Change in the Structure of
American Political Attitudes: The Nagging Question of Question Wording. American Journal of
Political Science 22:250-269.

Key, V. O., Jr. (1961). Public Opinion and American Democracy. New York: Knopf.

Kohlberg, Lawrence (1981). From Is to Ought: How to Commit the Naturalistic Fallacy and Get
Away with It in the Study of Moral Development. Pp. 101 - 189 of Lawrence Kohlberg, ed. Essays
on Moral Development. Volume I. The Philosophy of Moral Development: Moral Stages and the
Idea of Justice. San Francisco: Harper and Row.

Lane, Robert E. (1974). Patterns of Political Belief. In Jeanne Knutson. The Handbook of Political
Psychology. San Francisco: Jossey-Bass.

Luttbeg, Norman R. (1968). The Structure of Beliefs among Leaders and the Public. Public Opinion
Quarterly, 32:398-409.

Marcus, George E., David Tabb and John L. Sullivan (1974). The Application of Individual
Differences Scaling to the Measurement of Political Ideologies. American Journal of Political
Science 18:405-420.

Sullivan, John L., James E. Piereson, George E. Marcus and Stanley Feldman (1979). The More
Things Change, the More They Stay the Same: The Stability of Mass Belief Systems. American
Journal of Political Science 23:176-186.

Ward, Dana (1982). Genetic Epistemology and the Structure of Belief Systems: An Introduction to
Piaget for Political Scientists. Paper presented at the Annual Meeting of the American Political
Science Association, Denver.

FOOTNOTES

0. Amartya Sen did a number of works on the moral premises of measures of inequality, which I mean
to use as a second example of (an attack on a concept's) definitional validity. Here are some references
Jerry Gaus gave me to get me started:

Radcliffe lectures on Economic Inequality (1973)

Ethics and Economics (1987)

"Equality of What?" in the Tanner Lectures, entitled Liberty, Equality, and the Law (Cambridge,
1987)

1. These four claims are related to, but are not identical with, those presented in Fischer (1985).
Fischer's "technical verification" combines the claims of internal and external validity, while his
"situational validation", "systemic vindication" and "rational social choice" are all subsumed here under
the claim of substantive validity. As is usual in social-scientific literature, the claim of definitional
validity is taken for granted and thus ignored.

2. Such hermeneutic analysis need be undertaken only when the researcher is unavailable, unable, or
unwilling to explicate her research's claims to substantive validity.

3. I discovered this claim by the simple expedient of calling up Professor Covington and asking what he
thought the significance of his research was. If he had not been available, an analysis of this work (and
his works as a whole, and related works, etc.) would have been required to infer his purpose.

4. An exception is the standard caution made in methods texts that statistical significance does not
imply substantive significance. The text's claim applies to the field of methodology; it is less true of the
field of policy analysis, where many analysts (e.g., Fischer 1985; DeLeon 1987) have emphasized the
role of normative claims made by different evaluators (DeLeon 1986). The increasing social-scientific
attention to value relevance arises from the increasing delegitimation of dominant social-scientific
paradigms. Formerly, researchers in the (say) pluralist tradition could carry out research whose
normative intention, while present, was implicit in their tradition and did not have to be spelled out.
(One need not claim, as some critics do, that the failure to spell it out is a deliberate mystification.) The
advent of nontraditional perspectives has resulted in a spate of analyses which differ from those of more
traditional work and thus have to address more explicitly their normative intentions. What we are
observing in this efflorescence of normative analysis is not an increase in normative concern but rather
the development of an alternative normative position.

5. These three criteria motivate the discussion in Campbell et al. (1960:190) of "functional
relationships" among beliefs.

6. England (1984) discusses several justifications usually given for ideological thought, including some
not given here.

7. These justifications no longer appear in belief systems work, but this is probably due to the technical
concerns becoming more important, not the substantive concerns becoming rejected or unimportant.

8. Despite its name, this method still fixed individuals in a shared configuration space. The "individual
differences" lie only in the different weights different respondents attach to the various dimensions of
this common space.

9. For example, Bishop, Tuchfarber, and Oldendick (1978) attack the external validity of Nie and
Andersen's (1974) findings by arguing that the supposed increase in ideological constraint may be due
to changes over time of the survey questions employed. See also the many criticisms made by Lane
(1974); Marcus, Tabb, and Sullivan (1974); Bennett (1975); Barton and Parsons (1977); and Ward
(1982).

10. Only "to a first approximation", because the survey questions employed are more relevant to some
ideological positions than others. This is not an issue of definitional validity, however. The researchers'
question selection constitutes a moral claim that these are [a representative sample of] the only
questions that matter (and, by implication, that the associated ideological positions are the only
positions that matter). This claim does not conflict with the claims of definitional significance, so
Ward's (1982) criticism of it is not of its definitional validity but instead of its substantive validity - that
is, an attack on the claim itself. Ward's argument thus passes out of the domain of methodology and into
the domain of moral discourse.

Lest this be taken as a rejection of Ward's argument rather than a clarification of it, let me say that I
agree with it.

11. Thus in Kohlberg's (1981) work, "universalizability" is such a criterion; it demands that moral
principles be justifiable from the position of any moral agent. At issue is not any particular set of justice
principles but rather the method of judging such principles.

12. Converse (1964:227-229). A similar procedure was applied to a parallel sample of 1958
Congressional candidates, using somewhat different question wordings. As mentioned earlier, the
comparability of results between the two samples is attacked by Bishop, Tuchfarber, and Oldendick
(1978). One might also attack the validity of the various statistical manipulations; Converse
(1964:258n) himself suggests one possible objection. However, the point of the present paper is that the
results from both samples are meaningless because of faulty conceptualization of belief structure.

13. Note that, because of the correlation measure used, Guttman-scale questions have correlations of 1 -
as do linear-relationship questions, of course.

14. Actually, the tau-gamma correlation is not quite as easily interpretable as this, employing a notion of
concordant and discordant rank-order pairs. Still, the Pearson product-moment correlation image serves
well enough for the following points. Further, mean inter-item correlation measures are closely related
to measures of single-factor structure; the discussion here is phrased in terms of the latter rather than the
former.

17. The last of the examples above also shows that it might be more useful to look first at belief
structures for their receptiveness to democracy, and only afterwards for their unidimensionality. Under
the circumstances given, the hapless villagers will clearly never experience democratic government.

18. This aspect of Durkheim's position is discussed by Abrams (1983) in the context of the latter's
broader analysis of social science's conflict over how to conceptualize the relationship between human
action and institutional constraints. Kohlberg (1981:esp. pp. 106-114) presents a succinct argument
against Durkheim's position. Note that my claim is not that group consensus is irrelevant to moral
discourse but rather that it cannot in itself be the ultimate moral authority.

19. This critique is not as general as the previous ones, because it depends on a phenomenon peculiar to
the Pearson product-moment correlation. Use of a different measure could in some circumstances
obviate this critique. However, note that most of the constraint analyses, including those using
factor-analytic/structural equation models, depend on product-moment correlations.

20. Currently, Piagetian psychology and schema theory appear to be the only approaches with an
individual-level view of belief structure.