I’ve followed with interest the controversy stemming from Jon Haidt’s address at SPSP. One issue that has not been discussed is how the political biases of the field have severely undermined some of the research. I propose that we have a serious problem. Most research in social psychology does not touch on politics and has no obvious political implications. However, some of the research in sub-fields like political psychology and attitudes has deviated sharply from valid scientific methods. Researchers sometimes embed ideological assumptions into their hypotheses, constructs, and measures, in ways that make their studies invalid or even meaningless. Regrettably, I can’t properly make my point without evaluating the work of noted social psychologists. I’m willing to do so here, and in future settings, because a) I think this is a serious problem for the field â€“ these biases may ultimately weaken our very standing as a science, and b) these practices have gone unchecked for years, and a frank and open consideration of them is long overdue.

My first example of the phenomenon is the Napier and Jost (2008) Psych Science article “Why Are Conservatives Happier Than Liberals?”

In this article, the authors want to show that conservatives are happier than liberals because they “rationalize inequality” (by which they mean economic or financial inequality, such as unequal incomes). This is already an unanswerable research question. Why? To rationalize is to explain away an uncomfortable reality, often by making excuses for it. It is dissonance reduction. Thus, a basic precondition for conservatives to rationalize economic inequality is that economic inequality be uncomfortable for them. However, economic inequality is particularly uncomfortable only for leftists. Conservative ideology does not feature economic inequality as an injustice or a problem to be solved. (Libertarians are also largely unconcerned about it.) Therefore it’s logically impossible for conservatives to rationalize it, since they aren’t particularly bothered by it. (Jost’s own data confirm that conservatives are relatively unconcerned about economic inequality.) A research program centered on conservative “rationalization” of something that only liberals care deeply about has no apparent way forward.

So how did the authors conduct the research? In Study 2, they operationalized the rationalization of inequality with a one-item measure: 1 (hard work doesnâ€™t generally bring successâ€”itâ€™s more a matter of luck) to 10 (in the long run, hard work usually brings a better life). High ratings on this item were cast as rationalization of inequality. In other words, the authors took endorsement of the efficacy of hard work and called it rationalization, then plugged it in as a mediator between conservatism and happiness (note that this belief about the efficacy of hard work is a constituent conservative belief â€“ we might find that other conservative beliefs work just as well as “mediators” here) . There was no attempt (in either study) to capture or measure any actual process of rationalization â€“ they simply applied the label to conservatives for endorsing this standard conservative view on hard work. (It may be worth noting here that hard work actually does pay off, as I assume anyone who has mentored graduate students can attest â€“ this is observationally self-evident and supported by massive amounts of data. So people are being labeled as rationalizers for simply endorsing an obviously true statement.)

Since no process of rationalization was exposed in Studies 1 or 2, and since it makes no sense that people who don’t have a serious problem with economic inequality could be accused of rationalizing it, the article’s results are essentially meaningless. The data don’t tell us anything related to the hypotheses. This is what I mean by a lack a validity â€“ the data do not represent the construct, and given the nature of this construct, it’s unlikely that any data could. This research is a scientific non-sequitur: From (1) Conservatives are happier than liberals, and (2) Conservatives believe that hard work pays off, we conclude (3) Conservatives are happier than liberals because they rationalize inequality. Our only way out is if we treat the following statement as an objective fact: Economic inequality is unjust. If we treat this ideological claim as a fact, as a description of reality, then we might assume that all people are motivated to rationalize such economic inequality as exists in their communities, and proceed from there. But of course we cannot take this ideological claim as fact. It’s a philosophical position held by one particular political ideology, and many people would disagree with it. Social scientists are in no position to ratify the truth or falsity of such philosophical positions.

In other work, Jost uses words like legitimize or justify, in addition to rationalize. The question might be something like “how do conservatives legitimize the status quo system?” All of these verbs are ideologically loaded, and the questions which rest on them are not answerable by social science. To ask why anyone legitimizes the status quo is to presume that the status quo is unjust and thus requires legitimization, rationalization, or justification. This assumption is fully an ideological/philosophical assumption, and has no place, nor any real utility, in framing scientific research.

Here are some analogous research questions: Why do liberals legitimize gay marriage? Are liberals less happy than conservatives because they rationalize abortion? These are exactly the same sorts of questions, and fully as invalid as the above. They presume that gay marriage is wrong and needs to be legitimized, or that abortion is wrong and must therefore be an object of rationalization. But of course, liberals don’t grant that gay marriage or abortion are wrong, so there is nothing for them to legitimize or rationalize. A research program thus framed would have nowhere to go. If a scientist presented research framed by these conservatively-biased, loaded questions, we would immediately recognize it as scientifically invalid. But framed from a leftist perspective, such loaded questions have escaped scrutiny.

The field should discard ideologically-loaded constructs like these â€“ constructs that have no scientific meaning because they rest on ideological assumptions, rather than observable facts.

A second example of how research is framed in biased ways: If we look at the Jost lab website, we find that many of the researchers frame their research around leftist ideological assumptions. To take just one example, Irina Feygina describes her research as focused on “the effect of motivation to justify the socioeconomic system on denial of environmental problems, such as ecological destruction and global warming, and resistance to implementing imperative pro-environmental changes to the status quo.”

This is alarming. Social psychologists know what the imperative “pro-environmental” changes are? How? When did we discover the correct human values and ideals, or imperative policy reforms? Environmentalism is a political ideology, and as such it rests on various philosophical assumptions and values (e.g. a conception of the natural world as sacred; a view of human activities as unnatural; resources as static and collectively-owned; and a propensity to value the preservation of status quo ecologies more highly than some increment of human prosperity). Reasonable people might embrace or reject environmentalism, in whole or in part, for any number of reasons. We cannot treat environmentalism as self-evidently correct, any more than we can treat conservatism or Kantianism as self-evidently correct.

Imagine if a researcher focused her research on “resistance to imperative pro-Christian changes to the status quo”, or “resistance to imperative pro-business changes to the status quo”. I assume you get my point. We should not be in the business of investigating why people “resist” the truth of our personal ideologies, and a researcher so motivated will likely struggle to maintain an appropriate scientific posture.

I offer a principle from all this: If a research question requires that one assume that a particular ideology or value system is factually true, then that research question is invalid. Stated differently, if a research question has no meaning unless we assume that a given political ideology is factually true, then that research question is invalid (and cannot be meaningfully answered).

Critical to any science is the generation of testable hypotheses. The practices I’ve highlighted above will consistently yield untestable hypotheses, because they rest on the assumption that liberalism is true, which will never be testable. It is, after all, a question of values and value judgments, which are not subject to empirical validation (at least not by our methods). Modesty requires that we allow for the possibility that reasonable people might embrace values that differ from our own. Notably, researchers who employ such ideologically loaded hypotheses are very likely to find what they are looking for. For example, suppose I wanted to show that conservatives are happier because they “rationalize” war. Mirroring the Napier and Jost method, all I would have to do is ask conservatives if they think some countries are a threat to the USA, and label their affirmative responses as “rationalization of war”. I would plug it in as a mediator between conservatism and happiness, which would likely work out. I could then publish my findings in a journal, concluding that conservatives are happier than liberals because they rationalize war, and garner some good media coverage. Apparently, no one would stop me. But would I respect myself in the morning? No, because such an article would have no standing as a work of science, and its conclusions would be completely unsupported by the data.

Among the sciences, social science operates with the most flexibility in constructs, methods, and measurement. This makes us especially vulnerable to bias (see John Ioannidis’ work for more on this). I think we should be vigilant, ambitious, and idealistic about keeping our science clean. I assume nothing but the best of intentions on the part of the researchers I’ve critiqued, and I don’t at all enjoy publicly critiquing them. Nevertheless, I submit that the issues I’ve raised here are not minor — these are serious violations of the valid practice of social science. Our credibility and even our very standing as a science are at issue, and will be questioned by politicians, taxpayers, and scientists in other fields if these practices continue. Admittedly, this sort of validity issue has not been well-elaborated in our training or the literature. Yet I’m confident most researchers will agree that what I’ve offered here is a straightforward extension of construct validity and the features of testable hypotheses. The biases at issue represent a (correctable) blind spot in our field, and an unsurprising one, given the large overrepresentation of liberals.

No user commented in " How political biases sometimes yield invalid research "

Leave A Reply

Advertisements

BloggerNews On The Air

We are pleased to announce our latest endeavor, Blogger News is now sponsoring some radio shows on Blog Talk Radio. You can check our full schedule, and listen to previous broadcasts here, and we hope that you will join us on the air in this new venture.