Monday, January 10, 2011

Unraveling the "obviousness" bias in psychology

In my last post, I argued that the pseudoscience of parapsychology (and in particular the publication of a paper in the Journal of Personality and Social Psychology claiming to provide evidence of precognition) hurts the perception of psychology as a science. This may seem like an obvious argument to make; Bem's paper was published in the flagship journal of social psychology, so it is easy to make the logical jump that this article is representative of the kind of research most social psychologists do. Therefore, the reasoning goes, social psychology is not a real science at all; to quote one comment on the media coverage of the Bem article,

Psychology is such a joke. A demonstration of future events influencing present events would be one of the most important (if not *the* most important) findings in the history of mankind. Yet this demonstration doesn't end up in Science or Nature, but is published in the Journal of Personality and Social Psychology? And some wonder why psychology is still considered pseudoscience....

However, all sciences have their occasional crackpots (in physics, John Baez humorously proposed the Crackpot index to deal with the profusion of physics cranks on Usenet forums). Why should psychology be any more affected by pseudoscientific claims for precognition than physics is by pseudoscientific claims for perpetual motion?

According to a recent paper by Keil, Lockhart and Schlegel, there may be a very good reason that psychology as a field is affected more than physics. Keil and his colleagues argue that psychology findings are afflicted by what I'm calling an "obviousness bias" -- they are viewed as intrinically less difficult to figure out than the findings of other sciences.

Keil and his colleagues took questions from physics, chemistry, biology, psychology, and economics. These questions were pre-tested with adults to be equally difficult and span a wide range of topics (see below).

The questions used in Keil, Lockhart, and Schlegel (2010) study 1.

The authors then asked people of various age-groups, from kindergarten up through adulthood, to rate the questions on difficulty from 1 (very easy to understand) to 5 (very hard to understand). The key finding is that while kindergartners viewed the questions from the various sciences as equivalently difficult to understand, children from 2nd to 8th grade viewed psychology problems as much easier to understand than the other problems. This bias disappeared in adulthood (see graph below).

A graph from Keil, Lockhart, and Schlegel of the mean difficulty score assigned to scientific questions

Note that children have absolutely no basis on which to judge how easy it is to understand why cooked eggs go from liquid to solid or why it is hard to understand two people talking at once. Children have no experience on which to draw when judging the difficulty of these questions and therefore, presumably, rely on heuristics to answer these questions. Thus, the fact that younger children rated psychology questions as less difficult than natural science questions points to a heuristic that makes psychology facts seem more obvious than other sorts of facts.

The authors found an identical pattern of judgments of questions within the various sub-disciplines within psychology; children judged questions from "harder" sub-disciplines, such as neuroscience and cognitive psychology, as more difficult to understand than questions from "softer" sub-disciplines, such as social psychology and personality psychology. Moreover, when the authors asked adults how easy it would be to figure the questions out just by living and watching things, the adults showed the bias as well -- they rated the psychology questions as easier to figure out than the natural science questions.

While the authors do not explore directly whether the obviousness bias translates into psychological findings being taken less seriously than the findings from other disciplines, the authors do show that people assume they can figure out psychological findings on their own. This could very easily lead to a perception that less scientific rigor is required for research psychology than for other sciences -- after all, anyone can do psychology just by experiencing life. This means that anything that reflects on the integrity of psychology as a scientific discipline -- like pseudoscientific work on precognition -- is likely to be especially damaging for psychology.

2 comments:

If science is not about encouraging paranormal claims, it is also not about repressing scientific evidence about previously unknown psychological phenomena in order to reinforce institutional beliefs and status quo, especially when the methodologies used to produce this evidence are consistent with mainstream processes in the field.

The peer-review process is one that is *supposed* to encourage challenges to existing theory in order to prove them and allow for changes in viewpoints and knowledge. Unfortunately, many journal editors and academics feel that journals and research should go on reproducing the prior views (and career paths) of the institutions and educators that produced them. This is a cancer on science, and has turned searching for the truth into dogma in the same way that religion has consistently turned simple teachings into iron-clad commandments.

If one can't see the inherent value of a controversial paper like this one that challenges dominant beliefs-- and which, incidentally, has followed the formulas and methodologies of the 'respectable' papers published alongside it-- and the biggest concern that it makes your field 'look bad', then you have essentially become part of the institution. You now are interested mostly in preserving the views and science that produced your views (a simultaneous example of confirmation bias and social reproduction). Yet, this is not how science is produced. Science is produced at the margins. Risks are sometimes necessary. For something like ESP, there is no 'safe' way to conduct this type of research. No matter what you do, it *will* be controversial, it will make some people immediately and automatically dismissive. But that doesn't make it junk science. Junk science is about methodology, not about subject matter.

While I agree that science is about rigorous method, not content, scientific evidence must be interpreted in light of all the other evidence that has been previously established. With respect to Bem's work, for example, if precognition does exist, why does it not affect people's behavior outside the laboratory (e.g., at the slot machine?).

Even if we accept Bem's evidence at face value, one of the weaknesses of his "theory" of precognition is that does not postulate a plausible mechanism through which precognition could occur. The reason that this is problematic is that the current understanding of biology, neurochemistry, chemistry, and physics does not provide any reasonable ways for people to precognitively experience emotional reactions to an event that has not occurred. This means that any evidence for precognition must be truly extraordinary -- hence my skeptical reaction to the extremely small effects obtained in the Bem experiments. If the evidence is not extraordinary, the most likely interpretation of the Bem effects is that they are due to data mining (which you can see in part in the search for interactions and other effects in the Bem paper).

Finally, there actually are some technical problems with the Bem paper. I will not review all of them here, but I will direct your attention to this critique: