Weak Evidence – Folklore, Pseudoscience and Superstition

Last week, in this blog, I covered the topic of scientific research. In a post, I described the most common forms of research as well as how they differ in terms of the strength of evidence that they provide. I thought it was important for lay people to understand that all research isn’t created equally. The fact that a researcher has advanced degrees, practices at a prestigious institution or has won a Nobel Prize, is no guarantee that his findings are accurate and relevant. Therefore, understanding the value of different forms of research, makes you a more informed professional, with an improved ability to critically assess scientific findings.

After I wrote this post, I realized that there is a larger and more insidious threat to the acquisition of knowledge. While I was focused on the reliability of scientific research, many professionals have belief systems and knowledge acquisition methods that are decidedly non-scientific. This post will assess those methods and compare them to scientific study.

Before we do that analysis, let’s start with a bit of level setting. Scientific research methods have been developed over the last several centuries to improve the quality of the acquisition of knowledge. They ride on top of a couple of basic philosophical ideas. First, we live in a world of objective facts that represent a set of consistent physical laws of the universe. Second, our human nature frequently blinds us to these facts leading us to false beliefs and judgements. Research methods, such as randomized controlled trials, allow us to put aside our personal biases and establish more accurate findings. While scientific findings may get revised over time, rigorous, formal research methods represent our best shot at understanding objective facts at any point in time.

All formal scientific research methods share the following characteristics, which differentiate them from non-scientific methods of understanding:

Hypothesis driven – An explanation for a phenomenon that will be tested as part of the research. The hypothesis will explain why the phenomenon occurs or will be able to predict its occurrence. A good hypothesis should be consistent with the existing body of knowledge

Testable – A hypothesis needs to be testable. That is, there needs to be an observable and measurable process that allows researchers to determine if the hypothesis is in fact correct.

Falsifiable – If must be possible to also demonstrate through testing that a hypothesis is incorrect.

Transparent – All aspects of the research process are documented and available for other researchers to review, critique and attempt to replicate.

Repeatable – Through transparency, it must be possible for other researchers to attempt the same experiments and either confirm, refute or revise the original findings.

Impartiality – The researchers should not have a conflict of interest (e.g. paid by sponsors) or strong personal bias (e.g. looking to validate political beliefs) that will influence their findings.

Revisable – The spirit of scientific research holds that all findings are provisionally true. That is, they represent the best possible understanding and explanation available at the time. Subsequent research can always overturn or modify existing findings.

Methods of acquiring knowledge can be placed on a spectrum, with varying degrees of accuracy and usefulness. Formal scientific methods occupy the upper end of this spectrum. There’s also a significant group of ideas with less methodological soundness (and in some cases, zero value) that are quite popular. Let’s look at these ideas in order of effectiveness and value:

Expert Opinion – While not representative of a formal study, expert opinions can be a useful source of knowledge. Frequently, a well qualified individual (or panel) will write articles (or editorials) with useful observations or new theories. While these ideas do not have the weight of scientific experimentation, they can be thought provoking and provide new paths of exploration. These opinions can serve as an early hypothesis that is later tested by a formal research method. The key here is not confusing opinion with verified facts. An insidious variant of expert opinion is a testimonial. A testimonial is a claim, sometimes offered by an expert, in support of a particular product. As testimonials are frequently sponsored by an interested party (typically the seller of the product), they represent a serious conflict of interest and should hold little weight.

Folklore – A catch-all term that reflects commonly understood ideas, practices and stories. As opposed to scientific concepts, folklore is not formally validated. In many cases, it is simply a tradition, or long held belief, frequently without a verifiable origin. In everyday life, an example of folklore would be an old-wives tale. Folklore may originate from actual research but become disconnected from its source over time. In the workplace, folklore is often a management practice that is commonly believed to be effective. An example would be the Hawthorne Effect, an oft cited behavioral phenomenon uncovered in factory experiments in the early 20th century. One of the main lessons of the experiments was that simply changing an aspect of a worker’s environment could result in improved productivity. Since the time of the study, the effect has been hotly disputed by scientists. It remains, however, a staple idea in corporate settings, uncritically conveyed in its original form. Another form of folklore is a saying or maxim. An example is “It’s better to be feared than loved”, attributed to Machiavelli from his work The Prince. Although it is merely a quote, from a 16th century political treatise, it is often used to rationalize aggressive management behavior, as if it were a hard scientific law. Another common type of folklore in the corporate world is the founder’s story. This is typically a short, complimentary tale about the legendary individual who started the firm. It is meant to convey a cornerstone principle of the firm, emblematic of the core culture. Folklore, is not inherently inaccurate or misleading. It can be useful, as in the founder’s story that provides guideposts for individual action. It is, however, a limited and weak form of knowledge that has not been validated nor recently updated.

Pseudoscience – A particularly insidious method of knowledge acquisition is pseudoscience. It is a decidedly unscientific set of methods that is cloaked in an air of scientific authenticity. Many New Age nutritional supplements and healing remedies are examples of pseudoscience. They use scientific sounding terminology to explain their effectiveness. They don’t, however, have the hallmark characteristics of true scientific methodology, such as falsification and repeatability. For example, an herbal extract may be touted as being able to balance energy levels. But there is no scientifically valid experiment that demonstrated this ability. Unfortunately, many people are unable to recognize pseudo-scientific claims, regarding them as actual science. An example in the workplace (I can already hear the gasps!) is Myers-Briggs Type Indicator. It is a widely used personality assessment, designed to identify how people perceive the world and make decisions. It uses many techniques common to conventional science such as testing, categorization and theory driven explanations. It is based, however, on the observations and theory of the psychologist Carl Jung. It is not based on sound scientific experimentation. Most psychologists do not feel that it represents an evidence based, valid assessment of personality type. Much like testimonials, pseudo-scientific claims are often made by interested parties who stand to gain from the public acceptance of their ideas. A telling hallmark of pseudoscience is the close association of the idea with a product or service.

Superstition – At the bottom end of the knowledge acquisition spectrum is superstition. While some pseudo-scientific beliefs can have elements of factual correctness, superstitions by definition represent non-factual thinking. Superstitions are false beliefs in supernatural or magical functioning of the universe. They ignore and contradict the accepted body of current scientific knowledge. Superstitions can represent negative power or fears, such as, “Something bad will happen if I walk under that ladder”. They can also involve positive outcomes such as crossing your fingers to ensure good luck. Some would argue that beliefs such as Astrology, Numerology and Crystals, are effectively superstitions. While they could be incorporated in my definition for pseudoscience, they do contain the hallmarks of superstitions; defying physical laws and conferring power to practitioners. As with it relatives in the “weak evidence” family, superstitions lack any resemblance to true scientific methods.

Understanding the characteristics of science and being able to recognize non-scientific practices is a valuable life skill. It allows you to more accurately assess the value of the different claims you encounter on a daily basis.