Everyday Skeptic, Part 3: Credible Evidence

Note: The video portion of this article could not be completed due to time constraints. It will be added on a future date.

While it is very possible to dismiss many claims using the rules of logic covered in the last installment of the series, you are still likely to encounter claims that, while adhering to the rules in their format, still warrant a bit of skepticism from you, the reader. This is because not all claims have credible evidence. Just to review from last week:

A statement is true if it is a positive claim and there is credible evidence to affirm it, while there is no credible contrary evidence that makes the statement untrue.

The most critical piece of any argument, given that it is a logical, positive statement, is supporting evidence. What remains to be added to our skeptic’s toolkit is a way to tell if evidence is credible, flimsy, or false. Some of the most common bits of misinformation are believed because readers either could not tell how bad the evidence really was, or were unable or unwilling to evaluate the credibility of evidence on their own. In this week’s article, we will go through what sort of things make good, affirmative scientific evidence, leaving the fallacies for their own shrine of dishonor next week.

The Scientific Method

The scientific method was developed and refined over the course of many centuries leading up to the modern day as a means to determine the objective reality of various physical processes. It accomplishes this using specific experimental processes, data gathering techniques, and rules of reasoning. When evidence fails to follow the guidelines of the scientific method in its acquisition and evaluation, it loses its objectivity, and therefore becomes less usable to others.

I. The experimental process

Not all scientific information requires experimentation. Most facts can be derived from observation of physical objects and phenomenon that are already occurring naturally. For instance, no experiment is needed to determine the average height of trees in the Sequoia National Forest; one only needs to go measure the trees and do the math. Scientific experiments are usually used to determine causal relationships.

1. Background.

Usually, when you are performing a scientific experiment, you are not doing it blind or striving for a totally random outcome. You start by reviewing literature of those who have come before you, or observing something in nature or the human world. You may see hints at a pattern, or may wonder about an area of science or a physical process that has yet to be tested. You use this background information to formulate the next part of the process:

2. Hypothesis.

A hypothesis, in its simplest form, is an educated guess about a causal relationship or the outcome of an experiment. A good hypothesis contains a single phenomenon to be tested, and is by its nature falsifiable, which means that an experiment can prove it right or wrong. Once the hypothesis occurs to the researcher, he formulates the next part:

3. Experiment

The experimental process sets about to observe a phenomenon and collect data using objective measurements. An objective measurement is usually done in such a way as to remove the bias and subjectivity of the researcher from the results. Instruments that measure temperature, mass, and time are necessary to make the data gathered from the experiment objective. Subjective data, such as asking people to talk about experiences, is usually not considered scientific. The data must also be quantifiable, or can be expressed through numbers. Lastly, a good experiment contains ideally only one variable, or thing that is being tested(more on that later), and contains a control, or something unaltered to compare the experiment to. Once the experiment has been done, and all the relevant data gathered, the researcher can move onto the next part:

4. Analysis

The analysis phase is when the researcher looks at the data gathered through the experiment and sees whether it proves his hypothesis or falsifies it, or sometimes, fails to do either. Analysis should always involve raw data, not extractions of data like statistics. Statistics are used to condense the analysis for another person to quickly understand outcomes. Once the researcher has analyzed the data, he then shares his:

5. Conclusion

The conclusion is the publishing part of the research, where a scientists shares what he has learned through experimentation with the rest of community or world. A good scientific publication contains not only the conclusion of whether the hypothesis was proven or not, but also contains the methods used for the experiment, so other researchers can try it for themselves and affirm the conclusion. A null hypothesis (one which the experiment has disproven) is just as important as a proven hypothesis.

Be extremely critical of experiments that lack all the components above, particularly if they lack any objective data gathering methods.

Example

A great example of the correct use of the experimental process was last week’s example of Galileo Galilei dropping two objects of different mass from the top of the leaning tower. It shows the process clearly:

1. Background. Galileo noticed that objects of different mass appeared to fall at the same speed. He set about to see whether this was truly the case or if he was mistaken in his observations.

2. Hypothesis. Galileo hypothesized that two objects of very different weight would fall at the same speed. He used a cannonball and a musket ball, the two having a very large difference in weight, while otherwise being of similar material and shape.

3. Experiment. Galileo dropped the two objects from the leaning tower to see if they fell at the same speed. They were dropped at the same time and hit the ground at the same time, proving that they fell at the same speed.

4. Analysis. Galileo analyzed the experiment, and quantified the simple data. He found that it affirmed his hypothesis.

5. Conclusion. Galileo shared his conclusion, along with his methods, with his community. Specifically, he re-enacted the experiment for students.

II. Variables

Variables in science consist of anything which varies from one subject or experiment repetition to the next. Often the variable in the experiment is what is being tested. In Galileo’s experiment, the variable was the mass (or weight) of the balls dropped. Everything else was set, or not subject to alteration. The balls were of similar shape (round), and of similar material (lead). If those had been varied as well, it would have been difficult to determine if weight differences led to the conclusion rather than something else.

It is very important in scientific research to as few variables as possible and so isolate the phenomenon being observed. It is very hard to draw causal relationships from data that has a lots of variables. In Galileo’s experiment, if he had varied the material and shape of the objects he dropped, he might have arrived at a different conclusion. If, for example, he had dropped a cannonball, which is round, and a modern type of bulled, which is aerodynamic, the modern bullet might have struck the ground first. This would have proven his hypothesis false, when in reality a more carefully controlled experiment would have proven it to be true.

The purpose of refined, limited, and objective experimentation is to make causal relationships obvious and to avoid the fallacy of “correlation equals causation” (an idea explored more fully in the weeks to come). Just because two things happen at the same time or in proximity to one another does not mean one caused the other. Using variables helps a researcher to see a definite relationship between things. When the variable is altered, and it changes outcomes, one can assume that the variable has a causal relationship to the outcomes.

Also important in the use of variables in many experiments is a control. A control is something that is unaltered, which is compared to the part of the experiment which is varied. For instance, if we wanted to see how much more energy was released in gasoline as the octane values were increased, we would need a sample of gasoline without any octane added. This control helps to establish that the variable in the experiment is, indeed, having some effect that would not otherwise occur. A control group in a drug trial will be one that has an affliction, but receives no drugs, either as a placebo or otherwise. It exists so that a scientist can make a comparison to things in their natural state.

III. Data points

Ideally, an experiment will be set up in such a way that there are only two relevant data points – those that affirm the hypothesis, and those that nullify it (make it false). In Gallileo’s experiment, either the balls fell the same or they didn’t. If they fell differently, the degree by which they varied did not matter, as it proved the hypothesis false. In lots of experiments there may be a large amount of data gathered, but each piece will still either go with or against the predictions of the hypothesis. Usually before an experiment is done, you decide what outcomes will prove or disprove the hypothesis.

Some data, specifically data that involves human perception such as drug trials, is difficult to quantify into only two data points. This doesn’t mean it is invalid, but that the analysis and conclusion will be more complex and have a less definitive conclusion that Galileo’s simple experiment.

IV. Cause and effect in studies, surveys, and other kinds of research

As stated, lots of research does not involve experimentation, it involves gathering information on things that already exist. This is particularly true when it comes to studying society, people, geology, and astronomy. Sometimes relationships can be inferred, sometimes very strongly, but causality is much more difficult to prove without experimentation. A few simple rules of logic should be applied when considering causal relationships of studies:

1. Cause always precedes effect. This means that whatever causes something else must happen before whatever is being affected.

2. A cause must be a sufficient condition for the effect.What this means is that a cause (even if it is more than one thing making up a cause) must be able to trigger an effect all on its own. If an action fails to trigger an event, or triggers it only randomly, it is not sufficient, or may lack a necessary condition in the chain of events. A necessary condition is one that is not causal, but is required for the causal relationship to work.

As an example, suppose we pulled the trigger of a gun. After we pull the trigger, the gun fires. The cause (pulling the trigger) proceeds the effect (the gun goes off). Pulling the trigger is a sufficient condition for the gun going off, since every time we pull the trigger, the cause and effect occurs. However, eventually the trigger gets pulled and the gun does not go off. It is out of ammunition. Loaded ammunition in this case is a necessary condition, but not sufficient, as it does not cause the gun to go off all by itself. The trigger must still be pulled. The real causal relationship would be: Pulling the trigger of a loaded gun causes it to fire.

When you are looking at observational studies, be sure to look for the way that they infer cause. Is there a sufficient condition, or is there just a flimsy relationship between two things? Be skeptical of all causal claims lacking experimentation.

V. Conclusion

What should be taken away from this week’s article is an idea of what makes good, credible evidence. Evidence that fails to meet the simple standards here is not going to be credible, and you should be willing to discount it. It’s a lot to take in, but when you are presented with evidence, ask yourself the following:

1. Does the research follow the scientific method?

2. Is the data objective, or is it informal, subjective personal experience?

3. Is there a control group? Are methods disclosed? How many variables are there?

4. Is there a causal relationship deduced with logic, or is there only correlation?

Asking these simple questions will help you to determine how objective and scientific the evidence is. Remember, research needs to meet scientific standards. Grey areas are not of much use.