College upperclassmen still fail at scientific reasoning

No, leaves do not contain a miniature Mr. Fusion—a survey highlights how …

Most of us develop a sort of intuitive logic about how the natural world works. Unfortunately, a lot of that informal reasoning turns out to be wrong, which complicates scientific education. But as students make their way through the science education pipeline, they should gradually start moving beyond the informal reasoning of their earlier years. Or at least that's what we'd like to think; instead, a new survey of college students, some in advanced biology classes, indicates that most end up with a confused mix of formal and informal reasoning.

The clearest example of the chasm between a typical intuition and scientific reasoning comes from the world of physics. Imagine a marble rolling around a curved track that comes to a sudden end. Physics tells us that, as soon as the marble is off the track, it'll continue moving in a straight line until it runs into something else. But many people use informal reasoning and conclude that the marble will continue to follow a circular path even after it escapes the track. In other contexts, it involves an interventionist view of the world. As the people behind the survey put it, "When using informal reasoning, students look for 'actors' that drive 'events' and are aided by 'enablers.'"

Scientific education, then, needs to convince people to move past their intuitions (at least if they want a more accurate picture of how the world operates).

The new survey tested for informal reasoning in the biological sciences, using over 500 students at a variety of colleges, enrolled in classes ranging from introductory biology to advanced ecology. The results show that, even as the students are immersed in things like trophic pyramids and the Calvin cycle, they don't always come to grips with basic things like conservation of matter and energy.

For example, most students could describe how the process of photosynthesis involves removing carbon dioxide from the atmosphere and combining it with water to form carbohydrates, which then get used to build the cellulose that forms most of the plant's bulk. But, when asked to actually trace what's going on on the organismal scale, many students have problems recognizing that a substantial proportion of a tree's solid bulk originated from a gas, instead suggesting that most of it was brought up from the soil.

The problems work in the opposite direction, too. When respiration breaks down solid matter, most of it is released in gaseous form, as carbon dioxide and water vapor. But the students often had a problem with recognizing that a solid compound could be chemically converted into a gas.

In the same way, they seem to believe that every leaf contains a miniature Mr. Fusion, since they think that some of a plant's mass comes from the transformation of one atom into another. Nearly 70 percent "chose 'sunlight' as a possible source of atoms in chlorophyll molecules," according to the results. They had problems with other aspects of energy, too, not recognizing that moving up a trophic level in an ecosystem generally entailed the loss of energy to the environment—"only 44 percent of students thought the top of a food web would have 'less available energy than the trophic levels below it.'"

Other problems noted by the authors include the use of energy as a fudge factor to make things balance out, and a tendency to substitute a wall of scientific terminology for actual understanding.

In all, the majority of the students used a mix of scientific and informal reasoning on the surveys, with about 20 percent using informal reasoning alone. More depressing still, the authors designed a short course to help introduce formal scientific logic, but it didn't help much. The course shifted more people towards relying on scientific reasoning, but the percentage of students who relied on it exclusively rarely exceeded 30 percent, and was sometimes in the neighborhood of 20, depending on the topic (on average, it went from 12 to 27 percent).

What's the root of this problem? The authors ascribe a lot of it to language. It's quite common to hear people describe fat as just melting away or vanishing, which doesn't encourage anyone to try to balance the books on where all those atoms actually go to, much less get them thinking in terms of their release as carbon dioxide and water vapor. The same problem persists in the language commonly used by biologists. We frequently refer to energy as "lost" when it's no longer available to an organism, but that doesn't mean it's not still there, typically in the form of heat.

The end result, the authors conclude, is that "faculty are unknowingly speaking a different language from their students." They think that when they mention lost energy, the students know what they're talking about, or that their students' poor choice of wording doesn't represent a failure of logic. As a result, they see little reason to speak more carefully or devote instructional time to clearing up misconceptions. And, even if they wanted to, most biology textbooks consider principle-based reasoning beyond their scope.

But, even if instructors and textbooks were ready, the limited improvements that resulted in the targeted interventions used in this study show that overcoming the tendency towards informal reasoning can be a significant challenge. And the authors don't necessarily know what to suggest beyond starting the process early and keeping it a consistent focus of the education system.

But, when asked to actually trace what's going on on the organismal scale, many students have problems recognizing that a substantial proportion of a tree's solid bulk originated from a gas, instead suggesting that most of it was brought up from the soil.

Your "substantial proportion of a tree's solid bulk" is brought up from the soil in the form of water. While I know what you were getting at, I'd expect more care with your wording given the article's topic...

Even as a grad student in physics, it's possible my snap answer to the marble question might have been that it would continue moving in a circle. Probably would depend on how tired I was. But it probably wouldn't take me to long to figure out why that was wrong. I'd say there's a difference between not being able to give the answer immediately and not being able to give it at all. But then I went to a liberal arts college, being able to spit out a right answer wasn't enough, you had to understand WHY your answer was right.

P.S. A lot of bio students do not seem particularly bright--I apologize for the generalization, and I'm sure there's plenty of intelligent bio students, but the ones I've dealt with did not seem too special. And don't get me started on the premed students I had to tutor for intro physics. One girl wanted velocity * sin(angle) to come out with units of degrees! I was just flabbergasted at that one.

P.P.S. Fun fact: You know who does well on the verbal section of the GRE? English majors...and physics majors.

We teach people to regurgitate and never challenge them to think. This needs to start at an early age, encouraging students/kids to understand why instead of being content to know "what".

When you get past basic science, "why?" becomes impossible to answer.

I'm of the opinion that most people simply aren't wired (due to some combination lack of interest, lack of capability, or lack of willingness to learn) to understand concepts at anything past a rudimentary level and it's a waste of time trying to teach them. For the most part that's perfectly fine as long as people understand what it is they don't know. The trouble comes when you have people with more ambition than talent taking science classes, failing to understand what they learned, and then going around spouting off uninformed opinions. The public discourse is filled with half-educated simpletons spouting off about issues of economics, history, and science and a variety of subjects with barely any understanding of what they're talking about.

I don't get how they're so willing to defer to the judgements of doctors when they have an ailment or acquiesce to hiring accountants and tax professionals to file their taxes, but are so unwilling to acknowledge when they're out of their element in other fields and leave the high-level discourse up to the experts. Call someone out for failing to understand any economics past their 101 level college class and they'll either ignore you or give you some line of BS about how they're entitled to their opinion.

In sciences it's even worse. There are no worse advocates for a legitimate science education than New Atheist idealogues for scientism who, 90% of the time, have no fucking clue how any of the "science" they're deferring to actually works. I've been in so many ID arguments where the "science" guy has clearly never done a Punnet square in his life and makes statements that sound more like Larmack than Darwin.

I am tired of the "curved track" anecdote. It doesn't represent much to me...

That humans answer on intuition when asked esoteric (if admittedly basic) questions is not quite what's wrong with science education in this country.

Yeah the curved track one is a bit iffy. Depending on how you visualize the board, the ball is likely to have some spin on it when it exits the track and actually will hook slightly along the direction of the curve. This is a more appropriate understanding of the situation for everyday use than the technically correct interpretation is. It can impede performance on a science test, but doesn't actually hurt them in real life.

Scientific education, then, needs to convince people to move past their intuitions (at least if they want a more accurate picture of how the world operates).

First they need the tackle of actually educating. I took 3 1/2 years of undergraduate biology, chemistry and physics classes and I remember very little. It was always a matter of memorizing what information was needed to get somewhere in the order of magnitude on a multiple-choice bubble exam. And the questions were extremely pedantic. It was the only way professors, who aren't concerned about teaching since it's 30% of their job description, could differentiate 1,600 students who all wanted to be doctors, dentists, veterinarians and such.

In the end I have a degree in biomathematics and statistics, because I actually learned more in science oriented math classes and had to apply concepts instead of memorizing and regurgitating retrosynthesis of basic compounds, IR spec charts, or the year that Ernest Rutherford proved that atoms have mass and weight.

Imagine a marble rolling around a curved track that comes to a sudden end. Physics tells us that, as soon as the marble is off the track, it'll continue moving in a straight line until it runs into something else. But many people use informal reasoning and conclude that the marble will continue to follow a circular path even after it escapes the track.

I think unenthusiatic teachers are partly responsible for this too especially in physics. I've sat through a lot of physics courses where the prof made me want to blow my head off because it was sooo boring. I wonder what the success rate is for that MIT course where the teacher actually demonstrates things and students aren't forced to visualize from textbooks.

Maybe I am overgeneralizing this, but I feel that this article seems to state that the study authors seem to be complaining that biology students are not well versed in chemistry and physics, and not that they can't think scientifically. If a student honestly doesn't know a marble will continue in a straight line coming off a curved track and doesn't understand the physics behind it why should it be surprising they use their intuition to figure out a credible answer? I supposed the correct scientific answer from them should have just been "I don't know".

Even their website is heavily influenced by chemistry. I am not trying to be dismissive of this, but I do think it is disingenuous to claim students are lacking in scientific reasoning because they do not have a full grasp on the influences of chemistry and physics on the life sciences.

This is not a surprise to me at all and it shouldn't be to anyone else either. I regularly find myself explaining things to someone just to run in to what i call "the wall of stupid", which is a set of preconceived notions that the person believes for no logical reason.

There doesn't seem to be a way to break down this wall with most people so the only real solution I've found is to give up and walk away. No matter how well thought out your arguments are, or how flimsy there arguments are there they are unsinkable ducks. There are of course exceptions to this rule, those people who choose to evaluate things based on evaluating available evidence scientifically but those people are few and far between.

This is the same psychology that allows people to believe in religion. Although I know this won't sink in, you can't prove something exists just by highlighting that you can't disprove it.

After doing undergrad, gradschool, Ph.D., and postdoct in a physical science, I'd say that in many cases, scientific thinking is something that has to be beaten into you by actually doing science, and being forced to do it. Usually that process completes somewhere between the last year of undergrad and the third year in grad school, and only for those people who actually do science. In empirical science it seems to take a few experiments that really do not come out how you think they will before you start questioning your assumptions and inherent correctness on a systematic basis.

If you then leave science, and then go do something else like engineering where things are a bit more predictable, or probably finance / PR where the mechanisms behind some processes are more or less unknowable, then the rigor you learned will likely slowly leave you as various shortcuts make their value evident.

In my opinion, the lack of scientific thinking in the general populace is simply because it is the least efficient way to think about things. It takes a lot of mental effort to question your convictions all the time and a monstrous amount of work to continually run controls on stuff that you are pretty sure are doing what you think they are. The scientific method, while in my opinion the most rigorous way to work on a problem, is against human nature.

Maybe I am overgeneralizing this, but I feel that this article seems to state that the study authors seem to be complaining that biology students are not well versed in chemistry and physics, and not that they can't think scientifically.

This is a very good point - there's a huge difference between not knowing something, and being unable to apply it "scientifically" once you do know it.

But, when asked to actually trace what's going on on the organismal scale, many students have problems recognizing that a substantial proportion of a tree's solid bulk originated from a gas, instead suggesting that most of it was brought up from the soil.

Your "substantial proportion of a tree's solid bulk" is brought up from the soil in the form of water. While I know what you were getting at, I'd expect more care with your wording given the article's topic... ;)

Yeah, that bugged me too. I assumed they were talking about the fibrous matter of the plant, not water and I still would have guessed the wrong answer. However, it has nothing to do with lack of scientific reasoning. I know that plants take in matter through their roots (nutrients), and I know they take in matter through respiration. But I have no idea what the proportion of these two amounts are, because we never talked about that. Simply knowing that some carbon is being fixated via photosynthesis isn't enough to deduce that nearly all the carbon in the plant comes from that.

More depressing still, the authors designed a short course to help introduce formal scientific logic, but it didn't help much.

Really it's amazing that a short course even helps at all when you realize it's matched up against roughly a couple decades of ingrained "common sense". I think that is the good news, that progress can be made at all when you aim at the issue.

I was a bit flabbergasted at how the article content and the article title related. Not knowing a few key facts about physics does not mean that you're failing at scientific reasoning, necessarily.

Personally I'd consider scientific reasoning to be more than making a prediction based on a known law of physics, and would possibly include things like making good experimental designs and drawing logical conclusions based on your data and stuff like that. Then again, given most people's lack of understanding of statistics, the article may still very well be true because most people don't understand statistics and how they assist conclusion-making.

But, when asked to actually trace what's going on on the organismal scale, many students have problems recognizing that a substantial proportion of a tree's solid bulk originated from a gas, instead suggesting that most of it was brought up from the soil.

Your "substantial proportion of a tree's solid bulk" is brought up from the soil in the form of water. While I know what you were getting at, I'd expect more care with your wording given the article's topic...

Yeah, that bugged me too. I assumed they were talking about the fibrous matter of the plant, not water and I still would have guessed the wrong answer. However, it has nothing to do with lack of scientific reasoning. I know that plants in matter through their roots (nutrients), and I know they take in matter through respiration. But I have no idea what the proportion of these two amounts are, because we never talked about that. Simply knowing that some carbon is being fixated via photosynthesis isn't enough to deduce that nearly all the carbon in the plant comes from that.

Pavon, are you a biology student? If so, how far along?

I mean no disrespect whatever, but, depending upon your answers to these two questions, it sure sounds like you are a perfect illustration of the issue highlighted here.

All one needs to do, in order to recognize the correct proportion of the sources of carbon in a tree, is to ask oneself "how much carbon (roughly speaking), as opposed to water and trace elements, are drawn up a tree's roots?" Do trees mostly "feed" upon the "nutrients" of the ground, making the fixing of carbon via photosynthesis almost negligible? Or is the fixing of carbon via photosynthesis central to the metabolic viability of a tree?

and a tendency to substitute a wall of scientific terminology for actual understanding.

IMO, This right here. Often times I that even when someone does understand, their terminology can be translated to simpler terms.

I think two excellent example of this are plane on a conveyor belt and 2 identical cars colliding head on vs double speed into a brick wall. I have to admit I got bit by the cars one but eventually I came around. Fortunately for the laymen Mythbusters took car of both of these

What's the root of this problem? The authors ascribe a lot of it to language.

Fuel for the Sapir-Whorf fire?

My experience in speaking with friends who TA undergraduate natural science classes is that the language that students use is often not scientifically appropriate. An example in Chemistry would be students who write that sodium and chloride want form ionic bonds because they need to have stable electron configurations. While this is an acceptable way to roughly explain interactions between elements to a lay person, a scientist would not be able to use words such as "want" or "need" when describing these phenomena as elements, so far as we know, don't have desires or needs. There is no compulsion or force of will; they just react as such under appropriate conditions.

Perhaps what is students need is instruction not just in thinking scientifically, but speaking scientifically. Without the appropriate vocabulary and understanding of the appropriate use of scientific language, students of the sciences may not realize the mistakes they make in conflating colloquial language with the more precise scientific language.

Can someone explain the curved track example? Most people think the marble will continue to follow the curve of the track even when the track ends and the ball starts rolling on a flat surface? I feel like I am missing something about that example that leads people to believe that...

We teach people to regurgitate and never challenge them to think. This needs to start at an early age, encouraging students/kids to understand why instead of being content to know "what".

When you get past basic science, "why?" becomes impossible to answer.

Just because it's "hard" to explain doesn't mean it's "impossible". Feel free to come up with a theory and prove or disprove it... but don't just sit on your ass and say that it's too hard so I'm just going to believe whatever is said.

The problem is, as the OP said, no one bothers to think or question anything. That's why we have the idiots at FoxNews, for example, who spout off lies and people just blindly believe them. No one bothers to be the skeptic and ask why or whether the statement is reasonable or whatever.

Look at the science of highway traffic engineering. Every single fact to date shows that highway speed limits are not being followed, yet people think "speed kills". If anyone bothered to THINK about it a bit, you'd see that the "common sense" of "speed kills" is scientifically incorrect.

Can someone explain the curved track example? Most people think the marble will continue to follow the curve of the track even when the track ends and the ball starts rolling on a flat surface? I feel like I am missing something about that example that leads people to believe that...

I can see how not understanding the difference between torque and force would lead someone to think the marble would continue to go in a circle, perhaps they think the momentum is circular so it would continue moving in a circle.

As for the experiment I think it's just a circular track and then one piece is removed.

schwinn8 wrote:

Look at the science of highway traffic engineering. Every single fact to date shows that highway speed limits are not being followed, yet people think "speed kills". If anyone bothered to THINK about it a bit, you'd see that the "common sense" of "speed kills" is scientifically incorrect.

Technically speaking speed would cause you live longer, that is relative to other people going slower than you...

We teach people to regurgitate and never challenge them to think. This needs to start at an early age, encouraging students/kids to understand why instead of being content to know "what".

When you get past basic science, "why?" becomes impossible to answer.

Just because it's "hard" to explain doesn't mean it's "impossible". Feel free to come up with a theory and prove or disprove it... but don't just sit on your ass and say that it's too hard so I'm just going to believe whatever is said.

A scientific theory doesn't answer the "why?"; only the "what?" and the "how?".

The marble on a curved track is a terribly poor example. If the track has a wall to keep the marble on the curve, and if the wall and marble have mutual friction, then the marble will come off the track with spin opposite the direction of the curve (clockwise track = counterclockwise spin). The spin will tend to make the marble continue following the curve of the track, though the marble path will rapidly straighten as friction with the table takes off the "English".

@ PavJ - darn it. missed your prior comment. sorry.

Similarly, "intuition" enables us to catch a ball affected by crosswind, drag, spin turbulence, and many other forces - such that a typical calculation of position for the ball would not match its actual location.

Scientific reasoning is great when it accounts for all variables.

Any scientists out here want to design an experiment that controls for all variables?

But, when asked to actually trace what's going on on the organismal scale, many students have problems recognizing that a substantial proportion of a tree's solid bulk originated from a gas, instead suggesting that most of it was brought up from the soil.

Your "substantial proportion of a tree's solid bulk" is brought up from the soil in the form of water. While I know what you were getting at, I'd expect more care with your wording given the article's topic...

Yeah, that bugged me too. I assumed they were talking about the fibrous matter of the plant, not water and I still would have guessed the wrong answer.

It is around even, especially when you are talking wet (AKA living) weight vs dry weight of trees, I believe, once you factor in the H component in the carbohydrates. Watermelons perhaps less so. But it isn't a matter of which is the higher percentage, it is ballpark of percentage understanding.

BTW, not being a biology major, are these biology students doing balance equations of these chemical reactions? To they talk about specific amounts of water vs gases vs nutrients (N, P, and so on) in class? Which are expended, which are stored.