Last week I promised I’d write about something worthwhile this time, so as promised: one of the most worthwhile things I’ve ever read. This piece is not typically seen as “math education research,” but I think it’s of vital importance for us.

Clever Hans

You don’t have to read the whole original book by Oskar Pfungst; there are plenty of good summaries of the research online (for instance here and here). This amazing story gets talked about in comparative psychology but I think it has special significance – often missed – for math educators.

Take-home lesson: never underestimate your ability to fool yourself into believing your students understand something when really what they are doing is watching you. To force them to engage the material it is often necessary to restrict their access to you or systematically confound the signals they get from you.

I think this is a central issue for modern math teachers. We need to explicitly develop ways of question-posing and interacting with our classes and individual students that hide or disguise our intentions for how they are supposed to respond. This needs to be part of the core training of math teachers, much more than it already is.

Clever Hans was a horse owned by Wilhelm von Osten, a high school teacher in Germany around the turn of the twentieth century. von Osten attempted to teach several animals basic arithmetic and German. The other animals failed but Hans, a Russian stallion, seemed to get it. He indicated answers by tapping his front right hoof. He was able to produce correct answers to a shocking array of questions including the four basic operations, some square roots, calculating days of the month, spelling words, etc. He was regularly displayed by von Osten in public. His feats caused an international stir. Nobody who hadn’t seen it could believe it (von Osten must be cueing the horse somehow, right?), but skeptics were converted when they saw and interacted with the horse themselves. A commission convened by the German board of education found in 1904 that no trickery was involved. von Osten didn’t even have to be present.

Psychologist Oskar Pfungst solved the puzzle: no trickery was involved, to be sure. But if the questioner didn’t know the answer to the question, then Hans couldn’t answer it. (For example, if Pfungst whispered one number to it, and von Osten another, and then asked for their sum, Hans would answer incorrectly.) What was going on was that whoever asked the question was cueing the horse totally subconsciously by tiny imperceptible body movements when the answer was reached. The horse took these movements as an indication to stop tapping that hoof.

Once Pfungst became aware of these movements, and trained himself to control them in himself, he was able to cause the horse to answer a question with any answer he desired. For example, he could say “tap 14″ but then cause the horse to tap 8 by using the body language he had found. Later, in laboratory experiments, he assumed the role of the horse, and “read the mind” of his subjects. He would instruct them to think of a number, and then begin tapping his hand. If he saw the characteristic body language indicating the “right answer” had been arrived at, he would stop tapping. The subjects were amazed by his ability to guess their number and had no idea they were sending tiny signals.

In comparative psychology they’ve taken the lesson of this story, and they’re diligent about preventing the “Clever Hans effect” from interfering with their experiments. In math education we still need to. If a horse can have all of Germany and beyond believing that it can extract square roots, when what it’s really doing is taking subconscious cues from its trainer, think of what human children can do.

This is not idle speculation, but a very real dynamic that shows up at least occasionally, in one form or another, in almost every math class I have ever watched or taught. It can happen at two levels:

To some students, getting cues from the teacher rather than thinking about the math has become such an ingrained habit that it is their entire modus operandi when doing math with an expert around (teacher, tutor, parent, peer, etc.). They usually have lost faith in their ability to have math actually make sense to them, and this cueing has become their sole survival strategy. They regularly fail tests and have come to accept this, but they would rather fake you out than admit ignorance – understandable, since they often carry a feeling of stupidity arising from their lost faith, and admitting ignorance would mean exposing this supposed stupidity. So they watch you and produce the right answers more often than not. Not with any sense of trickery or getting over on anyone – often, they’re barely conscious of the game. But it’s how they survive.

Most of us have had students like this in our class whether or not we recognized it. I’ve worked with several such students as a private tutor and this context puts their habit into very stark light. The first day I meet such a student, the very first problem I have them work on, they’ll make a guess quickly and watch my reaction to see if they’re on the right track. I’ve trained myself to respond noncommittally to everything:
“Is it 1/3?”
“Why do you say so?” (Even if 1/3 is correct.)
“Oh, no no, 1/4?”
“Why do you say so?”
I also often hide my face when posing a problem, or (even better) move out of my student’s visual field. These are not the only moves in this situation, but something similarly radical is needed: such a student will never engage the math itself unless all recourse to his or her standby survival method has been methodically denied. I believe that a necessary part of the repertoire of every math teacher is a set of moves designed to hide or confound the cues we send. The above are just a few suggestions.

In Pfungst’s research, when Hans was fitted with blinders and posed a question by a questioner outside his visual field, he made strenuous efforts to see the questioner. Similarly, when we begin to work on not telegraphing the answer we are hoping for – developing a good poker face, for example, or responding with “what makes you say so?” whether the proffered answer is right or wrong – students who are committed Clever Hanses will make every effort to get the answer out of us anyway. They will ask questions, make guesses, keep the game going till we let something slip. One of the lessons of the story is that it’s virtually impossible not to let something slip. Pfungst found that people communicated answers with minute physical signals they weren’t aware of. Even people who were made aware had trouble not sending them. So when working with a committed Clever Hans you can’t just trust yourself not to give up the goods. It’s often necessary to physically vacate the space between the student and the problem. When working with an individual kid this can even mean leaving the room.

Now even kids who are not in the situation described above, who have not come to rely solely on observation of the expert, can be drawn into a Clever Hans game if the teacher is telegraphing the answers too intensely. In most classrooms, for example, the question “Is this enough information to solve the problem?” does not receive a straight answer. Since this question is not usually asked when there is enough information, it is generally safe to conclude that the answer is “no.” Furthermore, this conclusion is a lot less work than actually thinking about the problem. The teacher obviously wants to hear “no” so let’s say it and be done with it.

Similarly, any yes-or-no question posed by a teacher to a class is a setup for a Clever Hans game. It is a lot less work to guess one answer blindly (or wait for someone else to do so) than to actually think about the question. The teacher’s response to that one blind guess gives away the game and the class moves on with the question “answered,” but the very real possibility that no one in the room actually thought about it.

With this in mind I believe it’s a very productive exercise to scrutinize our lessons (working with the whole class, a group, or an individual) with the question “how far was it possible to get by just following my lead?” It’s especially powerful to videotape class and then watch the videotape with this question in mind – you see a lot more that way. It’s also worthwhile to observe other teachers with this same question in mind, because the dynamic is much easier to recognize from the outside than in the heat of it.

I have one other suggestion related to all this: it helps to be legitimately open to students’ thought processes whether or not they initially sound like what we had in mind. This is something I’ve had to work on. I have throughout my career been repeatedly surprised by the discovery that nearly every time a student offers an idea authentically (i.e. not as just a random guess), it makes some sort of sense. Maybe not complete sense, and maybe it’s not at all where I was headed. But if I can curb my initial reaction of “this kid is totally confused” long enough to actually take in the train of thought, there is almost uniformly some worthwhile reasoning inside it. Then even if I need to say “we’re going to stick to the topic,” I can do so after acknowledging the reasoning. The connection to Clever Hans is that if we want them actually thinking, we have to make sure our questions are legit. This gets communicated by acknowledging people for treating them as legit. If the only answers we acknowledge are ones that fit our preexisting image for what the answer is supposed to be, this communicates that the question wasn’t authentic, and it’s probably easier to try to guess what the teacher is up to than to engage it authentically.

To summarize: the lesson of Clever Hans is of central importance for our profession. We want our students thinking about math, not watching us for cues. But it is natural to subconsciously cue them as to what we want to hear. So a necessary part of becoming a math teacher is developing techniques to deny access to or confound these cues. Vacating the students’ visual field while they work is one important method. Another is responding the same way (e.g. “why do you say so?”) whether we believe the student is on the right track or not. But all of us need to be thinking about this.

The study authors claim to find that introducing math concepts through abstract representations did a better job than introducing them in a concrete way in causing students to be able to generalize the concepts to a new situation. However:

* The study authors confuse the ability to produce correct calculations with the understanding of the concepts taught; and more importantly,
* The study authors used a lesson that was poorly adapted to concrete situations.

I conclude that their results don’t have the scope they claim. At most, they show that throwing in distracting information can make math harder to learn, but we already knew that. A better conclusion is that if you’re going to use concrete examples to teach something, be real about it. Don’t present a concrete setting and then dive straight into formal properties; use the setting to bring out the properties by having students think naturally about the setting.

The details:

First of all, the study authors declare in the online supporting material that “in the present research, study participants learned the concept of a commutative mathematical group of order three.” If that doesn’t mean anything to you, don’t worry, I’ll explain it very clearly in a moment. But this sentence already suggests an important flaw in the study design that I actually think is a flaw in a lot of curriculum design, so I’ll highlight it:

“The commutative group of order 3″ is a math object you learn about in a first course on abstract algebra. The “training” the authors used to introduce the idea to the students basically describe the group’s formal definition. (I’ll show you exactly what they do below.) But the students’ learning was assessed by a test that measured the ability to do computations inside the group. The thing taught and the thing tested were not the same at all. I say this is a flaw in a lot of curriculum design: we want students to have a deep understanding of mathematical objects, like fractions. But they’re often just tested on their ability to produce correct computational answers. Knowing how to add 1/2 and 1/3 is not the same as knowing what 1/2 and 1/3 are. Lots of kids never really learn the meaning of the objects (or even register that this is the important part about learning math) because they know they are only going to be accountable for the computational technique. The irony is that in the end, this makes the computational technique much harder to learn. But we set them up for this as long as the only things we ever test them on are the computations.

Now the bigger problem with the study is how the “training” used was inappropriate to concrete situations. But before I can show you this you have to understand something about the concepts and skills supposedly being taught. So, that explanation I promised (skip the next section if you already know this stuff). I am actually going to use one of the “concrete” situations from the study! But I am writing the “training” a little differently.

You know how tennis balls come 3 to a container? Suppose you work in a tennis ball factory packing balls. Every time you get 3, you can pack them away into a container and send it down the assembly line.

So, suppose in front of you is 1 tennis ball, and then along the conveyer belt come 2 more. You pack them away, send them along, and then you have – none!

How about if you have 0 balls, and 1 comes along? Then of course you can’t do anything because you don’t have enough for a container. So you’re stuck with 1.

Or if you have 1 ball, and 1 comes along? Again, you don’t have enough for a container, so you’ve now got 2. But if 1 more comes along, it and the 2 make 3, you can pack them up and send them along, and so now you’ve got 0 again.

Or – this is “the hard one” – you have 2 balls, and 2 come along. You pack up a container, send it along, and you’re left with 1 ball.

I think you’ve probably got the gist by now. This is an alternative arithmetic in which the numbers never go above 2 because every time you get 3 balls you just pack them up and send them along. So, 1+1=2 like normal, but 1+2 = 0 and 2+2 = 1.

Okay, that’s it. You can now do arithmetic in the “commutative group of order 3.” You can probably out-perform all of the study subjects in Kaminski et al.’s study on the tests they took at the end.

Compare this to the training set in the exact same “concrete situation” from the study (this is found in the online supplemental materials):

“A tennis ball manufacturing company is having trouble with their ball-making machine. Instead of producing batches of three balls to fill a container, it is producing batches of zero, one or two balls represented as {no balls}, {1 ball}, and {2 balls}. Consequently two or more batches need to be produced to fill a container. In doing so, the number of extra balls produced needs to be determined.

“Rules for finding the number of extra tennis balls:

“Rule 1. The order of the batches doesn’t matter. The number of extra balls will be the same. For example, if this batch {no balls} is made first and then this {1 ball}, then this much {1 ball} is extra. The same thing happens if this batch {1 ball} is made first and then this {no balls}. We will have this much {1 ball} extra.

“Rule 2. If this batch {no balls} is made with any other single batch, the other amount is always extra. Here are a couple of examples: If this {no balls} and this {1 ball} are made, then this {1 ball} is extra. If this {no balls} and this {no balls} are made, then {no balls} is extra.

“Rule 3. If {1 ball} and {2 balls} are produced, then one container can be filled and {no balls} is extra.

“Rule 4. If {1 ball} and {1 ball} are produced, then we cannot fill a container. So, {2 balls} is extra.

“Rule 5. If {2 balls} and {2 balls} are made, then one container can be filled and {1 ball} is extra.

“Rule 6. If more than two batches are produced, the order in which they are made doesn’t matter. The extra will be the same. For example, if {1 ball} and {no balls} and then {2 balls} are made, then {no balls} is extra. The same amount is extra if {no balls} and {2 balls} and then {1 ball}.”

Now if you actually have an abstract algebra background, you see what they’re going for here. Rule 1 is saying the group operation is commutative, rule 2 is saying that {no balls} is an identity, rule 6 is sort of a bastardization of the associative property, rule 3 declares that {1 ball} and {2 balls} are inverses, and the other two rules are examples of arithmetic in the group.

But my point is what they’re not going for. They’re not making use of the concrete situation to help the reader understand what’s going on. In fact, the very first “rule” leaves the reader with the sense that we might as well stop imagining tennis balls right now. If we were really talking about tennis balls, I wouldn’t have to tell you that the order in which they come doesn’t affect how many you have. Everybody knows this. The excessive formalism divorces the story from any concrete reality.

Rule 2 makes the divorce deeper. If we were really talking about tennis balls, I wouldn’t call {no balls} a “batch.” So by the time you come to rule 3 (when finally, if barely, the concrete situation is referred to in a sensible way – “one container can be filled…”) you have already completely given up on the idea that what they are talking about has anything to do with reality as you know it.

So here’s the lesson: if you’re thinking about using a concrete example to introduce a math idea, don’t worry. The results of the study don’t really mean “it’s better to leave the apples etc. in the real world” as the NYT writeup asserted. The safe conclusion is this: don’t be phony about it. If you are going to use taxi meters to develop linear functions, use the kids’ knowledge of how that situation actually works to bring out the math ideas. If you’re using temperature to explain negative numbers, use it with examples that actually make sense when you think about temperature. (For example, you can use temperature to explain why 5 – 7 is -2 but not why 5 – -7 = 12.) Being excessively formal about something that’s supposedly concrete reality makes the kids stop listening to their own logical reasoning and common sense and start just trying to guess what you’re up to.

I spent the whole day reading other people’s blogs psyching myself up to do this, so now I only have a few minutes to do it, so this one’ll be short.

In April 2008 the NY Times published a little article announcing a study by researchers at Ohio State University claiming to find that “it might be better to let the apples, oranges and locomotives stay in the real world and, in the classroom, to focus on abstract equations.” The study was published in no less a forum than Science Magazine. Knowing that something must be terribly wrong, I found myself inexorably drawn to read the Science article for myself.

The article summarizes the author’s findings that when they taught a certain mathematical concept through concrete examples, students did not do a good job applying the concept in a novel situation, but when they taught it through an abstract representation, the students successfully transferred the concept to a new situation. They concluded that “If a goal of teaching mathematics is to produce knowledge that students can apply to multiple situations, then presenting mathematical concepts through generic instantiations, such as traditional symbolic notation, may be more effective than a series of ‘good examples.'” (p.455)

The bad news is that the actual details of the study design are not found in the article. The good news is that they are found in a supplemental thingydoo on the Science website. The bad news the article costs $15 to download. The good news is that the supplemental materials don’t cost anything. Even better, since I paid for and printed out everything and it’s next to me right now, you can benefit from what I found out without even leaving this site.

If you’re a math teacher you already know there must be something rank in here somewhere. The authors are actually claiming to have scientifically shown that it’s better to introduce a math idea in a way totally disconnected from reality than to base the idea on anything the students already understand about the universe. Well, a look thru the online support material makes the rankness available to all:

First of all, when the authors say they gave students abstract training in one condition and concrete training in a different condition, what they mean is that in both conditions, the students stared at words on a computer screen. In the abstract condition, the words went like this:

“On an archaeological expedition, tablets were found with inscriptions of statements in a symbolic language. The statements involve these three symbols: {circle} {diamond} {flag} and follow specific rules.

“Rules for combining symbols.

“Rule 1. The order of the two symbols on the left does not change the result.

For example {diamond}, {flag} -> {diamond}

is the same thing as {flag}, {diamond} -> {diamond}

…”

In one of the three concrete conditions (the other two were similar), the words went like this:

“A pizzeria takes orders for one, two or three slices represented on individual cards as {1/3 of a pizza}, {2/3 of a pizza}, and {3/3 of a pizza}. Multiple orders are placed at a time; and the cook systematically burns a portion of each group order. Antonio needs help to determine how much pizza is burned. There is never more than 1 whole pizza burned. So the burned amount will always be {1/3}, {2/3} or {3/3}.

“Rules for finding how much pizza is burned stated by Antonio:

“Rule 1. What I order first or second doesn’t matter. The same amount gets burned. For example, if I order this {3/3} first and then this {1/3}, then this much {1/3} is given to us burned. The same thing happens if I order this {1/3} first and then this {3/3}. We get this much {1/3} burned.”

In other words, the instruction in the two conditions was substantively identical, except that in the “concrete” situation there was some additional distracting information.

But the whole point of using concrete examples in teaching math is to connect the mathematical structure to a comprehensible reality in the students’ world, and this was not done at all, in either condition. So the question that the study authors, and the NYT article, addressed themselves to (“Is it better to use concrete or abstract teaching methods?”) wasn’t answered. The question that got answered was “Do people learn better when you don’t throw in distracting info?” (Oh, you mean they do? You’re kidding, right?)

I’ve already made myself late but I want to tell you more detail just to have it on record, so I’ll post again soon.

This blog is a math education research digest. I’ll post weekly about something I read – a study about teaching, learning, cognition; a book or article about pedagogy; anything that provoked my thinking about math education. My hope is that people will find it thought-provoking and useful for reflection, and possibly also useful as a source of information about research.