The power of ‘best bets’

The other day I read Greg Ashman’s post Why Education is like smoking which talked about the way teachers often generalise from anecdotes in the same way that when smokers are confronted with statistics about the health risks of smoking they might say things like, “Well, my nan smoked 400 cigarettes a day! She may have had bright yellow fingers but she lived to the ripe old age of 130!” Or whatever.

Teachers do this sort of thing all the time. We say things like, “Well, the research may say x, but I find y works so much better for me!” Maybe it does, maybe it doesn’t but ignoring probability is a poor way to make decisions. As I explained here, relying on individual teachers’ intuitions is rarely a good indication of what will be most effective. Of course, no one can say with any certainty what will be the most effective approach to take on Monday period 2 with Philip in 9b. The physicist Wolfgang Pauli put it like this: “Like an ultimate fact without any cause, the individual outcome of a measurement is, however, in general not comprehended by laws. This must necessarily be the case.” But this doesn’t mean that we can’t say what is more or less likely to effective with a majority of children in a range of circumstances.

Then I read Gary Davis’s blog on why giving teachers access to education research could have unforeseen negative consequences. He argues that teachers, like most other human beings, are poorly equipped to understand probability. He gave this example of where we often go wrong:

The average heights of adult men and women in the UK are 5 ft 10 in. and 5 ft 4 in, respectively.

A scientist picks a random sample of either men or women from the population.

Which of these two options is more likely to be a sample of men?

Sample A consisting of a single person whose height is 5 ft 10 in. or

Sample B consisting of 6 people whose average height is 5 ft 8 in.

If you were tempted to pick Sample A then you’d be in the majority. You’d also be wrong. Probability is counterintuitive and its prediction don’t work at the level of individuals. Predictions and generalisations only really work when applied to large samples. The general rule of thumb is, the large the sample the more likely it is to regress to the mean. In the case of height, if we have enough men, although an individual could range anywhere from 21 ½ inches to 8′ 3”, their average height will be 5’10”.

What are the consequences of this sort of mistake in education? Let’s imagine we have a class of 30 4-5 year olds. We need to teach them to read and we can choose from 2 different methods. Your colleague in the next door classroom is a licensed Method 1 trainer and tells you that research shows Method 1 has an 80% success rate. She also shows you some negative press about Method 2 which indicates that if could result in up to 4 of the children in your class failing to learn to read. It’s an easy choice.

Now, let’s say every one in your class learns to read using Method 1 and you conclude it’s a winner. What’s your experience worth? If some snarky Method 2 proponent points out that their preferred method has the better probability of resulting in fluent readers, what will you say? Will you be convinced if they whip out their calculator and show you both probabilities as percentages? Or will your intuition and go with the evidence of your own experience and what ‘feels right’? Most of us like easy answers and we tend to ignore anything too complicated sounding and go with our guts. Even if, on average, we’re wrong more than we’re right.

Our best bet would be to go with ‘best bets’, but it’s really hard to accurately compare different studies. Tables of effect sizes can seem compelling but they conceal as much as they reveal. We could see what the best bets look like by comparing the bell curves produced by collating evidence on competing interventions. The trouble is, these are almost impossible to find unless you have the wherewithal; to create your own. This is something I have suggested the EEF ought to make available. As of yet, they haven’t.