Mirror, mirror, on the web

Dr Jonathan Sharples’s opening gambit was pretty odd and pretty interesting. For some reason he wanted us to decide between real and fake names for shades of paint. Imagine if this was a key curriculum objective (bear with me). How would you teach children to pull this off successfully? Perhaps you might ask them to visualise the shade – does it make sense? What colour would it be? Would it appeal to someone likely to be purchasing paint? We might describe the deployment of such questions as a ‘meta-cognitive strategy’.

The first three paint names popped up on the screen: Elephant’s Breath, Churlish Green and Norwegian Blue. Unlikely as the other two sounded, I immediately knew that the fake colour was “Norwegian Blue” because that is the breed of parrot in Monty Python’s dead parrot sketch. Which, when you think about it, is a pretty nifty demonstration of the fact that direct knowledge trumps the use of meta-cognitive strategies every time.

Sharples was speaking at the Evidence for Learning (E4L) Evidence Exchange. He heads-up the Education Endowment Foundation (EFF) in England and the recently founded E4L has a license to use the EEF knowledge base in Australia. E4L also intends to run the kind of randomised controlled trials (RCTs) that the EEF runs in England.

Part of the E4L (and EEF) strategy is to produce a toolkit offering a meta-analysis of different education interventions alongside an effect size stated in months of additional progress. This has the potential to enable educators to think critically about different practices but I wonder whether it is going to do that. At the Evidence Exchange, it became apparent that when people look into this toolkit, they see themselves. Like it’s some kind of mirror.

There were a lot of people who are clearly doing great work and achieving some amazing things but it hardly seems due to the toolkit. Instead, they have looked to it for verification. We heard about the use of feedback, for instance. Although this initiative pre-dated the toolkit. There was a fascinating talk by a Victorian primary school principal on mastery learning. However, he made it clear that it was a subject he had been interested in since 1970 and the work of Benjamin Bloom. So you look into the toolkit, see something you recognise, feel validated and off you go.

Unless you are from the Grattan Institute.

Grattan researcher, Jordana Hunter gave a presentation on ‘targeted teaching’ or what is normally known as ‘differentiation’. The Grattan folks had apparently read some literature that showed that this approach was effective (I disagree and you might want to read my analysis). They then asked around, found two schools who were implementing differentiation effectively and wrote about how these schools did this. This is an odd methodology because there is no comparison group of less effective schools. It means that we cannot make causal inferences about what these schools did that made them successful.

And whatever the literature was that they had read that convinced them of the need for such an approach, it was obviously very different to the literature read by the people who constructed the E4L toolkit. The toolkit analyses an approach known as ‘individualised instruction’. It’s hard to pin down exactly what this is and exactly how far the instruction is personalised because it is based upon meta-analyses of differing interventions, but the E4L verdict is clear:

“Individualising instruction does not tend to be particularly beneficial for learners. One possible explanation for this is that the role of the teacher becomes too managerial in terms of organising and monitoring learning tasks and activities, without leaving time for interacting with learners or providing formative feedback to refocus effort. The average impact on learning tends overall to be low, and is even negative in some studies, appearing to delay progress by one or two months.

Empirical research about individualised instruction as a teaching and learning intervention in Australian schools remains limited, and the few Australian-based studies on individualised instruction also tend to focus on either ‘gifted’ or ‘struggling’ students.

The available Australian research suggests that it is not the most effective or practical intervention and shows that teachers face practical difficulties employing this intervention, such as curriculum restrictions and significant increases in their workload.”

Maybe this is something very different to the kind of differentiation being promoted by Grattan. But given that the Evidence Exchange was about E4L then exactly where is the support for the Grattan method in the toolkit? No matter. Nobody seemed to notice this odd discrepancy. After all, differentiation is in the Australian Professional Standards for Teachers so it must be effective.

When Sharples looks into the mirror he sees meta-cognition. There is an extraordinarily vast array of things that the toolkit groups together as ‘meta-cognition and self-regulation’ which spans affective strategies aimed a student motivation and resilience, all the way to prosaic methods for planning. The measured effects are not restricted to cognitive ones. When you take all this into account, it is hard to interpret the extra 8 months of progress this whole mass of different things is suggested to produce.

On his own graph of effects versus costs, Sharples has reduced this to simply ‘meta-cognitive’ and he is clearly a fan.

Meta-cognitive strategies are particularly suited to the kinds of interventions that organisations like the EEF run. Dan Willingham has referred to meta-cognitive reading strategies as a ‘bag of tricks’ and with good reason. They are not skills in the sense that a sequence of deliberate practice will make you improve at them. They are useful hacks that, once known, produce a one-off hike in performance. If you take a student who can’t structure a piece of persuasive writing and teach them the ‘firstly… secondly… thirdly…’ hack then you will see an immediate and significant jump against a standardised persuasive writing test. But how significantly have you improved their writing skill? The slower, much more liminal, curriculum-centred process of building vocabulary is far harder to capture in this way.

I was surprised to see Sharples focus on The Philosophy for Children study. This has been much criticised and with good reason (see here and here). Briefly:

The principal researcher on the study has a philosophical issue with conducting tests of statistical significance and so didn’t do one. I would have thought that EEF studies were public goods and so individual researchers should not be able to impose their tastes on them in this way.

The outcome measure that they said they were going to use before the trial did not show any effect and so is not the one they used in their report. Instead, once they had the data they decided to do a different analysis looking at progress since KS1 results. This is the well-known problem of researcher degrees of freedom – analyse a study enough ways and eventually you will find something that looks like an effect.

I asked Sharples about the lack of statistical significance and he suggested that they have rerun the numbers and the results stand-up. I look forward to reading this paper.

Sharples also displayed a list of four trials that he said could broadly be categorised as meta-cognitive (Although I think he said that there were six – I might be wrong). He claimed that all of these trials showed a positive result, the implication being that, whenever they had tested meta-cognition, it had worked. But isn’t the “Let’s Think Secondary Science” program a meta-cognitive intervention? And didn’t that fail?

Like this:

Related

24 Comments on “Mirror, mirror, on the web”

“Part of the E4L (and EEF) strategy is to produce a toolkit offering a meta-analysis of different education interventions alongside an effect size stated in months of additional progress.”

This is probably the holy grail of educational research but it’s doomed to failure for precisely the reasons you outline above. The apparent inability of educational researchers to look at their own beliefs critically is at least part of the reason why education is in such turmoil and teacher retention rates are so poor. The driver for this ongoing fruitless quest is, of course, the fetishisation of data and the subsequent obsession over programs of ‘intervention’ that are, at best, a sticking plaster over the underlying issues of how children develop skills and retain knowledge and, for that matter, are motivated to do so.

Teaching and learning have been human activities for thousands of years. If there was a magic bullet, we would certainly have found it by now. (Nice to see individualised learning getting a kicking, btw. It’s a ticket to an early grave.)

The last paragraph you wrote reminds me of what a friend of mine (not a teacher) told me when I was facing an appraisal: “we have been teaching children for thousands of years, it cannot be that different today”.

One of the problems that education in England has is the politicization and the excessive influence that non-educationalists have. I was trained in the VAK learning styles school of thought because that is what the government backed up in those days and I was going to teach in a state school. (I’ve read since that it might be David Milliband, writing one of Labour’s policy documents, the person we need to thank for this approach to personalised learning).

I did my training in a very reputable institution in London, but they were spouting the fashionable nonsense of the times: “all children want to achieve, so if they disrupt your lesson it is because the lesson was not well planned” and my favorite, as I teach foreign languages: “you don’t need to teach grammar explicitly because students will acquire it through exposure, the way they acquired their first language” (never mind that this hypothesis goes against experience and decades of research in Second Language Acquisition).

My impression about governments in education is that they hear something that sounds cheap and effective and build policy round slogans. Labour gave us “assessment for learning”, that promised improvement in results and often ended up in a set of gimmicks that even Dylan William has written about. With Labour out and a new government questions such as “what has AfL ever done for us?” started, and we have now “visible learning” (Hattie’s eye-catching title) that, misinterpreted again, has caused the canonization of marking as feedback (because it is visible and it can be measured by visiting SLTs and Ofsted inspectors).

Excellent points jd – this is exactly why no one should have spent time on vaccines for polio or the germ theory of disease. Medicine had been around for thousands of years before people came along touting these magic bullets. So maybe excellent is not the best word to describe your points.

If you don’t know anyone who suffered from polio you might rethink your views on research. At minimum avoid the false dichotomy of we are looking for a magic bullet or we should stop looking for improvements.

Also fetish may not mean what you think it means and if it does perhaps you’ll recognize your emotional appeal to dismiss data driven science as exactly the problem of an inability to look at one’s own beliefs critically.

The key part of jddunsany’s comment was “human activities”. Not chemistry, not medicine, not physics. Education does not fall in the category of a “science”. Education is a product of human interaction, with all the uncertainties of character, idiosyncrasy and prior knowledge and experience which affects all other human interactions, whether it’s selling encyclopedias or striking it lucky at a pick-up bar. And success at the latter activities, by the way, has infinitely more in common with successful teaching than many of us might like to admit. Rules of thumb and the accumulated experience of millennia leave fads backed up by paper-thin evidence for dead.

There never will be a perfect, cure-all way to teach, in the way that there is a vaccine for polio. Obviously some lessons are better than others, some teachers are better than others, but do you really think that the mountains upon mountains of data, papers, studies, interventions, programs and the like over the past fifty years or so have done more to aid the professional growth of new teachers than trial and error, advice and ideas from supportive colleagues, and most importantly of all, a genuine enthusiasm and devotion to their subject and their pupils?

Mike,
When you cite “fads” with “paper-thin evidence” I don’t think you are referring to any sort of scientific evidence based education practice. To keep the analogy going it is like saying medicine is not scientific because of homeopathy.

I’d recommend to you Ben Goldacre’s books because what you are saying is exactly what doctors said when some started pushing for more scientific evidence based medicine. That is that it would be a waste of time as they already new from thousands of years of dedicated doctors what works best.

You only have to pick any of the fads you probably rightfully dismiss. Pick almost any one and you will find enthusiastic teachers who love it and are apply it in the education of the students they are devoted to.

Again the analogy works – there will never be a perfect cure all for all disease. Even antibiotics turn out to have some down sides and spend some time in a hospital waiting room if you think other areas of human endeavor have found perfect solutions.

Waving such an obvious false dichotomy as your argument – that science must either have a perfect, cure all or it is a waste of time doesn’t help you sound like someone whose judgment on how to teach a child anyone should trust.

Yes, I have read Goldacre’s books and enjoyed them. They gave me, in fact, a much better understanding of the necessity of double-blinding and consequently an understanding of something which had confused me considerably: why so, so many of the impressive-sounding studies which I was instructed to wade through during my Dip.Ed. were flatly contradicted by my own (and many others’) practical experience. I’m referring here mainly to the sorts of things which Greg mentions regularly on this blog; the preference for discovery learning over direct instruction, the deification of group work, differentiation, mixed-ability classrooms and all the rest. When I looked back over the sorts of studies I read through at university, it all became very clear – the confirmation bias stood out a mile.

I don’t recall saying that all educational research was a waste of time, but I made a comment on another of Greg’s posts (don’t remember which one unfortunately) about the very, VERY specific cases in which I think it could be of some value. My basic point is that interventions in medicine are in a completely different category to education for exactly the reasons that Goldacre mentions in those books. The majority of educational studies today are, in fact, the equivalent of homeopathy, to my mind. Designed to reach a predetermined conclusion, with a very significant undercurrent of self-promotion.

As for your final comment, all I can say is that my students over the past fifteen years haven’t generally thought so.

Mike,
I may be tarring you with the same brush I would use for they commenter you were supporting. Jd sounds like he or she has no place for evidence in his assessment of what choice is optimum.

Reading Greg’s book and blog and some of those he references there does seem to be a lot of scientific evidence to inform educational approaches. Here a scientifically arrived at negative is as informative as a positive. What not to waste time on seems to be quite well proven in the case of VAK learning styles, PBL and other ideas. Similarly direct instruction seems to have solid evidence to support it so there are positive results too. Another one that seems to be based on science is retrieval practice.

The problem with your claim that we can trust you because of your assessment of how your students like your teaching is it is exactly the same sort of claim we would hear from someone who was doing completely the wrong thing in the honest belief that it was the best option. I can’t tell if that is your confirmation bias and if that is all you are going on nor can you.

There are only two options for education – either it is so messy (cf Tim Hartford) that no one can tell even with wide margins what balance of approaches is optimal or it is possible to find that some approaches should dominate and some be largely or completely avoided. If it is the first case then the important thing is to make sure no child gets stuck with a single teacher or group with the same approach for any extended time (probably weeks not months).

“…What not to waste time on seems to be quite well proven in the case of VAK learning styles, PBL and other ideas. Similarly direct instruction seems to have solid evidence to support it so there are positive results too. Another one that seems to be based on science is retrieval practice…”

Yes, as it happens I agree with all of that.

What I would suggest, though, is that teachers sixty years ago were also aware of all that, even if they wouldn’t put it in such terms. They relied instead on experience, mentoring from colleagues and their knowledge of their subject areas.

As I stated in my earlier post, I’m not dismissive of all “scientific” evidence in this area. But I think that a very large proportion of it is to be approached with healthy (if not extreme) scepticism. And I also believe, for reasons largely explained in my first post above, that even applying the word “science” to educational research is actually mistaken and probably harmful.

“…The problem with your claim that we can trust you because of your assessment of how your students like your teaching is it is exactly the same sort of claim we would hear from someone who was doing completely the wrong thing in the honest belief that it was the best option. I can’t tell if that is your confirmation bias and if that is all you are going on nor can you…”

Not sure why you’re so determined to make this personal, but over time a teacher gets a reasonable idea of what works for them (and their students) and what doesn’t, and most teachers are willing to learn from their mistakes. Kids enjoying one’s classes is not the only criterion of course – there’s results, parental satisfaction, etc., etc. Of course I still have plenty to learn (as do most teachers), but my chief focus will be improving/consolidating my knowledge of my own subject area, rather than taking notice of the latest fad to filter its way through the DET. Long experience with different cohorts of kids, by the way, is the natural antidote to confirmation bias in the teaching profession.

Mike,
I am not the one making it a personal opinion. I am the one saying it should be objective facts. You brought in your personal opinion as evidence and I’m just saying why that is not a basis for any one to accept your point of view. Because they will then have to accept the opposite point of view for the same reason that someone’s personal experiences says it is the truth.

“If you take a student who can’t structure a piece of persuasive writing and teach them the ‘firstly… secondly… thirdly…’ hack then you will see an immediate and significant jump against a standardised persuasive writing test. But how significantly have you improved their writing skill? The slower, much more liminal, curriculum-centred process of building vocabulary is far harder to capture in this way.”

Couldn’t agree more.

As I mentioned on one of your other posts, I can see this happening with my daughter this year, and it’s frustrating.

Greg,
I was surprised at your compliant about hacks. Aren’t these going to be very hard to distinguish from subject matter expertise? Isn’t knowing the times tables just a hack? There is no skill involved in automatic recall of what 7×8 is. Yet it is a useful ability to have in order to get on with more interesting mathematics. Why wouldn’t this apply to a hack such as organizing an argument? It may be that the writer doesn’t understand the big picture but knowing how to break down the problem with a simply hack lets them focus on the other bits such as forming good paragraphs.

I think Greg’s point is that the “hack” then tends to be valued over the more important and long-term skill of improving vocabulary, overall sentence structure and grammatical precision (hence his reference to “a standardised persuasive writing test”). It’s easier for a teacher to tick a box rather than to measure improvement in other, perhaps less tangible areas. As I said earlier, I’ve seen this first-hand with the way my daughter is being taught this year – all that Genre Theory drivel still has a huge influence in primary schools.

Sure hacks could be valued more than deeper knowledge but doesn’t this sound exactly like the argument made by those who don’t value learning the tables and instead want to develop deeper understanding of math.

I propose that the approach to correctly value all hacks is to make them guilt free table stakes. If all students know the hacks then any competitive situation such as exam results requires that they have something more than the hack.

By definition a hack has to be efficient. That is it is quicker to learn and use the hack than use some deeper thinking or knowledge to get to the same result. So hacks are always a good idea for someone sitting an exam.

What hacks are efficient depends on the situation. For example in an exam where there are four possible texts to be examined and students must answer questions on two out of three options in an exam then the hack for students is to spend time on only three rather than all four while studying. (Here four are listed in the curriculum and three will be known at exam time.)

I can’t say whether learning three books more thoroughly than four less thoroughly is a good use of time but that is the implicit message in such a setup. If that is bad then fix the exam setup don’t avoid the simple hack. In fact if the exam is setup this way then it would be better to state in the curriculum that the obvious way to approach the exam is concentrate on three books not four.

Once it is explicit it is no longer a hack it is the stated intention of the setup that deeper knowledge of three books is the goal. Having four books is just to given people some options to address a wider range of interests. It becomes an anti-hack to spread your study time too thinly and try to cover all four books.

If you want learning to be more than hacks then share yours, blog about your favorites. Remove any guilt associated with using them. Force those setting exams to counter any unhelpful hacks or force students and teachers to do more than the hacks to distinguish themselves.

“…Sure hacks could be valued more than deeper knowledge but doesn’t this sound exactly like the argument made by those who don’t value learning the tables and instead want to develop deeper understanding of math…”

No, again that’s a false comparison. Learning the times tables is an essential part of a hierarchical body of knowledge – that’s the key point. It leads on to other things and forms the basis for other things. Learning to parrot the hackneyed “Firstly…secondly…finally…” stuff for a “persuasive text” is not really part of any hierarchy of knowledge when it comes to literacy, it’s just a convenient way for teachers to tick a box. It doesn’t contribute to anything beyond itself, and yet it becomes a marking/grading criterion. This is not how things should work.

“…What hacks are efficient depends on the situation. For example in an exam where there are four possible texts to be examined and students must answer questions on two out of three options in an exam then the hack for students is to spend time on only three rather than all four while studying…”

I still don’t see why your assertion is true. Why is knowing 7×8 with automatic recall essential? Sure knowing a few cases of single digit multiplication is useful but all the tables? At some point this becomes a hack to make life easier rather than a form of deep knowledge or essential for gaining such deep knowledge.

If I do advanced calculus with my Year 12s and quickly need to know 7×8 then the tables will still be really helpful, reducing my cognitive load and enabling me to remain focused on the question. If I try to write a Year 12 persuasive essay using ‘firstly, secondly, thirdly..’ then I will be punished by the examiner for doing this.

Sorry Greg, Mike,
I still don’t see it. Sure the times tables remain useful in the same form as they were first learned. But mathematics is an extreme case in terms of material being built up without the initial material becoming superseded.

The firstly, secondly, etc hack is just one way for a writer to provide the reader with a structure and signal it clearly. On its own there is nothing bad about it. The problem is it is overused. But that is going to happen to whatever is chosen as the first choice to teach.

You could keep on teaching new hacks which are alternatives to signal structure. Individually each would look like the firstly, secondly hack. But eventually the mental act of picking which to use from a large set becomes the automatic thinking of a good writer.

If someone wrote instructions for teaching writing that said first demonstrate and practice using a specific structure and phrasing, next practice using that one and another one, then three and so on. Where at each point when a new one is introduced some practice is done with it, then examples and practice are done using all the options learned so far. Would this be a terrible way to teach writing?

[…] Furthermore, some RCTs can be pretty badly designed or analysed. The new RCT factories such as the Education Endowment Foundation in England and I3 in the U.S. tend to find positive effects that don’t always stand up to closer scrutiny. […]