Having looked at ridiculous data crunching for the past few posts, I’m now going to look at unproven conclusions drawn from dodgy assumptions about dubious data. If the study of Statistics tells you one thing, it is to Beware of Demons. In particular, beware of dubious and dangerously incorrect assumptions which lead to dubious and dangerously incorrect conclusions. Data crunching means nothing if you start from a false premise. In particular, my ire has been provoked by entirely unjustified statements made in Ofsted reports. Teachers have, unsurprisingly, often complained that Ofsted has got it wrong about their school. What I aim to show is that Ofsted has often got it wrong before they enter a school, and that their underlying assumptions about what schools do are demonstrably false, dangerously inaccurate and career-destroyingly bad. But first, a bit of anecdote. I can’t be up on my high horse all day, and everyone loves a story. So here’s mine. My first class was a Year 3 one, with 30 children in inner city London. When I say inner city, I mean fifty plus home languages in a one form entry school, no grass to be seen anywhere nearby and a lot of cultures clashing. As an NQT, I worked stupidly hard, much harder than I worked in my training, which had been hard in turn, and teaching was much harder than my previous career. I struggled with behaviour and classroom management, as many new teachers do. It took up a huge amount of my time as I learned to be a teacher. The thing I didn’t really do was help any of my children to read. I can say that now, because I have helped children to read since. But to start with, I didn’t have a clue and I didn’t do very much at all to help my Year 3 class develop their reading. The odd thing was that the class did get better at reading; at least, the not-at-all-optional SATs they did at the end of the year indicated that, somehow, they’d got better at answering questions in reading assessments. Moving on a few years, I had another class which had in it two really tricky children. One was a buddle of neuroses and didn’t like school at all. This child spent significant proportions of the year upset and withdrawn, and in many, many meetings with combinations of me, my head teacher and parents. The other was quite happy, but unwilling to do anything other than the bare minimum, coasting through the year. At the end of the year, when the dreaded n-a-a-o SATs came around, guess what? They had both made excellent progress, and I submitted their levels with a bemused smile. I got the credit for their progress, but I was fairly sure that it had very, very little to do with me. They had learned something somewhere, even if it was by osmosis, chance, watching television, whatever... but, again, they didn’t learn it from their teacher. I have also worked with children with and for whom I have spent huge amounts of time, drawing them out, moving them on and generally developing them tremendously. That hasn’t always translated as progress in measurable assessments, however, and some have not lit up the assessment trackers on my watch.Enough anecdote. Let’s look at some research.

Adding to an already rich cake mix

The Sutton Trust has been looking into the ways in which the ‘well off families’ gain advantage for their children. Their recent ‘Parent Power?’ report confirmed the extent to which many families pay for private tutors. Private tuition, or ‘shadow education’ in the jargon term, is widespread in England. As long ago as 2005, the Sutton trust found that 18% (and 34% in London) of 11-16 year olds had had private tuition. In 2010, they found the figures to be 23% (38% in the capital), and last year they reported that 24% (40%) of 11-16 year olds had ever received private tuition. Now these figures are of course not what is actually happening, but there is the indication of an upward trend here. The research measures something which is a specific choice by families, and the same organisation has done the research three times. The trend is for children aged between 11 and 16 to be more likely to have received ‘shadow education’ outside of school recently than a while ago. Perhaps 7 out of 30 (12 out of 30 in London) have received some additional teaching; and more children are being tutored now than were previously. The observant reader will have spotted that I have used 30 as a denominator, and will have anticipated what I am about to say. This research suggests that in a given class of students, it’s reasonable to expect some of them to have had additional teaching at some point.Shadow education costs money, as does gaming the system A further point for consideration is the amount parents spend on shadow education. The ‘Parent Power’ report provides examples of hourly tuition fees in February 2013 of £20 per hour for Primary pupils outside of the South East to over £37 per hour for A level Science in the South East. It’s a reasonable assumption to say that the vast majority of parents of children claiming Free School Meals are not in a position to pay anything like these amounts. The Government’s figures suggest that round 18% of Primary School children are eligible and claim free school meals. A further 3% re eligible but do not claim FSM. So just of 21% of children in primary are likely to be too poor to be ble to afford to pay for any private tuition. If just 1% do scrape together enough money to give their child additional help, that leaves a round 20% or one in five children who have to be excluded from any group of children who might use the services of a private tutor. Factor that into the Sutton Trust’s figures of 24% (London 40%) of children receiving outside help, and it then appears that just over a quarter of children in schools in affluent areas with extremely low numbers of children receiving FSM outside London receive tuition. So at a school like, for example, St Oswald's CofE Aided Primary School, Chester, where just 3% of children claim FSM, or Christ Church Primary School, Hampstead, where just 5% do, the ‘school effect ‘ is boosted by more than a quarter of children (as much as a half in London) receiving extra tuition. And conversely, a school with very high levels of FSM will have almost no outside help from tutors at all.You paid how much for a house near a school? The Sutton Trust’s figures suggest that, in 2012, “parents face a ‘premium’ of around £91k to buy a house near a ‘top’ state primary school, and £77k to live near one of ‘the best’ secondary schools”. It would be reasonable to assume that those who can find this much cash to pay for a house near a school are likely to be part of a group more keen to see their children do well, and provide shadow education over and above the figures quoted above.

As the Sutton Trust noted when their report was released in December last year, "Around one in three (32%) professional parents with children aged 5 to 16 has moved to an area which they thought had good schools, and 18% have moved to live in the catchment area of a specific school."

A distorted national picture

All of this suggests that children in different schools are given very different amounts of support by their parents. You couldn’t take a child out of Christ Church Primary School, Hampstead, and replace her with a child from a deprived area elsewhere in the country and expect that their families would suddenly change the amount of money they were prepared or able to use to support their children’s education outside school.

So what picture have I painted here? A large and significant minority of children are being helped by those not directly employed by their school. Some children’s parents are prepared to pay a huge premium over and above average house prices to access a particular school. One in three professionals move to ensure that their children are not placed randomly in their state school.

In summary, only an idiot would suggest that children’s educational progress is exclusively explained by those who teach them and the schools in which they work.

Here come the idiots, having ‘analysed some data’

Here are some quotes from recent Ofsted reports:

“Over time, pupils have not always received good enough teaching to help them make better than expected progress relative to their starting points.“Canon Popham Church of England Primary and Nursery School, Edenthorpe, Doncaster.17–18 December 2013.

“Over time, the quality of teaching has varied too much. Some has been inadequate.“Sigglesthorne Church of England Voluntary Controlled Primary School, Sigglesthorne, Hull.27 January 2014.

“Over time pupils have not made good progress because the quality of teaching has not been good enough. “Moorhouse Primary School, Milnrow, Rochdale.27 January 2014.

“Teaching requires improvement because it has not promoted consistently good progress, especially in mathematics and among more-able pupils”Hilltop Primary School, Hilltop Road, Frindsbury, Rochester, Kent, ME2 4QN.14 November 2013.

“Teaching requires improvement because it has not promoted consistently good progress, especially in mathematics and among more able pupils”Churchfields Primary and Nursery School Churchfields Road, Beckenham, BR3 4QY.21-22 November 2013.

Note that these statements refer to 'progress'. In this context, 'progress' is a specific measure of the change in assessed point scores between Key Stage 1 and which ever year of Key Stage 2 these children are in.It doesn't mean 'have got better,' as many lay people might assume. It means that someone has looked at dodgy data and decided that the children in the school were being failed by their teachers.

Really? You could only make these statements if you actually believed that the academic attainment at a school was solely the responsibility of the teachers in the school. And, well, who knows, maybe that’s what these people think.

But that mindset is demonstrably wrong, and the fact that Oftsed, the DfE and Senior Management Teams accept this orthodoxy causes huge amounts of damage to children’s education in England, for reasons documented elsewhere.

Hundreds of teachers who have not had this nonsense levelled at them by Ofsted have are just as likely to have got lucky and had children with supportive parents who have helped their children to be able to learn, either through understanding the things which children need to do well at school or by paying in some way to get another child or adult to help their child to learn how to do well in school.

There may be other reasons why some schools make great progress, and heaven knows we’ve all heard of the desperate ways schools game the system. It’s my contention that, because school is the icing on the cake, there will be a cake underneath whatever icing the school is able to whip up. The idea that teaching “has not promoted consistently good progress” is simply insulting and wrong.

The 'Over Time' Scandal

The phrase 'over time' is a giveaway too. This is Ofsted shorthand for 'I've looked at the data and drawn some conclusions which are not even wrong.' These statements could only be made if you had looked at some data and drawn these conclusions, even though those conclusions are clearly not valid as the data analysis is beneath contempt.

Oh, and while I’m at it…

Did you notice anything about the last two reports above? Here’s a clue: They were both written by the same Ofsted Inspector. I have spent two months every year for the past decade writing reports for children, and I would never, ever have tried to get away with using exactly the same wording in two different reports, even if the two children were extraordinarily similar.

And I haven’t included the full sentences, because the suggestion at the end of the sentence, highlighted in italics below, is extraordinary:

“Teaching requires improvement because it has not promoted consistently good progress, especially in mathematics and among more able pupils, as a result of historic weaknesses in planning and assessment.“

Good progress has been hindered by ‘historic weaknesses in planning and assessment ‘.Remember that these children have made progress, it just hasn’t be judged ‘good.’ The only way this be true in both schools, is that this statement reflects the inspector’s prejudice about what causes a lack of ‘good progress’, namely teaching and teaching alone. What’s more, that lack of ‘good progress’ is a judgement which has to have come from - hopelessly inaccurate and not even wrong - analysis of both schools’ data in RAISEonline.

So this lie – I cn’t really describe it as anything else - about teachers’ planning and assessment has been concocted based on interpretation of data, which as I have shown in my articles on OSDD, Performance Tables and RAISEonline, is fatally flawed and not even wrong.

It goes on:

“The school’s own evaluation of the quality of teaching shows that some in the past has had a negative impact on pupils’ performance (). Teaching has now improved (). This is improving pupils’ achievement, but pupils are not yet making good progress over time in all years and subjects.”

“The school’s own evaluation of the quality of teaching shows that some in the past has had a negative impact on pupil performance. Teaching has now improved and this is improving pupils’ achievement, but pupils are not yet making good progress over time in all years and subjects.”

The first paragraph is from Hill Top’s report and the second from Church Fields. Or is it the other way around? Who knows? Because this reflects not even wrong analysis of data, and it is insulting in the extreme.

The school’s evaluation of teaching has had “a negative impact on pupil performance” in two different schools, inspected a week apart? In both schools it has “now improved” but children aren’t making enough progress, and this is because of planning and assessment which is, by implication, not good enough?

How dare they? How dare they suggest that teaching has not been good enough? When the person who has taught that child has spent an average of 60 hours a week, typically working from 8am to 6pm, every day, for six to eight weeks at a time? Has spent hours planning, preparing and assessing; marked work; written reports, each and every one tailored to each child (at least in a way that no parent could accuse any one of having cut corners or mad assumptions about the child); planned trips and completed endless Risk Assessments for no-existent risks and follow up forms so a box is ticked; run clubs at lunch time, clubs after school, chaperoning children to netball, football, cricket and rugby matches and umpteen other sports events; taken children to choir performances and organised and escorted trips to the theatre; neglected their own children, family and friends; worked till midnight far too often and woken up at 4am trying to find a solutions to the endless issues which crop up with small children; and spent their every waking hour thinking about their job…

It couldn’t possibly be because a small cohort of children are being compared - in a obviously and stupidly invalid way - with a national cohort where anything between 24% and 40% of children are being coached by providers of shadow education, and parents who value educational achievement and have the means to do something it make a huge difference on behalf of their children?

No, in these schools, someone has the nerve to swan in and say teaching has not been good enough based on data analysis which is - in my new favourite term - not even wrong. And then write a report which could clearly have been written without even stepping inside the school. That’s not good enough and it's not even wrong, Ofsted. You should be ashamed of yourselves. Teachers' careers are destroyed by your judgements, which are made on outstandingly bad analysis and prejudice.

As the great Bill Hicks said of those who work in marketing, “There is no rationalisation for what you do.” He went further, and I am tempted to do so as well. I’ll just say that I’m planting seeds too. Holding schools accountable for factors beyond their control and misjudging them using bad data is not even wrong. And this scandal has to stop. School is simply the icing on a very big cake, and those who think otherwise are deluding themselves.

Let me attempt to paraphrase: teachers work very hard, but there are many other factors which determine how much progress children make. These other factors might be known, but are cryptic for any specific child. Ofsted judgements are based on inappropriate analysis of data of uncertain worth, and are unhelpful with respect to improving outcomes.

My immediate thoughts are these: what can teachers do to optimally positively effect outcomes? How can the accountability systems be changed to encourage such behaviour? What data (if any) is it worth schools collecting, and how can it be used to improve outcomes?

Reply

Icing on the Cake

24/3/2014 12:48:12 pm

Thanks for this useful precis, Chemistrypoet. A good summary.

Good questions too.
Q1) What can teachers do optimally positively effect outcomes?
A: Well, John Hattie that pretty much anything you do makes a differences, as long as you intend it to (I summarise, and ignore criticism of his data, and other considerations apply). I'd broadly expect that this is the case, however. Just care and nurture and the cake will bake itself. Teachers who have some clue as to what they are doing, and are supported and nurtured by their SMT should add excellent icing.
Q2} How can the accountability systems be changed to encourage such behaviour?
A: Develop a better understanding of the limitations of big data and move on from the Data Disaster. Then get managers to manage their staff without Ofsted hovering above everyone. Stop Ofsted grading schools and have them collaborate instead of judging.
Q3) What data (if any) is it worth schools collecting, and how can it be used to improve outcomes?
I'm tempted to say that, until children are much older, any assessment/test data is not worth collecting and diminishes education. No point collecting data if you can't do anything with it, so don't.

All this said, these are probably questions for wider, more considered responses after the current consensus on RAISE and similar data analysis has been challenged and overthrown!

Reply

Chemistrypoet

24/3/2014 02:22:32 pm

Indeed. More considered responses will be required....someone needs to start thinking about them, though.....

SBM

24/3/2014 12:15:46 pm

Very powerful
You might want to review the end part for a wayward apostrophe though

Reply

Icing on the Cake

24/3/2014 12:19:20 pm

I'm off to write "it's" and "its" a hundred times... Thanks for proof reading - much appreciated!

Reply

Andrew

3/4/2014 03:15:52 pm

Did you mean the it's or the teacher's?
It's the latter for me that's wrong.

Reply

icing on the Cake

4/4/2014 01:14:29 am

The Teachers' apostrophe should now be correct - thanks for reading and correcting!

igb

28/3/2014 03:39:31 am

"It couldn’t possibly be because a small cohort of children are being compared - in a obviously and stupidly invalid way - with a national cohort where anything between 24% and 40% of children are being coached by providers of shadow education"

That's a grotesque over-statement of what the Sutton Trust polling found, and given we're all interested in using data accurately, sauce for the goose should be sauce for the gander.

What the Sutton Trust found (http://goo.gl/trTCPz) is that 24% of people aged 11-16 in maintained schools answered yes to this question:

"When, if at all, have you received private or home tuition?”

That's a shockingly weak piece of questioning, and the results are meaningless. Does it include my sitting down with my children and spending an hour looking at some homework with them? Does it include violin lessons? Does it include religious instruction provided by mosques, including the teaching of Arabic? Does it include the Saturday morning "Chinese School" near here, in which expatriates gather once a week and encourage their children to improve their Chinese? Does it include painting at your local arts centre, Saturday morning drama groups, ballet? Does it include outreach schemes run by universities, after-school revision classes run by teachers out of the kindness of their hearts, paid-for after school language teaching? What about swimming lessons? What about tutors visiting ill or excluded children at home or in hospital?

Now your response will be "oh, it's obvious what it means: private instruction, on a paid-for basis, whose content overlaps the national curriculum in order to help you with your school work". But it doesn't say that, and if you're a child from a council estate whose parents would no more pay twenty quid an hour to drill you in your times tables than fly to the moon, and aren't actually even aware that such things even happen, but know your parents do pay subs at the football coaching sessions, what would you answer?

There's anecdotally quite a lot of tutoring going on, although if you factor out the tutoring associated with the legacy 11+ systems in some areas I suspect it drops a lot. It would be very interesting to do some rigorous fieldwork and find out how much. But by asking questions like that of 11 year olds? Seriously?

Reply

Icing on the Cake

28/3/2014 07:45:51 am

Thanks for your long and detailed response, igb, it is very much appreciated. As you note, we should all be interested in data being used accurately, and you are quite right to look more closely at the data to test if conclusions drawn from analysis are reasonable.

Having had a closer look at the Sutton Trust research again, I can see clearly your point that the question posed ("When, if at all, have you received private or home tuition?”) can be seen as a weak piece of questioning. I’m not going to argue too much, because as I noted, “These figures are of course not what is actually happening.“ But yes, I had looked at their research and made the assumption that everyone is clear that "When, if at all, have you received private or home tuition?” referred to academic tuition provided by a non-family member. And you are right to question that assumption.

So is there any other research which might shed light on the level of shadow education?

I have found a Judith Ireson and Katie Rushforth’s report “Mapping and evaluating shadow education” which was published in January 2005. (http://www.esrc.ac.uk/my-esrc/grants/RES-000-23-0117/read)

This study reports that of 3515 pupils who were questioned, 27% had received tutoring, and the questions asked specifically in which subjects they had had tutoring for, so it clearly wasn’t sports or ballet or the other paid-for activities you suggest. 35% or so of children who said that their fathers were in senior and professional occupations had had tutoring.
The study also surveyed parents, and 39% of university-educated fathers and 24% of vocationally educated fathers said that a child had received tutoring.

This seems to suggest that my figure of ‘around a quarter’ of children have received private or home tuition is possibly an underestimate for professional homes, but I’ll grant that this may or may not be correct and it’s not possible to be definitive. The Ireson and Rushforth report also suggests that some children in receipt of Free School Meals have also had private tuition, but that doesn’t, I think, weaken my basic argument. There is a lot of shadow education about.

Shadow Education is clearly an important influence in the education of a minority of children in England, and I think that this adds weight to my contention that it is unfair to compare any given school to the national cohort and say that the school Requires Improvement because its data says so.

Thanks again for responding and raising very valid criticism – if only more people did question things which are presented to them, schools might not be in the data-driven mess in which they find themselves...

Reply

Paul Atkin

6/8/2014 05:51:32 am

Two small scale anecdotal points.
1) When my son was in Year 6 he told me that he was the only child in the "top" two groups in his class who was NOT getting home tuition.
2) When I did an analysis of the children who took the Year 1 phonics screen in my school I found that the biggest single reason for a "fail" was being born in the summer. Some of the children concerned were also tested with a similar "non-word" reading assessment and came out at or above age related expectations; which underlines the point about the dangers of reifying one single set of results.

Reply

Slow on the uptake...

29/10/2014 02:49:12 pm

I've just arrived at this post from one of your others, and it seems as if what you're saying has huge implications beyond its role as evidence of shoddy data. Are you basically saying that it's possible a great many schools are not doing enough to give the children in them a satisfactory education, and that a shadow private sector and parental support is papering over the cracks in lots of cases?

Reply

Jack Marwood

29/10/2014 03:21:56 pm

I wouldn't want to imply any broad criticism of schools, which are the icing on the cake of factors centred on pupils, which might include parents who opt to pay for shadow education for their child. So, no, I don't think schools are not doing enough, but that many schools are being unfairly compared to other schools working in completely different sets of circumstances.

Reply

Slow on the uptake

30/10/2014 10:43:14 am

I'm sorry, my comment was badly phrased: I don't blame the schools. Logically, though, if some schools are *only* achieving good results because of shadow education, as you've suggested, then those particular schools, whether they know it or not, are not doing enough on their own to get the results that they have got, because if the results are partly dependent upon tutoring, the counterfactual situation would be that without the tutoring, the school's results would be lower.

It's only since my own children have been at school that I've realized the scale of the disadvantage children face if they are not heavily supported at home (formally or informally) as well as taught at school. I don't feel that all parents are aware of this. It's one of those great unsayables of this country's class system; I wonder what would happen if it were actually generally admitted.

Although the unfairness of the comparisons between schools is obviously very concerning, the result of this unfairness is worse, in that no-one, it seems, can see which schools are *really* effective.

What does it take for schools to be able to get good results, in the real world with fallibly human teachers, without home tutoring to lift those results? How many schools are provably doing this? If there are any, what's their secret, and is it replicable...? (Replicable in this case being possibly a synonym for affordable.) If there aren't any, or what's being done isn't replicable/affordable, should we instead, as a society, recalibrate what we consider to be the satisfactory level of *school*-based education?

I also wonder how English home tutoring levels compare with that in other countries. In Shanghai, from googling I get the impression it might be commonplace.... eg http://www.leadershiptutors.com/ So to what extent is out system being compared to systems with higher levels of tutoring than our own? It's potentially the school comparison issue write large.

Apologies for banging on about this - I'm just trying to get my head around it!

Slow on the uptake

30/10/2014 11:27:48 am

NB I grew up being told tutoring was Bad. Basically unsporting. It seems attitudes have changed among some people - but perhaps not all?

Reply

Slow on the uptake

30/10/2014 01:26:25 pm

NB #2 I just looked on Mumsnet for discussions of home tutoring. Ouch.

Reply

Your comment will be posted after it is approved.

Leave a Reply.

Author

Me?
I work in primary education and have done for ten years. I also have children
in primary school. I love teaching, but I think that school is a thin layer of icing on top of a very big cake, and that the misunderstanding of test scores is killing the love of teaching and learning.