Only those stuck under rocks, or beavering away in dark classrooms with no time to see the way the wind is blowing, can fail to have noticed that Ofsted has been under the cosh of late. As the inspectorate tries to move on from a period in which it has had few friends and many vocal critics, it has taken a long, hard look at what it does and how it does it. To its credit, it has made some substantial changes. From numerous rewritings of the School Inspection Handbook (SIH), to a complete reorganisation of the Inspector pool and a newly integrated inspection framework, Ofsted 2015 looks very different to the 2010 equivalent. Whilst the fear of inspection still remains in many schools, and unease about the organisation has been widespread, Ofsted has made great efforts to open up and face criticism head on, and for this they should be congratulated. Whilst further evolution is clearly required, not least to head off change being imposed centrally (one only has to read Dominic Cummings’ splenetic criticisms to get some sense of the mood within Whitehall), of late Ofsted has been more Frazier than Voldemort. In that vein, I’ve been having occasional discussions with Sean Harford and some of his statisticians following a meeting last year, and we found the time to have another gathering at Ofsted HQ on Friday 4th September. I corralled together a group of data-types including Jamie Pembroke, Yorkshire Steve and Peter Atherton, and Sean Harford and some Nameless Ones sat down with us to share developments within Ofsted and to discuss the Inspectorate’s use of numbers. I will start by saying that, from all that Sean Harford said to us, he has once again shown that he is determined to deal with any instances of Inspectors not following the SIH, and he is open to anyone contacting him to let him know of any concerns they might have should this be the case. In short, if you have a concern, ask Sean. Since the reorganisation of the Inspector pool late last year, all inspectors have had training on the use of numbers in schools. I’m using 'numbers' here, because Sean was very clear that the Inspectorate is encouraged to think about ‘information’, rather than ‘data’, when reviewing the current position of a school prior to and during inspection. Yes, Inspectors will look at outcomes for students and will look at historical information which is available prior to inspection, and they will also look at current information which schools can provide when Inspectors come to call. As Sean is fond of saying, this should be a signpost, not a direction. As many will know, the SIH is very clear that Inspectors will not expect performance and pupil-tracking information to be presented in a particular format:

Whilst the SIH notes that this directive was published during the 2014-5 school year, it was not in the SIH. Now it is, and it should be clear to both schools and Inspectors that schools can present in-school information in any format they like. Inspectors have had extensive training on the new framework, and Sean was at pains to underline the increased focus on ensuring that inspectors understand what information might actually be telling them.

At this point, I’m going to be Primary-specific before getting to Secondary-specific issues. Primaries have borne the brunt of the turmoil caused by the removal of levels without any statutory replacement, and the Kafka-esque nightmare of dealing with two completely different curricula for – on the one hand - years 1, 3, 4 and 5 and – on the other - years 2 and 6 on the other. For all the criticism of levels-as-assessment-criteria – and goodness know there were lots – simply removing them was an extraordinary decision which has had and will continue to have long term consequences for the sector. In many ways, the crisis this created – and it is an entirely manufactured crisis, whether by design or accident – has overwhelmed the more salient debates about whether learning is visible and whether we can track and compare progress using some sense of linearity at all. But I digress.

Within all the chaos caused by removal of a bad assessment system and the imposition of a new curriculum, primaries have, unsurprisingly been extremely wary of the ‘freedoms’ these changes have imposed, and schools have ended up, for the most part, simply replicating or retaining levels in the absence of any clarity from those charged with central education policy.

Whilst an Assessment commission has been set up, has had time to consult, and has come up with what looks like a fair summary of the mess the Primary sector finds itself, the unpublished report is now overdue and schools are once again having to hope that Inspectors like what they see when they come to call.

As Sean Harford emphasised, Inspectors should be looking at how schools have developed their curriculum and assessment systems in response to the removal of levels and the new curriculum. It remains to be seen how Inspectors on the ground react to the multi-faceted landscape which we now have. Sean Harford is positive that Inspectors will be open to a multitude of approaches, and we will have to hope that this is the case.

At Secondary level, the removal of levels has also caused many sleepless nights, although not quite to the extent it has in Primary. It has, however, coupled with the move to Progress 8, had a huge impact on curriculum design. Progress 8 has also had a major impact on the subjects schools offer to their pupils. I’m no secondary expert, so I’ll leave this to others, but we did discuss the positives, and some teething troubles, of Progress 8 and Assessment 8.

As many readers will know, one of my long standing issues with Inspection is with the use of test score data, and the way this impacts inspection. I’ve written, often intemperately, about the misuse of statistics as the government seeks to compare schools both with each other and with a national picture. The main criticism I started with was the use of tests of statistical significance with RAISEonline, and the effect a series of green and blue boxes has on those whose understanding of numbers may not be too secure.

The last time I met Sean, I was encouraged to hear that some of the concerns I had raised were understood by statisticians with whom he works. It’s also been good to see some public debate about using significant tests to compare non-random samples. Whilst there is always the quiet warning that you should be careful what you wish for (which was amplified via a stadium sized PA system for those of us who thought the use of levels for assessment was a bad idea – look what happened there…), I’m pleased that the discussion about Green/Blue is happening, and it’s clear that the debate will continue within Ofsted and the DfE.

One encouraging development is the new Inspection Dashboard, unveiled recently within RAISEonline. Whilst the detail needs scrutiny, it presents information in a much more clear format than previous summary reports. There are issues, however. The use of the terms ‘significantly above’ and ‘significantly below’ are still Not Even Wrong, and there is still work to be done to help those who struggle with the whole concept of a Null Hypothesis Significance Test to understand what they actually mean. But that discussion will have to happen on another day.

One serious criticism of the previous Inspection Framework was that the 'Attainment of Pupils' grade drove both the 'Quality of Teaching' and the 'Overall Effectiveness' grades of a school. This lead in many cases to teaching being lauded or damned based on the pupil outputs of often very small cohorts of children, especially in Primary. It remains to be seen how closely the new ‘Quality of Teaching, Learning and Assessment’ links to the new ‘Outcomes for new pupils’ grade.

Whilst Sean was at pains to insist that both the old ‘Achievement of Pupils’ and the new ‘Outcomes for new pupils’ aren’t directly driven by the amount of green or blue in a school’s RAISEonline report, the Inspectorate – along with many people in education - still clearly hold teachers directly accountable for the outcomes of the pupils they teach. This is a point which simply isn’t based on current research, for reasons which I discussed in my session at ResearchED the day after the meeting. Whilst I disagree with Sean on this, his is certainly the more mainstream view as things stand, and I await the first set of Inspection reports under the new framework with interest.

All I will say is that we have removed the grading of lessons because it became clear that observations of teaching and teachers were unreliable and not grounded in research. If research says that pupils outcomes are based on a multitude of factors, of which teaching input is a relatively small and fairly consistent factor, and that we really aren’t sure what great teaching looks like, then judging teachers and teaching is unreliable and not grounded in research. The following extracts from the Sutton Trust’s ‘What is Great Teaching?’ make this clear:

But, as I say, much of this is for another day. In the meantime, we should see Inspectors coming into schools having looked at some historical information beforehand. They shouldn’t be expecting in-school data in any particular format, and they should be open to a school’s internal analysis of its current position. That’s what Sean Harford expects, and that’s what we all should see.

Thanks, as always to the team at Ofsted and the DfE who made the time to meet with us, and to all those who provided input prior to the meeting, not least Becky Allen of FFT Datalab who has been a fantastic sounding board for my thoughts on Inspection, data and information about schools.

Thanks for this useful oversight. It remains to be seen whether inspectors will still rely on published test results too much. As you say, teaching is an inexact science and relying excessively on test results to decide teacher quality is misleading (and unfair for those teaching bottom sets).

Reply

Jack Marwood

9/9/2015 08:21:12 pm

Thanks, Janet. Yep, the proof will be in the pudding, but I'm hopeful that we've turned a corner. I look forward to the next few months to see how things are in the ground as the new SIH rolls out.

Reply

Just another teacher

9/9/2015 12:35:21 pm

Seems like a bit of lip service and whitewash to me. As expected.

The sig above and below stuff. Why do they not understand this?

Similarly with holding teachers responsible for outcomes, and the persistence with RUBBISHOnline.

I still have pretty much zero confidence that any inspector coming into my school will have the first clue about what the data actually means, and what can be inferred from it.

Not even wrong. Makes me laugh every time I read it. And still they carry on.

Reply

Jack Marwood

9/9/2015 08:23:39 pm

As you will know, I'm deeply sceptical about the general understanding of what RAISE actually tells you, but I do think that there has been some movement of late. The proof will be in the eating, of course, and we'll have to keep a close eye on what happens next...

Reply

Peter nye

14/9/2015 07:19:54 pm

Jack, thank you for this summary of your meeting. I suspect the outcomes judgement will not be directly driven by the amount of green or blue in a school’s RAISEonline report. It will probably be driven by number of red dots in the new inspection dashboard. I am puzzled as to why Ofsted have invested in this new data tool given that the format will become obsolete after the 2015 version is produced within the next few weeks.

Thank you for the interesting read. It is sad to note that you are facing many of the same foundational issues as we are in the US. The focus has to shift to meeting the needs of the student. I meet you at my door at the first of the school year and work with you and move you as far as I possibly can before you leave me at the end of the school year. A much harder job to do than many realize, especially when one thinks about all of the external factors that educators today have in play. And with the diversity of students that are in our classroom, making content accessible to all is not for the faint of heart!

And, as commented above, many who are in charge of "inspecting" may have not really been in a classroom on a daily basis to see firsthand the challenges and demands that the teacher faces!

As educators, let us continue to encourage each other for the sake of our students!

Reply

Jack Marwood

19/9/2015 01:16:38 pm

Thanks for this Tammy,

And yes, it appears that both the USA and England face many of the same issues, with similar foundational issues. Things seem to be tentatively heading in a better direction, but we wait to see whether we're seeing real change or not...

Reply

Your comment will be posted after it is approved.

Leave a Reply.

Author

Me?
I work in primary education and have done for ten years. I also have children
in primary school. I love teaching, but I think that school is a thin layer of icing on top of a very big cake, and that the misunderstanding of test scores is killing the love of teaching and learning.