Today I’m blogging without data, or even much evidence. We have had a few interesting commentaries from education bloggers on longer school days (here and here), but the twitter debate fell quickly into criticisms about impacts on family life, which need to be challenged. I believe the provision of non-compulsory extended schools is an essential reform to support families – as families by definition will include women who want, or need to work. And it is still women who are overwhelmingly expected to meet their children at the gates of schools well before the working day is over.

Schools are public institutions if they take tax-payers money (yes, even church schools and academies). Their buildings are ideal places for children to spend time, whether that time is spent learning, playing, doing extra-curricular activities, eating, and yes Tom… even just baby-sitting. Instead of the patchwork of finding childcare, after-school clubs, ad-hoc play dates and other activities to fill the rest of the day, let’s start using these expensive buildings to serve the modern needs of families.

I would like the government to mandate that publicly funded schools must open their buildings from at least 8am to 6pm, without restrictions on number of places available in their extended day. Schools can make reasonable charges to either children or providers for extended day activities, and there should be no requirement on either pupils or teachers to attend.

My reason why is the reality of combining children with work and a career.

Before I had children I had been prepared for the fact that the pre-school years would screw up my career a bit, but assumed that once children went to school, childcare would become more straightforward. I couldn’t have been more wrong.

Until my child is five I can (pay of course) to send her to a nursery where I can pick and choose for her day to start at 8am, 9am, 12 noon or 12:45pm and for her day to end at 12 noon, 12:45pm, 3:45pm or 6pm. Whilst there she takes music lessons with a specialist, learns to swim, spends many hours playing inside and outside, is introduced to letters and numbers, and so on. She loves it and I love to work while she is there.

I could keep her in this institution after the age of five because the nursery happens to be attached to a private school that has been forced to extend its working day to meet the desire and/or the needs of my generation of women who mostly want to work. But unfortunately my ideological beliefs and desire for her to be educated in her local community make this impossible.

I don’t live in a part of the world where state schools generally open for 10 hours a day or where there is a pool of affordable labour looking for work from 3pm-6pm each day. So, I am resigned to spending many afternoons each week standing at the school gate, driving my children to extra-curricular clubs, sitting reading my twitter feed while the club is running, driving them home and preparing their tea while they watch TV. Yes, it is great for families to spend quality time together, but this doesn’t feel like good quality time to me. If it was so great, I suspect it would be shared more evenly across the sexes. I believe it would be better for my child and for my career if schools were required to house a wide variety of fee-charging after-school clubs (youth clubs, sports, individual and collective music activities, ballet and gym, homework support, maths catch-up, one-to-one tutoring, arts and crafts, drama, etc…) for families who wanted to use this additional care. We could, as taxpayers and citizens, choose to subsidise this provision for some groups of children or even require attendance for others, but this would be controversial and step far beyond straightforward support for working parents.

I enjoyed Paul Kirby’s blog this week on lengthening the school day, but I don’t think he successfully made the case for longer mandatory opening hours for all children. Let’s look at Kirby’s arguments and some of the problems in the evidence, even where he has cited it. He says:

“The evidence on the impact of classroom hours on attainment is strong”, citing Kipp schools in the US. But unless we are to invite Kipp to come and run our schools, the correct evidence base is really rather weak

“It wouldn’t cost any money because we can redeploy Teaching Assistants to run additional activities without detriment to children they currently support”. This is not a view of the evidence that Peter Blatchford, the academic who is most quoted in this area, would subscribe to.

“Teachers currently work shorter hours than other workers so should be required to work a normal day”. This is a particularly ill informed prejudice. Teachers would work shorter hours, if children needed no pastoral care, lessons needed no planning and books needed no marking (see here). I worked far longer hours as a teacher than I did as an investment banker.

(And even if we had the money, we cannot substantially increase the size of the teacher workforce without comprising quality of new recruits.)

Postscript: Since I wrote this blogpost the government has published an interesting research report.

Last weekend, Laura McInerney and I hosted a rather experimental TouchPaper problems party. Her blog here tells you what happened on the day and a few party goers have started writing up their own thoughts on the day (here, here and here). Here are five observations from the perspective of a rather-out-of-touch-with-the-classroom academic:

1. I have never done so little preparation for event I have hosted. Laura was in the wrong part of the world; I was busy shelving research projects for my impending maternity leave. So, we could either host a day with close to zero preparation or not at all. I asked the Director of IOE, Chris Husbands, to give us a room, drinks and wifi while Laura wrote a set of blog posts to light the TouchPaper for each group. We outsourced all intellectual preparation to our fantastic group leaders in return for a couple of drinks and dinner. Without their help we would have been sunk. Thank you.

2. Smart, motivated adults don’t need as much event structure as young, easily-distracted children. Giving the groups complete discretion as to how they spent four hours produced an amazing diversity of activities. Some of the approaches reflected the size of the group (consensus is that 3 is too small and 8 is too large). One group spent an hour largely silent writing and reading pink post-it notes:

And while a couple of party-goers commented that they would have liked more structure in the day, more appeared to value the flexibility afforded to groups to do exactly as they pleased.

3. To answer the TouchPaper problems, group work needs to be complemented by intense individual endeavour. Group work enables sophisticated discussions about the interpretation of the TouchPaper problem and how it can be broken down into necessary sub-questions. It also works well for research design. But somewhere between conceptualisation and research design somebody needs to review relevant literature.

4. Research summaries that instruct teachers what to do in the classroom are a poor substitute for intense engagement in a research question. Educational research naturally wants to get teachers to use their studies to inform classroom practice (e.g. see current BERA and EEF efforts). Teachers are busy people and it is easy to infer that ‘digested’ research summaries and toolkits are the only practical approaches to achieving this. But these summaries are necessarily reductionist (e.g. you should give your pupils’ feedback on their work) and I wonder whether they alienate teachers who appreciate the complexity of deciding best practice across a variety of settings. Teachers live an incredibly hectic and stressful life during term-time, yet more than one party-goer said that engaging in the TouchPaper problems on a Saturday was exactly what they needed to re-fuel and reaffirm their love of teaching.

5. My dream school would employ these 40-odd party-goers, from the PGCE student through to the headteachers and ex-teaching drifters like this and this. I was blown away by some of the conversations I eavesdropped on (and occasionally baffled by our complexity group):

Admittedly we made no effort to recruit a representative sample of teachers (see point 1). Our party was made up of a self-selected group of the most engaged of the edu-twitterati and bloggers. Any future TouchPaper problems party might have to be organised differently to reflect diversity of engagement in educational policy-making and research amongst more typical teachers.

Back in September, the ResearchEd conference hosted a vast range of speakers suggesting how research might be more effectively used in education. My own contribution was a presentation of 7 problems which, if answered, would help teachers understand important things about their job. (See the full talk here)

The list was called the “TouchPaper Problems” – a reference to the blue paper with which one lights fireworks. I created each problem because I felt it would give information useful in classrooms. They are difficult questions though. Each one will require several layers of theory-testing and consideration before they can be considered ‘solved’. This sort of public problem-solving approach in the past motivated mathematicians and engineers to solve some of the most fundamental problems in their sector. My theory is simple: we should do the same in education.

When Joe Kirby and I presented data on whether Teach First is working at ResearchED2013 earlier this month, the best audience question came from Arthur Baker who wanted to know whether the improvements in average school attainment following Teach First participation meant that the most disadvantaged students were benefiting, or not.

This question struck a particular chord with me because I had recently read Machin and Silva’s chapter in The Tail, an excellent collection of essays about underachieving pupils in English schools. They showed that, although the early sponsored academies had indeed improved average GCSE outcomes (see Machin and Vernoit for evidence on this), there was no improvement in GCSE attainment for those starting secondary school at the bottom of the Key Stage two distribution. So, a policy designed to improve life chances for those at risk of being left behind in under-performing schools seemingly did not impact on those within the schools who were most educationally disadvantaged.

To return to Arthur’s question: what sort of pupils benefit most from a school’s participation in Teach First? We didn’t answer this in my original paper with Jay Allnutt, but I’ve re-opened up our datasets to have a look. In the paper we compare the performance of a particular pupil across their core subjects of GCSE English, maths and science to see how well the pupil performs in the department(s) who chose to take Teach First participants, compared to those who did not. We showed that departments taking on Teach First participants were more ineffective than others within the same school before joining Teach First (scoring about 10% of a GCSE grade worse), but that by years two and three of participation they were outperforming their neighbouring departments who did not join Teach First (by about 15% of a GCSE grade).

To mirror the analysis of Machin and Silva on academies, I looked at how lower and higher ability children responded to the presence of Teach First participants in the department. The chart below shows that the overall pattern of impact of Teach First is similar for children across the ability distribution. Those scoring in either the top 10% or bottom 10% at Key Stage two do not appear to be particularly suffering in their attainment within the department who will soon join Teach First. (We can only speculate as to why – e.g. very high achieving pupils have strategies and other support systems for overcoming any teacher quality issues; very low prior attainment pupils have specific circumstances that dominate GCSE grades across all subjects.) In years two and three after joining Teach First, it appears that improvements in attainment are shared across the ability spectrum. It is hard to make claims as to why this should be so, not least because Teach First themselves have not systematically collected information on what year groups and which ability sets within year groups their participants are teaching.

Finally, to answer the question that Arthur Baker was most interested in, this chart shows that the responsiveness of free school meals pupils to the placement of Teach First participants in departments is very similar to the response of other pupils. If anything, the free school meals pupils appear to benefit a little more (and this difference is statistically significant in year two).

Others have blogged about researched2013 and how great it was. I want researched2014 to be just like researched2013 because it was perfect. But just in case Tom Bennett and Helene decide to tamper with the current model, here are seven (it’s always 3 or 7) minor modifications I’d vote for:

Many participants were tweeters, so wifi is pretty important. But if we get wifi next year our phone batteries will run down, so we’ll need a phone charging station too.

Let’s have a consistent hashtag next time. How about #resed2014 compromise, or am I just encouraging further hashtag proliferation? (And what about a screen somewhere showing the tweets as they appear?)

Participant lists are really useful for people who like stalking at or after the conference

If we’d had twitter avatars and handles on our name badges I would have been able to spot Andrew Old!

It was fine having no lunch break. Lunch breaks just produce huge queues. How about charging everyone for a lunch bag in the ticket price that they can pick up and eat anytime?

Dulwich College worked pretty well but it was so hard to get to! Some of us visited a Dulwich village pub afterwards. I couldn’t stay long, but met even more new people and learnt about Croydon schools and English GCSE controlled assessment! Can we have a venue nearer a nice pub please?

Laura McInerney‘s (say “mac-in-errr-knee”) session on #touchpaper problems is the one I’m still thinking about because it was participatory and left us something very specific to think about. I thought about running a more interactive session myself, but chickened out. Conventional talks are great, but perhaps a little more audience participation (or homework!) might be nice for next year?

I hope we don’t have to wait until September 2014 to engage with researched stuff again. It is probably unreasonable to expect Tom Bennett to do everything for us (please Tom?), so I guess we each have to make some sort of tiny contribution. What is yours going to be?

Last week I published a paper I wrote with Jay Allnutt about the impact of Teach First on GCSE attainment. We received a large amount of feedback on the paper, via a seminar presentation at BERA conference, comments on a blog I wrote, twitter and email. Rather than simply present these research findings at researched2013, I showed the audience some of feedback we received on the paper from the education community to reflect on how research can move forward understanding of major education policies. This blog gives an approximate overview of my presentation at researched2013, following on from Joe Kirby’s @joe__kirbyblog of his talk.

The paper I published with Jay Allnutt showed that schools participating in Teach First improved their GCSE results. In their second year of participation, the improvements were in the order of one grade in one of a pupil’s best eight subjects (5% of a pupil s.d.) or a two percentage point gain in the % achieving five or more GCSEs at A*-C (incl. Eng and maths).

Making these sorts of claims was not straightforward because…

…Schools were not selected at random to join Teach First

I think all new education policies should be randomised in their implementation, providing it is possible. Teach First received substantial government funds so has a duty to taxpayers to demonstrate its effectiveness. This was a major failing on the part of Civil Servants who agreed to fund it (Sam Freedman @samfr spoke eloquently about this issue). Rather than recruit school’s from 20 local authorities in London in the first year of the programme, recruitment should have taken place across a randomly drawn lot of deprived schools or alternatively local authorities should have been randomised into the programme.

This did not happen, producing a very serious identification problem because Teach First schools look very different to others – they are concentrated in London, are relatively deprived, were likely to have had particularly severe teacher recruitment problems and may have had headteachers who were particularly dynamic or risk-taking.

We try to deal with non-random selection in our estimation by finding a set of schools that look identical to the Teach First schools, except that they were not participating in the early years:

And rather than simply comparing GCSE performance between our Teach First schools and a matched control group of schools, we run regressions that model changes taking place year-by-year at every school:

The first set of responses we received to our papers asked…

Are there confounding factors?

“it seems possible that managerial teams that are ‘early adopters’ [of Teach First] are a different calibre to ‘followers’ who catch up with trends after it starts to look cool … [so it might be] … higher managerial chutzpah of the early adopters doing others things nothing to do with TF that cause results to rise”

[blog comment]

For this type of criticism to be valid, the timing of the adoption of ‘higher managerial chutzpah’ must have to exactly coincide with the participation in Teach First. (Why? Well, we match on change in GCSE scores so their superior performance cannot have preceded the programme and we perform falsification tests, which demonstrate that Teach First did not have an impact in the years before the school joined the programme.) But more importantly, if our findings are entirely due to ‘higher managerial chutzpah then we would not witness positive effects of Teach First for those departments who received TeachFirsters, compared to those in the same school who did not.

The next set of feedback we received claimed that…

The effects are too largesmall

It seems most unlikely four [Teach First] teachers can raise the attainment of a school or department single handed by teaching their pupils better, not least as they simply do not teach enough pupils.

[blog comment]

Does ‘5% of a standard deviation’ represent significant and positive impact?

[@David_Cameron76 on twitter and in the audience at researched!]

This poster asked the far more important question…

The effects are too large small, given costs

Are these estimated marginal benefits worth the massive public subsidy that @TeachFirst receives? I don’t think so

[@jpjsavage on twitter]

We don’t answer this question in our research paper, but I hope we’ll be able to in a new Nuffield-funded project I’m working on which is led by Ellen Greaves at IFS. We’ve been asked to look at the relative costs and benefits of all the different teacher-training routes. There are two major impediments we face: no national database of school and departmental participation in PGCE programmes exists and SCHOOL DIRECT WAS NOT RANDOMISED (@samfr – this happened your watch did it not? Why no randomisation to stagger roll-out?).

Some commenters on our research did not believe our findings because…

Your research is clearly biased

…Clearly you have identified what you see as a benefit of this approach (albeit based on the work of someone who clearly has such a positive view of what Teach First are doing that he has gone to work for them)…

[Jonathan Savage as blog comment]

“You’ve seen improvements in these schools and have just gone looking for explanations…”

[seminar audience member]

I have no interest in Teach First in the sense that I wasn’t paid by them to do the research, I trained as a teachers via a traditional PGCE and the wider ‘narrative’ of the research I do isn’t underpinned by a theoretical view that programmes such as Teach First are likely to be successful. My co-author was trained by Teach First, worked in a secondary school whilst he conducted this analysis, and now works for Teach First. If people believe our research is biased it is hard for us to persuade them otherwise. It isn’t enough to claim that scientific and quantitative methods are less susceptible to bias, but this isn’t true (see this article on how doing maths is influenced by political beliefs). We should have been required to declare our methods – our outcomes variables and regressions we planned to run – before we conducted the research, as EEF evaluators are required to, but we didn’t.

We also had the usual comments from people who ask…

Can we ever ‘know’ anything?

You identify so many potential caveats, influencing factors, limitations to the validity of the methodology and other potential problems it is hard to take your findings seriously.

[Jonathan Savage as blog comment]

Paper ‘estimates’ benefits. Others claim them as facts.

[‏@jpjsavage on twitter]

Can you really claim direct causal links? Your paper is rightly tentative the headlines are not.

[@egwilson on twitter]

There are philosophical questions about what knowledge and causality are; I’m not the right person to blog on these things. But my problem with the commenters above is that by invoking the ‘it is not causality’ argument, they are taking a value position that there is still no evidence that Teach First works. For me, research on policies should simply aim to shift the balance of probabilities on whether they work or not. Does our research alter the balance of probabilities that Teach First works? I think it does, though we can argue about how much. Am I convinced enough of my findings that I’d be happy to suggest to a headteacher that they join the programme? I think I would (though if they were really reluctant I definitely wouldn’t push it!).

Some commenters rightly ask what the ‘thing’ is that we’ve identified, or…

How should we interpret the effect?

Could you account for the vacancy-filling nature of the programme? That is, a mathematics teacher is better than no mathematics teacher?

[comment by email]

(My response: This is just part of the Teach First effect so we don’t want to ‘account’ for it in our estimation!)

I don’t think you can compare to other ITE methods, as not part of study?

[@BeckyFrancis75 on twitter]

(My response: Correct)

What if the ’cause’ is involvement in “ITE” for a school/department & not “TF“?

[@JohnClarke1960 on twitter]

(My response: Great question! And we’ll be able to say more about this once we’ve completed our Nuffield project)

Have you proved that Teach First impacts on poor kids within these schools, as opposed to other kids?

[a question at researched2013]

(My response: Great question! Let me crunch the data and get back to you)

Now we start getting onto the really important questions, as raised by @samfr in a blogpost…

How might Teach First plausibly change the pupil experience?

And we had a few more great questions from the audience at researched2013:

Have PGCE courses seen a deterioration in applicants due to Teach First?Why is Teach First more appealing than a PGCE and what role does the salary versus student loans play in this?How can we know if Teach First or PGCE applicants have the qualitatively superior characteristics?Is a major advantage of Teach First the two year length of the programme, compared to just one year for a PGCE?

Today, Jay Allnutt and I published a new piece of analysis (PDF) showing that schools taking on Teach First participants have achieved gains in their GCSE results as a result of the programme. Our analysis tracks the performance of these schools in the first three years after they join the programme and compares them to changes in progress at a set of schools that look identical, except for their Teach First participation in that year.

We make sure this comparison set of schools have the same pupil demographic profile, the same prior levels and trends in GCSE performance, are in the same region of England and are all schools who will choose to join Teach First at some point in the future (formally this is known as a matched difference-in-differences panel estimation). Overall, school-wide gains in GCSE results are in the order of an improvement of about…