Having prised the mask off the Fischer Family Trust, and looked at the FFT Governor Dashboard, it's time to have a look at what has made the FFT infamous in schools throughout England - supplying schools with dubious data, primarily for use when setting targets. Playing Mystic Meg has made the FFT a household name, at least in the homes of countless teachers and senior managers who have been force-fed its dubious rubbish at taxpayers’ expense. Peddling stories of the past and tales of the future, conjuring up ‘estimates’ and foisting target culture onto an unsuspecting educational world has cost bucket loads of cash and wasted huge amounts of teachers' time and effort. It has to be said that the ‘estimates’ crunched by the FFT are so loose, so woolly and, even according to the FFS itself, so hedged with caveats the size of Belgium that they are worse than useless. They give the impression of foretelling the future much as any sideshow charlatan might. Worse still, this rubbish is paid for by you and me, at an estimated cost of £15 million over the last 13 years, and is another substantial cog in the money-extracting Data Driven Disaster machine leeching English education.Take some data and construct Castle Doom The FFS currently run an entity called FFTLive, a cartoonishly colourful website which looks like this:

According to their blurb, it is ‘a powerful online reporting system used by schools, LAs and Academy Sponsors. We process data for all schools and pupils in England and Wales and provide online reports which analyse pupil results and pupils' progress across all subjects and key stages, comparing performance to similar schools and the national average. FFTLive provides estimates of future pupil performance using FFT’s unique models which have been developed over 10 years.’ There is a fair bit of info on the FFT website about its history and magic, which I suggest you read for yourself. The highlights are briefly: 2001: FFT Founded by Mike Fischer of RM Plc and Mike Treadaway, ICT Advisor2004: DFE awards National Pupil Database contract to FFT2005: FFTLive launched 2006: RMFFT win contract to manage NPD and Performance Tables2013: FFT launch Governor Dashboard2014: Due to launch FFT Aspire in Autumn If you’d like to have a look at what FFTLive looks like for a school, you can log in using either of the following usernames 9992004X (Primary) or 9994002X (Secondary)and password ANON. (I found these here and here, by the way, in case you’re interested). There is far too much stuff available on the FFTLive website for me to go into in too much depth. Feel free to poke around yourself to see quite how much has been wrung out of the data. There are various guides which you can download (often called ‘Quick Start Guides’ accessed through ‘Help’ buttons), which are worth reading, although they don’t tell you anything at all about the methodology behind the data crunching. Here are some highlights before we get to estimates and target setting, the bit of FFT magic at which every teacher, parent and politician should take a very, very close look.Dashboards You can find the 4 page Governor Dashboard here, along with enormously data intense ‘self evaluation booklets’, which have an extraordinary 26 pages at KS1, 32 pages at KS2 and 16 pages at KS4 of stuff to plough through.Explore This has magic such as ‘opportunities and alerts indicators’ and ‘turbulence and context factors’ for which no methodology is given. I assume that we are simply supposed to accept the ‘analysis’ at face value, which I’m fairly sure we shouldn’t.Interactive reports Here you get into the murky world of ‘Reviewing Past Progress’ and ‘Supporting Target Setting (Estimates)’. ‘Reviewing Past Progress’ borrows the idea of ‘Value Added’ from economics, and, like many Data Disaster proponents, the FFT makes the highly disputed assumption that you can isolate a ‘teacher effect’ or ‘school effect’ from a ‘pupil effect’. I’ve shown before that most people in schools don’t have the knowledge, skills or understanding to question this assumption, which is entirely unjustified and makes Value Added Not Even Wrong. Suffice to say that it simply makes no sense to assume that a child’s educational development is 100% school and teacher and nothing else, much less to model an individual child's future performance based on the performance of entirely different children in the past, but that’s what happens here. It’s worth noting at this point that the FFT does two very separate things within FFTLive:

Assess the past

Predict the future

The methodology for both of these is highly suspect, and almost entirely opaque. I can make educated guesses about what RMFFT does in each area, but they haven’t made it easy to find out exactly what they do to data. Before looking at these two different but related aims of FFTLive, here are the final things to look at:Innovate New ideas for crunching data by ‘Reviewing Past Progress’ and ‘Supporting Target Setting (Estimates)’ similar to the current Interactive reports. This shows that the FFT has started to think beyond some of the issues I’ll highlight below, and that they are desperately trying to keep their teeth around the government’s DDD jugular. You can also export the data to perform more daft analysis yourself or have consultants charge you to ensure that you are a ‘Data confident school’, and the information section tells you a few things before tries to sell you training to become an Operating Data Thetan and explain that we are all actually ruled by lizards (this may not be true). So, there’s a lot here, but you don’t get to charge the government a lot of money for nothing, even if what you have produced has no value. And speaking of no value, let’s have a look at the Big Daddy of the FFT: reporting the past and guessing the futureLooking back with FFTLive All schools have to justify themselves to OFSTED when the inspectors come to call. These days, data is just about everything when being judged, and the FFT has been at the vanguard of the Data Driven Disaster. It has pushed a ‘Value Added’ model since its inception in 2001, and now all schools are expected to be solely responsible for the academic development of their pupils, as if children existed in suspended animation for the 80% of their waking hours they aren’t in school each week day. Value Added is, in essence, a (deeply flawed) measure of how much a school has added to a child’s academic development. It’s far from clear how all the FFT’s Value Added alchemy works. There is an indication of the thinking of the FFT in some of the data which is crunched in FFTLive, however. In reading at KS2, for example, some children have ‘Actual Levels’ of 5.1, 5.3, 5.7, which may be 5C, 5B and 5A; but then some children have 4.2, 4.3, 4.4, 4.7 and 4.9, which can’t correspond to 4C, 4B and 4A. Some ‘Actual Levels’ are coded in blue, which is apparently ‘lower than estimate by half a level or more’ Some are green for ‘higher than estimate by half a level or more’. So what are these estimates which these 'actual levels' are measure against? Well, in order to calculate how much ‘value’ a school had ‘added’, the FFT required an estimate of a given pupil’s future test results. This had to be a single number, which could then be compared to what a pupil actually got in the tests at Key Stage 2, 4 or 5. As far as I can guess, and based on the way in which RMFFT create estimate models for the OSDD, RAISEonline and Performance Tables, data for previous students is crunched to produce a model which has fixed coefficients to produce a linear line of best fit using regression analysis. Deep breath, non-mathematicians. It’s not so bad, really. Basically, this means this:

This is for KS2 to KS5, but it was produced by Mike Treadaway, who clearly understands how dubious the models he pushes actually are. As Mike explains, this is actually a line of best fit for data which looks more like this:

And as you can see from this, the line is a huge over simplification of what actually happens between one key stage and another, because that’s how regression works. The analysis works, just, at a group level, provided the data is identically and independently distributed, so a group of children with a mean of x at one Key Stage can be assumed to be likely to get a mean of y at another. But any statistician worth their salt would make it clear that any line of best fit is just that, and only works at the group level, and looking at y and reading off xfor a given child is clearly the work of a fool. Showing an ‘Estimated level’ of 4.6 was Not Even Wrong, because the student could get literally anything between in a wide data range and not surprise anyone with a vague idea of how grouped data works. To Mike Treadaway’s credit, he acknowledges this. But then he goes on to use it anyway to assess how well a school has ‘added value’ to children. I’ve demolished the whole ‘estimates’ nonsense before here, but that doesn’t make this any less irritating or wrong.Predicting the future, or not Most people probably know the FFT for its futurology, which we’ll look at next. The ‘Supporting Target Setting (Estimates)’ is the FFT data most teachers are presented with when setting targets with their senior management teams. Until 2009, teachers were given lists of ‘Estimated levels’ a child might get in their Key Stage 2 SATs, as used in the Value Added models above. They looked like this:

Someone at the FFT clearly realised that this was incredibly daft at an individual pupil level, since children were getting all kinds of different results and the estimates clearly made no sense for individuals. In 2009, the FFT (having used this Not Even Wrong model for eight years) amended the way they produced estimates for individual children, whilst, as I showed above, continuing to use the dubious ‘single number’ estimates to calculate 'Value Added'. Shamefully, many schools still used the single number estimate because they'd become used to it. Many may still do so. In their defence, I doubt many teachers would have understood the deep-seated problems with FFT futurology, but it clearly demonstrates the danger of bad data use in education. Currently, primary schools get FFT Estimates which look like this:

This is similar to the data currently presented to teachers in secondary which looks something like this:

The secondary estimates are presented in a slightly more palatable version of the older way of presenting estimates still used in primary (a good example of the old secondary version is on David Didau’s Learning Spy blog here), in that the percentages are given as a cumulative possibilities. 41% chance of a B+ looks a bit less appealing when you realise that the model actually means the student is most likely to get a C. Either way, this stuff shows you two important things: In Primary, the ‘Estimated Levels’ tell you nothing.In Secondary, the ‘Estimated Levels’ tell you nothing. In case it isn’t obvious why this is the case, I’ll repeat: A student could get literally anything between the lowest and highest level available and not surprise anyone with a vague idea of how grouped data works. You might get a B, then again you might not. You might get level 4, then again you might not. The estimate tells you nothing which you, as a child’s teacher or parent, couldn’t work out for yourself. There are umpteen other things wrong with this model, but here are a few to start with:

What data is used to produce the regression models for the estimates? All of it? Complete data points only? Partially complete data?

Is the data in the model, and therefore each estimate, changed each year that a child is in a key stage? If not, why not? If it is, what does it suggest?

What exactly is the methodology used to produce this magic?

Examining the Educational Tea Leaves

Once again, its hard to know where to start. So much energy has gone into this stuff - and at least £15 million over the years by my reckoning - and it doesn't tell you anything whatsoever that someone working in a school couldn't tell you given the opportunity. The 'Value Added' fiction is just that - the models are so deeply flawed as to be meaningless. The 'Estimates' are so woolly that they add little to the professional judgement of the staff on the ground.

I haven't even gone into the vagaries of FFTA, FFTB and FFTD, as you can find information about them elsewhere. I can't find any criticism of the kind I've made here about the fundamental error of using grouped data analysis to predict individual outcomes, which is why I've written about this here. I hope that this article provokes the debate as to whether using data in the way RMFFT does has any justification, and I'd like to hear your thoughts in the comments below.

Thirteen years of FFT analysis has shown that trying to summarise every diverse school community in England is witchcraft of the highest order and, at individual child level, is little better than examining patterns in tea leaves. The cost, both financially and on the diminished education of children by the limited focus on badly assessed levels, is simply not worth paying. Examining tea leaves is ultimately pointless, because they tell you nothing you couldn't have worked out for yourself. And in this case, having looked closely at the tea leaves, we need to stop throwing our money away on yet more worthless data driven nonsense and completely rethink the way we assess 'achievement' and 'progress' in English schools..

There is much in this article that I agree with - most notably the shocking number of schools who do not understand the extreme limitations of using FFT data on an individual student level and still insist upon stating, as if fact, the GCSE grade target for every individual student, in each individual subject in school.

My own school did pretty much that but, upon my appointment to the school, it was the first decision I made - to scrap this approach.

A large majority of staff thought the targets set for students were akin to dark magic and thus developed a mistrust of all data in school.

We now provide the chance table to teachers (not in the cumulative form that your show in the article but in the much more user-friendly non-cumulative format) so they have an idea of what 'broadly similarly able' students have achieved nationally previously.

These chance tables, given the immense volumes of student level data nationally, are basically statements of "How did students with these same KS2 results in Ma & En achieve in your subject at GCSE".

There are some issues with students with very quirky results (high 5 in Maths but level 2 in English) where there are not enough similar students nationally to give me any sense of reliability of the chance tables but these are relatively few and far between in our cohorts.

For some schools/departments/staff seeing how 'broadly similar students' have performed nationally can raise expectations. Students moving from a level 5 in Maths to a GCSE grade C - an odd one here or there, yes, but if most of our level 5 mathematicians end up with GCSE grade C's then I should be concerned.

And I have seen some schools/departments/staff who thought level 5 moving to a grade C was 'just about right' on average. FFT blows that low aspiration boat right out of the water.

Yes there are a myriad of other factors that come into play in determining the progress a student makes but this data can steer some staff away from low expectations - if every student in their classes, year on year, achieves GCSE grades at the lower end of the chance tables is it more likely that all the students had different 'negative' factors impacting on their progress or that the constant factor, their teaching, may be an issue?

Similarly in the positive sense - did all students have different 'positive' factors meaning they gained higher grades than students with broadly similar starting points or is it more likely that the constant, your teaching, has contributed to this.

Now we must be very careful not to make judgments, or take decisions, based on this data alone but it can be very useful indeed at pointing towards some questions to be asked.

That can work equally well when looking at large cohorts (why do our boys do so well compared to similar boys nationally but our girls do not; why did our students do so well in English compared to similar students nationally but not so in science?) - but the cohort size must be reasonably sized (the notion of looking at this type of data for 5 or 6 students of a given 'type' is just ridiculous) and the pattern repeating over a period of time (rather than chasing one off statistical 'flukes').

The key is, as you state, about using the summary data with decent sized cohorts of students and not just handfuls here and there. For this reason I worry about it's use in Primary, especially medium sized or small Primary schools where the numbers in the various cohorts can not be anywhere near large enough to be used.

At secondary level most schools are large enough for the summary data to be of use - including at subject level although we mist be aware that progression from KS2 Ma & En is likely to be less reliable for some subjects, such as those with performance elements, which FFT themselves state.

So some schools, many schools, misuse the FFT data but they, on the whole, are ignoring FFT's very clear guidance on how the data should be used - notably as a tool to be used by trained staff to aid in the target setting process. There are many examples of products that are mis-used/abused but we don't get rid of all them.

I would balk at your direction of thought that seems to suggests FFT is worthless for schools. I would argue that £15 million over 13 years is extremely good value for what it can give us.

It's important not to overstate what it can do but equally important not to understate what it can do which, I feel, your blog does in places.

The bottom line is that data never tells us the answers - it can only point us towards some questions to ask. As long as FFT is used in that way I have no problem with it.

Reply

Icing on the Cake

6/5/2014 05:28:47 am

Thansk very much for this detailed response, Steve. I'll make time to respond fully, but just a quick note to say that it's good to debate what the FFT does with data, and what schools do with the data provided by the FFT.

I'm interested in views from both sides of the debate and welcome further comments anyone has on the specific criticisms I've made above.

Reply

Icing on the Cake

8/5/2014 05:24:38 am

I've finally managed to find time to sit down and write a proper response to your comments above.

Firstly, it's good to see that you share my disquiet at the way FFT data is understood in schools, and I am glad to see that your school benefits from a critical approach to what data can tell you.

I work in Primary, and my experience in five very different schools is that FFT analysis is not understood even at a basic level, despite FFT clear guidance on using the analysis. One of the many problems is that those in school simply haven't been in a position to understand how the data is manipulated - I hope my blog is shedding some light on the magic which has been used to do this, and the questions which should be asked about it.

I am also heartened to see your criticism of the use of summary data in the small cohorts typical in Primary Schools. You won;t be surprised to hear that I echo these thoughts.

Given that Primary Schools vastly outnumber Secondaries, this suggests that - before even considering the indepth criticism I have made - FFT analysis is worse than useless for most small schools simply because the numbers are too small.

I see that you say that FFT should be used by trained staff. Whilst this might be possible in Secondary - and I'd question who is in a position to train staff in this in a balanced way - it's simply beyond the budget of Primary Schools to have a 'data manager' as far as I am aware, and therefore, by your reasoning, Primaries should not use (or be forced to use) data they don't undertand.

I plead guilty to the charge that I have somewhat overstated the case against the FFT. In my defense, this is a internet blog, and using some hyperbole to get attention people's attention is not unusual. I'll try to tone it down, but the uncritical acceptance of FFT analysis does get to those on the ground, like me, eventually. I'm finally in a position to let rip, so I have done so.

I would really like someone to find fault with my analysis of the statistical reasoning behind both the FFT's assessment of the past and predictions for the future, but they haven't been forthcoming as yet. Until then, I stand entirely behind my view about examining tea leaves, and I hope I've provoked debate amongst those who are at the sharp end of education.

Thank you for a fantastic set of articles. I have recently suffered somewhat in appraisal terms because target grades/levels. I contacted FFT to ask them what they thought about their 'estimates' being used as 'targets', on which to judge teachers. Mine was a rant. Theirs was a polite reply considering, Although sympathetic they gave me nothing that I could take back to my Head at appraisal meetings. They seem quite happy for management to continue using their data as they do. It would be interesting to know what FFT think of your articles. I assume I can refer my Head to your site as it is in the public domain. I wish I had found your site sooner (my personal research on FFT didn't get very far). I found your site from your excellent article on triple marking. Thanks I am going to read your whole site in the next few weeks

Reply

Jack Marwood

10/5/2015 03:40:46 pm

Thanks for this - always good to get feedback. And I'm glad you found this, as one of the reasons I wrote it was because I couldn't find out much about the FFT anywhere else and I wanted to help teachers in my position.
As far as I can tell, the FFT do go to some length to make it clear that their estimates are just that, and not targets, but the underlying assumptions mean that schools can and do simply use them as expectations rather than possibilities based on prior results.
Feel free to quote my writing - it's here in the public domain for exactly that reason. And feel free to comment and criticise too - I'm more than happy to explore any holes in my reasoning and research - the more discussion about numbers in education, the better as far a I'm concerned...

Leave a Reply.

Author

Me?
I work in primary education and have done for ten years. I also have children
in primary school. I love teaching, but I think that school is a thin layer of icing on top of a very big cake, and that the misunderstanding of test scores is killing the love of teaching and learning.