Fish oil exam results fail all the tests

You'll remember the Durham fish oil "trial" story, possibly the greatest example of scientific incompetence documented from a local authority. Initially it said - to blanket media coverage - that it was running a trial on fish oils, giving pills to 3,000 children to see if it improved GCSE performance. I pointed out, along with several academics, that the experiment was incompetently designed, and would only produce false positive results. The council responded that this was okay, as it hadn't called it a "trial".

This was untrue: it had, repeatedly, in press releases and interviews, and anyway, whatever you called it, this was still a stupid experiment. Durham's response was to edit the online version of its press release to remove the word "trial". Then I asked what it was going to do in this experiment, how it was going to measure results, and more. It refused to give me this information - for an experiment by a public body performed on thousands of children - so I used the Freedom of Information Act. It still refused. Then hundreds of you wrote to its information commissioner, using the FoI, and it refused again, accusing us of running a "vexacious campaign".

Then the GCSE results for Durham came out. They weren't too great, so I asked for the results of the "trial". Durham refused to give me this information. Then it announced, bizarrely, in a formal response to a written question: "It was never intended, and the county council never suggested, that it would use this initiative to draw conclusions about the effectiveness or otherwise of using fish oil to boost exam results."

This was, once again, untrue. Durham's own press release had clearly said it was giving out the pills "to see whether the proven benefits it has already brought children and young people in earlier trials can boost exam performances too".

The council's chief schools inspector, Dave Ford, said: "The countywide trial will continue until the pupils complete their GCSE examinations next June, and the first test of the supplement's effectiveness will be when they sit their 'mock' exams this December."

Suddenly this trial did not exist. Now, bafflingly, in defiance of its previous denials, Durham has released some results. "Detailed analysis of the outcome of the initiative," it says, "shows that pupils who took the Omega-3 supplement did better than those who did not." Hardly. Let's try to disentangle what it thinks it has done.

"Initially, just over 3,000 year 11 pupils began the study, taking the Omega-3 tablets at school and at home. By the time GCSE examinations came around, 832 pupils had 80% or greater compliance." This is appalling. 2,168 of its subjects dropped out of the trial: it must count these people in the results. It does not. This makes the rest of its claimed results even more meaningless.

"Mr Ford and his colleagues then sought to identify the same number of year 11 pupils who had not taken the supplement and match them to those who had, according to school, gender, prior attainment and social background." It originally said it was going to compare children's predicted GCSE performance (whatever on earth that means) against actual performance. It refused to say how it would analyse this, despite hundreds of requests.

Who are these 629? I thought it was 832? But more importantly, by selectively only looking at the results from the pupils who were most highly adherent to the capsules regime, it has skewed its sample. It has discovered that school performance is better in children who are more highly adherent to a school regime involving pills, and who are, in all probability, also more adherent to everything at school, harder-working, better-performing, and so on.

This result has nothing to do with the pills. It is laughably incompetent science. Fish oil pills are the biggest selling food supplement product in the UK, and still no one has ever tested them properly. Durham council had the children, and the pills, necessary to perform a decent piece of research. The only thing it was missing was the rigour.

The real question now is this: why has Dave Ford performed an incompetent experiment on thousands of children? And, more importantly, why has Durham council let him?

Sign up for the Guardian Today

Our editors' picks for the day's top news and commentary delivered to your inbox each morning.