Beware of statisticians using sleight of hand – especially when it comes to religion

A new study claims faith schools hold countries back. The detail reveals a very different picture

Academics are entitled to our opinions. But we are not entitled to use sleight of hand to give our views a spurious air of academic authority. Unfortunately, in my own field of statistics, this happens all the time – not least when it comes to religion.

Take, for instance, a recent widely-reported study by Professor Gijsbert Stoet, Professor of Psychology at Leeds Beckett University. Prof Stoet argued in the respected journal Intelligence that the government should stop funding faith schools, because, he says, they contribute to lower test scores in maths and science.

In the accompanying press release, he advocates “a stronger secular approach” to education, and calls for the government to reduce funding for faith schools.

This is a surprising conclusion, not least because most research is very positive about the effect religious schools (and Catholic education in particular) has both on educational outcomes as well as other measures of social welfare. For some reason, this research gets very little attention in Prof Stoet’s paper.

So how did he reach such unusual conclusions? As it turns out, he and his team collected data on test scores in maths and science in 76 countries. They compared the scores to levels of religious adherence in each country.

He found, first, that test scores are lower in countries with high levels of adherence; and second – based on a much smaller sample – that test scores are lower in countries which allocate more time to religion in schools.

Now, every first-year statistics undergraduate knows that that correlation does not necessarily imply causation. Correlation may be due to other factors which make both variables change together.

This study is especially vulnerable to the problem, since the countries vary widely, from Canada to Saudi Arabia, from the UK to Ghana. The differences in political structure, and the historical development of education policy, introduce a host of possible factors which make any conclusions about causation highly dubious.

The researchers acknowledge the problem, but their response is hardly reassuring. They introduce just one control (the UN Human Development Index) – which most researchers would feel is inadequate.

The other problem is that Prof Stoet uses cross-sectional statistical techniques: that is, the researchers estimate correlations across many countries at single points of time. This is far less reliable than panel data, in which you combine data to compare country-specific changes over time.

To their credit, the researchers report all their data in the paper, so I have been able to do precisely this analysis (termed a fixed effects panel data estimation). What I found was, if anything, the opposite of Prof Stoet’s conclusions: both science and maths test scores are estimated to decrease when a country becomes less religious. But this effect is not statistically significant: we cannot rule out the possibility that there is no effect at all.

Given the much smaller sample of data, the criticisms above apply even more strongly to the attempt to correlate test scores with time allocated to religion. Indeed, in my judgement it is unreasonable to draw any inference from this result.

So there are strong grounds for suspecting that the key results in the paper do not hold up to proper scrutiny.

No academic paper is perfect, and perhaps others would critique my critique. But the real problem here is with the press release accompanying the research. Prof Stoet claims that his research suggests that test scores would be improved by a more secular approach to education, that religion should be taught as a subject rather than something to be practised, and that the British government should reduce expenditure on faith schools. In passing he makes a further sideswipe at religion: “It is already known that faith schooling leads to segregation of communities.”

Let’s be clear: these are the personal opinions of Professor Stoet and have nothing at all to do with his research paper.

His paper compares the levels of religious adherence in different countries. This is separate from how secular the education system is. For example, the UK is reported as being less religious than the US, and has higher reported test scores – exactly the correlation found by Professor Stoet. But the UK, unlike the US, gives very significant state support to faith-based schools. If you had to make a policy recommendation based on these countries, it would be to increase state funding for faith schools! Of course the research does not consider state funding of faith schools at all, so there are no grounds for referring to this in the press release.

I feel somewhat guilty resorting to such an over-used phrase but there really is no alternative: the press release is fake news. An academic publishes a piece of research on one thing, and then issues a press release making policy recommendations on something completely different. It happens more often than you might think. Readers beware.