Tuesday, 24 May 2016

Who wants the TEF?

I'll say this
for the White Paper on Higher Education "Success
as a Knowledge Economy": it's not as bad as the Green Paper that
preceded it. The Green Paper had me abandoning my Christmas shopping for
furious tirades against the errors and illogicality that were scattered among
the exhausted clichés and management speak (see here, here,
here,
here
and here).
So appalled was I at the shoddy standards evident in the Green Paper that I
actually went through all the sources quoted in the first section of the White
Paper to contact the authors to ask if they were happy with how their work had
been reported. I'm pleased to say that out of 12 responses I got, ten were
entirely satisfied, and one had just a minor quibble. But what about the
twelfth, you ask. What indeed?

When justifying
the need for a Teaching Excellence Framework (TEF) last November, Jo Johnson
used some extremely
dodgy statistical analysis of the National Student Survey to support his
case that teaching in some quarters was 'lamentable'. I was pleased to see that
this reference was expunged from the White Paper. But that left a motheaten
hole in the fabric of the argument: if students aren't dissatisfied, then do we
really need a TEF? One could imagine the
civil servants rushing around desperate to find a suitably negative statistic.
And so they did, citing the 2015 HEPI-HEA Student Academic
Experience Survey as showing that "Many
students are dissatisfied with the provision they receive, with over 60% of
students feeling that all or some elements of their course are worse than
expected and a third of these attributing this to concerns with teaching
quality." (p 8, para 5). The same report is subsequently cited as
showing that: ".. applicants are
currently poorly-informed about the content and teaching structure of courses,
as well as the job prospects they can expect. This can lead to regret: the
recent Higher Education Academy (HEA)–Higher Education Policy Institute (HEPI)
Student Academic Experience Survey found that over one third of undergraduates
in England believe their course represents very poor or poor value for money."
The trouble is, both of these quotes again use spin and dodgy statistics.

Let's take the 60% dissatisfaction statistic first. The executive
summary of the report stated; "Most
students are satisfied with their course, with 87% saying that they are very or
fairly satisfied, and only 12% feeling that their course is worse than they
expected. However, for those students who feel that their course is worse than
expected, or worse in some ways and better than others, the number one reason
is not the number of contact hours, the size of classes or any problems with
feedback but the lack of effort they themselves put in." So how do we
get to 60% dissatisfied? This number is arrived at from the finding that 12% said that their experience had been
worse than expected, 49% said that it had been better in some ways and worse in
others. So it is literally true that there is dissatisfaction with 'some or all
elements', but the presentation of the data is clearly biased to accentuate the
negative. One is reminded of Hugh in 'The Thick of It' saying "I did not
knowingly not tell the truth".

But it gets worse: As pointed out on the Wonkhe
blog, among 'key facts' in a briefing note accompanying the White Paper,
the claim was reworded to say“over 60% of students said they
feel their course is worse than expected”.
The author of the
blogpost referred to this as substantial misrepresentation of the survey. This
is serious because it appears that in order to make a political point, the
government is spreading falsehoods that could cause reputational damage to
Universities.

Moving on to perceptions of 'value for money', there are
two reasons for giving this low ratings
- you are paying a reasonable amount for something of poor quality, or
you are paying an unreasonable amount for something of good quality. Alex
Buckley, one of the authors of the report replied to my query to say that while
the numeric data were presented accurately, crucial context was omitted. This made
it crystal clear it was the money side of the equation that concerned students.
He wrote:

"Figure 11 on page 17 of the 2015 HEPI-HEA
survey report shows that students from England (paying £9k) and students from
Scotland studying in Scotland (paying no fees) have very different perceptions
of value for money. And Figure 12 shows that the perceptions of value for money
of students from England plummeted at the time of the increase in fees. Half of
2nd year students from England in 2013 thought they were getting good or very
good value for money. In 2014, when 2nd years were paying £9k, that figure was
a third. (Other global perceptions of quality - satisfaction etc. - did not
change). There is something troubling about the Government citing students' perceptions
of value for money as a problem for the sector, when they appear to be
substantially determined by Government policy, i.e. the level of fees. The
survey suggests that an easy way to improve students' perceptions of the value
for money of their degree would be to reduce the level of fees - presumably not
the message that the Government is trying to get across."

So do students
want the TEF? All the indicators say no. Chris Havergal wrote yesterday
in the Times Higher about a report by David Greatbatch and Jane Holland in
which students in focus groups gave decidedly lukewarm responses to questions
about the usefulness of TEF. Insofar as anyone wants information about teaching
quality, they want it at the level of courses rather than institutions, but, as
an ONS
interim review pointed out, the data is mostly too sparse to reliably
differentiate among institutions at the subject level. Meanwhile, the NUS has
recommended boycotting
the National Student Survey, which forms a key part of the metrics to be
used by TEF.

This is all rather
rum, given that the government claims its reforms will put
students at the heart of higher education. It seems that they have
underestimated the intelligence of students, who can see through the weasel
words and recognise that the main outcome of all the reforms will be further
increases in fees.

It's widely
anticipated that fees will rise because of the market competition that the
White Paper lauds as a positive stimulus to the sector, and it was clear in the
Green Paper that one goal of the reforms was to tie the TEF to a regulatory mechanism
that would allow higher fees to be set by those with good TEF scores. Perhaps
less widely appreciated is that the plan is for the new Office for Students to
be funded largely by subscriptions paid by Higher Education Providers. They
will have to find the money somewhere, and the obvious way to raise the cash
will be by raising fees. So students will be in the heart of the reforms in the
sense that having already endured dramatic rises in fees and loss of the
maintenance grant, they will now also be picking up the bill for a new
regulatory apparatus whose main function is to satisfy a need for information
that they do not want.