More Shanghai Needed

I’m in Shanghai this week, a guest of the Center for World-Class Universities at Shanghai Jiao Tong University for their biannual conference. It’s probably the best spot on the international conference circuit to watch how governments and institutions are adapting to a world in which their performance is being measured, compared and ranked on a global scale.

The THES and QS rankings are both commercially-driven exercises. QS actually used to do rankings for THES, but the two parted ways a couple of years ago when QS’s commercialism was seen to have gotten a little out of hand. After the split, THES got a little ostentatious about wanting to come up with a “new way” of doing rankings, but in reality, the two aren’t that different: they both rely to a considerable degree on institutions submitting unverified data and on surveys of “expert” opinion. Shanghai, on the other hand, eschews surveys and unverified data, and instead relies entirely on third-party data (mostly bibliometrics).

In terms of reliability, there’s really no comparison. If you look at the correlation between the indicators used in each of the rankings, THES and QS are very weak (meaning that the final results are highly sensitive to the weightings), while the Shanghai rankings are very strong (meaning their results are more robust). What that means is that, while the Shanghai rankings are an excellent rule-of-thumb indicator of concentrations of scientific talent around the world, the QS and THES rankings in many respects are simply measuring reputation.

(I could be a bit harsher here, but since QS are known to threaten academic commentators with lawsuits, I’ll be circumspect.)

Oddly, QS and THES get a lot more attention in the Canadian press than do the Shanghai rankings. I’m not sure whether this is because of a lingering anglophilia or because we do slightly better in those rankings (McGill, improbably, ranks in the THES’s top 20). Either way, it’s a shame, because the Shanghai rankings are a much better gauge of comparative research output, and with its more catholic inclusion policy (500 institutions ranked compared to the THES’s 200), it allows more institutions to compare themselves to the best in the world – at least as far as research is concerned.

One Response to More Shanghai Needed

I welcome engaged criticism of the rankings, that’s how we ensure that we continue to improve, but I think your case against the Times Higher Education World University Rankings would be better made if:

a) You cited our results correctly. You say that McGill is “improbably” ranked in our top 20. This would indeed be improbable, because McGill is actually ranked 28th in the world by THE in 2011-12. See: http://bit.ly/thewur
b) You were more aware of our offering: this year we published a list of 400 institutions (not 200 as you state). This is not far from Shanghai’s 500, and we actually give firm ranking positions to the top 200, giving more detailed data on more institutions than Shanghai.
c) You got our name right. The magazine which publishes the rankings is known as “Times Higher Education”. We were once called “THES”, but we have been known as Times Higher Education – “THE” – since 2008.

It is correct that THE decided to end its partnership with QS in 2009, in order to produce a more robust piece of work with a new partner, Thomson Reuters. This is something we have achieved and we are delighted with the results of the new THE-Thomson Reuters partnership.