Rankings Watch

A reader of this blog, Richard Holmes, has been keeping a close track of the latest worldwide university rankings (Shanghai Jiao Tong, Newsweek and of course, the now infamous, THES rankings). He's written some insightful stuff on his blog on the methodology used by the different organizations that compile these rankings. I'd like to highlight some of his analysis here.

Firstly, a comparison of the Newsweek and THES Top 100 finds that many of Asian universities that were present in the THES are absent or experienced significant falls in the Newsweek rankings. Ditto for many of the Australian and European universities. Why this discrepancy? From Richard's post here, we find:

"The Newsweek ranking is also notable for what it leaves out. It does not include the THES peer review which accounted for 50 per cent of the ranking in 2004 and 40 per cent in 2005 and the rating by employers which contributed 10 per cent in 2005. If we compare the top 100 universities in the THES ranking with Newsweek’s top 100, some very interesting patterns emerge. Essentially, the Newsweek ranking tells us what happens if we take the THES peer review out of the equation."

"So what is going on? Basically, it looks as though the function of the THES peer and employer reviews was to allow universities from Australia, Europe, especially France and the United Kingdom, and Asia, especially China, to do much better that they would on any other possible measure or combination of measures."

A possible underlying reason on why universities from these regions (Australia, Europe and Asia) were ranked higher on the THES ranking compared to the Newsweek ranking? Richard gives one:

"Is it entirely a coincidence that the regions that are disproportionately favoured by the peer review, the UK, France, China and Australia, are precisely those where QS, the consultants who carried the survey, have offices and are precisely those regions that are active in the production of MBAs and the lucrative globalised trade in students, teachers and researchers?"

It seems that one of the linchpins of the THES ranking system is the peer review score. QS, the organization which does the compilation of the results, have been very reluctant to release detailed information in how they obtain these peer review scores. Richard, doing some digging on this own, has unearthed some of the methods used by QS to obtain their sample size for the peer review.

He refers to two websites - one from the Special Astrophysical Observatory of the Russian Academy of Science and the other from the University of Auckland, New Zealand. Both sites tell us something more about how QS solicits and entices respondents to answer the QS survey. Richard comments:

"But the most interesting thing is the bit at the top of the Russian page. The message was addressed not to any particular person. but just to "World Scientific Subscriber" . World Scientific is an online collection of scientific journals. One wonders whether QS had any way of checking who they were getting replies from. Was it the head of the Observatory or some exploited graduate student whose job was to check the e-mail? Also, did they send the survey to all World Scientitific subscribers or just to some of them or only to those in Russia or Eastern Europe?"

"So now you know what to do if you want to get on the THES panel of peer reviwers. Subscribe to World Scientific and, perhaps, a few other online subscription services or work for an institution that does. With a bit of luck you will be recognised as a real smart person and get a chance to vote your employer and your alma mater into the Top 300 or 200."

It was interesting to read the following from the University of Auckland website:

"We note that the Times Higher Education Supplement’s ranking of The University of
Auckland rose from 67th in 2004 to 52nd in 2005. But we note also that this survey establishes its rankings by appealing to university staff, even offering financial enticements to participate (see Appendix II). Staff are likely to feel it
is in their greatest interest to rank their own institution more highly than others. This means the results of the survey and any apparent change in ranking are highly questionable, and that a high ranking has no real intrinsic value in any case. We are vehemently opposed to the evaluation of the University according to the outcome of such PR competitions."

Every ranking methodology has its own shortcomings. By understanding better how these rankings are compiled, we can better understand why some universities are present or absent in these rankings, why some of them move up, why some of them move down, how some aspects of these rankings can be 'gamed' or manipulated, and so on.

I highly doubt that our local universities will drop out of the Top 200 in the 2006 THES rankings. Given that QS representatives have made several visits to Malaysia and the region to market their services, it is likely to be in their interest to 'maintain' a least a few Malaysian universities in the top 200. Hopefully, the new VCs in UM and UKM would be less prone to the disease of 'bannertidis', which we saw in the previous UM VC.

A good university would be able to stand on its own feet, whether or not that is reflected in such worldwide rankings. That is what our local universities should be working towards, instead of short term shortcuts to 'game' the system and obtain a higher ranking.

I'd like to thank Richard Holmes for pointing me out to his website and his efforts at illuminating us as to the methodology of the THES ranking system.