Including five different rankings into a single “Rankometer” offers an opportunity to make some comparisons between the different ranking systems.
We use the data used in Rankometer to answer three questions:
- Are all rankings the same?
- Do certain ranking systems favour certain countries?
- Which universities do best in Rankometer compared to other ranking systems?
Are all Rankings the Same?
While there has been criticism that world university rankings basically all measure the same thing – research1 , there are some notable differences between the five rankings included in Rankometer. The following is a very brief summary, a full description of the methodology can be found on the respective ranking publisher’s website.
Reputation Surveys: QS and Times Higher Education (THE) both use reputation surveys as part of their ranking methodology, whereas the other three rankings don’t. The QS World University Rankings uses surveys to measure academic reputation (40%) and employer reputation (10%). THE uses teaching reputation (15%) and research reputation (18%) in its World University Rankings.
Institutional Data: QS, THE and the Academic Ranking of World Universities (ARWU) use data such as the number of academic staff in their rankings. While the ARWU only obtains this data from third parties, QS and THE sometimes use data directly published or submitted by the universities.
Strictly Research: The Leiden Ranking only uses data about scientific publications contained in the Scopus database. In that sense it’s the ‘purest’ research ranking. QS, THE and ARWU also use various sources of scientometric data.
Weblinks: Webometrics only uses the reputation of an institution’s website based on the number and quality of other websites that link to it. This is similar to how search engines like Google rank search results. The assumption is that a good website with high quality content says something about the quality of a university.
In the table below you will find the correlations between the different rankings (Pearson correlation R2 based on ranks). The lowest correlation between rankings is the QS World University Rankings with the Leiden Ranking (R2 = 0.27). The highest correlation is between the QS-WUR and THE-WUR (R2 = 0.66) which incidentally both use reputation surveys. Also relatively highly correlated are ARWU and Webometrics (R2 = 0.61), which is a bit surprising as their methodologies are very different.
1See: E. Hazelkorn & A. Gibson (2017) “Global science, national research, and the question of university rankings” Palgrave Communications, 3(1), 1–11 and M.M. Vernon, E.A. Balas & S. Momani (2018). “Are university rankings useful to improve research? A systematic review” PloS One, 13(3), e0193762.
To illustrate the large differences between ranking systems at a more granular university level, the following table provides an overview of the universities with the largest differences. The largest gaps are observed for the University of Buenos Aires (935 places), Moscow State University (927 places) and Humboldt University of Berlin (921 places). Note that 1001st place is the lowest placing possible within the Rankometer methodology.
|438||University of Buenos Aires, Argentina||66||201||1001||937||378|
|260||Moscow State University, Russia||74||93||174||1001||211|
|293||Humboldt University of Berlin, Germany||117||1001||80||304||226|
|408||Yeshiva University, United States||341||201||1001||96||785|
|378||National Autonomous University of Mexico||100||201||801||1001||144|
The large differences show the value of having a composite ranking like Rankometer rather than relying on a single ranking system.
Do Certain Ranking Systems Favour Certain Countries?
There have been claims of rankings bias for as long as there have been university rankings, but Rankometer provides an opportunity to test if certain rankings really do seem to favour universities from particular countries.
Using regression analysis with country dummies and assuming that Rankometer is the ‘true benchmark’ of university quality (and an independent variable) with the component rankings as dependent variables, the following countries (with 10 or more universities among Rankometer’s top-500 institutions) appear to rank higher or lower in a statistically significant way.
To interpret the results in the table, first have a look at the Rank coefficient (at the bottom), which shows that in some ranking systems universities are likely to score lower because of the inclusion of many relatively small and more narrowly focused institutions. This is very clear in the Leiden ranking (Rank coefficient of 1.4) and is least in the QS-WUR (Rank coefficient of 1.0)
Now moving to comparing the influence of countries as a factor in university rankings, it is important to note that a “-” means that universities are likely to have a higher rank in a particular ranking system than in Rankometer and a “+” means that a university is likely to have a lower rank (at 95% significance level or higher). Thus United States universities tend to score lower in QS-WUR and THE-WUR, but higher in the Leiden and Webometrics rankings and about the same in ARWU relative to their average (Rankometer) rank.
2Results to be presented in a forthcoming research paper. A link to the paper will be posted here.
It is interesting to note that most ranking systems rank their ‘own’ universities higher: this is true for ARWU (China), THE-WUR (UK) and Leiden (Netherlands). It is also notable that of 11 countries included in the table 9 have a “+”, thus ranking lower in QS-WUR on average than they do in Rankometer, with Japan being a notable exception.
Which Universities Do Best in Rankometer Compared to Other Ranking Systems?
Universities which score relatively well in all five rankings perform well in Rankometer, but poor performance in one or two rankings can place a university significantly lower in Rankometer. Because of this some “great all-rounders” actually have a higher rankometer rank than they do in any other ranking system, in some cases improving their rank by more than 30 places!
|University||Ranko||Increase||Other Highest Rank|
|University of Wollongong, Australia||166||+30||196||QS-WUR|
|University of Pavia, Italy||371||+30||401||ARWU|
|University of Vienna, Austria||126||+23||149||Webometrics|
|University of Aveiro, Portugal||449||+23||472||Leiden|
|University of Adelaide, Australia||83||+21||104||Webometrics|
|Renmin University of China, China (Mainland)||479||+21||500||Webometrics|
|Murdoch University, Australia||480||+21||501||THE-WUR|
|Macquarie University, Australia||175||+20||195||THE-WUR|
|University of Amsterdam, Netherlands||41||+14||55||Leiden|
|University of Nottingham, United Kingdom||85||+14||99||QS-WUR|
|University of Sheffield, United Kingdom||80||+13||93||QS-WUR|
|Cardiff University, United Kingdom||139||+12||151||ARWU|