The Citation Culture

Link: http://www.nytimes.com/2010/11/15/education/15iht-educLede15.html?scp=1&sq=times%20higher%20education%20ranking&st=cse

  The problematic nature of one-dimensional rankings of university performance has been underlined in a recent story in the New York Times. The story focuses on the University of Alexandria that was ranked as nr. 147 in the world ranking of the recent Times Higher Education Ranking.  In other words, the university is placed as a strong player in the second tier of the universities that are globally relevant. According to the New York Times, Ann Mroz, editor of the THES, even congratulated the Egyptian university to have made it "into this table of truly world class". 

However, the reason for this high position is the performance of exactly one (1) academic: Mohamed El Naschie, who published 323 articles in the Elsevier journal Chaos, Solitons and Fractals of which he is the founding editor. His articles frequently cite other articles in the same journal by the same author. On many indicators, the Alexandrian university does not score very high, but on the number of citations indicator the university scores 99.8 which puts it as the 4th most highly cited in the world. This result clearly does not make any sense at all. Apparently, the methodology used by the THES is not only problematic because it puts a high weight on surveys and perceived reputation. It is also problematic because the way the THES survey counts citations and makes them comparable across fields (in technical terms, the way these counts are normalized) is not able to filter out these forms of self-promotion by self-citations. In other words: the way the THES uses citation analysis does not meet one of the requirements of sound indicators: robustness against simple forms of manipulation.

Even within the narrow terms of the technical expertise of bibliometrics, this is a fatal flaw. So far, the THES web site has not (yet?) mentioned the problem and if you click on the University of Alexandria in the ranking, nothing warns you that this is the result of one extremely "productive" scholar. The THES was quite proud when they had outsourced citation analysis to Thomson Reuters. They thought they had acquired world class methods. Clearly, they were wrong. I do not want to be smug about this, but as we know quite well at CWTS, and sometimes have had to learn the hard way in the past, a company that is able to provide large scale data does not have to be good in analyzing these data in the most sophisticated way possible. The THES should surely reconsider its citation analysis methodology.