The Citation Culture

We recently migrated our blog to a new platform. Please visit us at:

 http://citationculture.wordpress.com

We also moved the complete archive of posts and comments to this address.

Kind regards,

Paul Wouters and Sarah de Rijcke

The widespread use of rankings of journals in research institutes and universities creates a disadvantage for interdisciplinary research in assessment exercises such as the British Research Excellence Framework. This is the conclusion of a paper presented at the 2011 Annual Conference of the Society for the Social Studies of Science in Cleveland (US) by Ismael Rafols (SPRU, Sussex University), Loet Leydesdorff (University of Amsterdam) and Alice O’Hare, Paul Nightingale and Andy Stirling (all SPRU, Sussex University). The study is the first quantitative proof that researchers working at the boundaries between different research fields may be disadvantaged compared with monodisciplinary colleagues. The study argues that citation analysis, if properly applied, is a better measurement instrument than a ranked journal list.

The study is quite relevant for research management at universities and research institutes. Journal lists have become a very popular management tool. In a lot of departments, researchers are obliged to publish in a limited set of journals. Some departments, for example in economics, have even been reorganized on the basis of having published in such a list. The way these lists have been composed does vary. Sometimes a group of experts decides whether a journal belongs to the list, sometimes the Journal Impact Factor published by ISI/Thomson Reuters is the determining factor.

The study by Rafols et al. has analyzed one such list: the ranked journal list used by the British Association of Business Schools. This list is based on a mix of citation statistics and peer review. It ranks scholarly journals in business and management studies in five categories. “Modest standard journals” are category 1, “world elite journals” are category 4*. This scheme reflects the experience researchers have with the Research Assessment Exercise categories. The ranked journal list is meant to be used widely for a variety of management goals. It is used as an advice for researchers about the best venue for their manuscripts. Libraries are supposed to use it in their acquisition policies. And last but not least, it is used in research assessments and personnel evaluations. Although the actual use of the list is an interesting research topic in itself, we can safely assume that it has had a serious impact on the researchers in the British business schools community.

The study shows first of all that the position of a journal in the ranked list correlates negatively with the extent of interdisciplinarity of the journal. In other words, the higher the ranking, the more narrow its disciplinary focus. (The study has used a number of indicators for interdisciplinarity by which different aspects of what it means to be interdisciplinary have been captured.) Rewarding researchers to publish first of all in the ranked journal list may therefore discourage interdisciplinary work.

The study confirms this effect by comparing business and management studies to innovation studies. Both fields are subjected to the same evaluation regime in the Research Excellence Framework. Intellectually, they are very close. However, they differ markedly with respect to their interdisciplinary nature. Researchers in business schools have a more traditional publishing behaviour than their innovation studies colleagues. The research units in innnovation studies are consistently more interdisciplinary than the business and management schools.

Of course, publication behaviour is shaped by a variety of influences. Peer review may be biased against interdisciplinary work because it is more difficult to assess its quality. Many top journals are not eager to publish interdisciplinary work. This study is the first to show convincingly that these already existing biases tend to be made even stronger by the use of ranked journal lists as a tool in research management. The study confirms this effect by comparing the performance based on the ranked journal list with a citation analysis. In the latter, the innovation studies research is not punished by its more interdisciplinary character which does happen in an assessment on the basis of the journal list. The paper concludes with a discussion of the negative implications in terms of funding and acquiring resources for research groups working at the boundaries of different fields.

The paper will be published in a forthcoming issue of Research Policy and has been awarded the best paper at the Atlanta Conference on Science and Innovation Policy in September 2011.

Reference: Ismael Rafols, Loet Leydesdorff, Alice O'Hare, Paul Nightingale, & Andy Stirling, “How journal rankings can suppress interdisciplinary research. A comparison between innovation studies and business & management,” Paper presented at the Annual Meeting of the Society for the Social Studies of Science (4S), Cleveland, OH, Nov. 2011; available at http://arxiv.org/abs/1105.1227.

Recently, the new Times Higher Education World University Rankings 2011-2012 saw the light. The ranking revealed that Harvard University is no longer number one on the list. Incidentally, the differences with Caltech - now highest - are minimal. The main reason for Caltech’s rise are the extra revenues it drew out of industry. Caltech’s income increased by 16%, thereby outclassing most other universities. Harvard scored a bit better when it comes to the educational environment. Other universities also rose on the list as a result of a successful campaign to obtain (more) external financing. The London School of Economics, for example, moved from 86 to 47. The top of the ranking did not change that drastically though. Rich US-based universities still dominate the list. 7 out of ten universities highest on the list, and one third of the top 200, are located in the US.

This illustrates the THE ranking’s sensitivity to slight differences between indicators that, taken together, shape the order of the ranking. The ranking is based on a mix of many different indicators. There is no standardized way to combine these indicators, and therefore there inevitably is a certain arbitrariness to the process. In addition, the THE ranking is partly based on results of a global survey. This survey invites researchers and professors to assess the reputation of universities. One of the unwanted effects of this method is that well-known universities are more likely to be positively assessed than less popular universities. Highly visible forms of maltreatment and scandals may also influence survey results.

This year, the ranking’s sensitivity to the ways in which different indicators are combined is aptly illustrated by the position of the Dutch universities. The Netherlands are at number 3, with 12 universities in the top 200 and 4 in the first 100 of the world. Given the size of the country, this is a remarkable achievement. The result is partly caused by a strong international orientation of the Dutch universities, and partly by previous investments in research and education. But just as important is the weight given to the performances of the social sciences and humanities in a number of indicators. Compared to last year, the total performance of Dutch universities most likely did not increase that much. A more likely explanation is that the profile of activities and impact are better covered by the THE ranking.

The latest THE ranking does make clear that size is not the most important determinant in positioning universities. Small specialized universities can end up quite high on the list.

When it comes to critical analysis of the role of computers, data visualization, simulations and modeling in the sciences, there’s a lot to be learned from humanities scholars. I’m currently teaching a course on the role of computer-generated images in contemporary science and visual culture at Utrecht University. Yesterday I learned that the New Media department hosts two very interesting events. Today, Tuesday October 18, there’s a workshop on software applications as active agents in shaping knowledge. The two keynote speakers are Dr Eckhart Arnold (University of Stuttgart), expert in the field of simulation technologies, and Dr Bernhard Rieder (University of Amsterdam), who researches how computers and software organize knowledge.

A week later, on October 25, Setup will host an event on data visualization at the Wolff Cinema movie theatre in Utrecht. Some of the most striking recent data visualization projects will be displayed on screen, and the following questions will be addressed: what makes data visualizations so appealing? Do they bring across the same message as the ‘raw’ data they originate from? Ann-Sophie Lehmann (associate professor New Media en Art History, UU) will discuss the visualizations and will throw light on some of the effects they have on viewers. One question that came to my mind is what this particular context (a movie theater) does to the (reception of) the visualizations, compared to a web-based interaction on a laptop or PC, for instance.

On November 16, 2011, the Rathenau Institute and the VU University Amsterdam organize a symposium on Dynamics of Academic Leadership. The symposium addresses the conditions that are necessary for high level performance and creativity in research, and the implications for research management and policy. Paul is one of the invited speakers. He will discuss some of the programmatic aspects and preliminary results of a large European FP-7 project: Academic Careers Understood through Measurement and Norms (ACUMEN). ACUMEN is aimed at understanding the ways in which researchers are evaluated by their peers and institutions, and at assessing how the science system can be improved and enhanced. The project is a cooperation among several European research institutes, with Paul as the principal investigator and CWTS's Clifford Tatum as project manager.