Harvard no longer number 1 in ranking

Recently, the new Times Higher Education World University Rankings
2011-2012 saw the light. The ranking revealed that Harvard University is no
longer number one on the list. Incidentally, the differences with Caltech – now
highest – are minimal. The main reason for Caltech’s rise are the extra
revenues it drew out of industry. Caltech’s income increased by 16%, thereby
outclassing most other universities. Harvard scored a bit better when it comes
to the educational environment. Other universities also rose on the list as a
result of a successful campaign to obtain (more) external financing. The London
School of Economics
, for example, moved from 86 to 47. The top of the ranking
did not change that drastically though. Rich US-based universities still dominate
the list. 7 out of ten universities highest on the list, and one third of the
top 200, are located in the US.

This illustrates the THE ranking’s sensitivity to slight
differences between indicators that, taken together, shape the order of the
ranking. The ranking is based on a mix of many different indicators. There is
no standardized way to combine these indicators, and therefore there inevitably
is a certain arbitrariness to the process. In addition, the THE ranking is
partly based on results of a global survey. This survey invites researchers and
professors to assess the reputation of universities. One of the unwanted
effects of this method is that well-known universities are more likely to be
positively assessed than less popular universities. Highly visible forms of maltreatment
and scandals may also influence survey results.

This year, the ranking’s sensitivity to the ways in which
different indicators are combined is aptly illustrated by the position of the
Dutch universities. The Netherlands are at number 3, with 12 universities in
the top 200 and 4 in the first 100 of the world. Given the size of the country,
this is a remarkable achievement. The result is partly caused by a strong
international orientation of the Dutch universities, and partly by previous
investments in research and education. But just as important is the weight
given to the performances of the social sciences and humanities in a number of
indicators. Compared to last year, the total performance of Dutch universities
most likely did not increase that much. A more likely explanation is that the profile
of activities and impact are better covered by the THE ranking.

The latest THE ranking does make clear that size is not the
most important determinant in positioning universities. Small specialized universities
can end up quite high on the list.

Understanding Academic Careers

On November 16, 2011, the Rathenau Institute and the VU University Amsterdam organize a symposium on Dynamics of Academic Leadership. The symposium addresses the conditions that are necessary for high level
performance and creativity in research, and the implications for research
management and policy. Paul is one of the invited speakers. He will discuss some of the programmatic aspects and preliminary results of a large European FP-7 project: Academic Careers Understood through Measurement and Norms (ACUMEN). ACUMEN is aimed at understanding the
ways in which researchers are evaluated by their peers and institutions,
and at assessing how the science system can be improved and enhanced. The project is a cooperation among several European research institutes, with Paul as the principal
investigator and CWTS’s Clifford Tatum as project manager.

Science mapping: do we know what we visualize?

For example, what often gets
glossed over in these endeavors is that visualizations of scientific
developments also
prescribe how these developments should be known in the first place.
Science maps are produced by particular
statistical algorithms that might have been chosen otherwise, calculations
performed on large amounts of ‘raw’ data
stored in databases, and for this reason they are not simply ‘statistical
information presented visually’.
The choice for a particular kind of visualization is
often connected to the specificities and meaning of the underlying dataset and
the software used to process the data.
Several software packages have been specifically
designed for this purpose (the VOSViewer supported by CWTS being one of them).
These packages prescribe how the data should be handled. Different choices in
selection and processing of the data will lead to sometimes strikingly
different maps. Therefore, we will increasingly need systematic experiments and
studies with different forms of visual presentation (Tufte, 2006).

At the same time, a number of interfaces are built into the mapping
process, where an encounter takes place with a user who approaches these
visualizations as evidence.

But how do these users actually behave?
To our knowledge hardly any
systematic research is done on how users (bibliometricians, computer
scientists, institute directors, policy makers and their staff, etc.) engage
with these visualizations, and which skills and strategies are needed to engage
with them.
A
critical scrutiny is needed of the degree of ‘visual literacy’ (Pauwels, 2008)
demanded of users who want to critically work with and examine these
visualizations. The visualizations
embody
technical or
formal choices that determine what can be visualized and what will remain
hidden. Furthermore, they are also shaped by the broader cultural and
historical context in which they are produced.

Unfortunately
t
here is a
tendency to downplay the visuality of science maps, in favor of the integrity
of the underlying data and the sophistication of transformation algorithms.
However, visualizations are “becoming increasingly dependent upon technology,
while technology is increasingly becoming imaging and visualization technology”
(Pauwels 2008, 83). We expect that this interconnection between data selection,
data processing and data visualization will become much stronger in the near
future. These connections should therefore be systematically analyzed, while
the field develops and experiments with different forms of visual
representation.

As said, science
mapping projects do not simply measure and describe scientific developments –
they also have a normative potential.
Suppose, in an hypothetical example, that the director of a research institute wants to
map the institute’s research landscape in terms of research topics and possible
applications, and wants to see how the landscape develops over the next five
years. This kind of mapping project, like any other description of reality, is
not only descriptive but also performative. In other words, the map that gets
created in response to this director’s question also shapes the reality it
attempts to represent. One possible consequence of this hypothetical mapping
project could be that the director decides on the basis of this visual analysis
to focus more on certain underdeveloped research strands, at the expense of or
in addition to others. The map that was meant to chart the terrain now becomes
embedded in management decision processes. As a result, it plays an active part
in a shift in the institute’s research agenda, an agenda that will be mapped in
five years’ time with the same analytical means that were originally merely
intended to describe the landscape.

A comparable example can actually be found in
Börner’s book: a map that shows all National Institute of Health (NIH) grant
awards from a single funding year.
The project comes with a website, giving
access to a database and web-based interface. The clusters on the map
correspond to broader scientific topics covered in the grants, while the dots
correspond with individual grants clustered together by a shared topical focus.

Here, too, it
would be informative to analyze the potential role these maps play as policy instruments
(for instance, in accountability studies). This type of analysis will be all
the more urgent when
bibliometric maps are increasingly used for the purposes of research
evaluation. The maps created on the basis of bibliometric data do not simply ‘visualize
what we know’. They actively shape bibliometric knowledge production, use and
dissemination in ways that require careful scrutiny.

Still using the Hirsch index? Don’t!

“My research: > 185 papers, h-index 40.” A
random quote from a curriculum vitae in the World Wide Web. Sometimes,
researchers love their Hirsch index, better known as the h-index. But what does
the measure actually mean? Is it a reliable indicator of scientific impact?

 

Our colleagues Ludo Waltman and Nees Jan van
Eck have studied the mathematical and statistical properties of the h-index.
Their conclusion: the h-index can produce inconsistent results. For this
reason, it is actually not the reliable measure of scientific impact that most
users think it is. As a leading scientometric institute, we have therefore
published the advice to all universities, funders, and academies of science to
abandon the use of the h-index as a measure of the overall scientific impact of
researchers or research groups. There are better alternatives. The paper by
Waltman and Van Eck is now available as a preprint
and
will soon be published by the Journal of the American Society for Information
Science and Technology
JASIST.

 

The h-index is a measure of a combination of productivity and citation
impact. It is calculated by ordering the number of publications by a particular
researcher on the basis of the total number of citations they have received.
For example, someone who has an h-index of 40 has published at least 40
articles that have each been cited at least 40 times. Moreover, the remaining
articles have not been cited more than 40 times each. The higher the h-index the
better.

 

The h-index was proposed by physicist Jorge Hirsch in 2005. It was an
immediate hit. Nowadays, there are about 40 variants of the h-index. About one
quarter of all articles published in the main scientometric journals have cited
Hirsch’ article in which he describes the h-index. Even more important has been
the response by scientific researchers using the h-index. The h-index has many
fans, especially in the fields that exchange many citations, such as the
biomedical sciences. The h-index is almost irrresistable because it seems to
enable a simple comparison of the scientific impact of different researchers. Many
institutions have been seduced by the siren call of the h-index. For example,
the Royal Netherlands Academy of Arts and Sciences (KNAW) in the Netherlands inquires
about the value of the h-index in its recent forms for new members. Individual
researchers can look up their h-index based on Google Scholar documents via
Harzing’s website publish or perish. Both economists and computer scientists
have produced a ranking of their field based on the h-index.

 

Our colleagues Waltman and Van Eck have now shown that the h-index has some
fatal shortcomings. For example, if two researchers with a different h-index
co-author a paper together, it may lead to a reversal of their position in an
h-index based ranking. The same may happen when we compare research groups.
Suppose we have two groups and each member of group A has a higher h-index than
a paired researcher in group B. We would now expect that the h-index of group A
as group is also higher than that of group B. Well, that does not have to be
the case. Please note that we are now speaking of a calculation of the h-index
based on a complete and reliable record of documents and citations. The
problematic nature of the data if one uses Google Scholar as data source is a
different matter. So, even when we have complete and accurate data, the h-index
may produce inconsistent results. Surely, this is not what one wants using the
index for evaluation purposes!

 

At CWTS, we have therefore drawn the conclusion that the h-index should not be used as measure of scientific
impact in the context of research evaluation.

Evaluating e-research

We had a very interesting discussion last week at the e-Humanities Group of the Royal Netherlands Academy of Arts and Sciences. The problem I presented is how to evaluate e-research, the newly emerging style of scientific and scholarly research that makes heavy use of, and contributes to, web based resources and analytical methods. The puzzle is that current evaluation practices are strongly biased towards one particular mode of scientific output: peer reviewed journal articles, and within that set in particular those articles published in journals that are used as source materials for the Web of Science, published by ISI/Thomson Reuters. If scholars in the sciences, social sciences and humanities are expected to contribute to e-science and e-research, it is vital that the reward and accounting system in the universities do honour work in this area. Here is the link to the presentation "Evaluating e-Research".

La habitación de Fermat

Recently, we saw a somewhat crazy Spanish movie, "La habitación de Fermat". In the story, a couple of mathematicians and an inventor are invited to a mystery play on the basis of their capacity to solve puzzles. In the end they are locked up in a room that becomes smaller each time a puzzle is not solved in time. As a result, they threaten to become crushed and need to mobilize all their considerable brainpower to survive. I will not reveal who did it but the motivation is interesting. It has everything to do with the extreme competition in some fields of science. It all revolves around the solution to  Goldbach’s Conjecture. A young mathematician claims that he has discovered the proof and one of the older guys, who has been working on this problem for over thirty years, feels very threatened. This is exacerbated by the arrogance of the upstart and the brazenness with which he is giving interviews.The movie is full of dialogues that dwell on how it is to live in researc. In the closing part of the movie the group is boating back home. One of the mathematicians has gotten hold of the proof, not written by himself, and agonizes over whether he should publish it as his own or not. One of the others solves the problem by throwing the proof in the river. "What?", the guy shouts, this is a world disaster!" His companion rows on, looks around himself, and points to the fact that nothings seem to have changed. We see the proof drifting away, and the world is oblivious.