How relevant is an author’s citation index?

As I was scratching my head trying to find new criteria to add to the NSERC formula for “binning” Canadian scientists, I contemplated adding an “Author’s Citation Index”.  The thought didn’t last long.

To start with, there are many of them (Math.Sci-net, h-index, ISI, and soon PoP for Google Scholar).  Each has been subjected to serious critiques.  Then, I remembered that one of my own most cited papers is far from being one of my best –at least in my own opinion, which should matter here!

I also recalled how a Texan friend of mine once told me that the “Functional Analysis” paper containing the Johnson-Lindenstrauss lemma is much more cited in computer science journals than in the mathematics journals surveyed in the MathSci-Net Database for citations. Should Google Scholar be then the new measuring stick?

It is safe to say that it is definitely not in the culture of mathematicians to consider the citation index as a good measure for evaluating fellow researchers. There must be some reasons other than those I stated above.

This is, however, not the case for other scientific disciplines. I am always struck by how much certain subjects rely on such indices, whenever I serve on interdisciplinary panels, the latest being the selection committee for the Killam Prize of the Canada Council for the Arts. Various citation indices were often at the heart of the evaluation of our colleagues outside the mathematical sciences. Referee reports often included citation numbers for various papers. I admit finding this helpful at times, especially when you are faced with files displaying 500+ publication titles, some involving more than a dozen co-authors.

In any case, the question of what a citation index means for a discipline or for a researcher cannot be that easy to answer. For example,  the average citation index varies enormously across the disciplines, and this also begs for an explanation. Another obvious observation is that these averages are shrinking rapidly over the years, regardless of the discipline, as one can see in the following spreadsheet for average citation rates, taken from the 1998-2008 Thomson Reuters’ Essential Science Indicators database. Are we converging towards any other steady state besides zero?

Mathematics is –as usual– on one end of the spectrum. The other end in this case is Molecular biology. One explanation is that Mathematics papers typically list few references, whereas those in other fields rely more on other publications.

Can someone clarify these puzzles for us? I also recall there was lots of controversy about the popular “journal impact factor”. This story will be for another day.

This entry was posted in R&D Policy. Bookmark the permalink.

4 Responses to How relevant is an author’s citation index?

  1. Greg Martin says:

    Don’t read too much into that table – the right-hand half at least. It takes years before papers garner their eventual citations.

  2. Pingback: Tweets that mention How relevant is an author’s citation index? | Piece of Mind --

  3. The obvious answer is that researchers should be allowed to propose their own metrics which can inlude the number of downloads, the comments received, and so on.

    The systematic application of an index like the h-index may have undesirable effects. For example, the h-index encourages younger researchers to work within the research program of established researchers. What we want are researchers who go on their own, crafting their own ideas. Hirsch revised the h-index to take into account the effect of these collaborations in “An index to quantify an individual’s scientific research output that takes into account the effect of multiple coauthorship”. See But I fear that any one index will fail us.

  4. Eugene Barsky says:

    Science librarians in UBC and elsewhere frequently see professors who ask for help calculating h-index, eigenfactor, and in particular checking the impact factor of journals.

    We do feel that it is also our role to show how easy it is to manipulate these metrics. Here is a very good piece from SIAM about that –

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s