Back in the old days, when people travelled in aeroplanes – and further back, before airports were detention camps for the tired, the angry and the disoriented – flying was an adventure eagerly anticipated. While exercising my memory the other day I recalled one flight made astonishingly happy by an air hostess; that job title tells how distant was this recollection, but I can see her before me now, hear the Irish lilt of her voice, see her dancing green eyes and a smile that told me that she knew the absurdity of the world she lived in.
She came to me in answer to a press of the call button, a call I’d made on discovering that my in-flight meal – penne in tomato sauce – was seeded with squares of cellophane.
I’ll come back to that because I’m writing here about citation metrics. Citation metrics are interesting creatures with much to tell us, but as measures of quality they are not intellectually defensible, and to use them in this way is corrupt and corrupting. Others have expressed this more forcefully, so I will leave my comments as this mild rebuke while conceding that I have a profile on Google Scholar and another on Publons that I sometimes try to avoid looking at. However I have come across small joys on both Google Scholar and Web of Science recently, though probably not for reasons either company would be particularly proud of.
Google Scholar has been kind to me lately, as evident in this screen shot from my profile.
You will see that 2012 was an annus mirabilis for my citations. This citation distribution surprised me. Apparently, I’d been completely oblivious to my most ‘impactful’ year. What possibly could have happened to send my annual citations soaring from 550 in 2011 to a staggering 2,710 in 2012?
After phoning Rhodri and investigating a little further, we soon established that the steep rise in citations was explained almost entirely by citations to two papers that I had published in 1995, each of which was cited about 1,000 times in 2012. These papers (on growth hormone secretagogues) are splendid and I’m glad that they were belatedly recognised, but I was curious about who was citing them.
But wait, why would they be causing this kind of commotion 17 years later? Why didn’t I keep getting cited at this rate for at least a couple more years? And what happened to send my annual citations plummeting to the now measly-looking 482 in 2013?
Google Scholar let me in on this secret. I’ll just show you one of the citing publications:
OK. Perhaps the citation counts for these papers might be misleading. According to Web of Science, these papers received just 2 citations from papers published in 2012 – hmm. In fact, Rhodri, after scanning through the Google results happily informed me that, at most, four of the papers Google Scholar recorded as citing these papers in 2012 were real. OK. I’ll show you more:
Ah, to be cited by a paper entitled “Plastic Kids Electric Toy Car for Children Electric Toy Car”. Exactly what is causing this problem on Google Scholar we haven’t been able to figure out, but if anyone has any explanation – please do get in contact.
Obviously this is an extreme example of mistakes in Google Scholar, but I’ve found different problems previously with other bibliographic databases. In 2016 I published an especially splendid paper:
Leng G, Sabatier N (2016) Measuring Oxytocin and Vasopressin: Bioassays, Immunoassays and Random Numbers. Journal of Neuroendocrinology Oct;28(10):10.1111/jne.12413. doi: 10.1111/jne.12413.
I am particularly proud of this paper. The words ‘and random numbers’ were included in the title at the strong suggestion of an insightful referee who suggested that it should be made compulsory reading for anyone using an immunoassay. But according to Web of Science (as reported from an author search), it has been cited only twice. Now I know this is wrong because I’ve cited it more than twice myself. So Rhodri suggested that I run a “Cited Reference Search” for it on Web of Science. The results are below:
Web of Science records at least 69 citations to my splendid paper but has decided to hide 67 of these from my author profile. (The list above includes 13 to another paper I published in the same volume of The Journal of Neuroendocrinology – also a fine paper but a bit more ‘niche’ than ‘random numbers’; these 13 are not included in the 69). It seems that because ‘random numbers’ is cited in different (sometimes incorrect) formats it appears as many different publications, only one of which is attributed to me.
This may be a common issue. PubMed cites my paper as “Oct;28(10):10.1111/jne.12413. doi: 10.1111/jne.12413.” Exactly what parts of this an author should give in a reference is not obvious, so it is to be expected that references are published in diverse formats. We expect Web of Science to somehow sort this out and mostly they do but …. obviously …. they make mistakes.
I am not complaining, and to explain why, let me return to the in-flight meal with which I opened this blog.You will remember that I was poised to protest at the presence of squares of cellophane threaded through my airport meal. The dancing green eyes of the air hostess alerted me to the absurdity of this. OK, there were cellophane squares, but they added texture and interest. My siren was too polite to ask me what I had expected from an in-flight meal (flaccid texture, mainly bland with mildly unpleasant aftertaste yes, squares of cellophane no). She brought me a replacement meal (no pasta or cellophane) and a small bottle of champagne (mainly bland with mildly unpleasant aftertaste but alcoholic). And she smiled at me in a way that made me feel glad to have tasted cellophane.
I’m not complaining, because I expect citation metrics to be flawed. I think it is intellectually feeble to imagine that they are a reliable measure of quality, I expect that if I need to judge the quality of a paper then I have to read it. When I read papers in my own field, my assessment of their quality generally bears little relationship to how often they have been cited or to where they were published, and this is often true of my judgements of my own papers as well as those of others that I read.
So my experience tells me that citation metrics are worse than useless as an index of quality – worse because they have been so often abused that they are increasingly actively distorted by the corrupt behaviour of journals and authors, or simply nonsense due to human (or computational) error. But others may feel more strongly – my memories of green eyes are perhaps keeping me temperate in my choice of words.
Gareth Leng | December 2020.