14 August 2010

Cite more papers, get more citations?

Update, 18 August 2010: An attempt to replicate the result discussed here finds serious issues with the statistics.

ResearchBlogging.orgNature News is reporting some interesting results presented as a paper at a meeting of the International Society for the Psychology of Science & Technology last week: articles published in the journal Science with longer "Works Cited" sections are themselves more frequently cited.
A plot of the number of references listed in each article against the number of citations it eventually received reveal that almost half of the variation in citation rates among the Science papers can be attributed to the number of references that they include. And — contrary to what people might predict — the relationship is not driven by review articles, which could be expected, on average, to be heavier on references and to garner more citations than standard papers.
The same authors did a similar analysis of papers published in the journal Evolution and Human Behavior over 30 years, and found similar results [PDF]. Here's the relevant figure from that paper:

Cite more, be cited more. Figure 2 from Webster et al. (2009) [PDF].

The lack of a "review effect" is surprising, but I don't think this overall result is. Academia, as much as we might describe it as cutthroat, also runs on reciprocal altruism. Authors notice when their papers are cited, and are more likely to cite papers that build on or relate to their own work. I'd be interested to see the network of citation underlying the pattern Webster et al. have found—I suspect that there's a lot of clustering around disciplines and sub-disciplines and sub-sub-sub-disciplines that contributes to all this mutual back-scratching citing.

Updated, 15 August 2010, 2126h: Fixed the link to the original Nature News article, which turns out not to be access-restricted.

Reference

Webster, G.D., Jonason, P.K., & Schember, T.O. (2009). Hot topics and popular papers in evolutionary psychology: Analyses of title words and citation counts in Evolution and Human Behavior, 1979-2008. Evolutionary Psychology, 7 (3), 348-348 Other: http://www.epjournal.net/filestore/ep07348362.pdf
Posted by Jeremy Yoder at 17:59
Filed under: , , ,

8 comments:

  1. I also wonder if papers including more references tend to be more cited because they are apt to be the result of more in-depth research and are subsequently of higher caliber. It may be interesting to see if it is possible to tease out differences in this trend and determine how often it is a product of quality versus mutual citing.
    ReplyDelete
  2. Matt, I don't know how you'd go about judging "quality," although I think there must be effects of actual paper content. Brief papers reporting raw observations (species accounts, natural history notes, &c) should have fewer citations, and may be of interest to a smaller range of other researchers and therefore receive fewer citations than longer papers that tackle "big picture" questions. On the other hand, papers describing new analysis methods, software, &c are often, in my experience, short and have accordingly few citations—but they can be citation magnets.

    Of course, since the newest dataset is from Science, the papers under analysis probably don't include a lot of those kinds of articles. Science papers are pretty homogenous in length, even across disciplines.
    ReplyDelete
  3. If this is indeed reciprocal altruism, here's a testable hypothesis: There should be a greater correlation with number of different authors cited than with the raw number of citations.
    ReplyDelete
  4. Could this be explained by papers with more citations being more likely to be found in journal search engines? If I find an article I like is cited by another paper I am more likely to investigate that paper. The more citations in a given article, the more likely I am going to at least look at a couple of them. Then if I write an article myself, I will include both of the previous papers.
    ReplyDelete
  5. Zen - Actually, the 2009 paper does find such a correlation, though it's weaker than the correlation between citations made and number of citations received: r = 0.2 for number of authors v. citations received, r = 0.44 for citations made v. citations received. Not sure what's in the new dataset.

    Kestrel - Good question. I've been trying to think about my own research habits in relation to this, and it's actually tricky wrapping my head around how I might find papers in a pre-Google world, and whether it'd produce a different pattern. Was there a way to back-search articles citing a given article before electronic databases? The citations made v. citations received pattern apparently goes back to 1900 in the Science dataset.
    ReplyDelete
  6. This comment has been removed by the author.
    ReplyDelete
  7. Whoops. Misread Zen's comment. The correlation I mention is for number of authors on the paper itself—papers with more authors have more citations. Number of different authors cited should be straightforward to do, too. But I don't find it in the 2009 paper.
    ReplyDelete
  8. Very interesting paper and thanks for the review.
    Original research papers (as opposed to reviews) tend to have a limited number of citations. But as commented by Matt, papers with longer citations may be considered (by the potential reader) as having involved a solid investigation of previously published research and, ergo, more authoritative.
    Indeed, I am very curious to see if all of this can be related to what you aptly state as 'mutual back scratching citing'- wouldn’t be surprised.
    ReplyDelete

Subscribe to: Post Comments (Atom)