Reflection on: “RIMS: The Research Impact Measurement Service at the University of New South Wales”

Background

Seven years ago, I was approached by my then colleague, Richard Wartho, to co-write an article about a service we had been delivering for a couple years at UNSW Library called the Research Impact Measurement Service (RIMS). Although a number of Australian university libraries had started working in the bibliometrics space around this time, we believed that RIMS was quite unique in the way it was delivered and in the breadth of its offerings. By 2009 when our article was published in AARL (Drummond, 2014), we had been delivering a comprehensive service for a full two years.

Early impact

Following the publication of the article, the service generated a lot of interest locally in Australia and in New Zealand. In the next few years, UNSW Library was approached by a number of other university libraries for advice on developing their research support services around publishing impact. This resulted in a decision to host a workshop at UNSW, aligned with a Sydney-based conference in early 2011, to explain our service approach. The workshop also included some practical scenario-based sessions, using anonymised but real-life examples of queries and service requests from our own academic community.

Feedback from this workshop was excellent and the resulting support requests from numerous libraries in Australia and New Zealand made us realise that we were seen as leaders in this emerging research support space. It had clearly struck a nerve.

Citation impact analysis

As a fundamental part of RIMS at UNSW Library, we encouraged researchers to analyse their citation impact. It can be illuminating for a researcher to quantify and qualify publishing impact by looking at the nature and breadth of their influence. Although it is difficult to infer anything significant from analysing the impact of just one article, it is nevertheless interesting to understand the influence of our original RIMS article.

Commencing in the same year of publication (2009), citations for our article started to accrue. By mid-2016, 39 verifiable references had been made to our article in the scholarly literature. Fifty per cent of these citations took around five years to accrue, with the peak number of annual citations occurring in 2014 (see Table 1). It would appear from this example that literature in the discipline of library science may take some time to disperse.

Table 1. Breakdown of citations by year.

Citing year	No. of cites	Cumulative cites
2009	1	1
2010	5	6
2011	3	9
2012	5	14
2013	7	21
2014	9	30
2015	5	35
2016	4	39

An analysis of the geographic location of the citing authors showed that in the first two years only one citing article had a non-Australian author. That article was published in a medical librarianship journal by an author from the USA. Over the next two years, the article started to reach an international audience, being cited by authors and co-authors from the USA, UK, NZ and Spain.

In the following two years, the number of non-Australian citing authors finally exceeded local authors. Overall, the geographic reach of the article was quite extensive, being cited by authors from Europe, North America, South Africa, China and Russia (see Table 2). The themes addressed in the article seemed to resonate in the broader academic and research library community, indicating we are all facing similar issues in service provision to researchers. However, from this specific example, it appears to take a number of years for Australian published literature to reach an international audience.

Table 2. Geographic location of citing authors.

Country of citing author	Number
Australia	19
USA	10
Spain	3
China	2
UK	2
Norway	1
Russia	1
Germany	1
South Africa	1
NZ	1
Switzerland	1

The subject distribution of citing articles was both expected and surprising (see Table 3). The majority of citations (64%) were from articles relating to aspects of librarianship. A small percentage of articles (13%) were about LIS and bibliometrics more broadly. Somewhat surprisingly, 23% of the citing articles, nine in total, were focussed on areas such as scholarly communication and higher education and were written by non-librarians. Presumably, as Australian Academic and Research Libraries (AARL) is indexed by Scopus, Web of Science and Google scholar, our article has been exposed to a wider audience.

Table 3. Subject distribution of citing articles.

Citing article subject area	Number of articles
Librarianship	25
Higher education	6
LIS	3
Scholarly communication	3
Bibliometrics	2

Reflection

In reviewing the impact of the 2009 RIMS article, it has been interesting to reflect on a number of issues around librarianship, scholarly communication the changing impact environment.

It is somewhat ironic that the article is still being cited in 2016, as the service no longer exists in its original format. In 2014, an article was published in AARL (Drummond, 2014) in which I discussed the evolution of our research impact services from their inception in 2007. In some ways, the original service became a victim of its own success. Although extremely popular, it became unsustainable in the format described in 2009 and had to evolve. As the research and IT environments in which we operated were also evolving, we responded to the opportunities provided. The affordances of new automated systems allowed us to transform our service from a very manual one to an advisory service, using smart technology to deliver similar outcomes.

This has reinforced to me the need for library services to respond in an agile manner. Although not unique to our industry, libraries often seem to become fixed in their service offerings without challenging themselves to respond to emerging trends and environments.

Similarly, the whole research impact measurement environment may not be keeping up with changes in scholarly communication. It is perhaps constructed on old publishing paradigms that will not be as meaningful in the future. As Castellani and colleagues note (Castellani, Pontecorvo & Valente, 2016), ‘the role of the scientific journal in the exchange of scientific knowledge is decreasing’ in favour of more informal means of communication. However, it is still the bibliometric measures based on formal journal publications that are used to ‘evaluate the careers of the researchers and the quality of the research centers’.

A broadening view of research impact, particularly evident in worldwide research assessment exercises, has challenged some of the standard approaches to analysing publication impact. The emergence of alternative metrics (altmetrics) is evidence of an evolving impact landscape, although it is acknowledged that altmetrics measure other dimensions such as web attention, rather than the traditional academic impact of bibliometrics.

Through delivery of RIMS, it has become clear that librarians should exercise caution in reporting bibliometric measures. It has been illuminating to see how publication impact measures can be misused and how research evaluation can influence researcher behaviour. From the obsession about the h-index to the use of journal metrics to evaluate an individual’s performance, it is apparent that a simple ‘number’ is very appealing to research administrators and even to the researchers themselves.

In spite of these issues, bibliometrics when used appropriately can educate and inform. At UNSW Library, we have used them as a starting point for discussions with researchers about making their research visible; connecting their outputs; and developing their research profile. As one UNSW researcher remarked, it can be quite ‘transformative’ to analyse your citations and review your influence and publishing strategies.

In reflecting on the readiness with which researchers accepted our support in this area, it is apparent that our expertise on issues in scholarly communication is valued and adds value to the work of our researchers and our institutions. Librarians should not be afraid to speak up and influence the debate.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes on contributor

Robyn Drummond is currently the Director of Information Services at UNSW Library, Sydney, Australia. She was appointed to this position in 2015. Through her work over a number of years in academic services at UNSW Library, she has developed an ongoing interest in library support for researchers and has been instrumental in developing innovative services in this area. Robyn has held a variety of library and information management roles in the academic, research and government sectors over 20 years. She is an active participant in the activities of the informal research librarian network in Australia and is an associate member of ALIA.