A few weeks ago (the 28th of November to be exact), I attended ‘Towards the REF: Defining Bibliometric Requirements for Research Assessment’. The event was the latest in a series organised by King’s College London and supported by HEFCE. It was very well attended (200+); perhaps indicative of the uncertainty that still surrounds it. With the RAE results only days away it is probably bad timing on my part to be blogging about the REF; however, it is a reality and we (researchers and research administrators) do need to be thinking about it. I thought it was well worth circulating some of my notes from the day, as well as jotting down some thoughts about the part ORO has to play in all of this.
The day started off with a useful update from Graeme Rosenberg of HEFCE. One of the key things I picked up here was a move away from a twin-track approach, i.e. with Sciences largely assessed by bibliometrics and Arts & Social Sciences more by peer review. In fact, Graeme told us, the idea is to have a ‘family of tools’ that can be used for all disciplines, with bibliometrics being just one part of that ‘family’. Of course a burning question for HEFCE here is how to combine all elements of the ‘toolkit’ to provide a single indicator of research quality.
HEFCE seem to have great confidence in the potential of bibliometrics to contribute towards the REF. The basic idea is to establish a citation rate per paper, normalised against the average for that field, and then aggregate to produce an indicator. The indicator could then be used by expert panels as part of a wider portfolio of evidence.
A pilot study involving around 20 institutions was set up and has recently been completed. HEFCE expect to publish the results in the summer of 2009. Thereafter, further consultation will take place in Autumn 2009, with outcomes in early 2010. Another bibliometrics exercise will then follow, and the full REF exercise itself will take place in 2013 in order to drive funding from 2014 onwards.
Jonathan Adams from Evidence Ltd – the consultancy contracted by HEFCE to oversee the bibliometrics pilot exercise – gave a progress report presentation. A key theme that came out of what Jonathan was saying was the definite need for institutions to have either an institutional repository or a robust central publications database (a point reflected in a later presentation by Stuart Bolton – a consultant to JISC and HEFCE employed to look at the ICT implications of the REF). It is clear that either of these two methods is going to be crucial for collecting data for the REF. Also, it seems an advantage is to be had if your repository or database is somehow linked to your HR systems. ORO, of course, is linked to PIMS, which undoubtedly puts us in quite a strong position.
Jonathan highlighted some key issues that need to be addressed by analysis of the pilot exercise data. For example: will the REF cover all staff or selected staff? Will papers be linked to institution or individual researcher? Do you include staff that were present at an institution but have since moved on (and vice-versa)? Will the REF look at all publications by an individual or selected publications?
Wendy White and David Arrell from Southampton and Portsmouth Universities respectively spoke of their experiences of being involved with the REF pilot. Southampton have a very well established institutional repository and used it to gather together all the information needed, whereas Portsmouth do not and relied upon their RAE database. Wendy mentioned the importance of Southampton’s mandate in making sure their data was rich enough to gather all the information required. David, on the other hand, implied Portsmouth still have a decision to make in terms of whether or not to develop their repository or go with some other kind of publications database. Again, the message is that one or the other seems necessary.
Also of note from the day was Dr Henk Moed’s appraisal of citation data sources. Dr Moed is a bibliometrics expert from the University of Leiden and was commissioned by HEFCE to compare the two major commercial databases available for performing bibliometrics analysis: Thomson’s Web of Science and Elsevier’s Scopus. The basic conclusion of the work was that Scopus is a more than adequate substitute for Web of Science in terms of subject coverage. HEFCE are yet to decide which database to go with, or indeed whether to use a combination of the two.
All in all, the main take-home message for me was that HEFCE are still not clear on the exact detail of the REF (selected researchers or all researchers; selected publications or all publications; institutionally-linked or researcher-linked publications), but they did recommend that having a publications database or institutional repository in place would make it a lot easier for institutions to make their REF submission. So, it is looking as though ORO will have a dual role to play when it comes to the REF:
1. Open Access. As described above, a large part of the REF’s ‘family’ of research assessment tools is going to be bibliometrics. That is, how well cited your work is will be considered as a measure of quality, whether you agree with it or not. It follows, therefore, academics need to be thinking about citations; specifically, what can be done to maximise them. Making the full text of your work openly accessible through your institutional repository can help with this. It breaks down subscription barriers and makes your research visible to fellow academics that might not otherwise have access to it through their institution.
2. Research administration. Although the scope of the REF is not yet known, it is clear that having a central publications database or institutional repository available to collate the information required for our submission is key. As far as administration is concerned, the ‘worst case scenario’ would surely be having to submit all publications for all researchers present at the institution during the REF-defined period. If this possibility becomes a reality, making sure ORO is properly populated now has to be a priority.