Open Research Online

A few weeks ago (the 28th of November to be exact), I attended ‘Towards the REF: Defining Bibliometric Requirements for Research Assessment’. The event was the latest in a series organised by King’s College London and supported by HEFCE. It was very well attended (200+); perhaps indicative of the uncertainty that still surrounds it. With the RAE results only days away it is probably bad timing on my part to be blogging about the REF; however, it is a reality and we (researchers and research administrators) do need to be thinking about it. I thought it was well worth circulating some of my notes from the day, as well as jotting down some thoughts about the part ORO has to play in all of this.

The day started off with a useful update from Graeme Rosenberg of HEFCE. One of the key things I picked up here was a move away from a twin-track approach, i.e. with Sciences largely assessed by bibliometrics and Arts & Social Sciences more by peer review. In fact, Graeme told us, the idea is to have a ‘family of tools’ that can be used for all disciplines, with bibliometrics being just one part of that ‘family’. Of course a burning question for HEFCE here is how to combine all elements of the ‘toolkit’ to provide a single indicator of research quality.

HEFCE seem to have great confidence in the potential of bibliometrics to contribute towards the REF. The basic idea is to establish a citation rate per paper, normalised against the average for that field, and then aggregate to produce an indicator. The indicator could then be used by expert panels as part of a wider portfolio of evidence.

A pilot study involving around 20 institutions was set up and has recently been completed. HEFCE expect to publish the results in the summer of 2009. Thereafter, further consultation will take place in Autumn 2009, with outcomes in early 2010. Another bibliometrics exercise will then follow, and the full REF exercise itself will take place in 2013 in order to drive funding from 2014 onwards.

Jonathan Adams from Evidence Ltd – the consultancy contracted by HEFCE to oversee the bibliometrics pilot exercise – gave a progress report presentation. A key theme that came out of what Jonathan was saying was the definite need for institutions to have either an institutional repository or a robust central publications database (a point reflected in a later presentation by Stuart Bolton – a consultant to JISC and HEFCE employed to look at the ICT implications of the REF). It is clear that either of these two methods is going to be crucial for collecting data for the REF. Also, it seems an advantage is to be had if your repository or database is somehow linked to your HR systems. ORO, of course, is linked to PIMS, which undoubtedly puts us in quite a strong position.

Jonathan highlighted some key issues that need to be addressed by analysis of the pilot exercise data. For example: will the REF cover all staff or selected staff? Will papers be linked to institution or individual researcher? Do you include staff that were present at an institution but have since moved on (and vice-versa)? Will the REF look at all publications by an individual or selected publications?

Wendy White and David Arrell from Southampton and Portsmouth Universities respectively spoke of their experiences of being involved with the REF pilot. Southampton have a very well established institutional repository and used it to gather together all the information needed, whereas Portsmouth do not and relied upon their RAE database. Wendy mentioned the importance of Southampton’s mandate in making sure their data was rich enough to gather all the information required. David, on the other hand, implied Portsmouth still have a decision to make in terms of whether or not to develop their repository or go with some other kind of publications database. Again, the message is that one or the other seems necessary.

Also of note from the day was Dr Henk Moed’s appraisal of citation data sources. Dr Moed is a bibliometrics expert from the University of Leiden and was commissioned by HEFCE to compare the two major commercial databases available for performing bibliometrics analysis: Thomson’s Web of Science and Elsevier’s Scopus. The basic conclusion of the work was that Scopus is a more than adequate substitute for Web of Science in terms of subject coverage. HEFCE are yet to decide which database to go with, or indeed whether to use a combination of the two.

All in all, the main take-home message for me was that HEFCE are still not clear on the exact detail of the REF (selected researchers or all researchers; selected publications or all publications; institutionally-linked or researcher-linked publications), but they did recommend that having a publications database or institutional repository in place would make it a lot easier for institutions to make their REF submission. So, it is looking as though ORO will have a dual role to play when it comes to the REF:

1. Open Access. As described above, a large part of the REF’s ‘family’ of research assessment tools is going to be bibliometrics. That is, how well cited your work is will be considered as a measure of quality, whether you agree with it or not. It follows, therefore, academics need to be thinking about citations; specifically, what can be done to maximise them. Making the full text of your work openly accessible through your institutional repository can help with this. It breaks down subscription barriers and makes your research visible to fellow academics that might not otherwise have access to it through their institution.

2. Research administration. Although the scope of the REF is not yet known, it is clear that having a central publications database or institutional repository available to collate the information required for our submission is key. As far as administration is concerned, the ‘worst case scenario’ would surely be having to submit all publications for all researchers present at the institution during the REF-defined period. If this possibility becomes a reality, making sure ORO is properly populated now has to be a priority.

It is often said that making your work available in an open access repository like ORO provides wider exposure to your research and a better chance of being well cited. I firmly believe this, and I refer again to a recent study that seems to support this hypothesis. More peer-reviewed research from information scientists and bibliometricians is certainly needed, but there is also a lot to be said for anecdotal evidence as well.

Ray Frost is Professor of Inorganic Materials in the School of Physical & Chemical Sciences at the Queensland University of Technology (QUT), Australia. In a recent post on QUT’s Library News for the School of IT we read that Professor Frost has found that regularly depositing his work in QUT ePrints (QUT’s equivalent of ORO) actually saves him time, simply because he no longer has to deal with numerous requests by email for individual personal copies of his papers. Furthermore, the amount of citations Professor Frost’s work receives has increased markedly since 2005, which also happens to be the year he began to make his work available open access through QUT ePrints.

So, depositing a record of your publications in ORO is good, but attaching the full text as well is even better. We always provide a link to the published version, but this is only of use to people who have a subscription to the journal in which you have published, either personally or through their institution. Just because we are lucky enough to have access to a vast number of journals here at the OU, that doesn’t mean to say University X down the road or University Y overseas will. Depositing the full text of your research in ORO breaks down subscription barriers and opens it up to many more people that otherwise would never have been able to read it. And, as suggested by Professor Frost, you may also subsequently reap the rewards in terms of citations as well.

This morning I popped along to the Physics & Astronomy leg of the CEPSAR coffee morning to give (what was supposed to be) a short presentation on ORO. As it turned out there were quite a few questions and quite a bit of discussion, meaning all in all I was there for about 45 minutes instead of my allotted 15. I don’t think anyone minded though. I didn’t spot anyone falling asleep anyway.

After initially fearing the worst when a few people put down their coffees and scarpered when the title slide of my presentation appeared on-screen, I was pleased to see 15 or so people grab a chair and settle down to listen to what I had to say.  Bearing in mind we’ve also recently presented to PSSRI and E&ES, I think we can consider CEPSAR well and truly ORO’d for the time being.

Knowing that phisicists and astronomers are very used to using subject-based repositories and services like arXiv and ADS I paid quite a lot of attention to addressing the issue of ‘why deposit in yet another database?’ Actually, no one asked this question directly, but I still felt it was worth making this particular audience aware of the additional advantages to be had by depositing in your institutional repository. For me, there are three answers to this question:

1. To help showcase your institution’s research. Bearing in mind that I still come across people that weren’t aware the OU actually does any research, this is perhaps more important for us than many other institutions. Nevertheless (and I guess this depends on how allied the individual researcher feels to their current employer), I believe all academics should help where possible to contribute to their institution’s ’shop window’.

2. To be able to output from and link to ORO. For example, RSS feeds for your department, research centre, or research group; or perhaps a link through to your publications list on ORO from your own personal website. The point here is that ORO is not a static dumping ground. We’re thinking about and working on new ways to develop ORO as a service all the time. Unless your publications are in you won’t be able to benefit from these developments.

3. The citation advantage of Open Access. I spoke about recent research that suggests a significant citation advantage to making your work available Open Access. Yes this can be achieved by depositing your work in arXiv, but I suspect you improve your chances even further by also depositing in your institutional repository. Basically, don’t put all your eggs in one basket.

I was asked, quite rightly, about the time that all this takes – depositing in multiple repositories. To address this I included in my presentation a demo of how easy it is to import journal articles into ORO using the DOI (Digital Object Identifier), which nearly all online journal articles possess these days. In doing so, I also allayed fears that we expect our depositors to enter manually every single author of the paper, which, for physicists and astronomers can be quite a significant amount. The DOI import function will only pull in the first author, but that’s enough to deposit; library staff that are checking and verifying submitted items will do the rest.

All in all I felt most people seemed to find the presentation useful. Not only were there some interesting questions and discussion points, but some useful feedback as well. For example, when I mentioned that we are in the process of developing a cover sheet for full text items on ORO one member of the audience suggested we brand each page of the document (as arXiv does for instance), as many people might just bin the front page from a print-out. I’ll certainly discuss this further with my ORO colleagues as we develop this over the next few weeks.

Just in case anyone has subscribed to feeds from this blog and/or visited the site a few times to see what’s new and been disappointed to not find anything, I just thought I’d write a quick post to explain why we haven’t written much so far. Basically, we’re in the process of revamping the ORO front page, and, when it’s done, there will be a direct link to this blog from the main ORO homepage. Until then, we’re assuming that not many people will know about the blog; so, when ORO has its shiny new look we’ll begin posting more regularly. Thanks for your patience!