Open Research Online

As many people reading this post will know, HEFCE (the Higher Education Funding Council for England) have recently published their second consultation on the assessment and funding of research. This document, which sets out proposals for the Research Excellence Framework (REF), has helped cement in my mind the areas in which institutional repositories (Open Research Online [ORO], in the case of us here at the Open University) will play a crucial role.

The first, and perhaps least exciting role that institutional repositories (IRs) can and should play in the REF is an administrative one. So, the physical gathering together of publications for the submission process itself. Last time, for the Research Assessment Exercise (RAE) 2008, we (the Open University) were one of few institutions, I believe, to use our IR to populate HEFCE’s spreadsheet. I imagine, given that IRs are now much more mature and prevalent, that this will be more common-place across institutions for the REF.

HEFCE’s consultation document finally spells out exactly how they expect bibliometrics and citation analysis to contribute to the REF’s assessment process. The detail can be found in Annex C of the document, but essentially it will inform the process for certain subjects, and it seems to have been left fairly flexible as to how the panels can use the information. So, it’s a reality. Maximising citations is now in the interest of the REF-submitted researcher… and this is the second role that IRs can play. I’ve blogged in the past about evidence (anecdotal and non-anecdotal) for IRs helping to maximise citations, and so I’m not going to repeat the detail again. However, what I will say, is for those still sceptical about the citation advantage of open access, is that really a reason not to deposit in your IR? Even if there is only a small chance that your paper may pick up just one extra citation, surely that is worth the minute it takes to deposit the paper in your IR? There is nothing to lose by depositing, but potential citations to lose by not. To me, it’s a no-brainer.

Finally, I want to talk about impact. Not impact in the context of citations, but impact of research to society, the economy… to UK plc… beyond the realms of academic circles. This, according to HEFCE, will constitute 25% of the assessment in the REF. Not only does research need to create that impact in the first place, but we also need to be able to evidence it in our REF submissions. In my mind, this is rapidy becoming the most important aspect for the institutional repository to affect. Specifically, what proportion of those people in UK industry and society will have access to the academic journals, books, and proceedings in which you publish? How will they get to know about the (hopefully) world-leading research you are producing? And thus, how will that knowledge be transferred to society, for the benefit of the economy and UK plc, as HEFCE are so keen to see? Well, one way is to make sure as much of your research as possible is made openly available through your IR. Then, hopefully, all that work that is all too often locked up behind journal subscription barriers can begin to filter out and have the positive effects that the UK government want to see. This, of course, is the very essence of the open access “movement”, and I for one hope that HEFCE’s emphasis on “impact” in the REF will do wonders for it.

So, there is no doubt in my mind an institutional repository is an essential component in the engine that is “research assessment”. Indeed, to take the analogy further, it is like the petrol tank for the REF: keep it filled up with research output and you will travel greater distances than without it.

Last Thursday I attended a ’Repository Softwares Day’, organised by the Repositories Support Project (RSP). Held at the Museum of Science & Industry in Manchester, the event comprised a good mix of presentations and exhibits from key software developers such as EPrints, DSpace, Fedora, and so on.

Microsoft were there, talking about their complete cycle of solutions for the scholarly community. So, from tools to assist academics in researching and writing their paper, through to publishing platforms for hosting e-journals, and then finally their open-source repository software.

In terms of the repository end, I was left wondering whether there is room for more software – certainly in the UK, where EPrints and DSpace are very well established. Of more interest, in my opinion, was hearing about their article authoring add-in for Word 2007. Installing this enables the user to create very well structured technical documents (e.g. journal papers) in a way that captures additional metadata and semantic information at the authoring stage. The add-in also makes use of SWORD (Simple Web Service Offering for Repository Deposit), meaning an author could potentially deposit their article in whatever repository they choose from within Microsoft Word at the click of a button, assuming the repository is SWORD-compliant. This carries benefits for both the author (through ease of deposit) and Repository Managers/Administrators (possibly more full text). We will certainly be looking at making ORO SWORD-compliant in the coming months so as to take advantage of these features.

Another tool that I came away from the day feeling quite excited about is SNEEP (Social Networking Extensions for EPrints). I’d read bits and pieces about this plugin for EPrints (the software underpinning ORO) prior to attending this day, but I was really grateful of the opportunity to see an actual presentation on it. Basically, installing the plugin would give us three new features for ORO: the ability to comment, to bookmark, and tag individual eprints. The various permutations of who can and can’t add / see comments and tags are explained in the SNEEP Wiki pages; again, I expect us to look into the possibility of installing SNEEP for ORO in the coming months.

The final major point of interest from the day for me was hearing and learning more about the various CRIS (Current Research Information System) solutions on offer. I’m going to mention Symplectic here, not because I’m endorsing the product, but simply because I attended their presentation, so it’s the one I feel most informed about currently. I was particularly impressed by Symplectic’s Publications Management System, which automatically gathers publications information from key databases such as Web of Science and ‘asks’ academics by email whether the publications it has found belong to them. If the academic clicks ‘yes’ then the article can automatically pass through to their repository, giving them the option to attach full text beforehand. More needs to be known, but one can see how a system like this could take away a lot of the data entry needed to populate a repository – an element typically cited by academics as the biggest barrier to depositing their work. However, the depositor is still making a conscious decision to put their work in their repository, but at the click of a button rather than by filling in lots of data fields manually.

All told, this was an extremely informative and thoroughly enjoyable day!

Now that RAE2008 is over, like all UK universities we now have the REF firmly on our minds. Actually, it’s probably fair to say it was on the minds of many long before now. But it does appear to me to be all of sudden much higher on the agendas of not only research administrators but our academics as well - and not just those involved heavily in research strategy. Indeed, as we’re talking agendas, I’ve been to quite a few departmental meetings recently where the REF has been the main item for discussion.

At one such meeting last week I was extremely impressed by just how deeply a particular department of ours here at the OU is thinking about bibliometrics. In particular, they have not only begun to look at their citation counts for journal papers, but also non-journal material as well (edited books and book chapters, for example). This struck me as being very prepared and organised indeed, as up until that point (like most people, I think) I had been taking it for granted that this type of output will not be considered as part of the REF’s bibliometrics exercise. Actually, having come away from that meeting and reflected on things I still believe that to be the case, but it did make me think long and hard about the issue, which was certainly of great value.

At the same time as pondering this topic, and formulating plans to blog about it, I also came across a recently published paper on a proposed new method for bibliometric analysis of books. Although I confess at only having read the abstract, introduction and discussion (a bad habit from my student days!) this paper only served to strengthen my views that we are some way away from a reliable measure for non-journal material. Basically, the so called ‘libcitation’ approach described in the paper uses at its core a count of the libraries holding a given book. The theory is that in deciding what books to acquire for the audiences they serve, librarians make an informed decision based on the reputation of the authors and the prestige of the publishers that can be used as a basis for gauging impact. This seems to me miles away from the relative robustness of citation analysis of journal papers through Thomson’s Web of Science or Elsevier’s Scopus.

Non-journal material can of course be analysed through Web of Science and Scopus, but, as the authors of the above paper point out, these databases will only capture citations to books and book chapters by journals – not by other books and book chapters. Academics in the arts and some social sciences might quite rightly argue that this simply isn’t fair – that most of the citations they care about about will not be from journals. What’s more, Web of Science and Scopus don’t necessarily have the greatest coverage of journals in book-oriented subjects anyway. Also, I carried out a quick citation analysis comparison on a set of non-journal articles in both Web of Science and Scopus and found massive differences. In quite a few cases books and book chapters were not being picked up as having been cited at all by Web of Science’s journals, whereas in Scopus those same items had received tens and sometimes hundreds of citations. Granted, there are some differences between the journal coverage of these two databases, but not that much.

Lastly, there is Google Scholar. Most of you will be aware that when you find an item on Google Scholar it proudly tells you how many times it has been cited. Given the ubiquity of Google it is very tempting to take what it says as gospel. Well, don’t. Most people working in academic publishing or with bibliometrics will tell you the same thing – Google Scholar has potential, but as yet lacks editorial quality control. Although I have recently been left wondering just how much Web of Science and Scopus might be letting through their doors in order to achieve the coverage required to get the REF gig, I have no doubt that the editorial value they add is essential for a reliable platform upon which to base citation analysis.

So, this is why I think the bibliometrics element of the REF will only look at journal output. I’m not saying that future REFs won’t – indeed, I’m sure it won’t be too long before we have an all-encompassing Web of Science or Scopus product that pays better attention to non-journal material. But, for the purposes of REF 2012/2013 I just think it would be too ambitious and would be too full of holes. By all accounts HEFCE are having a hard enough time arriving at how to fairly analyse journal paper citations, let alone anything else.

A few weeks ago (the 28th of November to be exact), I attended ‘Towards the REF: Defining Bibliometric Requirements for Research Assessment’. The event was the latest in a series organised by King’s College London and supported by HEFCE. It was very well attended (200+); perhaps indicative of the uncertainty that still surrounds it. With the RAE results only days away it is probably bad timing on my part to be blogging about the REF; however, it is a reality and we (researchers and research administrators) do need to be thinking about it. I thought it was well worth circulating some of my notes from the day, as well as jotting down some thoughts about the part ORO has to play in all of this.

The day started off with a useful update from Graeme Rosenberg of HEFCE. One of the key things I picked up here was a move away from a twin-track approach, i.e. with Sciences largely assessed by bibliometrics and Arts & Social Sciences more by peer review. In fact, Graeme told us, the idea is to have a ‘family of tools’ that can be used for all disciplines, with bibliometrics being just one part of that ‘family’. Of course a burning question for HEFCE here is how to combine all elements of the ‘toolkit’ to provide a single indicator of research quality.

HEFCE seem to have great confidence in the potential of bibliometrics to contribute towards the REF. The basic idea is to establish a citation rate per paper, normalised against the average for that field, and then aggregate to produce an indicator. The indicator could then be used by expert panels as part of a wider portfolio of evidence.

A pilot study involving around 20 institutions was set up and has recently been completed. HEFCE expect to publish the results in the summer of 2009. Thereafter, further consultation will take place in Autumn 2009, with outcomes in early 2010. Another bibliometrics exercise will then follow, and the full REF exercise itself will take place in 2013 in order to drive funding from 2014 onwards.

Jonathan Adams from Evidence Ltd – the consultancy contracted by HEFCE to oversee the bibliometrics pilot exercise – gave a progress report presentation. A key theme that came out of what Jonathan was saying was the definite need for institutions to have either an institutional repository or a robust central publications database (a point reflected in a later presentation by Stuart Bolton – a consultant to JISC and HEFCE employed to look at the ICT implications of the REF). It is clear that either of these two methods is going to be crucial for collecting data for the REF. Also, it seems an advantage is to be had if your repository or database is somehow linked to your HR systems. ORO, of course, is linked to PIMS, which undoubtedly puts us in quite a strong position.

Jonathan highlighted some key issues that need to be addressed by analysis of the pilot exercise data. For example: will the REF cover all staff or selected staff? Will papers be linked to institution or individual researcher? Do you include staff that were present at an institution but have since moved on (and vice-versa)? Will the REF look at all publications by an individual or selected publications?

Wendy White and David Arrell from Southampton and Portsmouth Universities respectively spoke of their experiences of being involved with the REF pilot. Southampton have a very well established institutional repository and used it to gather together all the information needed, whereas Portsmouth do not and relied upon their RAE database. Wendy mentioned the importance of Southampton’s mandate in making sure their data was rich enough to gather all the information required. David, on the other hand, implied Portsmouth still have a decision to make in terms of whether or not to develop their repository or go with some other kind of publications database. Again, the message is that one or the other seems necessary.

Also of note from the day was Dr Henk Moed’s appraisal of citation data sources. Dr Moed is a bibliometrics expert from the University of Leiden and was commissioned by HEFCE to compare the two major commercial databases available for performing bibliometrics analysis: Thomson’s Web of Science and Elsevier’s Scopus. The basic conclusion of the work was that Scopus is a more than adequate substitute for Web of Science in terms of subject coverage. HEFCE are yet to decide which database to go with, or indeed whether to use a combination of the two.

All in all, the main take-home message for me was that HEFCE are still not clear on the exact detail of the REF (selected researchers or all researchers; selected publications or all publications; institutionally-linked or researcher-linked publications), but they did recommend that having a publications database or institutional repository in place would make it a lot easier for institutions to make their REF submission. So, it is looking as though ORO will have a dual role to play when it comes to the REF:

1. Open Access. As described above, a large part of the REF’s ‘family’ of research assessment tools is going to be bibliometrics. That is, how well cited your work is will be considered as a measure of quality, whether you agree with it or not. It follows, therefore, academics need to be thinking about citations; specifically, what can be done to maximise them. Making the full text of your work openly accessible through your institutional repository can help with this. It breaks down subscription barriers and makes your research visible to fellow academics that might not otherwise have access to it through their institution.

2. Research administration. Although the scope of the REF is not yet known, it is clear that having a central publications database or institutional repository available to collate the information required for our submission is key. As far as administration is concerned, the ‘worst case scenario’ would surely be having to submit all publications for all researchers present at the institution during the REF-defined period. If this possibility becomes a reality, making sure ORO is properly populated now has to be a priority.

It is often said that making your work available in an open access repository like ORO provides wider exposure to your research and a better chance of being well cited. I firmly believe this, and I refer again to a recent study that seems to support this hypothesis. More peer-reviewed research from information scientists and bibliometricians is certainly needed, but there is also a lot to be said for anecdotal evidence as well.

Ray Frost is Professor of Inorganic Materials in the School of Physical & Chemical Sciences at the Queensland University of Technology (QUT), Australia. In a recent post on QUT’s Library News for the School of IT we read that Professor Frost has found that regularly depositing his work in QUT ePrints (QUT’s equivalent of ORO) actually saves him time, simply because he no longer has to deal with numerous requests by email for individual personal copies of his papers. Furthermore, the amount of citations Professor Frost’s work receives has increased markedly since 2005, which also happens to be the year he began to make his work available open access through QUT ePrints.

So, depositing a record of your publications in ORO is good, but attaching the full text as well is even better. We always provide a link to the published version, but this is only of use to people who have a subscription to the journal in which you have published, either personally or through their institution. Just because we are lucky enough to have access to a vast number of journals here at the OU, that doesn’t mean to say University X down the road or University Y overseas will. Depositing the full text of your research in ORO breaks down subscription barriers and opens it up to many more people that otherwise would never have been able to read it. And, as suggested by Professor Frost, you may also subsequently reap the rewards in terms of citations as well.