There have been several studies indicating that deposit in an institutional repository (like our own ORO) does provide a citation advantage.  The one I usually refer to when I’m promoting ORO is the one conducted by Lars Kullman “The Effect of Open Access on Citation Rates of Self archived Articles at Chalmers” which finds that “self-archived articles have a 22% higher citation rate than articles that were not self-archived“.  

It’s something I’ve often wanted to do in relation to ORO, but haven’t got round to before now, not least because it’s a tricky thing to investigate.  There are a number of things to bear in mind when evaluating the impact deposit in a repository may have on citations of any particular paper, they include:

1. The disciplinary variance of citations – some disciplines cite heavier than others and this might skew the results if a certain discipline is more heavily represented in the dataset being analysed.
2. Citations increase over time – obviously older papers that have been in circulation longer will accrue more citations.
3. Items deposited in the institutional repository as Open Access may also be Open Access in another repository (e.g. arXiv) or published Gold Open Access on the publisher’s website.  So it might not be their presence in ORO that is making the difference!
4. Author’s may self select which papers are made Open Access in a repository i.e. only the better papers are made openly available.  And these (you would hope!) are the ones that will get more citations.

And at this point I normally give up and go back to the nitty gritty of running ORO – I’m not a researcher.  However, we’ve recently been using the bibliometrics tool SciVal (an Elsevier product based on Scopus data) and evaluating how that can support individuals, research groups and the University to benchmark their research.  So I decided to run some data through it and see what came out, and this is what I found.

• Date range: 1996-2015
• Field Weight Citation Impact (FWCI) – a relative measure of citations normalised by discipline, publication type and published date where 1 is the average e.g. a FWCI of 1.48 means it has been 48% more cited than expected
• All tables ranked by FWCI
• Data available here: ORB_OROSciValData

OK, let’s unpick that a bit.

• OU research outputs is everything in Scopus/Scival that has been affiliated to an OU researcher.  SciVal/Scopus is only a subset of everything that is published and is skewed towards the sciences – coverage of the Arts and Social Sciences is not great!
• OU research outputs will capture all outputs from authors with an OU affiliation – these may not be research contracted staff (e.g. Academic-related staff or Associate Lecturers) e.g. some of these people may not be able to deposit in ORO.
• ORO currently captures around two thirds of everything that gets published by OU researchers.
• ORO captures research outputs of current OU researchers published before they joined the OU – so some ORO items will not have an OU affiliation and therefore not be in the SciVal/Scopus dataset.
• The ORO research outputs with DOI are only those items in ORO with a DOI – there are a lot of items in ORO that don’t have DOIs – but that’s the only way I could import them into SciVal to do the analysis!  I don’t know but maybe there would be a correlation between possession of a DOI and citability of an output.

So that’s some context, but the data there does seem to indicate that items in ORO get a citation advantage compared to all OU research outputs indexed in SciVal/Scopus.  So I dug a bit deeper and tried to get some data on those publications that were only on ORO or only in the SciVal/Scopus OU dataset.

That appears to be even more emphatic, but it may well be self selecting – people may only be putting on ORO the stuff they want people to see!

My appetite well and truly wetted I tried to see whether being Open Access in ORO differed from just being in ORO (i.e. metadata only).

Note: you’ll see the numbers don’t add up – that’s because there were a number of duplicates that appeared in both sets (i.e. they had been deposited in ORO as both open access and metadata only). 

Not much at first glance, it might appear that just getting the metadata visible to Google is enough to get the research output noticed and citable.  However,  the FWCI value for Open Access is a bit higher than metadata only, especially when considered in relation with Citations per Publication.  I’ve written here before how Arts and Social Sciences full text gets downloaded more than STEM and, given differences in citation practices, I wonder if that is happening here too?

Finally, I tried to tie this up by seeing whether appearance in high impact factor journals had a determining influence in the varying citation measures across these datasets.  So I used one of the metrics in SciVal (Publications in Top 10 Journal Percentiles% – SNIP) to see whether this might be the case.

And generally yes, there is a trend that those outputs published in journals in the top percentiles in their subject area do get a higher FWCI value.

However, there is one notable exception: those items that are Open Access items in in ORO have a higher FWCI value despite having a lower Journal percentile ranking than metadata only deposits . They also have very similar Journal percentile ranking but significantly higher FWCI value than all OU research outputs with DOI.

So I think there are a couple of things to take away:

• All items deposited in ORO do appear to get higher citations than those that aren’t, however this may be due to the fact that these items are more likely to be published in more prestigious journals in the field, rather than the fact they have been deposited in ORO.  So it’s likely that there is a self selecting element here – researchers may be more inclined to deposit items in ORO if they appear in more prestigious journals.
• Open Access items in ORO appear to contradict this general trend. Open access deposit in ORO may lead to more citations irrespective of the apparent quality of the journal they appear in.