Technology and Education Research Group

There was a session on bibliometrics, including a hands-on bit looking at citation records.

The consensus seems to be that the REF for the education area won’t use bibliographic metrics extensively in the way that other subjects will. I assume this is partly because the data isn’t well recorded and possibly that publication patterns are just not sufficiently rich to make the statistics reliable. So indications the REF is likely to be similar to the RAE in that each personal submission will be based on four world-class publications (I wish!). But to keep effort manageable the reviewing panel may only look in detail at one of these and they may use bibliometrics to choose.

Earlier Bridget had mentioned external grant income as important REF metric – again, avoids REF panel looking at detail since they can use the fact that external funding is acquired through peer review process, so existence of grant is some statement of quality.
 
Journal rankings were discussed. The impact factors given by databases didn’t seem to reflect views of people in the room. People are also submitting to journals that aren’t high impact or even to those that aren’t given a rating at all. (I wasn’t at all clear what this means – implication was that impact was so poor that not worth calculating. But I don’t think that follows, though it might mean that data was so thin that couldn’t be calculated and that would imply too few citations to measure.) Also emerged that some places people publish aren’t covered by databases – some open access journals may not yet be included and book chapters aren’t indexed. So such publications appear invisible.

Message appears mixed. ‘Best to publish in journals where you wish to be part of the conversation’. But on the other hand, given a choice, better to publish in prestigious journal.

Mention was made again that book chapters don’t appear in citations. (Actually, the big databases index journals and so only collect citations between journals and not out to books. But the new kid Google Scholar does count citations to books, although its journal citations probably are nothing like as complete.) I’ve been looking for my own publications (vanity, vanity) and found that the book chapters are all but invisible on the web, though journal publications much easily found. So putting them in ORO is a good thing – Google Scholar indexes ORO and demonstrably increases chance of finding stuff.

Still on a personal note, my citation record actually appears pretty good – if you look at it the right way. Papers I published twenty years ago are still being cited, the total number of citations is pretty good and some journals have high impact factors. Slightly unfortunate that it includes no recent papers and none in the CREET area! So I guess none of it will count 

[BTW I also noticed that Bridget Heywood doesn’t appear to feature in ORO  ]

The talk by Colin Smith (ORO manager) had some points of interest.

One is that ORO now belongs to the research office rather than the library, and now seems to figure largely in the OUs attempt to make sense of its research activity. But it still does have a role to play in providing open access to research, especially for those in less well-off institutions than the OU who have much poorer access to journals because of the high cost of subscriptions. A study was quoted which showed that papers available in open access repositories received a higher citation rate than those that weren’t open access (mean 9.04 citations for open access papers, 5.76 citations for ‘toll access’ papers).

Full text helps. The ORO people will check rights position as part of submission process so we don’t need to worry too much about that aspect. Suggestion was that final draft (ie that accepted for publication but before copy editing by journal editors) is most often OK. As part of ‘your publication routine’ you should submit to ORO when you return final draft to journal and include the final draft as full text. [Publication routine – I wish!]

I asked and was told that it was OK to use ORO for papers of any age and not only for work carried out at OU. So ORO could be used to maintain your full publication list in format you would expect for official OU CV and several things help make that sensible. The format of display of your publications from an ORO search has been improved. You can use a url to ORO from say your personal web page to send people to your publication list. The magic of RSS can be used to embed your publications list in a page. With a bit of cleverness in constructing a search (and then turning into RSS), combined publication lists are possible for a group of people, eg a TERG list. I think Colin said this was better done by ORO staff.

Adding old papers should be relatively painless since, rather than enter all details manually, you can give a DOI and ORO will suck in all bibliographic details from one or other database. [I’ve just been trying that – not quite as painless as promised, but a start.]

Colin said that publishing full text of book chapters was generally possible though ORO had to approach publishers on case-by-case basis. Given the invisibility of book chapters on the web and in citation indices otherwise, this has to be a good thing.

If you include your email address in ORO record, there will be a ‘request a copy’ button which allows you to distribute personal copies to anyone, which sidesteps copyright restrictions in any case (apparently – don’t blame me when you find yourself in the Scrubs!).

An issue was raised about non-refereed stuff. I got the impression there was pressure allow ORO to include other types of publication such as commissioned reports, internal reports etc to give them visibility. But probably also pressure to maintain a quality cachet. Sounds like this might take a while to resolve.

There was a talk about journals and reviewing but I didn’t get much to take away from this (sorry — after lunch ). There was general agreement that being a reviewer was good for the soul. But also comment that well-known names tended to get too many papers for review, often not close enough to their field.

Other take-home messages:

• Get questions right at start of research and publication quality will follow (Bridget said this)
• Read journals, choose where to be part of the conversation [but journals are so slow… people like Tony Hirst and Martin Weller know that the conversation is really going on in blogs, Twitter streams, etc]

On reflection, I thought it strange that issues that loomed large a year or so ago weren’t mentioned. Scholarship or research? Genesis research? Institutional research? Some of these I thought were quite problematic for CREET but they weren’t mentioned. Was this because everyone else (certainly not me!) now knows the answer? Or have they just subsided as attention has switched to this year’s concerns?

One of Bridget’s themes was the ‘story’ or ‘narrative’ of our research. She said several times that ‘it is no longer acceptable to say “I do research”’ and that ‘Research is no longer a hobby’ – your research must be clearly situated so that it answers a need of society. Ask not what your country can do for you, but what you can do for UK plc.

Although there was a good deal of talk about these ‘narratives’ and ‘stories’, I was left a bit confused.

I wasn’t at all clear on where this ‘narrative’ would actually exist. Narratives were written for purposes of the last RAE – but these were quite small pieces of text, written by a few people that I suspect put a post hoc spin on research that was probably never intended to be part of a coherent strategy. Tail wagging the dog is a phrase running through my mind – except I can’t even tell whether the tale is the tail or the dog.

And what form should such a narrative take? Is it words for the RAE/REF? The yearly OU research publication? Press releases? Words on faculty / department web sites? Talks that Bridget gives? Dinner party conversations? (Dinner party conversations did feature several times as examples). Or maybe these research ‘rooms’?

I don’t know whether the C&S dept attempt at using podcasts for dept web site is the sort of thing Bridget had in mind or not.

Back to the idea of there being a research ‘narrative’ and that this should be focussed on explaining how the research is answering a problem in society. If this is taken as a steer to the sort of language that should be used in writing grant bids, then fair enough. If the REF has to go through the same post hoc exercise to make sense of each subject areas submission, then fair enough. But if it means there will be some ‘management’ of what people choose to research, then it sounds a bit tricky.

Bridget did mention academic freedom and said that this new conception wasn’t a threat, but I wasn’t sure I got her argument which seemed to be on the lines of: yes, you will always be free to publish what you want/can, but you are not free to choose what you do with public monies so your activities must be accountable and so you will have to justify your research activity against these new criteria. So you are free to publish the results of work that you are not free to carry out…

To be fair to Bridget, she was at pains to point out that justifying most research should not be a problem, just a matter of applying the right spin (‘articulate the relationships between topics and themes’ and ‘narrating these stories to the public and UK plc’). The OU’s mission statement is so broad that phrases like ‘social justice’ can be used as hooks to situate many areas of educational research. ‘Open as to methods’ justifies anything to do with ed tech; ‘open as to places’ justifies anything to do with mobiles!

About conferences, Bridget said:

• attend key conferences
• plan conference strategy to promote OU research
• plan conversations with the movers and shakers
• invite externals to visit OU

BH specifically mentioned that she was inviting one of the external speakers Christine Borgman from OU Making Connections to the OU.

I went to the CREET event on Wed 17th June. This included a presentation (performance?) from Bridget Heywood on research strategy, a presentation about ORO and presentations on publications, one about processes of reviewing and the other about metrics. Sessions were recorded (for the Knowledge Network? CREET intranet?) but I haven’t tracked them down yet.

Bridget’s talk was impressive – she spoke for nearly an hour without PowerPoint or notes. Some repetition but no hesitation. I think it was intended as an upbeat pep talk but parts came over as a harangue (!). Detail is probably in Research Strategy document for Senate.

She started with an overview of the OU’s performance in RAE and general research prospects. There was a mixture of good and bad news. Good news is the improvement in grant success over the past few years and the increase in our overall research ranking. Bad news seems to lie in squeeze on institution’s own money to spend on research. In particular, we do very badly on externally funded studentships compared to other universities, so badly that it somehow affects overall research income via some HEFCE formula (I think!). The OU is using its own resources over the next few years to protect the number of studentships, but the message seemed to be that getting PhD students is a priority. Other than this, in the current financial crisis the OU is unlikely to be able to invest own money in research.

The PVC-level view is that the OU should have a number of big themes and assemble its research portfolio to support these. Themes mentioned were quite broad; I noted:

• A life of quality
• Digital identities
• Interfaces

And she was encouraging people to be creative in finding ways of linking activity to these themes.

Focus should be on obtaining a few major grants rather than lots of odd people doing their own thing to the tune of a few tens of thousands. This is because the effort and costs required to obtain grants makes in not worth doing for small amounts. [My cynical view is that part of the problem lies in OU bureaucracy – if that wasn’t so arcane, small grants wouldn’t be seen as a problem.]

So one of her take home messages was that we should collaborate more across the University to create cross-disciplinary bids. [I get muddled on the values attached to these words but I think generally ‘interdisciplinary’ is better than ‘multidisciplinary’; Bridget said ‘cross-disciplinary’ and I have no idea where that sits.] Perhaps surprisingly, she wasn’t that upbeat on inter-institution collaboration. She sees that as often meaning that the OU does the work, but someone else gets the money and the credit. So from OU plc perspective, it is much better if the project is hosted at the OU.

[Actually, from personal experience that applies within the OU as well. There is a big difference between being the principal investigator and the collaborator on a bid – the money, power and recognition follows the PI. Working across units isn’t easy; I’ve heard it said that Systems suffered from being too good a collaborating with other people.]

Collaborations can include visits, student exchanges etc, but must be evidenced by publications.

One of the ways in which Bridget will promote this is by some sort of embodiment of these themes as virtual(?) rooms which will at least list activity in different parts of the University. (I think there was some implication that this would somehow be done through ORO but that later that seemed a bit of a surprise to the ORO chap). Presumably this would be intended both to make ourselves aware of what we currently do and open up possibilities for collaboration, but also to be a public face for research activity. Yet another place besides department and faculty and project web sites…

Bridget mentioned external grant income as important REF metric since that was seen as a quality kite mark attached to research – if successfully funded, it will have been through a peer review process. So external funding is important to chase not just for its own sake, but for positive feedback in future funding. [But this seems problematic when set against the call to stop footling with small bids – it may be OK for the institution to be successful with a few huge bids, but that means only a few big name individuals get credit. The rest of the minions end up doing the work but don’t get the career or research recognition attached to being the PI. And I can’t see how they get on the gravy train – I would have thought it impossible to initiate a big bid unless you have a track record of success in smaller ones.]

Some other messages:

• need to be smarter
• need to collaborate
• increase visibility
• what did my research do to solve a problem?
• what contribution has my research made?
• what evidence of impact do I have about my research?

Tomorrow Thursday 18th June 2009. David Gorham Library (Venables N1015). 2.30 p.m. Martin Weller will give an EATING talk titled:

It’s not blogging or twittering – it’s identity creation

Martin says:

“I will look at how a distributed online identity is created through the use of tools such as blogging and twitter. The manner in which we share professional and personal elements, and combine these in different ways provides a powerful means for developing your personal network which may be your most useful asset. The ability to easily create, share and reuse also has implications for teaching.”

Karen, Helen and Judith have had a paper accepted for ALT-C 2009. The paper is titled:

Students’ experiences of wikis for a collaborative project: technology choice, evidence and change.

The paper concerns the use of wikis by student project groups in T209.

Tony Hirst and Joss Winn (from Lincoln University) have received funding from JISC for the JISCPress project. The project, led by Joss, will provide an online commenting environment for JISC documents.  It is based on the existing Write To Reply environment, developed by Joss and Tony using WordPress.

Many congratulations to Joss and Tony.

PhD student Paul Herring has recently had a journal paper published. Paul, who is supervised by Roger, Karen and Kieron Sheehy from FELS, has recently passed his probation.  Paul’s paper is:

Herring, P. (2009). “Can computer assisted learning help autistic children communicate?” SLD experience(53): 23-28.