Technology and Education Research Group

Attendees: Karen, Judith, Mirabelle, John, Giselle, Richard

As usual, we started with news and updates:

• Richard is giving a conference paper at an IEEE conference in Venice in May.
• Mirabelle and colleagues are to write a book chapter on feedback in the book Reconceptualising feedback in higher education: towards an agenda for effective change, to be published by Routledge.
• Judith, Giselle and their colleague Soraya have submitted a paper to the European Journal of Open Distance and eLearning (EURODL) for a special themed issue on creativity and open educational resources. Their paper is a case study of the video creation work in Block 5 of T215.
• Giselle, with Tina Wilson, is writing a book chapter on Web 2.0 and OER.
• Jon will be trialing his eSTEeM confidence-based marking project through T216, and is also intending to work on converting the T184 ECA into a part computer-marked exam with the objective of improving retention.
• Karen mentioned a possible new part-time PhD student, who has proposed a project about VLEs in schools.

 The meeting then moved to general discussions. First, stimulated by Jon’s discussion of his T184 project, we talked generally about examinations in OU modules. Giselle talked about the new VMware course for presentation in 2012B that she is currently chairing.

 We finished the meeting with a discussion on finance. It was agreed that Mirabelle’s Survey Monkey licence should be extended for a further year and that Karen’s licence should also be extended and made available for other TERG members to use. Karen asked members to identify whether there were any conferences they would like to attend.

Assessing ‘alternative media’ elements: is there a generic model?

(Submitted by Soraya, Judith and Clem)

During the production of T215 ‘Communication and Information Technologies’, the Module Team developed a new assessment model which is applied on each of the five TMAs.  This assessment model seems interesting because it is used for both text-based and multimedia-based assessment elements so it provides a consistent framework within which students can work and critique their own assessment outputs. We are interested in investigating how non-text assessment media ( such as video, PowerPoint presentations, podcasts, group working …etc) are assessed in other STEM modules and whether T215’s assessment model could be applied equally as well in these other contexts too.

 From previous discussions we had with other module teams across the faculty it emerged that they often struggle to develop an appropriate assessment model for non-text assignments. In this project we seek to:

• bring together the body of relevant knowledge that is emerging as new technology provides opportunities for more innovative assessment
• investigate the feasibility of developing a common model that can be applied across a range of assessments using different media on other modules.

 Our work initially would be to conduct a survey of the assessment models of STEM modules in the Open University that include alternative media assessment elements. We would then explore whether T215’s assessment framework could be applied in the assessment on these other modules or whether another generic model might emerge from our work. If a generic model can be developed module teams would not need to spend time developing new strategies for assessment resulting in improved efficiency and cost effectiveness in line with the OU’s strategic objectives.

This proposal has been accepted in broad terms but requires some development . The eSTEeM team consider it a little too restrictive and they advise us to “explore other domains where assessment of these materials is more commonplace”.  So we are asked to “redefine and broaden the scope of your initial research before converging on the STEM subjects”.

How do eLearning technologies affect students’ perception of support from their tutor?

Tutors have an increasing choice of digital communication technologies with which to support their students – for example, tutor group forums, on-line tutorials, social bookmarking, SMS messages, virtual worlds, podcasts and video casts – but little is known about the effect these technologies have on student perception of the support model. Does their use increase student satisfaction and feeling of community with their tutor and fellow students, or do students perceive them as an unwelcome distraction and an additional workload with little return?

This project will follow one tutor group of students studying T215 Communication and information technologies in 2011 as the tutor adopts an exclusively distance support model (no face-to-face tutorials). It will examine the level of student interaction using the eLearning support tools and their effect on student satisfaction. The key support tool will be Elluminate which the tutor will use to provide more frequent (though shorter) tutorial sessions and increased flexibility for support at critical points in the module than the traditional face-to-face model. The tutor will also use phone, SMS and the tutor group VLE forum.

The project’s investigation will be through analysis of records (tutor logs, forum messages, Elluminate recordings), a student survey and a focus group.  During the progress of the project the experience will be shared with other T215 tutors through the VLE tutor forum. At the end of the project the findings will be disseminated through conference and journal papers.

This proposal was rejected: it is considered too narrow in that it “concentrates on a small cohort of students and is highly dependent on their response”. The eSTEeM Scrutiny Group do, however, agree that “this is an important area”. Nevertheless, I do intend to continue with this project outside the framework of eSTEeM.

 Judith

Another proposal I submitted to eSTEeM:

Multiple choice questions (MCQ) are the basic fare of e-assessment. MCQs are robust and easy to implement, but are pedagogically not ideal: open questions are preferable but automated marking of free text answers is problematic.

A possible squaring of this circle is to appropriate the technique of confidence-based marking (CBM).

In CBM, a student both selects an answer and also their level of confidence: they score full marks for knowing that they know the correct answer, some credit for a tentative correct answer but are penalised if they believe they know the answer but get it wrong. There are several motivations for CBM: it rewards care and effort so engendering greater engagement, it encourages reflective learning.

I believe we can appropriate CBM and, with one simple change, enrol it for quite a different end. Here the MCQ is presented in two stages. Initially, the question is presented but with no answer options visible; instead the student must set their confidence level that they know the answer. Only then are the possible answers are revealed and the student answers as a normal MCQ. The marking scheme follows standard CBM practice.

Consider what this means. Mechanically the question remains a simple MCQ. This means that answer matching is trivial and robust, questions are easy to implement, and existing question banks can be reused. However, to the student, the question is effectively transformed from closed MCQ to an open question. They need to formulate an answer first before they can decide their confidence in their answer. This means that they will decide their answer in the absence of any positive or negative clues. They cannot work back from the answer. They will not be led into misconceptions. Instead they have little choice but to answer the question as set.

This project plans to:

• implement a CBM question type in OU Moodle Quiz and/or OpenMark

• trial introduction in course assessment, with controlled experimental design if possible

• record measures such as assignment scores, time spent on task

• survey or interview to probe attitudinal aspects.

The premise is that students using CBM will engage better with questions, improve their learning, and become more reflective learners. Measures will be collected to probe these questions, although since demonstrating learning gains is notoriously difficult it may not be possible to demonstrate clear-cut outcomes. However, since any CBM question can also be presented as standard MCQ, there is potential for controlled experimental design which is rarely the case for pedagogic interventions.

This has been more successful — the eSTEeM reviewers ‘would like you to take your project forward’ but they also ‘identified possibilities for refinement’…

I submitted the following proposal to eSTEeM:

Many OU courses suffer from poor retention. Folk wisdom is that exams and the end-of-module assessment represent a significant hurdle to students, who appear to be deterred by perceived difficulty and do not submit. On the other hand, CMAs and particularly iCMAs are typically attempted by most students.

Can retention therefore be improved by offering part of the end of module assessment in the form of an iCMA?

The specific context to be explored is T184 Robotics and the meaning of life, a 10-point 10-week course with currently a mid-course iCMA and a final written EMA. From 2011E the final assessment will be take the form of a further iCMA (corresponding to short-answer questions in earlier ECAs) and reduced written EMA (retaining programming and essay questions). Students will obtain detailed feedback on their iCMA after cut-off, rather than the anodyne performance profile currently provided three months later.

The hypothesis to be tested is that this change will result in improved engagement and confidence, feeding through to improved retention and progression measures.

Measures to be looked at:

• standard OU stats on submission, retention and progression (for comparison with pre-intervention presentations);

• online attitudinal survey;

• follow-up email contact (structured questions) with non-completing students.

Another course, TM190 The story of maths, is making the same change to its end of course assessment while other companion courses in the Relevant Knowledge programme (eg T183, T189) are not changing assessment. This provides points of comparison for completion measures although it is not planned to survey students on these other courses.

This has been put into the category of ‘interesting and may be worth following up’; will see what transpires.

The intervention is going ahead regardless, so the issue really is how best to evaluate it to capitalise on the opportunity. Unfortunately I missed the chance to survey previous presentations (ie before intervention) and a combination of low numbers and already very good retention/progress may make it difficult to spot any effect. Suggestions for how to proceed welcomed!

eSTEeM is a new Science and Maths, Computing & Technology initiative to support teaching and learning in (have you guessed it yet?) the STEM subject areas. In many ways it is the son-of-COLMSCT, although unfortunately hasn’t the same level of funding available. A call has been out for projects, and members of TERG have submitted proposals; results just in and details in following posts.

Jon Rosewell submitted a bid for JISC funding for an iSpot related proposal:

iSpot User-Contributed Keys

iSpot is a social networking website that connects beginners in natural history with experts who help identify organisms from photographs and descriptions. We also provide iSpot Bayesian keys, a novel form of
interactive key to help identify plants and animals. These keys are now available through the web on the desktop and on mobile phones. By using this new form of key, we have also made the creation of keys
much easier.

The proposed project will open up the authoring, reviewing and development of keys to a much wider public. We will do this by combining the commenting and discussion features already on iSpot with the existing authoring features for keys. This will make it possible for any user to start a new key and for others to comment and discuss, allowing groups of users to collaborate on improving keys. We will use experts from our existing network of collaborators to seed this activity in addition to encouraging the full iSpot community to create, use and comment on new keys.

By empowering many more individuals to create identification keys, we can increase their number, their taxonomic range and their use. This has the potential to feed through to more extensive collection of
biodiversity data.

Unfortunately this wasn’t successful, but another iSpot bid in the same funding call was funded. This was for a project called iSpot local; read more on Doug Clow‘s blog.

Leuven town hall

The E-xcellence project on quality in e-learning started on January 1st 2011. The OU contribution is led by Keith W, with Karen, Jon, Giselle and Wendy forming the rest of the team.

Keith and Karen went to Leuven, in Belgium, for the project start-up meeting on 17th and 18th January.

Leuven is a lovely old university town with ornate buildings and cobbled streets. The meeting was at the Irish College which was founded by Irish monks - this was also a lovely building.

The project meeting was attended by representatives from the many project partners and associates. There were attendees from Latvia, Poland, Lithuania, Equador … The project partners gave presentations on their particular workpackages. Keith presented for the OU, and included a great whistle-stop tour of ideas and new developments in e-learning.

The project dinner

On the evening of the first day there was a project dinner in a Leuven restaurant. This was a very pleasant event and good for getting to know the other people, and finding out about education, and other aspects of life,  in the different countries represented. 

The OU team will be developing a set of self-assessment guidelines and benchmarks for quality in e-learning. The project will also run a number of local workshops where an institution can carry out a supported review of their e-learning.

Attendees: Karen, Judith, Mirabelle, Wendy, Jon, and Giselle – with Elaine on the phone from Ireland.

We started with news and updates. Jon reported that the iSpot project had won a prize, and was in the running for a second one.  Jon was in the process of bidding to JISC for further funding for his part of  iSpot. He and ChrisB had aubmitted an abstract for the CAL conference, about the history of online conferencing at the OU. Giselle continues to work with Tina Wilson on ethics and OERs. They have papers to CAL and a book chapter under review. Giselle, Judith and Soraya have a conference paper accepted for the SRHE (Society for Research in Higher Education). Wendy’s tablet PC project continues well, using a Ning network for communication among the 9 tutors. She has papers with Birgit Loch from Australia, to be presented at the Ascilite and ATEC conferences there. Mirabelle has a paper about feedback at the ISSOTL conference, together with colleagues in the Spanish Department. The proposed book on feedback is being discussed with possible publishers. Mirabelle has run a regional staff development workshop in Scotland, and continues her evaluation work on T215. Karen’s book is now in print. She is working on a bid with John Pettit related to real-time conferencing (e.g. using Elluminate). The paper with John Wookthorpe and two T175 ALs, about wikis, is now published. Judith has been working with Helen, Frances and Karen to write a paper about Elluminate in T175. They are using a wiki for this, which is working well. Elaine is in discussions with Arts staff tutors for a bid/seminar related to research on assessment.

For ‘other news’ Giselle reported that all the HEA subject centres have had funding withdrawn, so are likely to close. Karen reported briefly on the 2-year, EU-funded E-xcellence Next project - looking at quality in e-learning – which will start in January. Keith Williams is the lead for the OU, with Karen also involved. Giselle and Wendy are interested in contributing to the project.

We moved on to group discussion of  a journal paper suggested by Mirabelle:

Nicol, David J. and Macfarlane-Dick, Debra(2006) ‘Formative assessment and self-regulated learning: a model and seven principles of good feedback practice’, Studies in Higher Education, 31: 2, 199 — 218.

Mirabelle led the discussion, beginning by clarifying the meanings of ‘formative’ and ‘summative’ assessment and ‘self-regulated learning’. After an interesting discussion, we agreed that it might be valuable to look at the seven principles and try to find examples of them.

We finished the meeting with a few points on group business e.g. finance, web presence. Karen reminded members to complete the OU audit of research. The first stage is to make sure all publications are in ORO (Open Research Online). She also mentioned the MCT and Science Faculty ‘Esteem’ initiative for STEM-related work. This will provide support for various kinds of projects.

iSpot, the OU’s social networking for nature website, recently won a ‘Panda’ award.

The Wildscreen Festival is the equivalent of the Oscars for film of the natural world. There are about 20 categories; iSpot won the ‘ARKive new media’ category. We were in good company: the Jury’s special prize went to the David Attenborough’s series Life (a BBC/OU/others production). So didn’t the OU do well!

The judges said of iSpot:

The winner takes maximum advantage of what the internet uniquely affords users – being both participatory and collaborative. The judges loved the user generated content, and the fact that users are able to take an active role in the curation and validation of the content. A virtual community of interest putting amateurs in contact with experts in an unintimidating environment.

More info in the OU press release.

BTW:  No, I didn’t get to go to the ‘glittering award ceremony’. But we are also shortlisted for the Times Higher Education award for ‘Outstanding ICT Initiative of the Year’ award and I will be going to that at the end of November — so fingers crossed!