Technology and Education Research Group

Assessing ‘alternative media’ elements: is there a generic model?

(Submitted by Soraya, Judith and Clem)

During the production of T215 ‘Communication and Information Technologies’, the Module Team developed a new assessment model which is applied on each of the five TMAs.  This assessment model seems interesting because it is used for both text-based and multimedia-based assessment elements so it provides a consistent framework within which students can work and critique their own assessment outputs. We are interested in investigating how non-text assessment media ( such as video, PowerPoint presentations, podcasts, group working …etc) are assessed in other STEM modules and whether T215’s assessment model could be applied equally as well in these other contexts too.

 From previous discussions we had with other module teams across the faculty it emerged that they often struggle to develop an appropriate assessment model for non-text assignments. In this project we seek to:

• bring together the body of relevant knowledge that is emerging as new technology provides opportunities for more innovative assessment
• investigate the feasibility of developing a common model that can be applied across a range of assessments using different media on other modules.

 Our work initially would be to conduct a survey of the assessment models of STEM modules in the Open University that include alternative media assessment elements. We would then explore whether T215’s assessment framework could be applied in the assessment on these other modules or whether another generic model might emerge from our work. If a generic model can be developed module teams would not need to spend time developing new strategies for assessment resulting in improved efficiency and cost effectiveness in line with the OU’s strategic objectives.

This proposal has been accepted in broad terms but requires some development . The eSTEeM team consider it a little too restrictive and they advise us to “explore other domains where assessment of these materials is more commonplace”.  So we are asked to “redefine and broaden the scope of your initial research before converging on the STEM subjects”.

How do eLearning technologies affect students’ perception of support from their tutor?

Tutors have an increasing choice of digital communication technologies with which to support their students – for example, tutor group forums, on-line tutorials, social bookmarking, SMS messages, virtual worlds, podcasts and video casts – but little is known about the effect these technologies have on student perception of the support model. Does their use increase student satisfaction and feeling of community with their tutor and fellow students, or do students perceive them as an unwelcome distraction and an additional workload with little return?

This project will follow one tutor group of students studying T215 Communication and information technologies in 2011 as the tutor adopts an exclusively distance support model (no face-to-face tutorials). It will examine the level of student interaction using the eLearning support tools and their effect on student satisfaction. The key support tool will be Elluminate which the tutor will use to provide more frequent (though shorter) tutorial sessions and increased flexibility for support at critical points in the module than the traditional face-to-face model. The tutor will also use phone, SMS and the tutor group VLE forum.

The project’s investigation will be through analysis of records (tutor logs, forum messages, Elluminate recordings), a student survey and a focus group.  During the progress of the project the experience will be shared with other T215 tutors through the VLE tutor forum. At the end of the project the findings will be disseminated through conference and journal papers.

This proposal was rejected: it is considered too narrow in that it “concentrates on a small cohort of students and is highly dependent on their response”. The eSTEeM Scrutiny Group do, however, agree that “this is an important area”. Nevertheless, I do intend to continue with this project outside the framework of eSTEeM.

 Judith

Another proposal I submitted to eSTEeM:

Multiple choice questions (MCQ) are the basic fare of e-assessment. MCQs are robust and easy to implement, but are pedagogically not ideal: open questions are preferable but automated marking of free text answers is problematic.

A possible squaring of this circle is to appropriate the technique of confidence-based marking (CBM).

In CBM, a student both selects an answer and also their level of confidence: they score full marks for knowing that they know the correct answer, some credit for a tentative correct answer but are penalised if they believe they know the answer but get it wrong. There are several motivations for CBM: it rewards care and effort so engendering greater engagement, it encourages reflective learning.

I believe we can appropriate CBM and, with one simple change, enrol it for quite a different end. Here the MCQ is presented in two stages. Initially, the question is presented but with no answer options visible; instead the student must set their confidence level that they know the answer. Only then are the possible answers are revealed and the student answers as a normal MCQ. The marking scheme follows standard CBM practice.

Consider what this means. Mechanically the question remains a simple MCQ. This means that answer matching is trivial and robust, questions are easy to implement, and existing question banks can be reused. However, to the student, the question is effectively transformed from closed MCQ to an open question. They need to formulate an answer first before they can decide their confidence in their answer. This means that they will decide their answer in the absence of any positive or negative clues. They cannot work back from the answer. They will not be led into misconceptions. Instead they have little choice but to answer the question as set.

This project plans to:

• implement a CBM question type in OU Moodle Quiz and/or OpenMark

• trial introduction in course assessment, with controlled experimental design if possible

• record measures such as assignment scores, time spent on task

• survey or interview to probe attitudinal aspects.

The premise is that students using CBM will engage better with questions, improve their learning, and become more reflective learners. Measures will be collected to probe these questions, although since demonstrating learning gains is notoriously difficult it may not be possible to demonstrate clear-cut outcomes. However, since any CBM question can also be presented as standard MCQ, there is potential for controlled experimental design which is rarely the case for pedagogic interventions.

This has been more successful — the eSTEeM reviewers ‘would like you to take your project forward’ but they also ‘identified possibilities for refinement’…

I submitted the following proposal to eSTEeM:

Many OU courses suffer from poor retention. Folk wisdom is that exams and the end-of-module assessment represent a significant hurdle to students, who appear to be deterred by perceived difficulty and do not submit. On the other hand, CMAs and particularly iCMAs are typically attempted by most students.

Can retention therefore be improved by offering part of the end of module assessment in the form of an iCMA?

The specific context to be explored is T184 Robotics and the meaning of life, a 10-point 10-week course with currently a mid-course iCMA and a final written EMA. From 2011E the final assessment will be take the form of a further iCMA (corresponding to short-answer questions in earlier ECAs) and reduced written EMA (retaining programming and essay questions). Students will obtain detailed feedback on their iCMA after cut-off, rather than the anodyne performance profile currently provided three months later.

The hypothesis to be tested is that this change will result in improved engagement and confidence, feeding through to improved retention and progression measures.

Measures to be looked at:

• standard OU stats on submission, retention and progression (for comparison with pre-intervention presentations);

• online attitudinal survey;

• follow-up email contact (structured questions) with non-completing students.

Another course, TM190 The story of maths, is making the same change to its end of course assessment while other companion courses in the Relevant Knowledge programme (eg T183, T189) are not changing assessment. This provides points of comparison for completion measures although it is not planned to survey students on these other courses.

This has been put into the category of ‘interesting and may be worth following up’; will see what transpires.

The intervention is going ahead regardless, so the issue really is how best to evaluate it to capitalise on the opportunity. Unfortunately I missed the chance to survey previous presentations (ie before intervention) and a combination of low numbers and already very good retention/progress may make it difficult to spot any effect. Suggestions for how to proceed welcomed!

eSTEeM is a new Science and Maths, Computing & Technology initiative to support teaching and learning in (have you guessed it yet?) the STEM subject areas. In many ways it is the son-of-COLMSCT, although unfortunately hasn’t the same level of funding available. A call has been out for projects, and members of TERG have submitted proposals; results just in and details in following posts.

Jon Rosewell submitted a bid for JISC funding for an iSpot related proposal:

iSpot User-Contributed Keys

iSpot is a social networking website that connects beginners in natural history with experts who help identify organisms from photographs and descriptions. We also provide iSpot Bayesian keys, a novel form of
interactive key to help identify plants and animals. These keys are now available through the web on the desktop and on mobile phones. By using this new form of key, we have also made the creation of keys
much easier.

The proposed project will open up the authoring, reviewing and development of keys to a much wider public. We will do this by combining the commenting and discussion features already on iSpot with the existing authoring features for keys. This will make it possible for any user to start a new key and for others to comment and discuss, allowing groups of users to collaborate on improving keys. We will use experts from our existing network of collaborators to seed this activity in addition to encouraging the full iSpot community to create, use and comment on new keys.

By empowering many more individuals to create identification keys, we can increase their number, their taxonomic range and their use. This has the potential to feed through to more extensive collection of
biodiversity data.

Unfortunately this wasn’t successful, but another iSpot bid in the same funding call was funded. This was for a project called iSpot local; read more on Doug Clow‘s blog.

Karen Kear and John Pettit (IET) recently submitted a bid to the ESRC small grants fund for an investigation of web conferencing in higher education and corporate training. The project, if funded, will look at how real-time communication tools, such as Elluminate, Adobe Connect and Webex, can be used to engage learners in tutorials and training sessions.

The ESRC small grants fund is now closed (government cuts) so it was a mad rush to get the application in! Keep your fingers crossed – we will hear in about 3 months.

Leuven town hall

The E-xcellence project on quality in e-learning started on January 1st 2011. The OU contribution is led by Keith W, with Karen, Jon, Giselle and Wendy forming the rest of the team.

Keith and Karen went to Leuven, in Belgium, for the project start-up meeting on 17th and 18th January.

Leuven is a lovely old university town with ornate buildings and cobbled streets. The meeting was at the Irish College which was founded by Irish monks - this was also a lovely building.

The project meeting was attended by representatives from the many project partners and associates. There were attendees from Latvia, Poland, Lithuania, Equador … The project partners gave presentations on their particular workpackages. Keith presented for the OU, and included a great whistle-stop tour of ideas and new developments in e-learning.

The project dinner

On the evening of the first day there was a project dinner in a Leuven restaurant. This was a very pleasant event and good for getting to know the other people, and finding out about education, and other aspects of life,  in the different countries represented. 

The OU team will be developing a set of self-assessment guidelines and benchmarks for quality in e-learning. The project will also run a number of local workshops where an institution can carry out a supported review of their e-learning.