Maths for Science assessment – the first 10 years

Ten years ago, at the end of January 2003, 186 Open University students submitted the second online interactive end-of-module assignment (EMA) for my little 10-credit baby, S151 Maths for Science (57 brave souls had submitted the first EMA, back in October 2002). Since then, something between 14, 000 and 15,000 students have taken Maths for Science. The new edition of the book has now been copublished with Oxford University Press, and it is currently being used by 315 students on the new version of S151 and also by 396 students as part of the new 30-credit module S141 Investigative and mathematical skills in science, compulsory in the latest version of our BSc Hons Natural Sciences. The new EMA for S151 opened for the first time last Monday and the iCMA for S141 (one component of the  assessment that assesses the content of Maths for Science) closed last Wednesday. It was Maths for Science that got me into eAssessment in the first place, so this seems like a pretty good moment to stop and reflect on the past 10 years.

In common I suspect with many other bloggers I’m a reflective practitioner. Indeed, I am so reflective that I sometimes find myself paralysed by indecision about the right way forward. I am aware that some see online interactive assessment as a fundamentally bad thing, reducing learning to low-level learning outcomes and giving extrinsic rather than intrinsic feedback. That upsets me, but I can see their point, in particular when coupled by the generally-held view that online assessment = multiple-choice quiz. I have argued strongly in this blog that there is a need (and, increasingly, an ability) to do eAssessment better and I don’t see eAssessment as a panacea – I prefer S141’s assessment strategy to S151’s, because the former combines online assessment with other forms of assessment, with each used when it is most appropriate.

And, when all is said and done, 10 years and more than 14,000 students on, S151 and its online assessment strategy has worked. The module has been highly successful by just about every performance measure and students like the instantaneous targeted feedback with a chance to have another go. The system has also provided me with extremely useful information about what students get wrong, with an indication of why. Our large student numbers give us the edge here – and when the assignment is summative so students are trying, and equivalent errors appear in each variant of a question, you know you’re onto something. I’ve been able to improve our teaching as a result.

So, in the broadest sense of the word, Maths for Science has been a formative experience. I’d like to thank those without whom the book, the module and the online assessment would never have happened. Ten years ago we were only just able to assume that sufficient of our students would have access to the internet (and some still don’t, in particular our offender learners). We were breaking new ground, all in order to give our students instantaneous and meaningful feedback. Since then we’ve evaluated, improved, disseminated – and learnt from others. Thank you.

Posted in e-assessment, Maths for Science, Open University | Tagged , , , | 2 Comments

Twelve Pipers Piping

Day 12. Monitor and improve. One of the advantages of eAssessment (whatever you mean by that term)  is the ability to monitor actual student behaviour. This is not the same as what students say they do, neither is it the same as student opinion. Student opinion is important of course, but there are real inconsistencies between what students say they do and what they say they want, and what they actually do. The best documented of the inconsistencies is the fact that students ask for more ‘feedback’ whilst not collecting their marked work. I could write at length on reasons and possible reasons for this fact – but for now, let’s concentrate on actual student behaviour. Continue reading

Posted in e-assessment, Twelve Days of Better eAssessment | Tagged , | Leave a comment

Eleven Drummers Drumming

Day 11. A broader definition of eAssessment. I can clearly remember the day, around 5 years ago, when I felt rather small at a conference because I was talking about my work with short-answer free text questions, and realised that everyone else was using the term ‘eAssessment’ to mean something rather different. The irony is that I’ve always had far broader interests in assessment  – I come from a tutoring background and my current ‘day job’, as an Open University staff tutor, means that I spend a lot of time getting our ‘correspondence tuition’ (marking and commenting on tutor-marked assignments) as good as possible. I have also been involved in the assessment of wikis and tutor group forum discussions. But somehow I didn’t see this as eAssessment… Continue reading

Posted in e-assessment, Twelve Days of Better eAssessment | Tagged , | Leave a comment

Ten Lords a-Leaping

Day 10. Check, check and check again. So you’ve chosen your question type, written your question and feedback and constructed your answer-matching. You may even have put your questions together into a quiz or interactive computer-marked assignment (iCMA) of some sort. So now it’s ready to go to students? WRONG. Checking eAssessment questions and whole quizzes is vitally important and this post considers things such as when to do the checking and who should do it. Continue reading

Posted in e-assessment, Twelve Days of Better eAssessment | Tagged , , | Leave a comment

Nine Ladies Dancing

Day 9. Using STACK to write questions to assess maths.  There may or may not be nine ladies dancing, but there is certainly one female blogger who would happily do a little dance around her house this evening. I’ve managed to prise another few hours this afternoon to write another STACK question and I am oh so happy with the result. I feel a bit embarrassed to be writing about STACK and, yes, it did take me several hours to write one question. I am a real beginner, standing on the shoulders of giants (Chris, Tim, Phil – that’s you). In writing STACK questions I am having to learn STACK, learn Maxima, learn Moodle, learn LaTeX, and – as I get onto questions that assess rather more maths than basic differentiation – I will be having to dig deep into my memory (and revise from the module materials which the questions are designed to support) from the days, something approaching 40 years ago, when I might have described myself as a mathematician. But I am loving every minute of it. Continue reading

Posted in e-assessment, STACK, Twelve Days of Better eAssessment | Tagged , , | Leave a comment

Eight maids a-Milking – part 2

Day 8b. Using PMatch for short-answer free-text questions. As promised, what I’d like to do is to give you an example of the answer matching for a real question, based on real student responses. Continue reading

Posted in e-assessment, short-answer free text questions, Twelve Days of Better eAssessment | Tagged , , , , | Leave a comment

Eight Maids a-Milking – part 1

Day 8a. Free text marking of short answer questions.  I’ve already talked about the automatic marking of short answers quite a lot on this blog so today, the first day of 2013, I’d like to do two slightly different things (1) Set our work in this area in context; (2) Give an example of the actual PMatch answer matching that we use for one question. Actually, we’ve been out walking and I wanted to get my other blog written up, so I will only do (1) today, letting the ‘Twelve Days’ slip and ending on 6th January (which some people consider to be the 12th Day of Christmas in any case) rather than 5th. A clever solution and another reminder that assessment questions need to be unambiguous (not like ‘What date is the 12th Day of Christmas?’) and to mark all correct answers as correct and all incorrect answers as incorrect – that is very important for short-answer free text questions too. Continue reading

Posted in e-assessment, short-answer free text questions, Twelve Days of Better eAssessment | Tagged , , , , | Leave a comment

Seven Swans a-Swimming

Day 7. Other question types. OK, so you’ve decided to limit your use of multiple choice questions to situations in which you have no option or where MCQs have something specific to offer. What options do you have for other question types? This will depend on which eAssessment system you are using, but I have taken examples largely from the OU’s ‘OpenMark’ system. OpenMark questions can be used in Moodle quizzes – and some of the question types that I’ve used extensively in OpenMark are now available as Moodle question types too. Continue reading

Posted in e-assessment, Twelve Days of Better eAssessment | Tagged , , , | Leave a comment

Six Geese a-Laying

Day 6. Making multiple-choice better. Although I don’t, in general, much like multiple-choice questions, I have to admit that they can sometimes work very well. In conventional face-to-face settings, the use of electronic voting systems (‘clickers’) can bring lectures alive as well as informing lecturers about student misunderstandings. And it gets better – we don’t have conventional lectures at the Open University, but we are making more and more use of synchronous conferencing tools such as Elluminate Live! (now Blackboard Collaborate elsewhere) – which has a quiz function. I have seen the quiz function used most effectively to make Elluminate sessions much more interactive. Continue reading

Posted in e-assessment, multiple-choice questions, Twelve Days of Better eAssessment | Tagged , , , , , , | Leave a comment

Five Gold Rings

Day 5. Go beyond multiple choice. I’ve been reading a lot recently about the pros and cons of multiple choice (selected response) and constructed response questions. If you’re a regular reader of this blog, you’ll realise that I am not a great fan of multiple choice questions. I’ve already given some of my reasons, but I’ll attempt to summarise here. Continue reading

Posted in e-assessment, multiple-choice questions, Twelve Days of Better eAssessment | Tagged , , | Leave a comment