e-assessment (f)or learning

Day 4. What gift was delivered on 28th December? Thinking about possible answers to this question should help you to identify the theme’s of today’s post, so perhaps I’ll stop writing now!

Some people say that the first day of Christmas is 25th Dec; leading to a 4th Day of 28th Dec and a 12th Day of 5th January. That’s the convention I’ve followed in this blog, but others would argue that the first day of Christmas is  26th Day, leading to a 3rd Day of 28th December and a 12th Day or 6th January. I wouldn’t like to say which of these is ‘right’ – it depends on your culture. So the question as asked is ambiguous – are you being asked about the gift delivered on the 3rd or 4th Day of Christmas? (more…)

One of the joys of trying to catch up with others who have been working in the field of assessment for much longer than me is finding books and articles that were written some time ago but which still seem pertinent today. I’d definitely put the following book into this category (and more thoughts from it will follow):

Gipps, C. and Murphy, P. (1994) A fair test? Assessment, achievement and equity. Buckingham: Open University Press.

For now, I’d like to highlight a particularly memorable quote from Gipps and Murphy, originally from the Times Educational Supplement back in November 1988,expressing sceptism about the ‘Code of Fair Testing Practices in Education’ in the USA. As a former Assistant Secretary of Education put it:

If all the maxims are followed I have no doubt the overall quotient of goodness and virtue should be raised. Like Moses, the test makers have laid down ten commandments they hope everyone will obey. This doesn’t work very well in religion – adultery continues.

So I’d like to emphasise that my ‘top tips’ in the previous post are not commandments! – apart perhaps from my final tip (monitoring the questions when in use) which I think ought to be made compulsory.

In general though, although the ‘top tips’ have worked well for me, and I hope that these are ideas that others might find useful, perhaps it is more important that question authors take responsibility for the quality of their own work, rather than mindlessly following ‘rules’ written by others. This wish reflects most of my practice, in writing e-assessment questions and in everything else so, for example, I far prefer helping people to write questions in workshops (when they are writing questions ‘for real’) than providing rules for them to follow. Sadly, I think a wish to improve the quality of our e-assessment may be leading to a more dictatorial approach – I’m not convinced it will work.

I’ve recently been asked for my ‘top tips’ for writing interactive computer-marking assignment (iCMA) questions. I thought I might as well nail my colours to the mast and post them here too:

•  Before you start writing iCMA questions, think about what it is appropriate to assess in this way – and what it isn’t.

•  Think about what types of iCMA question are most appropriate for what you want to assess. Don’t assume that multiple-choice and multiple-response questions are more reliable than free-text entry questions – they aren’t!

•  Write multiple variants of your questions – this enables you to use the same basic template in writing questions for multiple purposes, reduces opportunities for plagiarism and gives students extra opportunities for practice. However, in summative use, make the variants of similar difficulty.

•  For multiple response questions (where students have to select a number of correct options), tell students how many options are required (otherwise students get very frustrated).

•  Check carefully that each question is unambiguous. Does it use language that all students should understand? If you want an answer in its simplest possible form, is this clear?

•  Think carefully about what you will accept as a correct answer. Do you want to accept miss-spellings (e.g. ‘sulphur’ instead of ‘sulfur’), surplus text etc. If in doubt, have surplus text at the end of a response removed before the response is checked. Students are not happy if their response is marked wrong because, for example, they have indicated the precision of an answer by typing ‘to 3 significant figures’ at the end of a perfectly correct numerical answer.

•  Wherever possible, give feedback that is tailored to the error that a student has made. (Students get very annoyed when they are given general feedback that assumes they don’t know where to start, where to their mind they have made a ‘small’ error late in the process, e.g. given incorrect units).

•  If a response is partially correct, tell the student that this is the case (preferably telling them what is right and what is wrong).

•  Check your questions carefully at each stage, and – especially important – get someone else to check them.

•  Monitor ‘real’ use of your questions and look at student responses to them. Check that your variants are of sufficiently similar difficulty. Be prepared to make improvements at this stage.