At last week’s Quality Enhancement Seminar, I was asked for guidance on checking interactive computer-marked assignment (iCMA) questions and for guidance on writing questions that are easy to check.
We have detailed but rather OU- and OpenMark-centric guidelines for question checkers (if anyone from the OU would like a copy, please email me) but this has made me think about the more general points.
Firstly, checking iCMA questions is vitally important and I think it is reasonable that this might take as long as writing the questions in the first place. Checking all assessment material is important, but with tutor-marked assignments, you don’t need to check the answer matching, and there are tutors to mediate any unexpected answers. Not that this should be an excuse for not doing everything you can to make the questions as good as they possibly should be before students see them. There are two general points that apply to checking both computer-marked and tutor-marked questions : (1) try the questions for yourself as if you were a student; (2) if possible get someone else to check your questions too – a new pair of eyes will often see ambiguities etc. that you have missed.
Concentrating now on iCMA questions, I know that some people get someone else to print out the questions or to put the content in a Word file – for ease of checking and commenting. This might help you to check the questions at the technical level, and to spot some issues, but I think that it is also very important to check the questions in the form that they will appear to students. And if the technology is difficult to use, shouldn’t we be asking questions about its suitability for use by our students.
Secondly, do we need to check all variants of questions? Again, I’m afraid that experience leads me to a definite answer of ‘yes’. You can get a certain distance by checking just one variant thoroughly, but problems and nonsensical explanations will appear in some variants that are not there in others.
A third point that I’d like is that it is important that checking is an iterative process – so if you are checking someone else’s questions, don’t assume that all will be well with the revised version. Altering one thing has a horrible tendency to lead to problems elsewhere (see the lyrics of the Flanders and Swann song ‘The Gas Man Cometh’ below).
Moving on to the second part of the question I was asked, can I give guidelines on writing questions that are easy to check? I’m not sure that I can. I can give guidelines on writing questions (see an earlier post and again people at the OU who want more detail should email me) and obviously questions will be easier to check if the author’s instructions were clear and the question didn’t add unnecessary complexity. I think people who have logical minds and time to think things through properly (ha ha) probably write better questions than those who are muddled and in a rush. But I wouldn’t want ease of checking to be a driver towards over-simplistic questions or ones without targeted feedback.
Plenty here to think about. Or perhaps we should just enjoy ‘The Gas Man Cometh…’
The Gas Man Cometh – Flanders and Swann
- ‘Twas on a Monday morning the gas man came to call.
- The gas tap wouldn’t turn – I wasn’t getting gas at all.
- He tore out all the skirting boards to try and find the main
- And I had to call a carpenter to put them back again.
- Oh, it all makes work for the working man to do.
- ‘Twas on a Tuesday morning the carpenter came round.
- He hammered and he chiselled and he said:
- “Look what I’ve found: your joists are full of dry rot
- But I’ll put them all to rights”.
- Then he nailed right through a cable and out went all the lights!
- Oh, it all makes work for the working man to do.
- ‘Twas on a Wednesday morning the electrician came.
- He called me Mr. Sanderson, which isn’t quite the name.
- He couldn’t reach the fuse box without standing on the bin
- And his foot went through a window so I called the glazier in.
- Oh, it all makes work for the working man to do.
- ‘Twas on a Thursday morning the glazier came round
- With his blow torch and his putty and his merry glazier’s song.
- He put another pane in – it took no time at all
- But I had to get a painter in to come and paint the wall.
- Oh, it all makes work for the working man to do.
- ‘Twas on a Friday morning the painter made a start.
- With undercoats and overcoats he painted every part:
- Every nook and every cranny – but I found when he was gone
- He’d painted over the gas tap and I couldn’t turn it on!
- Oh, it all makes work for the working man to do.
- On Saturday and Sunday they do no work at all;
- So ’twas on a Monday morning that the gasman came to call…