Day 10. Check, check and check again. So you’ve chosen your question type, written your question and feedback and constructed your answer-matching. You may even have put your questions together into a quiz or interactive computer-marked assignment (iCMA) of some sort. So now it’s ready to go to students? WRONG. Checking eAssessment questions and whole quizzes is vitally important and this post considers things such as when to do the checking and who should do it.
In some circumstances I write draft questions on paper or using a simple word processor. It is worth reviewing questions at this stage, to spot any glaring problems, or getting someone else to take a look. Sometimes (especially when using OpenMark) I pass my wordprocessed drafts to someone else for programming, but I am increasingly (especially when using PMatch and STACK) actually writing the questions myself. I’m enjoying this, but it’s a weak link in the chain – when I’ve passed draft question details to someone else for programming, they have also been an extra pair of eyes; someone else to spot errors or just things that I might do better.
So my first general point is that, whilst question authors should take the responsibility for checking their own questions, it is much better if someone else can check them too, preferably someone who isn’t an expert in the same academic area as you, so who can see the difficulties much more from the student perspective. It helps if the checker knows the course material (again, to put them in the shoes of students). And some people are just better at checking than others. You need someone with an eye for detail, but who will also look at the ‘big picture’ – I have known checkers of tutor-marked assignments to go on at length about a small point, whilst missing a huge problem. And your checker needs to be willing to raise all their concerns – you (the author) may not agree, but comments from your checkers are worthy your consideration.
Your review of your draft questions is obviously not the end of the checking process. It is vitally important that you check the questions in their online form; this is, after all, the form in which your students will see the questions, and so will see things that you haven’t noticed previously. I don’t think I’ve said anything terribly contentious yet, but I know that some people disagree with my next conclusion: I think that checking your online questions should go far beyond simply checking that they have been programmed in the way you intended; I think the questions need checking from the student perspective; I’d take this so far as to say that the correct answers should not be supplied to checkers.
Also contentious (because I am making more work for people – as said proviously, eAssessment is not an easy option!) is the fact that, when a question exists in multiple variants, I think all the variants need checking. Sometimes faults only show up in one variant and, for summative use, you also need to check that there is not something (which might be as simple as having some variants which involve rounding up and some that involve rounding down) that is likely to make some variants more difficult than others.
My final point is that I think checking needs to be done iteratively – the iCMA checking process simply won’t fit into a ‘first draft, second draft, finished product’ mentality. Each time you make a change there may be a knock on effect on something else, and each time you may notice things that weren’t obvious before. I believe quite passionately that we owe it to our students to check and make changes, check and make changes, etc.etc., check – until our questions and iCMAs are as good as they possibly can be.
By ‘checking’ I am referring here to the checking of individual questions, but sometimes you also need to check the performance of a new question type or similar. Here, I would advocate watching students engaging with the question(s), perhaps in a usability lab or similar – watching people struggle with something that you thought was straightforward is a salutary experience!