I was talking at the eSTEeM Conference last week about the fact that, whilst our interactive computer-marked assessment (iCMA) questions are generally well liked by our students, occasional questions can cause problems (usually because we are not giving sufficiently targeted feedback, so students don’t understand why the answer they have given is wrong). Why not, someone said, have an ‘unsound’ button? Well, we used to have such a function for our short-answer free text questions (shown below in use back in 2007) and it wasn’t my choice to stop using it. This post considers the pros and cons.
The first thing to note is that there was no obvious relationship between questions that were reported as unsound and questions that actually had a problem. Students reported as unsound questions that we thought were fine (notwithstanding the fact that, in general, if students really think there’s a problem with a question, then I think there is a problem with the question, albeit that the answer-matching is usually fine and the difficulty is likely to be that the student has not understood the feedback). More significantly, at this stage we definitely had questions that were unsound, but students didn’t tick the unsound button.
When we started using the short-answer free-text questions for real, the Module Team were not keen on having an unsound function for this type of questions but not for others (fair enough) and others were not keen to introduce an unsound function on all questions, because we would then be obliged to investigate and respond every time the button was ticked (also fair enough). So, in the early presentations, where I was not sure about the answer matching for the short-answer free-text questions, we checked the marking of ALL responses. Clearly this is not sustainable.
In our old computer-marked assignments, if sufficient students ticked the unsound button, the question was zero weighted and we could do that for iCMAs too. However I like the idea of an ‘unsound’ button simply because of its potential to defuse the angst. It would give students a way of saying ‘this isn’t right’ without having to contact the module team separately. Some students do this of course (and a lot of curriculum manager time is spent in dealing with the queries) but these are the angry students who have an axe to grind – it all feels a bit unbalanced. But an unsound button might open the floodgates – students might tick it just because it is there. Tricky. An answer might be to provide an unsound button that notified the student’s own tutor rather than the module team (given that in 99% of cases, the student would be revealing a misunderstanding that their tutor could help with).
The underlying issue here is student confidence, or lack of it, in the computer’s marking. What we haven’t really felt able to do is to remind our students just how accurate computer marking is and, relatively, how inaccurate human marking is. Hmmm.