Depressingly, this post reports a similar finding to the last one.
For the question shown (which is one of a series of linked questions on the Maths for Science formative-only practice assessment), 62% of students are right at the first attempt but 22% remain incorrect after the allowed two responses. At the response level, whilst 60.2% of responses are correct, the other options are selected approximately equal numbers of time. The details are below:
P>0.1 12.4% of responses
0.1>P>0.05 14.0% of responses
0.05>P>0.01 60.2% of responses
P<0.01 13.5% of responses
So what’s this saying? I think it is reasonable to assume that many of the 12.4%-14.0% responses which selected the incorrect option were guesses. It is also then reasonable to assume that a similar number of responses which selected the correct option were also guesses. So overall, it is possible that as many as 50% of all responses were guesses (and most of the rest were right). And you wonder why I don’t like the use of multiple-choice questions without some way to discourage guessing!
In ‘e-assessment (f)or learning’, I have described two things which seem to imply less than full engagement with a question. The first of these (which occurs in selected response type questions, like multiple-choice) is as described here i.e.incorrect distractors being selected an almost equal – and large – number of times. The second is blank and repeated responses, as reported in a previous post. This occurs mostly for constructed response (i.e. free-text) type questions – so when students can’t guess, some of them just don’t bother. Not great.
Not surprisingly, both of these effects occur more for formative-only questions; indeed they are quite rare for summative questions. The effects are also more common for difficult questions and for questions that require more effort (e.g. doing a complex calculation, accessing a calculator, filling in multiple boxes in one question).
Students say that even if they don’t engage seriously with the question, they still use the feedback provided. But do they? The problem is, we don’t really know.