So are the variants of equivalent difficulty?

One glance at the figure from the previous post (reproduced to the right) makes it clear that whilst the variants of the question shown at the top are equivalent, those for the lower question are not.

Reasons why variants may be of differing difficulty include

• the variables selected may result in a more difficult mathematical task (e.g. rounding up instead of rounding down, understanding of negative exponents rather than positive ones)
• a graph to be interpreted may have a more awkward scale to read, or if different readings are to be taken from the same graph, some students may have to interpolate or extrapolate whilst others are taking readings where the graph crosses the grid-lines.
• the letters used may appear similar in lower and upper case and so be confused e.g. k and K are very similar whilst q is unlikely to be confused with Q.
• the words used in setting up the question may use language or describe a situation which is unfamiliar to the student.
• one variant may result in a common spelling mistake (I would argue that both spellings should be accepted as correct in any case).
• one variant may result in an answer that is difficult to guess, or unexpected.

Many people think that multiple-choice questions are more reliable in this regard, but this has not been my experience. It has been reported previously that even offering the same options in a different order can lead to different student behaviour. More seriously, if different variants are formed by picking different correct and incorrect options from a pool, some of the options are likely to be more difficult to assign as ‘correct’ or ‘incorrect’ than others. And if you’re not careful, two options will be closely related – the strategic student will quickly see that both are either correct or incorrect, or perhaps that if one is correct the other must be incorrect.

Perhaps the worst performing questions I have seen are those where students have to select, say, the two correct statements. Suppose that, for one variant, all but one of the statements is clearly correct or incorrect. If a student incorrectly assigns the final statement at the first attempt, they will quickly realise that this is the one they need to reassign at the second attempt. If two statements are in the grey area, the question becomes much more difficult.

So why did the different variants behave differently in the question whose behaviour is shown in the lower graph above? More detailed inspection shows that  variant 3 appeared to be easier than the other variants and variant 2  was more difficult. Variant 3 had a facility index of 77.8% whilst Variant 2 had a facility index of 66.3%. Variant 2 of this question is shown below: