So are the variants of equivalent difficulty?

One glance at the figure from the previous post (reproduced to the right) makes it clear that whilst the variants of the question shown at the top are equivalent, those for the lower question are not.

Reasons why variants may be of differing difficulty include

  • the variables selected may result in a more difficult mathematical task (e.g. rounding up instead of rounding down, understanding of negative exponents rather than positive ones)
  • a graph to be interpreted may have a more awkward scale to read, or if different readings are to be taken from the same graph, some students may have to interpolate or extrapolate whilst others are taking readings where the graph crosses the grid-lines.
  • the letters used may appear similar in lower and upper case and so be confused e.g. k and K are very similar whilst q is unlikely to be confused with Q.
  • the words used in setting up the question may use language or describe a situation which is unfamiliar to the student.
  • one variant may result in a common spelling mistake (I would argue that both spellings should be accepted as correct in any case).
  • one variant may result in an answer that is difficult to guess, or unexpected.

Many people think that multiple-choice questions are more reliable in this regard, but this has not been my experience. It has been reported previously that even offering the same options in a different order can lead to different student behaviour. More seriously, if different variants are formed by picking different correct and incorrect options from a pool, some of the options are likely to be more difficult to assign as ‘correct’ or ‘incorrect’ than others. And if you’re not careful, two options will be closely related – the strategic student will quickly see that both are either correct or incorrect, or perhaps that if one is correct the other must be incorrect.

Perhaps the worst performing questions I have seen are those where students have to select, say, the two correct statements. Suppose that, for one variant, all but one of the statements is clearly correct or incorrect. If a student incorrectly assigns the final statement at the first attempt, they will quickly realise that this is the one they need to reassign at the second attempt. If two statements are in the grey area, the question becomes much more difficult.

So why did the different variants behave differently in the question whose behaviour is shown in the lower graph above? More detailed inspection shows that  variant 3 appeared to be easier than the other variants and variant 2  was more difficult. Variant 3 had a facility index of 77.8% whilst Variant 2 had a facility index of 66.3%. Variant 2 of this question is shown below:

Inspection of the actual student responses to all variants quickly alerted the course team to the cause of the problem: the table that students use in the course material refers from mRNA codon to amino acid, but students are frequently looking up the anticodon instead. In the case of variant 3, this results in a stop codon not an amino acid, so presumably students realise they have done something wrong (thus the variant is easier than the others). In the case of variant 2, this results in Gln, which many students misread as Gin – so when they are told that their answer is incorrect they start by correcting Gin to Gln rather than looking for a more fundamental mistake. This causes variant 2 to be more difficult than the other variants.
For the future, variants 2 and 3 will be removed and targeted feedback will be added to all variants for answers that give amino acids found by looking up the anticodon rather than the codon.
The process of investigation followed for this question illustrates the importance of looking at the actual student responses when a problem has been identified by the statistical tools.
Michael Isherwood has written a paper outlining points to consider in writing iCMA questions in order to make them of similar difficulty. See:
Isherwood, M. & Jordan, S. (2010) iCMA questions – Equivalence of variants at
This entry was posted in variants and tagged , , . Bookmark the permalink.

One Response to So are the variants of equivalent difficulty?

  1. Pingback: e-assessment (f)or learning » Blog Archive » Confusing letters – a distance-learning problem

Leave a Reply

Your email address will not be published. Required fields are marked *