Bad questions

As part of a ‘Refreshing Assessment’ Project, the Institute for Educational Technology at the Open University is hosting three talks during June. The first of these, last Wednesday, was from Helen Ashton, head of eAssessment at the SCHOLAR programme at Heriot Watt University, with the subject ‘Exploring Assessment Design’. It was a good talk, highlighting many points that I bang on about myself, but sometimes we need to hear things from a different perspective (in this case, from Helen’s experience of authoring questions for use by a wide range of schoolchildren).

Amongst the things that stuck me from Helen’s talk were the following:

  • That you should think about what you want to do [in assessing students] before being driven by technology. You should start by thinking about the purpose of the assessment and the learning outcomes you are trying to assess.
  • Feedback to staff, by finding out how students answer their questions, is a valuable feedback mechanism.

Helen didn’t claim to be an expert in writing multiple-choice questions and I suspect she shares my tendency to dislike them. However she did talk about the design of good multiple choice questions and ways in which questions can be improved by rewording.  Helen gave an example (which I’ve seen before) of a question that doesn’t make any sense at all, but where  it is possible to correctly guess the answer. She said the question was from the work of Sharon Shrock and Bill Coscarelli, but I haven’t been able to find it.

However, in looking for Helen’s example, I stumbled across an excellent source of ‘tips for what  not to do when writing questions’,  in the shape of several guides, meant for students, of how to answer eAssessment questions. In particular, Robert Runt of the Education Department of the University of Lethbridge, has written a very interesting list of Tips for taking multiple choice tests. Two of his examples are similar to those given by Helen:

2. When you have to complete a sentence, see if one of the answers fits better grammatically than the others.

A dog is an A) animal B) machine C) mineral D) vegetable

In this question, the “an” gives you a clue to the right answer, because you know the correct answer has to start with a vowel. “An animal” works, but it would have to be “a machine” or “a mineral” or “a vegetable” to be a correct sentence. Since the question says “a dog is an ” rather than “a dog is a”, the answer has to be “animal” — because “animal” is the only one that fits with the “an” in the question.

4. Look to see if any of the answers is much longer than the others. The person making the test wants the right answer to be completely true. To make it completely true, they sometimes add more details to the correct answer, making it much longer than the others.

The freezing point of water is A) 32 degrees Kelvin B) 32 degrees centigrade C) 0 degrees Fahrenheit D) 0 degrees centigrade for pure water at sea level

The correct answer, of course, is D.

Robert Runt goes on to point out that ‘these tricks will not work on well designed tests’. Salutary.

This entry was posted in multiple-choice questions and tagged , , , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *