Finally for today, I’d like to talk about where I believe the limits currently sit in the use of short-answer free-text questions.
I have written questions where the correct response requires three separate concepts. For example, I have written a question which asked how the rock in photograph shown was formed. (Incidentally this is granite, photographed near Lands End in Cornwall, but I’d never say that in a question, otherwise students just Google the answer). A correct answer would talk about the rock being formed from magma (first concept), which has cooled and crystallised (second concept) slowly (slow because it has cooled within the Earth rather than on the surface) (third concept). I haven’t managed to write a successful question with a correct answer that includes more than three separate ideas, but that doesn’t mean to say it can’t be done.
A second consideration in authoring short-answer questions is the number of discrete correct and incorrect responses. Here I think the limit came in another question based on one of my photographs, with thanks to the colleagues shown. We used this photograph in training people to write answer-matching and the question was simply ‘Who is taller?’ That might sound like a very straightforward question (until you get the bright-sparks who say ‘The prettier one’), but writing a complete set of answer-matching for this question was a time-consuming and non-trivial task. Think about the correct answers: ‘The one on the right’, ‘The one with longer hair’, ‘The one carrying a brochure’, ‘The one wearing glasses’, [not ‘The one holding a glass’]..and so on…and so on.
The third limit is the serious one. Developing answer matching for the questions I’ve talked about was time-consuming but we got there in the end. However the correct answers are unambiguous – it is clear that the woman on the right-hand side of the photograph is taller than the one on the left. However in some subject areas, what is ‘correct is less well defined. I think that’s the real limit for questions of this type.