Despite my recent ‘rants of the day’, I think it is vitally important that we try our best to evaluate our assessment practice. There is some good, innovative practice out there, but it can still be very tempting to confuse practice that we consider to be “good” with practice that we know to be good, because it has been properly – and honestly – evaluated. And, at the danger of appearing a stick in the mud, innovation does not necessarily lead to improvement.
My quote for today is from the (fictional) Chief Inspector Morse:
“In the pub, with Lewis, he’d felt felt convinced he could see a cause, a sequence, a structure, to the crime… It was the same old tantalizing challenge to puzzles that had faced him ever since he was a boy. It was the certain knowledge that something had happened in the past – happened in an ordered, logical, very specific way. And the challenge had been, and still was, to gather the disparate elements of the puzzle together and to try to reconstruct that ‘very specific way’.” (from Colin Dexter’s “The remorseful day”, Chapter 22)
Honest evaluation will sometimes ‘prove’ what you expected; but sometimes there will be surprises. Sometimes good ideas don’t work and we need to reconsider. Sometimes a ‘control’ group does better than the experimental group and we need to think why. Sometimes students don’t engage with an assessment task in the way that we expect; sometimes students don’t make the mistakes that we think they will make; sometimes they make mistakes that we don’t expect.
Actually, in the long run, it is often the surprises that provide the real insights. And sometimes they can even save time and money. We would have gone on using linguistically-based software for our short-answer free-text questions had we discovered that pattern matching software was just as effective.
But whatever, we must find out…Chief Inspector Morse always got it right in the end!