e-assessment (f)or learning

The title of this post is the title of a paper I have just read. It was written in – wait for it – 1938. It’s a delightful little paper, but its findings are shocking. I came across it whilst valiently trying to find good reasons for using multiple-choice questions (which, you will remember, are not my favourite type of question). However, it turns out that multiple-choice (‘objective’) questions were first used because of the lack of objectivity of human-marked essay-type questions. (more…)

I’m pleased to hear that OCR have apologised for errors in adding up marks for GCSE and A-level papers last year. It doesn’t seem right that the whistleblower remains suspended, but I don’t know the details so perhaps I shouldn’t comment.

When I first heard about this latest case of human error in exam marking, I was amazed that we are still reliant on human arithmetic (though I know that addition of scores is meant to be checked by someone else – and that payment for this is included in the payment of GCSE and A-level markers). However I suppose that if markers were required to enter their scores into a computerised system of some sort (to enable the computer to check the arithmetic, or to do the arithmetic in the first instance) there would still be transcription errors – and it would take time and so slow down the marking process.

The important point is that, however much their work is checked and however much they are encouraged not to make mistakes and/or punished for making them (according to The Guardian, some examiners have had their contracts terminated), human markers are fallible. They are fallible when marking, they are fallible when adding scores. I’m a human, I make mistakes. The sooner we are honest with ourselves and admit that, the better.

Hot on the heals of my previous post, I’d like to make it clear that human markers sometimes do better than computers in marking short-answer [less than 20 word] free-text questions.

 I have found this to be the case in two situations in particular:

1. where a response includes a correct answer but also an incorrect answer;
2. where a human marker can ‘read into’ the words used to see that a response shows good understanding, even if it doesn’t use the actual words that you were looking for. (more…)

Back to short-answer free-text questions. One of the startling findings of my work in this area was that computerised marking (whether provided by Intelligent Assessment Technologies’ FreeText Author or OpenMark PMatch) was consistently more accurate and reliable than human markers. At the time, I was genuinely surprised by this finding and so were the human markers concerned (one of whom had commented to me that he didn’t see how a computer could possibly pick up on some of the subtleties of marking). However others, especially those who know more about issues of reliability in the human marking of exams etc., were not at all surprised. And, reflecting on my own marking of batches of responses (where I can start by marking something in one way and find myself marking it differently at the end) and on the fact that I make mistakes in my work (not just when marking – think about the typos in this blog) however hard I try, I can see that human-markers have weaknesses! (more…)

I’ve been lent a copy of Todd Farley’s book ‘Making the grades: my misadventures in the standardized testing industry’ (published by PoliPointPress in 2009). The blurb on the back of the book says ‘Just as American educators, parents and policymakers reconsider the No Child Left Behind Act and its heavy em