The title of this post is the title of a paper I have just read. It was written in – wait for it – 1938. It’s a delightful little paper, but its findings are shocking. I came across it whilst valiently trying to find good reasons for using multiple-choice questions (which, you will remember, are not my favourite type of question). However, it turns out that multiple-choice (‘objective’) questions were first used because of the lack of objectivity of human-marked essay-type questions.
Returning to the 1938 paper: The author found that
(a) The passing or failing (not merely the difference of a letter grade but the difference between credit and no credit) of about 40% of papers depended, not on what the student knew or do not know, but on who marked the papers.
(b) The passing or failing of about 10% of the papers depended, not on what the student knew or do not know, but on when the papers were marked.
And all of this was despite the fact that the markers were fully aware of the experiment, so they were perhaps more careful in their marking than would normally be the case.
So the current anxiety about the accuracy (or otherwise!) of human marking is not new. But don’t take my word for it – read the 1938 paper for yourself:
Ashburn, R. (1938) An experiment in the essay-type question. Journal of Experimental Education, 7(1), 1-2.