For various reasons I’ve been thinking a lot recently about confidence-based marking. (Tony Gardner-Medwin, who does most of the work in this area also calls it ‘certainty-based marking’). The principle is that you get most marks for a correct response that you are sure is right, fewer for a correct response that you are not sure about. But at the opposite end of the scale, you tend to get a penalty for an incorrect response that you were sure was right.
Confidence-based marking is one way of dealing with the ‘guessing effect’ for multiple-choice questions, but it can also be used to make students engage at a deeper level with questions. Jon Rosewell has an interesting eSETeM project in this area (getting students to rate their confidence before seeing the distractors) and Silvester Draaijer has recently blogged on the subject.
I’m not comfortable with the use of negative marking and, probably for this reason, I’ve never been very keen on the idea of confidence-based marking. However I can see that if you are using this approach to encourage deeper learning then there might be some mileage it in. One of the criticisms of confidence-based marking comes from the theory that girls tend to be less confident than boys, so are disadvantaged. But of course, when properly used, you aren’t necessarily penalised for lack of confidence, rather for lack of an appropriate level of confidence – and Tony Gardner-Medwin has shown that female medical students are not disadvantaged by the approach.
Several people, including Tony (who works at a Medical School) make the point that doctors need to have confidence in what they are doing. When I heard him speak, several years ago now, that made me very upset because I had recently recovered from a serious illness which my GP incorrectly diagnosed as a mental breakdown. He was sure he was right and his confidence caused a delay in reaching the correct conclusion (as well as knocking my confidence further; in addition to feeling ill I had to deal with the thought that I was cracking up!). However, on reflection, I can see Tony’s point. I want doctors to have confidence when they are right – but to have the decency to admit that they aren’t sure when the situation is less clear-cut.
Having blogged about one thing that’s rather more personal than my usual fare, I will now tell you something that I hadn’t told anyone until I was in the middle of a discussion of confidence-based marking’s pros and cons a couple of weeks ago. When I heard Tony Gardner-Medwin speak, he used some ‘every day’ examples to illustrate his approach. One of these asked a question about the speed limit on dual carriageways in the UK. I ticked the ’70 miles per hour’ box and indicated that I had high confidence in my answer. I was devasted when Tony said that the correct answer was ‘of course’ ’60 m.p.h’ – apparently I was wrong. For the next few months I crawled around the country – it took me some time to even have the confidence to check the highway code. And of course I wasn’t wrong! So, if you use questions of this type, make jolly sure that you are right.
That would have been the end of this little anecdote were it not for a wonderful insightful comment from someone else in the meeting when I was telling my story. Her comment was [the speed limit] ‘depends what vehicle you’re driving’. Oh yes! Life just isn’t as black and white as the world of multiple-choice questions would have us believe:
Sally! Great to have some forthright thoughts about CBM.
Wow, I wonder if I really did get the speed limit wrong at this meeting (6 years ago)! I don’t remember anyone saying so at the time. Your point is well made that it is crucial that Qs for CBM be unambiguous and the answers correct. Vigorous complaints about bad Qs are one of the benefits from using CBM!
The printed Q I distributed was indeed wary of ambiguities:
“UK speed limits for a car (no trailer) on the following roads (not otherwise signposted) are ..
(1) dual carriageways
(2) motorways
…”
Due authority had been consulted on matters such as trailers! So how sure am I about what I said about answers? Despite the 6 years, I’m sure I would have started not with Q1, but with “On motorways of course it is 70 mph, and I expect everyone has it right and has put down C=3” (or some such). Everybody does know that! The whole point of my using this example was that I knew lots of people (myself included till I looked it up) were uncertain about limits on dc’s and single roads – especially those of us old enough to have driven when they were 50 & 60mph for single and dual (1973-7). No way would I have said ‘of course’ about dual carriageways unless through an absolute slip of the tongue. Interestingly, this chain of reasoning about what I may or may not have said that day is just the kind of probing thought that is needed for certainty judgements about answers in a test. It is the process CBM is there to encourage in students while they are studying.
Yes, doctors need to be confident before they make a decision. Hence the importance of educating them to question their confidence and to reward them (as in CBM) for identifying and acknowledging uncertainty when appropriate. My favourite story along these lines is of a doctor treating me who looked up over his notes and his spectacles and said “You know, I haven’t the faintest idea what these test results mean.” And of course he rang someone up. This definitely boosted my confidence in him as a doctor. I’m sorry if you understood what I said in a way that jarred. You don’t like negative marking (which incidentally our students have no problem with, partly because they know how to avoid it by acknowledging that they are unsure). One hopes that your inept and over-confident doctor learned better to recognise uncertainties after the grossly negative consequence he inflicted on you (and hopefully himself) through misdiagnosis. It shouldn’t have to be that way. There are unconfident and overconfident personalities in every student cohort (at least marginally gender and culture related). Both extremes suffer penalties with CBM and I think it entirely good if CBM encourages them to confront what is in both cases a handicap. CBM rewards objectively sound and honest judgement about how likely each answer is to be correct. I don’t particularly want to encourage its use in summative assessment because there isn’t wide enough experience with it yet (though in trials it has improved both reliability and validity). The challenge for the moment is how best to present it and spread its trialing in self-tests, to aid student study.
Thanks again for your blog.
Tony, thank you so much for taking the time to post such a detailed and thoughtful comment. I suspect that I didn’t make it sufficiently clear in my original posting that I am very definitely coming round to the idea of CBM – especially in some scenarios. I also agree that doctors need to be confident, or at least to have knowledge of what they do and don’t know – one of the reasons I’d never have made a doctor!
For whatever reason there seems to have been a real resurgence in interest in the issues; Jon Rosewell’s project at the OU has a lot to commend it, and I’m looking forward to hearing the outcomes.
thanks again, and very best wishes
Sally