{"id":809,"date":"2012-01-19T17:53:22","date_gmt":"2012-01-19T17:53:22","guid":{"rendered":"https:\/\/www.open.ac.uk\/blogs\/SallyJordan\/?p=809"},"modified":"2012-01-23T18:25:11","modified_gmt":"2012-01-23T18:25:11","slug":"multiple-choice-vs-short-answer-questions","status":"publish","type":"post","link":"https:\/\/www.open.ac.uk\/blogs\/SallyJordan\/?p=809","title":{"rendered":"Multiple choice vs short answer questions"},"content":{"rendered":"

I’m indebted to Silvester Draaijer <\/a>for leading me\u00a0towards an interesting article:<\/p>\n

Funk, S.C. & Dickson, K.L (2011) Multiple-choice and short-answer exam performance in a college classroom. Teaching of Psychology<\/em>, 38 (4), 273-277.<\/p>\n

The authors used exactly the same questions in\u00a0 multiple-choice and\u00a0short-answer free-text response format – except (obviously)\u00a0the short-answer questions did not provide answer choices. 50 students in an ‘introduction to personality’ psychology class attempted both versions of each question, with half the students completing a 10 question short-answer pretest before a 50 question multiple-choice exam and half the students completing the 10 short-answer questions as a post-test after the multiple-choice exam. The experiment was run twice (‘Exam 2’ and ‘Exam 3’, where students didn’t know what format to expect in Exam 2, but did in Exam 3).<\/p>\n

In each case the performance on multiple-choice items was significantly higher (p<0.001) than performance on the same items in the short-answer test. The mean scores are given below:<\/p>\n\n\n\n\n\n\n\n
Exam 2<\/td>\nMean<\/td>\nExam 3<\/td>\nMean<\/td>\n<\/tr>\n
Group 1<\/td>\nSA pretest<\/td>\n36.67<\/td>\nGroup 2<\/td>\nSA pretest<\/td>\n31.81<\/td>\n<\/tr>\n
MC<\/td>\n75.56<\/td>\nMC<\/td>\n78.64<\/td>\n<\/tr>\n
Group 2<\/td>\nMC<\/td>\n82.27<\/td>\nGroup 1<\/td>\nMC<\/td>\n75.00<\/td>\n<\/tr>\n
SA posttest<\/td>\n66.36<\/td>\nSA posttest<\/td>\n53.33<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Note that I\u2019ve reported the data exactly as published even though I think they have given too many significant figures!<\/p>\n

The authors summarise the results as follows (p.275) ‘In short, students who were unable to answer several short-answer items were able to answer significantly more of the same items when presented in a multiple-choice format several minutes later…[and] students who were able to answer multiple-choice items were unable to answer a few of those same items in a short-answer format when presented minutes later.’<\/p>\n

Moving on to reflect on this result – first of all my reflection. This result seems very\u00a0plausible – if you provide prompts, students are far more likely to recognise the correct response than they are to work it out for themselves. \u00a0But it’s not the same result as I found when looking at a range of different free-text and multiple-choice questions. Not all\u00a0short-answer questions are harder than multiple-choice questions and not all multiple-choice questions are easier than short-answer questions, and we happened to have asked a selection of short-answer questions which, overall, were easier than our multiple-choice questions. See my previous posting <\/a>on this.<\/p>\n

Now the authors’ reflection. They say that it is a simplification to equate multiple-choice questions with recognition processes and short-answer questions with recall\u00a0processes.\u00a0They go\u00a0on to\u00a0say ‘students often need to understand and interpret information when answering multiple-choice questions in addition to merely recognizing correct answers’. I agree and think I’d take this point slightly further – I’m disappointed to see such a lot of emphasis on recognition and recall. They’re implying that recall is ‘better’ than recognition, which is probably true, but surely assessment should be about more than either recognition or recall. My memory is hopeless but I can still do physics (I do appreciate that recall is more necessary in some disciplines).<\/p>\n

In other aspects I agree wholeheartedly with what the authors say : ‘…we know that multiple-choice questions allow for a variety of strategies for identifying correct answers: recognition, recall, analysis and other test-taking strategies such as eliminating wrong answers or guessing’ …'[multiple choice questions] may inadvertently foster dependency’ (I think what they’re saying here is that students learn strategies for passing multiple-choice exams and so success in a multiple-choice exam doesn’t necessarily imply knowledge or understanding of the course)…’Performance on multiple-choice exams may provide inaccurate information to instructors concerning student learning and overestimate students’ learning of course information’.<\/p>\n

I know that there are bad multiple-choice questions and not so bad ones and I also know that other techniques (e.g. certainty-based marking) can help to reduce some of the validity issues with multiple-choice questions, but this paper definitely provides food for thought.<\/p>\n","protected":false},"excerpt":{"rendered":"

I’m indebted to Silvester Draaijer for leading me\u00a0towards an interesting article: Funk, S.C. & Dickson, K.L (2011) Multiple-choice and short-answer exam performance in a college classroom. Teaching of Psychology, 38 (4), 273-277. The authors used exactly the same questions in\u00a0 … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[77,86,1],"tags":[356,360],"class_list":["post-809","post","type-post","status-publish","format-standard","hentry","category-multiple-choice-questions","category-question-difficulty","category-uncategorized","tag-multiple-choice-questions","tag-question-difficulty"],"_links":{"self":[{"href":"https:\/\/www.open.ac.uk\/blogs\/SallyJordan\/index.php?rest_route=\/wp\/v2\/posts\/809","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.open.ac.uk\/blogs\/SallyJordan\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.open.ac.uk\/blogs\/SallyJordan\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.open.ac.uk\/blogs\/SallyJordan\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.open.ac.uk\/blogs\/SallyJordan\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=809"}],"version-history":[{"count":7,"href":"https:\/\/www.open.ac.uk\/blogs\/SallyJordan\/index.php?rest_route=\/wp\/v2\/posts\/809\/revisions"}],"predecessor-version":[{"id":815,"href":"https:\/\/www.open.ac.uk\/blogs\/SallyJordan\/index.php?rest_route=\/wp\/v2\/posts\/809\/revisions\/815"}],"wp:attachment":[{"href":"https:\/\/www.open.ac.uk\/blogs\/SallyJordan\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=809"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.open.ac.uk\/blogs\/SallyJordan\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=809"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.open.ac.uk\/blogs\/SallyJordan\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=809"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}