![\"\"](\"https:\/\/www.open.ac.uk\/blogs\/SallyJordan\/wp-content\/uploads\/2011\/01\/snowflake-before.png\" "\\"snowflake-before\\"")
<\/a><\/p>\nInitially we only used the questions formatively and we didn’t tell students how long we expected their responses to be. The length distribution of 888 responses to the snowflake question is given below:<\/p>\n
The distribution is bimodal with peaks on one word (corresponding to responses such as ‘balanced’) and three words (corresponding to responses such as ‘They are balanced’ and ‘equal and opposite’). There are a small number of very long responses.<\/p>\n The first time this question was used in low-stakes summative use, we again didn’t tell students how long we expected their responses to be, although the general instructions implied that they should\u00a0answer in a single\u00a0sentence. The length distribution (below) of 2057 responses\u00a0is again bimodal but the peaks are now on 3-4 words (e.g. ‘The forces are balanced’)\u00a0and 8-9 words (e.g. ‘The forces of gravity and air resistance are equal’).<\/p>\n There are more excessively long responses, such as the following one (64 words):<\/p>\n Gravity from the earth core is pulling on the snow flake and no other force is acting on it\u00a0 for example wind. When the snow flake reach the ground\u00a0 there are two forces acting on it. One is the gravitational pull from the earth and the upward force from the ground. Both these force are balance which means that the snow flake is statioary<\/em><\/p>\n The answer matching generally coped well with responses of all lengths, but very long responses are more likely to contain a correct response within an incorrect one, identified by Tom Mitchell as being the biggest problem area for the automated marking of short answer questions. For this reason, we decided to impose a filter that meant that only responses of 20 words or\u00a0fewer would be accepted for marking (if students gave an answer that was too long, they were told and given a ‘free go’ in which to shorten their answer. The warning ‘You should give your answer as a short phrase or sentence<\/em>. Answers of more than 20 words will not be accepted<\/em>.’ was added to all questions.<\/p>\n So what was the effect? You may have guessed.\u00a0The length distribution\u00a0of 1991 responses is\u00a0\u00a0shown below: There are still peaks at 4 words and 8-9 words and the filter has obviously been effective in removing the excessively long answers. But the general trend is towards longer<\/strong> answers (the mediam has increased from 8 words to 10 words); some students believed that we were telling them that we expected the answer to be 20 words long and so they gave responses that were unnecessarily long.<\/p>\n So we have removed the specific instruction about length. We now just tell students: ‘You should give your answer as a short phrase or sentence’<\/em> and they are only told that their\u00a0response needs to no more than 20 words if their original\u00a0response is longer than this. The latest length distribution (for 1919 responses) is shown below. The bulk of responses with length of 20 words or the high teens has gone, and the median has returned to 8 words. Phew!<\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n