e-assessment (f)or learning

I have posted before about the difficulties that students have with fractions and the problems that this causes. Many others recognise the same difficulty.

On a related point, students also have problems with reciprocation, and sometimes it is simply that they don’t recall that x^-2=1/x²

So the analysis of the question shown on the right should not be too surprising. The question is very well answered, but the ‘correct’ option that students are least likely to select is the one shown highlighted (top right). This is present in about 86% of all answers, but all the other correct options are present in more than 90% of answers.

The question shown was originally planned to assess students’ ability to estimate, but since we can’t prevent them from using a calculator, I adapted it to test calculator use as well.

43% of the analysed 14943 responses were correct, and the errors made were depressingly familiar:

8.1% were numerically correct but expressed to the nearest order of magnitude not to 1 significant figure.

3.1% were numerically correct but expressed to 2 significant figures.

In 2.9% of responses, the square root on the numerator had only been applied to the first number (4 in the example shown)

2.9% had used 10 to the plus 6 instead of 10 to the minus 6 in the denominator (which may have been caused by dividing rather than multiplying by 10 to the minus 6).

So it remains the simple things that cause the problems. Sig figs, calculator use, precedence…

I’ve said before that students are not good at giving answers to an appropriate number of significant figures. But what do they do wrong? The question shown on the left provides some insight.

The correct three options are B, C and E and they are the three most commonly selected. However options C and E are selected more frequently than option B. The two incorrect responses are selected an approproximately equal number of times – perhaps due to guessing; this question is in formative-only use.

Things get more interesting when you just look at responses from students who use all three responses before getting the question right – or who fail to get it right at all. Whilst 69.2% of these responses include Option C and 65.1% include Option E, only 56.4% include Option B – and an almost identical proportion of responses (56.0%) include the incorrect Option A. So leading and trailing zeroes cause problems.

I’ve posted before about the difficulties students seem to have with significant figures and rounding. This post adds some data, to give you an appreciation of the size of the problem.

The data are for the question shown on the left (from a summative iCMA, with all the implications that has as to students trying their best etc. – and the findings are very similar for all variants).  In order to answer the question the students have to write a number in scientific notation to three significant figures. In doing this, they have to round the number up. Of the responses analysed:

323 (90.2%) of responses were given in scientific notation

308 (86.0%) of responses were given to three significant figures

270 (75.4%) of responses had been rounded up.

(more…)

The question shown on the right is quite well answered even though it is on the formative-only practice assessment. Errors when they do occur are mostly as I’d have predicted – 4.5% of responses give the wrong sign in the power of ten and 2.5% of responses raise 2.6  rather than 10 to the correct power. Then a small number give answers that are 10 times too big or 1o times too small or fail to use the superscript function correctly.

However a comparison of the different variants of this question is interesting. A variant where students have to express 0.1578 in scientific notation appears to cause significantly more difficulties than the one shown above. At one level this is the opposite from what you might have expected, because it is all too easy to miscount the zeroes in 0.00026. It appears that ten to the minus one is a difficult power to understand.

It was the analysis of question such as the one above that led to one of my early insights into mathematical misunderstandings. I discovered then that a common answer to this question was 11809.8. This is caused by students finding 3¹⁰ and then dividing the whole thing by 5. So either students misunderstand the rules of precedence or they just can’t use their calculator. As you’ll see from above, I have since added targeted feedback in response to errors of this type.

A more recent analysis has shown (more…)

I’ve posted before about the fact that whilst students usually engage quite well with formative-only iCMA questions, when the going gets tough, they are inevitably more likely to guess than is the case when the mark counts. When I eventually get to the end of my course writing (and associated preference for blogging about things to do with maths misunderstandings, on the basis that this is relevant for the course writing too), I will talk about our changes of assessment strategy in the Open University Science Faculty. For now, I just want to reflect on the size of the formative vs summative effect. Don’t treat this too seriously, but I think the answer to ‘how big is the effect’ may be 3%. Read on.

Conside the question shown on the right. Variants of this question occur both in the formative practice assessment and in one of the summative end-of-module assessments.

In the practice assessment, 74.3% of responses were correct whilst 6.2% gave the number given in the question (so for this variant, they gave an answer of 4 – presumably guessing). In the summative equivalent, 77.7% of responses were correct whilst 3.3% gave the number given in the question.

The question shown on the left is actually quite well answered (81% of students got it right at the first attempt) especially since it is in the formative-only practice assessment rather than the summative end-of-module assessment.

However it is interesting that the most common error to all variants of this question (in 4.4% of responses) is to calculate the value of the fraction (3π/4 in the variant shown) rather than converting to degrees.

I find this particularly interesting because this error feels conceptually similar to  that made when students give the values at which a line has zero value rather than where its gradient is zero (see ‘Function or derivative?‘) or when they give twice a number rather than squaring it. I’m not sure whether there really is some conceptually similar misunderstanding or whether the students who make these mistakes  just haven’t a clue!

It is not a mistake that I start this post with a screenshot of the same variant of the same question that I was talking about last time.

I said that 8.2% of responses got the trigonometry or the algebra wrong. The problem is that 6.2% of responses are wrong simply because they can’t round correctly.

To five significant figures, the correct answer is 39.495 metres. As I’ve said before, people seem to be very poor at rounding this sort of number. The correct answer, to the requested two significant figures, is 39 metres, but those 6.2% of responses give the answer of 40 metres.

There are two lessons here

1. Improve the teaching on rounding (already done);

2. Don’t assume you know what students will do wrong in e-assessment questions. Look at real responses from real students. I don’t apologise for saying that yet again. If you don’t do this you will be left as I have been left here with a question that, for a sizeable fraction of students, is not assessing what the author (me!) intended.

Ask a university teacher of science about their new students’ mathematical difficulties and the chances are you’ll be told that students can’t rearrange equations. They may go on to tell you that this is the fault of poor school-teaching or of dumming down the school curriculum ‘these days’. I used to think that this argument was wrong on two counts. I still think that we should be looking deeper into the causes of our students’ misunderstandings rather than apportioning blame. But what about rearranging equations? (more…)