CAA 2012

Last week I attended the International Computer Assisted Assessment Conference in Southampton. This is the third consecutive year I’ve attended this conference and I enjoyed it, even if it was sometimes challenging to the point of being depressing.

So what is there to be depressed about?  Bobby Elliott from the Scottish Qualifications Agency said ‘CAA2002 would be disappointed in CAA 2012’ – not because of the conference itself, but because computer aided assessment has not achieved as much as was hoped 10 years ago.  Sue Timmis from the University of Bristol summed up the problem by saying by saying that, in reviewing the literature relating to the use of digital technologies in assessment, she and colleagues have not yet found evidence of a transformative effect. Steve Draper from the University of Glasgow and the Keynote speaker, raised another issue in saying that there is not much evidence of the effectiveness of feedback given from tutor to student.

So, on one level, has all of our work been a waste of time? I think I’d be slightly more optimistic if only because most of the conference attendees were interested in these issues, rather than talking about a wish to use technology whether or not that is the best solution from the students point of view. So at least our focus is on learning and teaching and we are looking for evidence of effectiveness rather than sailing on regardless – now we just have to get it right!

One good thing that came out of the conference is that John Kleeman told me about his Assessment Prior Art wiki – do take a look.

Posted in conferences | Tagged , , | Leave a comment

Self-assessment. Discuss!

Whether or not you’re coming to our seminar ‘Self-assessment: stratgies and software to stimuate learning’ on 11th June, there are lots of things worthy of discussion.

Many of the speakers have provided links that you might like to look at:

Gwyneth Hughes on Ipsative Assessment

Tony Gardner-Medwin on Certainty-based Marking and CBM in Moodle

Peerwise (Paul Denny is in New Zealand so we will be showing a video) and a JISC-funded project that is evaluating the use of Peerwise at Edinburgh University.

Jon Rosewell on ‘Assessing with confidence’

And, from Phil Butcher et al at the Open University, ‘eAssessment at the Open University with open source software.’

Continue reading

Posted in self assessment | Tagged | 1 Comment

Exam marking errors

I’m pleased to hear that OCR have apologised for errors in adding up marks for GCSE and A-level papers last year. It doesn’t seem right that the whistleblower remains suspended, but I don’t know the details so perhaps I shouldn’t comment.

When I first heard about this latest case of human error in exam marking, I was amazed that we are still reliant on human arithmetic (though I know that addition of scores is meant to be checked by someone else – and that payment for this is included in the payment of GCSE and A-level markers). However I suppose that if markers were required to enter their scores into a computerised system of some sort (to enable the computer to check the arithmetic, or to do the arithmetic in the first instance) there would still be transcription errors – and it would take time and so slow down the marking process.

The important point is that, however much their work is checked and however much they are encouraged not to make mistakes and/or punished for making them (according to The Guardian, some examiners have had their contracts terminated), human markers are fallible. They are fallible when marking, they are fallible when adding scores. I’m a human, I make mistakes. The sooner we are honest with ourselves and admit that, the better.

Posted in human marking | Tagged | Leave a comment

Is education lighting a fire or filling a bucket?

I have a huge amount of work to do, yet this afternoon I have found myself enthralled in a number of coincidental conversations about the role of ‘incentives’ in getting students to engage with assessment, or dare I say, with learning.

With permission, I am copying  parts of Phil Langton’s latest message:

I’m coming rapidly to the conclusion that the mistake we make is to fail to put ourselves in the students’ shoes.  We ask lots of questions and expect that by weight of numbers of sheer dumb luck the  students will begin to see the world as we do.

Wrong! Continue reading

Posted in incentives | Tagged , | Leave a comment

Formative thresholded assessment

Building on work that others have done, in particular my colleagues Joy Manners and John Bolton in the OU Department of Physical Sciences, we are encouraging a move across the Science Faculty to what we are calling ‘formative thresholded assessment’. Continue reading

Posted in formative thresholded assessment | Tagged | 2 Comments

JISC Assessment and Feedback Programme

I’m just back from my second attendance at a JISC Learning and Teaching Practice Experts Group meeting. It was excellent – it is inspiring to be surrounded by people who know such a lot about learning and teaching and, more importantly, actually care about the student experience.

Half the day was spent discussing some of the early outcomes from the new JISC Assessment and Feedback Programme. I was involved in an unsuccessful bid to this programme, but can truly say that I am happy not to have been successful (not sure what my colleagues think!).  I’m just SO busy and it is nice to learn from others rather than being in the thick of it. Continue reading

Posted in assessment design, JISC | Tagged , | 2 Comments

Self-assessment seminar

We have funding from the HEA for a seminar on self-assessment that we (Tony Gardner-Medwin and me) are running at the OU in Milton Keynes  on 11th June 2012.

It’s a really interesting topic and we have some excellent speakers lined up. How does testing affect learning – for good or ill? What part does ipsative assessment have to play? How about certainty-based marking? And there will be an update on the Moodle quiz engine. Click on OU_prog  for the  full programme or on http://www.ucl.ac.uk/~ucgbarg/OU_workshop.htm for more information (including booking information).

Do come!

Posted in self assessment | Tagged | Leave a comment

Learning from failure

I’m indebted to Phil Langton to pointing me towards an article by Steven Schwartz in Times Higher Education. The article, Get back in the saddle, explores the way in which failure and life’s knocks can lead to success in the future.

The theme is similar to that discussed in comments on a previous post of mine. I think it is certainly the case that we sometimes molycoddle our students. It doesn’t seem right to be deliberately hard or harsh in our grading or feedback, but I think we should be honest. If a student’s work is not up to the mark, we should tell them. It may be painful for all, but if the article is to believed (and it rings true) this approach may be just what is needed in order to encourage our students to succeed in the future.

Posted in learning from failure | Tagged | 2 Comments

x to the minus 1 = 1 over x. OK?

I have posted before about the difficulties that students have with fractions and the problems that this causes. Many others recognise the same difficulty.

On a related point, students also have problems with reciprocation, and sometimes it is simply that they don’t recall that x-2=1/x2

So the analysis of the question shown on the right should not be too surprising. The question is very well answered, but the ‘correct’ option that students are least likely to select is the one shown highlighted (top right). This is present in about 86% of all answers, but all the other correct options are present in more than 90% of answers.

 

Posted in mathematical misunderstandings | Tagged , , | Leave a comment

Yet more problems with significant figures, using a calculator for scientific notation and precedence

The question shown was originally planned to assess students’ ability to estimate, but since we can’t prevent them from using a calculator, I adapted it to test calculator use as well.

43% of the analysed 14943 responses were correct, and the errors made were depressingly familiar:

8.1% were numerically correct but expressed to the nearest order of magnitude not to 1 significant figure.

3.1% were numerically correct but expressed to 2 significant figures.

In 2.9% of responses, the square root on the numerator had only been applied to the first number (4 in the example shown)

2.9% had used 10 to the plus 6 instead of 10 to the minus 6 in the denominator (which may have been caused by dividing rather than multiplying by 10 to the minus 6).

So it remains the simple things that cause the problems. Sig figs, calculator use, precedence…

Posted in mathematical misunderstandings | Tagged , , , | Leave a comment