Archive for May, 2011

Random guess scores

Tuesday, May 31st, 2011

As an extension to my daughter Helen’s  iCMA statistics project, random guess scores were calculated for multiple choice, multiple response and drag and drop questions in a number of different situations (e.g. with different numbers of attempts, different scoring algorithms, different numbers of options to select from and different numbers of options being correct, students being told how many options were correct, or not).

The random guess score for a question is essentially the score that you would expect from someone who is completely logical in working though the question but knows absolutely nothing about the subject matter.

Helen’s report is here.

(more…)

Fair or equal?

Tuesday, May 31st, 2011

This post returns to ideas from Gipps and Murphy’s book ‘A fair test?’. We use words like ‘equality’, ‘equity’ and ‘equal opportunities’ frequently, in the context of assessment and elsewhere. Gipps and Murphy deliberately talk about ‘equity’ not ‘equal opportunities’ and the UK Government talk about ‘equality’ (the 2010 Equality Act came fully into force in April 2011) – all in an attempt to make their meaning more clear. I used to think I was really clued up on all of this (as a line-manager in the UK Open University, I ask a lot of interview questions relating to equal opportunities – and I was once told that the answer I gave to an interview question of this ilk was the best that the interviewer had ever heard). However, especially in the context of assessment, I’ve come to realise that things aren’t as simple as they might appear… (more…)

Not like Moses

Friday, May 6th, 2011

One of the joys of trying to catch up with others who have been working in the field of assessment for much longer than me is finding books and articles that were written some time ago but which still seem pertinent today. I’d definitely put the following book into this category (and more thoughts from it will follow):

Gipps, C. and Murphy, P. (1994) A fair test? Assessment, achievement and equity. Buckingham: Open University Press.

For now, I’d like to highlight a particularly memorable quote from Gipps and Murphy, originally from the Times Educational Supplement back in November 1988,expressing sceptism about the ‘Code of Fair Testing Practices in Education’ in the USA. As a former Assistant Secretary of Education put it:

If all the maxims are followed I have no doubt the overall quotient of goodness and virtue should be raised. Like Moses, the test makers have laid down ten commandments they hope everyone will obey. This doesn’t work very well in religion – adultery continues.

So I’d like to emphasise that my ‘top tips’ in the previous post are not commandments! – apart perhaps from my final tip (monitoring the questions when in use) which I think ought to be made compulsory.

In general though, although the ‘top tips’ have worked well for me, and I hope that these are ideas that others might find useful, perhaps it is more important that question authors take responsibility for the quality of their own work, rather than mindlessly following ‘rules’ written by others. This wish reflects most of my practice, in writing e-assessment questions and in everything else so, for example, I far prefer helping people to write questions in workshops (when they are writing questions ‘for real’) than providing rules for them to follow. Sadly, I think a wish to improve the quality of our e-assessment may be leading to a more dictatorial approach – I’m not convinced it will work.