Quote of the day – The Orangoutang score

Posted on March 10, 2013 by Sally Jordan

I was looking for something lighter for a Sunday afternoon and came across this. I haven’t read the original paper, just a reference to it, but I reckon that ‘Orangoutang score’ is more fun than ‘random guess score’.

‘The Orangoutang score is that score on a standardised reading test that can be obtained by a well-trained Orangoutang under these special conditions. A slightly hungry Orangoutang is placed in a small cage that has an oblong window and four buttons. The Orangoutang has been trained that every time the reading teacher places a neatly typed multiple choice item in a reading test in the oblong window, all that he (the Orangoutang) has to do to get a bit of banana is to press a button, any of these buttons, which, incidentally, are labelled A, B, C and D.’

Fry, E. (1971) The Orangoutang score. Reading Teacher, 24, 360-2.

Posted in quotes | Tagged | 1 Comment

Quote of the day – assessment anxiety (even for formative assessment)

Posted on March 8, 2013 by Sally Jordan

‘The formative assessment for Anne was not a supportive step toward summative assessment, but a significant hurdle in its own right; a moment of judgement of her aptitude for higher education and her identity. Therefore, for Anne, the formative process was one of anxious torment’ [pg515]

‘For our participants, as one assessment hurdle is jumped, another looms darkly in the distance.’ [pg517]

Cramp, A., Lamond, C., Coleyshaw, L & Beck, S. (2012). Empowering or disabling? Emotional reactions to assessment amongst part-time adult students. Teaching in Higher Education, 17(5), 509-521 [pg515]

Posted in assessment anxiety, formative assessment, quotes | Tagged , , | 2 Comments

Quote of the day

Posted on February 27, 2013 by Sally Jordan

‘Assessment is not working, or at least it is not working as it should. In our attempt to generate forms of assessment capable of addressing all the purposes for whish we use assessment, we have produced a Frankenstein that preys on the educational process, reducing large parts of teaching and learning to mindless mechanistic process whilst sapping the transformative power of education.’

Broadfoot, P. (2008) Assessment for learners: assessment literacy and the development of learning power. In Havnes, A. and McDowell, L. (eds).  Balancing dilemmas in assessment and learning in contemporary education. Routledge/Taylor and Francis. pp213-224.

Posted in quotes | Tagged | Leave a comment

Quote of the day

Posted on February 25, 2013 by Sally Jordan

Reading through my notes on some of the many assessment papers I have read, I’m finding a few of those ‘sit up and take note’ quotes; things (sometimes very obvious) that other people somehow manage to say so much better than I can. So, I bring you the first of an occasional series of ‘Quote of the day’:

…’ summative assessment is itself ‘formative’. It cannot help but be formative. This is not an issue. At issue is whether that formative potential of summative assessment is lethal or emancipatory. Does summative assessment exert its power to discipline and control, a power so possibly lethal that the student may be wounded for life?’

Barnett, Ronald (2007) Assessment in higher education: an impossible mission? In Boud, David and Falchikov, Nancy (eds) Rethinking assessment in higher education. London, Routledge. pg37.

Posted in quotes, summative assessment | Tagged , | 2 Comments

Selected response or constructed response?

Posted on February 23, 2013 by Sally Jordan

I have had an interesting debate with colleagues about whether questions in which you have to drag one or more markers to appropriate places on an image (see example below) are selected response or constructed response questions.

I am of the opinion that this is a constructed response question, because students are not given clues as to where the markers should go. It is fundamentally different to the question below (‘drag and drop onto image’) which is selected response because there are only a number of places where the labels can go.

However, during this debate, my colleague pointed out that the boundaries between constructed and selected response question types are not that clear cut. In a sense the top image is selected response because there are a finite number of pixels in the image. Similarly if you ask a numerical question in which you want an answer that’s in integer between 1 and 9, there are actually only 9 options available to you. For what it’s worth, I still think both of these are constructed response questions, but the debate is an interesting one.

Posted in constructed response, selected response | Tagged , | 3 Comments

The hidden curriculum

Posted on February 23, 2013 by Sally Jordan

This morning I’ve been reading an oldish paper (Sambell & McDowell, 1998) about work on the ‘hidden curriculum’, an even older phenomenon (Snyder, 1971). 

The hidden curriculum can be thought of in terms of the distinction between ‘what is meant to happen’ i.e. the curriculum stated officientally by the educational system or institution, and what students actually experience ‘on the ground’. Assessment is very important in determining the hidden curriculum.

Sambell & McDowell take this one step further. They point out that every student has a different hidden curriculum; the same assessment is interpreted differently not just by ‘staff’ and ‘students’ but by individuals. Students bring with them different range experiences, motivations and perspectives which influence their response.

In the light of this, how are we do work to improve the assessment experience (and hence the hidden curriculum) for all our students, especially in a large and diverse University such as the Open University? Dealing with each individual student’s previous experiences and perceptions is challenging. However I think we are still missing some very obvious tricks. We assume that our students share our understanding of what assessment is for, but many probably don’t. I am interested in doing some work to improve that understanding, perhaps by running online tutorials for students before their first assignment, to discuss the assessment process not the assessment itself. Then we might to similarly after students have got the first piece of work back, to discuss what they have learnt from it and from the feedback.

Anyone want to join in the fun?

Sambell, K. & McDowell, L (1998) The construction of the hidden curriculum: messages and meanings in the assessment of student learning, Assessment and Evaluation in Higher Education, 23(4), 391-402.

Snyder, B.R. (1971) The hidden curriculum. New York: Alfred A. Knopf.

Posted in hidden curriculum | Tagged | Leave a comment

SIREJ

Posted on February 19, 2013 by Sally Jordan

I’ve been on the road even more than usual recently, dealing with a domestic crisis of a rather unusual nature (a tree through the roof of my elderly in-laws’ house down in Sussex). They’re OK, but assessment work is low priority right now. Apart from thinking about it.

And when I think, I always come to the same conclusion – what we do isn’t really very good. There’s lots of theorising out there, but our assessment practice still isn’t great. So, after my previous timid posting, let’s think big. This is my personal manifesto for change. In order to improve our assessment practice, we need to be truly:

Student centred (which isn’t the same as saying that we should always do what students say they want, but our students and learning should come first. Always).

Innovative (or stictly, open to innovation, making change when that’s appropriate, but not when it isn’t).

Reflective

Evaluative

Joined up (trying to make our practice more coherent – I’m not saying that all practice should be identical, but rather that we should think about the big picture).

Think SIREJ – my initials with anger in the middle!

Posted in assessment design | Tagged | Leave a comment

Automatic generation of assessment items

Posted on February 11, 2013 by Sally Jordan

Last week I participated in a fascinating Transforming Assessment webinar on ‘The semi-automatic generation of assessment items: objectives, challenges, and perspectives’ . The presenter was Muriel Foulonneau from the Henri Tudor Research Centre, Luxembourg, and I was left feeling very much an amateur in a ‘room’ full of very clever professionals. The way in which assessment items and distractors are automatically generated is really very clever – with my ‘day job’ and so many wide-ranging interests in assessment, I feel I will never be able to keep up with what is going on.

However I was absolutely amazed to discover that one of the drivers for this work is the cost of producing test items – $1500-$2500 per item. How so? We are only talking about simple multiple-choice items here, how can it possibly cost that much?

Posted in Automatically generated questions, cost | Tagged , | 1 Comment

Feedforward and dialogue

Posted on January 26, 2013 by Sally Jordan

Until this morning, I thought the term ‘feedforward’ was something that had been invented recently – indeed, I thought it had its origins around the time of the FAST (Formative Assessment in Science Teaching) Project and the now famous Gibbs & Simpson (2004-5)literature review which was (I think! – I am now trying to be careful about what I attribute to whom) done as part of the scene setting for the FAST Project. I was certainly wrong about ‘feedforward’ – it turns out that the term was introduced by Mats Bjorkman in 1972 in the following paper:

Bjorkman, M. (1972) Feedforward and feedback as determiners of knowledge and policy: Notes on a neglected issue. Scandinavian Journal of Pyschology, 13, 152-158.

‘Feedforward’ is a useful term in that it makes us remember that we should be looking forwards not backwards. However, as I have argued previously, ‘feedback’ in its use in science and engineering is NOT backward looking. I have shown a picture of our current cold and snowy weather because it is prettier than a picture of my central heating thermostat (please bear with my rather peculiar logic which has led me to show this photo simply because central heating keeps us warm in cold weather). The thermostat uses information about the current temperature, to control the temperature in the future. This is feedback (actually ‘negative feedback’, but let’s not go there!) – a process that uses information.

So I don’t really like the term ‘feedforward’. I don’t think we need it. Let’s just remember that assessment feedback needs to be a process, most significantly a process that involves the student as well as the teacher, not just a flow of information from the teacher to the student. Then it will be forward looking. This is entirely in line with a mass of recent literature that argues in favour of assessment as a dialogic process.

Posted in dialogue, feedback, feedforward | Tagged , , , | Leave a comment

Diagnostic quizzes

Posted on January 25, 2013 by Sally Jordan

Following on from the discussion with Tim on my previous post, it occurs to me that our online quizzes that consistently attract the highest number of users are our diagnostic ‘Are you ready for?’ quizzes. Since it opened in April 2012, more than 6000 people have completed ‘Are you ready for S104?‘ and more than 13000 others have started but not completed this quiz. It’s open to the world, so take a look – are you ready? have you got sufficient time for study?

I wouldn’t claim that everyone who clicks on the link is seriously contemplating Open University study, but a lot are, and we have a huge amount of evidence that many students have found previous versions of this quiz to be useful, either informing them that they need to do more preparation before beginning their studies, or – just as important – reassuring them that they are ready.

But yet, I don’t think we make as much use of diagnostic quizzes as we should. It is all too easy to sign up for a qualification without any real understanding of the time required or the level of study. I think that we should be making engagement with a quiz of this type mandatory, as part of the registration process.

Posted in diagnostic assessment | Tagged , | Leave a comment