Dependability – the one-handed clock

Posted on September 8, 2010 by Sally Jordan

This is  my final post relating directly to the Earli/Northumbria Assessment Conference. Well that’s a relief I hear you say. It was an amazing conference for me, coming at just the right time in my thinking about broader issues in assessment.

This post continues the theme of ‘quality’. In the final keynote, instead of dealing separately with issues of validity, reliability, authenticity and manageability (in practical terms), Professor Gordon Stobart talked about a ‘one handed clock’ . You have to decide where in the 360-degree round to place the one hand when construct validity, reliability and manageability are equally spaced (at 120 degree intervals) around it.  This is a useful way of thinking, capturing the tensions I was trying to describe in my previous post.

Posted in quality | Tagged , , , , , , , , | Leave a comment

Do we know what we mean by ‘quality’ in e-assessment?

Posted on September 8, 2010 by Sally Jordan

This was the topic of my roundtable at the Earli/Northumbria Assessment Conference and I am very grateful to the 10 people who attended one of the two wonderful discussions we had on the topic.

The obvious answer is that, no, we don’t know what we mean by ‘quality’. We don’t even know what we mean by ‘e-assessment’, Having discussed this a bit, we moved on to discuss different aspects of ‘quality’. I note that both groups, whilst mentioning that validity and reliability are important, also emphasised the role of e-assessment in transforming learning, in particular through its ability to provide instantaneous feedback. Continue reading →

Posted in quality | Tagged , , , | Leave a comment

Why don’t the marks go up?

Posted on September 8, 2010 by Sally Jordan

Given tha amount of work that we put into formative assessment, why don’t students do better in summative assessment?

This was one of the recurring themes at the EARLI/Northumbria Assessment Conference, first raised in Liz McDowell’s keynote. Liz wondered if the explanation might be that we sometimes norm-reference our summative assessment, even when we claim we don’t do this.

Sue Bloxham (in discussion after Liz’s paper) and Gordon Stobart (in the final keynote of the Conference) saw the problem as a lack of alignment between formative and summative assessment.

If our formative interventions are not having any obvious impact, we certainly ought to be investigating this further.

Posted in formative assessment, summative assessment | Tagged , , , , | Leave a comment

Is less more? The Goldilocks of assessment

Posted on September 7, 2010 by Sally Jordan

After my tirade in one of the EARLI/Northumbria Assessment Conference sessions about the true meaning of ‘feedback’  (see the second posting in this Blog, 30th July 2010) here I go again, doing exactly what I mutter at other people for doing. Just for now please bear with me and take ‘feedback’ to mean the words we write on a student’s assignment.

Peter Rawlins from New Zealand  reported on an interesting study which found that whilst teachers said ‘there is no point giving more feedback; students won’t use it’, students actually found the feedback useful and used it. This finding rather contradicts some of my work (which has found that although students might say that they find feedback useful, they don’t necessarily make much use of it). However, perhaps the issue is ‘how much feedback is the most useful’. I have a feeling that less is sometimes more. Continue reading →

Posted in feedback | Tagged , , , | Leave a comment

Continuous or terminal assessment?

Posted on September 7, 2010 by Sally Jordan

I’m a bit slow on the uptake. I’ve now moved on from the EARLI/Northumbria Assessment Conference in Northumberland to ALT-C 2010 in Nottingham, with a day of walking, a day of writing and a day of interviewing in between. Before my thinking gets clouded with lots of things from ALT-C, I want to return to some of the issues raised at the EARLI/Northumbria Assessment Conference. In fact I want to return right to the beginning, to Royce Sadler’s keynote in which he suggested that all summative assessment should be at the end of a period of teaching, not during it i.e. that it should be terminal not continuous. Continue reading →

Posted in summative assessment | Tagged , , , , , | Leave a comment

Assessing achievement, not ‘being alive’

Posted on September 2, 2010 by Sally Jordan

I’m at the EARLI/Northumbria Assessment Conference at the Slaley Hall Hotel in Northumberland (UK). Yesterday Royce Sadler got the conference off to a fine start with a challenging Keynote ‘Close-range assessment practices with high yield prospects’. ‘Close-range’ refers to things that are within the reach of teachers; ‘high-impact’ refers to things that might have a substantial impact on learning. Professor Sadler focussed on the assessment of achievement with all the sense of satisfaction that this brings. Continue reading →

Posted in summative assessment | Tagged , , , , , | Leave a comment

Partial credit for correct at second or third attempt

Posted on August 30, 2010 by Sally Jordan

One of the features of OpenMark, the OU’s e-assessment system, is the fact that students are allowed several (usually three) attempts at each question, and receive hints which increase in detail after each unsuccessful attempt. This is the case even in summative use, where the marks awarded decrease in line with the amount of feedback that has been given before the question is correctly answered.

The provision of increasing feedback is illustrated in the figure below.

The way in which we give partial credit when a question is only answered following feedback contrasts with other other systems which give partial credit for partially correct answers (we sometimes do that too). Is one of these approaches better than the other? I have always liked our approach of giving increasing feedback, and it has recently been pointed out the me that it is also good if students can be encouraged to get a completely correct answer for themselves. However, I think it is important that we tell students if their answer is partially correct, rather than letting them think that it is completely wrong – and so sending them off on a wild goose chase!

I also think that one of the remaining issues for the use of e-assessment of this type, especially in assessing maths, is the fact that we don’t give credit for ‘working’, which of course is so much a feature of human marking.

Posted in partial credit, Uncategorized | Tagged , , , | 1 Comment

Overall impact of different variants of questions

Posted on August 29, 2010 by Sally Jordan

You may be relieved to hear that this will be my final posting (at least for a while) on our use of different variants of interactive computer-marked assignment (iCMA) questions. We know that, whilst the different variants of many questions are of equivalent difficulty, we can’t claim this for all our questions. How much does this matter? A typical iCMA might include 10 questions and contribute something between 2 and 5% to a student’s overall score. My Course Team colleagues and I have reassured ourselves that the effect of the different difficulty of some variants of some questions will, when averaged out over a large number of questions, have minimal impact on a student’s overall score. But is this true? Continue reading →

Posted in statistics, variants | Tagged , , , , | 2 Comments

So are the variants of equivalent difficulty?

Posted on August 29, 2010 by Sally Jordan

One glance at the figure from the previous post (reproduced to the right) makes it clear that whilst the variants of the question shown at the top are equivalent, those for the lower question are not.

Reasons why variants may be of differing difficulty include

  • the variables selected may result in a more difficult mathematical task (e.g. rounding up instead of rounding down, understanding of negative exponents rather than positive ones)
  • a graph to be interpreted may have a more awkward scale to read, or if different readings are to be taken from the same graph, some students may have to interpolate or extrapolate whilst others are taking readings where the graph crosses the grid-lines.
  • the letters used may appear similar in lower and upper case and so be confused e.g. k and K are very similar whilst q is unlikely to be confused with Q.
  • the words used in setting up the question may use language or describe a situation which is unfamiliar to the student. Continue reading →
Posted in variants | Tagged , , | 1 Comment

Investigating whether variants of a question are of equivalent difficulty

Posted on August 29, 2010 by Sally Jordan

We have devised a range of tools to determine whether or not the variants of a question are of equivalent difficulty. Continue reading →

Posted in statistics, variants | Tagged , , | 1 Comment