How long is short?

Posted on January 23, 2011 by Sally Jordan

I’ve been looking at student responses to our short-answer free-text questions. I’ll start by considering something simple; how long are the responses? Continue reading →

Posted in short-answer free text questions, student engagement, Uncategorized | Tagged , , | 1 Comment

Summative and formative are not opposites

Posted on January 20, 2011 by Sally Jordan

I promise that this will be my last post about the difference between summative and formative assessment per se.  It seems to be something that bothers people; maybe I’ve caught the bug!

I used to imagine a continuum that had formative assessment at one end and summative assessment at the other, and the  debate hinged around whether you believed that the formative (‘assessment for learning’) or summative (‘assessment of learning’) function was more important i.e. where on this continuum you sat. Bull and McKenna talked about a blurring of the boundaries of formative and summative assessment, and I agreed.

But actually I think you can go further than that. Surely summative and formative assessment are just different things. Summative assessment is about measuring; formative assesment is about learning. Surely both can happen at the same time (or neither! – on the basis that assessment isn’t formative just because it is meant to be). Formative assessment can be summative; summative assessment can be formative.

Posted in formative assessment, summative assessment | Tagged , | Leave a comment

Pedagogy and patterns

Posted on January 20, 2011 by Sally Jordan

While I’ve been absent from this blog I’ve been doing lots of analysis of student responses to our questions. More on that to follow. I’ve also been reading a variety of stuff, including the report on the  JISC-funded project ‘Scoping a vision for formative e-assessment’ and a selection of papers about the same work. I don’t agree with everything in this report, but I do absolutely agree that ‘no assessment technology is in itself formative, but almost any technology can be used in a formative way – if the right conditions are set in place’. I also agree that problems arise when practices are driven by ‘state-of-the-art technological know-how rather than pedagogy’.

The report identifies design ‘patterns’ of processes in formative assessment that can be supported by software tools. Continue reading →

Posted in formative assessment | Tagged , , , | Leave a comment

Why assessment isn’t really like central heating

Posted on December 13, 2010 by Sally Jordan

Back in the early days of this blog, I muttered about our misunderstanding of the word ‘feedback’ in the context of assessment and gave examples of positive and negative feedback from everday life. I talked about the thermostat on a central heating system as an example of negative feedback.

The trouble is, it isn’t quite that simple. Whereas a thermostat acts in a predictable way to close the gap between the actual and reference temperature (by causing the heating to switch off and on as appropriate),  humans respond to feedback interventions in complex (if understandable) ways. For example, Kluger and deNisi point out that if a student does well on an assignment, he or she may ‘raise the bar’  i.e. make their target level higher, effectively making the gap that has to be closed larger than would otherwise be the case.

Mind you, feedback in other contexts is also more complicated than may initially appear to be the case – global warming is a case in point!

Posted in feedback | Tagged | Leave a comment

Not all feedback interactions are helpful

Posted on December 13, 2010 by Sally Jordan

I’ve just finished reading Kluger & DeNisi’s meta-analysis of the impact of feedback interventions on performance. (Kluger, A.N. & DeNisi, A. (1996) The effects of feedback interventions on performance : a historical review, a meta-analysis, and preliminary feedback intervention theory, Psychological Bulletin, 119 (2), 254-284. It’s rather theoretical and I didn’t understand all of it, but there are some clear messages.

Firstly,  ‘feedback interventions’ (which they define as actions taken by external agent(s) to provide information regarding some aspect(s) of one’s task performance) are not the same as feedback. I think that’s a useful distinction.

Secondly, in the 131 papers included in the meta-analysis (representing 607 effect sizes, 12,652 participants and 23,663 observations), whilst on average feedback interventions had a moderate positive effect on performance, over 38% of the effects were negative. These reported negative effects are frequently ignored, as we continue to assume that feedback interventions  are always beneficial. The paper goes on to develop a ‘feedback intervention theory’, seeking to identify ways in which the effectiveness of feedback interventions can be improved. There is a crying need for more work in this area.

Posted in feedback | Tagged , | Leave a comment

Trying our questions

Posted on December 2, 2010 by Sally Jordan

With apologies, I thought I had put links on this blog to some Open University eAssessment questions, but this was not the case. I’ll add these links to the ‘E-assessment at the Open University’ link (right-hand side of blog) in a minute.

It’s actually a bit tricky, because we can’t make summative questions generally available. However the diagnostic quiz, Are you ready for S104? (https://students.open.ac.uk/openmark/science.level1ayrf.s104/) should give you the idea.

In addition, if you are specifically interested in our short-answer free-text questions, go to  https://students.open.ac.uk/openmark/omdemo.pm2009/ (though again note that our best questions are in summative use!).

Posted in e-assessment | Tagged , , , | Leave a comment

Summative assessment is not the same as giving marks

Posted on December 1, 2010 by Sally Jordan

The title of this post may sound contradictory. If we give students marks, the assessment is summative. Right? Not necessarily. It is perfectly possible to tell students their ‘mark’ for an assignment but for that mark not to count towards the final outcome (so this assessment is purely formative). Similarly, it is possible for a grade from an assignment to count towards the final outcome (and so be summative) even if the student is not told this grade. Continue reading →

Posted in feedback, summative assessment | Tagged , , , | 2 Comments

Correspondence tuition, or is it just tuition?

Posted on November 28, 2010 by Sally Jordan

We had a wonderful session  at an OU staff development meeting in Cambridge yesterday, discussing ‘Time well spent? Making the most of correspondence tuition’. Follow the links below for a copy of the handout on ‘theory’ and for a summary of our discussions. Continue reading →

Posted in correspondence tuition | Tagged , | Leave a comment

How students react to feedback from a computer

Posted on November 28, 2010 by Sally Jordan

Returning to Lipnevich and Smith’s interesting work (Lipnevich, A.A. & Smith, J.K. (2009) “I really need feedback to learn:” students’ perspectives of the differential feedback messages, Educational Assessment Evaluation & Accountability, 21, 347-367). And for the benefit of those who attended the session that Lesley and I ran at yesterday’s OU associate lecturer staff development meeting in Cambridge, yes that is how the punctuation is given in the title! And yes, I know that the first sentence in this paragraph isn’t a sentence at all, and the second and third both begin with ‘And’. 🙂

Lipnevich and Smith explored the way in which students react to feedback and one of their variables was whether the students believed the feedback (and in some instances the grade) to come from a computer or a human marker. Continue reading →

Posted in e-assessment, feedback | Tagged , | 1 Comment

Feedback, feedforward or feedaround?

Posted on November 22, 2010 by Sally Jordan

This is another of those ideas that others probably thought of years ago, but I’ve been a bit slow on the uptake. In summary, findings about the effectiveness or otherwise of feedback probably depend on what the feedback is meant to be used for. The usual OU scenario is a student receiving feedback on a tutor-marked assignment and (supposedly) using this feedback to improve for next time. But in other contexts, the feedback may be intended to enable the student to improve the same piece of work. And the ‘three attempts with increasing feedback’ that we provide on interactive computer-marked assignments perhaps has more in common with the second of these than the first. Continue reading →

Posted in feedback, Uncategorized | Tagged , | 1 Comment