e-assessment (f)or learning

The first keynote at the Physics Higher Education Conference was given by Professor Derek Raine of Leicester University, who is perhaps best known for using problem based learning to teach physics and interdisciplinary science. Thinking about Derek’s work leads me to the unsurprising conclusion that getting people to work things out for themselves leads to deep learning. Do we perhaps provide too much scaffolding?

Both Derek’s keynote and the second one (from Frederik Floether (just starting Part III at Cambridge and winner of the 2010 UK Physical Sciences Centre student essay competition) considered the future of physics education. Frederik described a situation in which students take learning into their own hands, using the internet to find things out for themselves. We need to encourage this sort of behaviour : ‘You can take learning into your hands and you can do it’. Frederik also described knowledge exchange in which ‘Everyone is a teacher and a learner’ – so Wikipedia has its uses!

I’m now home and conferenced out (not helped by problems on the East Coast Main Line on Friday evening). The Physics Higher Education Conference (PHEC) (in Glasgow this year) was considerably better than when I first attended the event. Some of the presentations, probably including mine, were a bit predictable and some (definitely not including mine!) were based on rather flimsy self-reported evidence of student behaviour. However there were also some excellent presentations, two excellent workshops and two excellent keynote talks.

A ‘byte-sized’ (5 minute) presentation from Vijay Tymms of Imperial College has really got me thinking. His title ‘Is repeated testing at school causing physics undergraduates to become more extrinsically motivated?’ really says it all. Students certainly seem to be becoming more extrinsically motivated in that they care more about their marks that about learning. But why is this? Vijay wonders if it might be the fault of all the testing that the National Curriculum has brought into schools? That’s certainly a possibility (as is ‘grade polishing’ by repeating AS and A-level modules). Both of these things could probably be investigated by a comparison with students from other countries. I also wonder if the increased emphasis that is placed on the importance of ‘good degree’ for employment might be significant.

I was bowled over by two posters at ALT-C: Matt Haigh’s ‘Changing the way we see test-items in a computer-based environment: screen design and question difficulty’ (session 096)and Silvester Draaijer’s ‘Design of a question design support tool’ (session 148).

Both Matt and Silvester have done things that others ought to have done years ago – but haven’t. Matt has looked at the way in which the appearance of a test items (e.g. does it include a picture) alters its apparent difficulty. Silvester has designed a tool to help academics write better e-assessment questions.

The quote I’ve used as the title of this post has been attributed to the late Professor Roger Needham at the University of Cambridge.

I can’t believe I’ve been blogging for two months and am only now mentioning our work with short-answer free-text e-assessment questions for the first time. I talked about this  yesterday at ALT-C at Nottingham University (the photo shows the Millenium Garden), in a demonstration linked to the launch of the JISC guide to ‘Effective Assessment in a Digital Age’ (in which we feature as a case study). I’m talking about the work again on Friday, at the Physics Higher Education Conference in Glasgow.

In summary: we have written computer-marked e-assessment questions in which students give their answer as a short sentence (up to 20 words). We give students three attempts at each question, with increasing targeted feedback for incomplete and incorrect responses. We have used responses from students in developing our answer-matching – I think that’s key. We’ve used both linguistically-based software (Intelligent Assessment Technologies’ FreeText Author) and the OU’s own OpenMark PMatch, which is entirely based on keywords and word-order. Both have worked well, achieving better marking accuracy than human markers. I’m not advocating  doing away with human markers, however perhaps we could use this type of e-assessment to relieve them of some of the drudgery, leaving them free to mark more sophisticated work and to support students in other ways.

This is  my final post relating directly to the Earli/Northumbria Assessment Conference. Well that’s a relief I hear you say. It was an amazing conference for me, coming at just the right time in my thinking about broader issues in assessment.

This post continues the theme of ‘quality’. In the final keynote, instead of dealing separately with issues of validity, reliability, authenticity and manageability (in practical terms), Professor Gordon Stobart talked about a ’one handed clock’ . You have to decide where in the 360-degree round to place the one hand when construct validity, reliability and manageability are equally spaced (at 120 degree intervals) around it.  This is a useful way of thinking, capturing the tensions I was trying to describe in my previous post.

This was the topic of my roundtable at the Earli/Northumbria Assessment Conference and I am very grateful to the 10 people who attended one of the two wonderful discussions we had on the topic.

The obvious answer is that, no, we don’t know what we mean by ‘quality’. We don’t even know what we mean by ‘e-assessment’, Having discussed this a bit, we moved on to discuss different aspects of ‘quality’. I note that both groups, whilst mentioning that validity and reliability are important, also emphasised the role of e-assessment in transforming learning, in particular through its ability to provide instantaneous feedback. (more…)

Given tha amount of work that we put into formative assessment, why don’t students do better in summative assessment?

This was one of the recurring themes at the EARLI/Northumbria Assessment Conference, first raised in Liz McDowell’s keynote. Liz wondered if the explanation might be that we sometimes norm-reference our summative assessment, even when we claim we don’t do this.

Sue Bloxham (in discussion after Liz’s paper) and Gordon Stobart (in the final keynote of the Conference) saw the problem as a lack of alignment between formative and summative assessment.

If our formative interventions are not having any obvious impact, we certainly ought to be investigating this further.

After my tirade in one of the EARLI/Northumbria Assessment Conference sessions about the true meaning of ‘feedback’  (see the second posting in this Blog, 30th July 2010) here I go again, doing exactly what I mutter at other people for doing. Just for now please bear with me and take ‘feedback’ to mean the words we write on a student’s assignment.

Peter Rawlins from New Zealand  reported on an interesting study which found that whilst teachers said ‘there is no point giving more feedback; students won’t use it’, students actually found the feedback useful and used it. This finding rather contradicts some of my work (which has found that although students might say that they find feedback useful, they don’t necessarily make much use of it). However, perhaps the issue is ‘how much feedback is the most useful’. I have a feeling that less is sometimes more. (more…)

I’m a bit slow on the uptake. I’ve now moved on from the EARLI/Northumbria Assessment Conference in Northumberland to ALT-C 2010 in Nottingham, with a day of walking, a day of writing and a day of interviewing in between. Before my thinking gets clouded with lots of things from ALT-C, I want to return to some of the issues raised at the EARLI/Northumbria Assessment Conference. In fact I want to return right to the beginning, to Royce Sadler’s keynote in which he suggested that all summative assessment should be at the end of a period of teaching, not during it i.e. that it should be terminal not continuous. (more…)

I’m at the EARLI/Northumbria Assessment Conference at the Slaley Hall Hotel in Northumberland (UK). Yesterday Royce Sadler got the conference off to a fine start with a challenging Keynote ‘Close-range assessment practices with high yield prospects’. ‘Close-range’ refers to things that are within the reach of teachers; ‘high-impact’ refers to things that might have a substantial impact on learning. Professor Sadler focussed on the assessment of achievement with all the sense of satisfaction that this brings. (more…)