e-assessment (f)or learning

I was due to be tutoring at the Open University residential school SXR103 Practising Science at the University of Sussex (shown left; the crane is a reminder of the huge amount of building work that is going on) for two weeks this summer. Unfortunately flu and laryngitis forced me to bail out after just one week. It was an excellent week, even if teaching is a bit challenging when you have no voice! – I’m extremely grateful to my colleagues, especially Keith, Chris and Anne-Marie, for helping to cover for me and to Dave for staying on an extra week in my place.

So now I’m home with unexpected time to catch up on some reading, writing and reflection (though my thinking may be even more befuddled than usual!).  I’ve thought previously that the assessment-related issues confronting those of us who work at the UK Open University, with its ‘supported distance learning’, are not that dissimilar to those working at conventional ‘face to face’ universities. Now I’m beginning to wonder. (more…)

It seems hard to believe that I’ve been blogging on assessment, especially e-assessment, and especially the impact of e-assessment on students, for a year now.

Even more amazing is the fact that there is still so much I want to say. Assessment and e-assessment  have been growth areas for the past 20 years or so (huge numbers of papers have been written and huge numbers of conference presentations given). In many ways we know so much…but yet we know so little.  I’m not an expert, just an amateur pottering around the edges of a large, overpopulated and sometimes contested territory. I find it difficult to get my head around many of the concepts. (more…)

In describing our use of  (simple) PMatch for answer matching for short-answer free-text questions, I may have made it sound too simple. I’ll give two examples of the sorts of things you need to consider:

Firstly, consider the question shown on the left. I’m not going to say whether the answer given is correct or incorrect, but note that the answer ‘Kinetic energy is converted to gravitational (potential) energy’ includes exactly the same words – and responses of both types are commonly received from real students. So word order matters.

The other thing to take care with is negatives. As I’ve said before, it isn’t that students are trying to trick the system. However responses that would be correct were it not for the fact that they contain the word ‘not’ are suprisingly common. So answer matching needs to be able to deal with negation.  (more…)

The title of this post comes from the title of a thoughtful paper from John Dermo and Liz Carpenter at CAA 2011. In his presentation, John asked whether automated e-feedback can create ‘moments of contingency?’ (Black & Wiliam 2009). This is something I’ve reflected on a lot – it some senses the ideas seem worlds apart. (more…)

For various reasons I’ve been thinking a lot recently about confidence-based marking.  (Tony Gardner-Medwin, who does most of the work in this area also calls it ‘certainty-based marking’). The principle is that you get most marks for a correct response that you are sure is right, fewer for a correct response that you are not sure about. But at the opposite end of the scale, you tend to get a penalty for an incorrect response that you were sure was right. (more…)

Just home from CAA 2011 (the International Computer Assisted Assessment Conference in Southampton). The attendance was quite low ( probably a victim of the current economic climate) but the conference was good, with some very thoughtful presentations and extremely useful conversations. I’ll post more in the coming days, but for the moment I’ll just reflect on John Dermo’s summary at the end of the conference. John had used wordle.net to create a word cloud from the papers. Amazingly the ‘biggest’ (i.e. most common) word in the cloud was  was ‘student’, whilst ‘technology’ was tiny. 

I did occasionally feel that some presenters were still seeing  technology as a solution in need of a problem (and also seeing ‘evaluation’ as something that we do to convince others that what we’re doing is the sensible way forward – surely honest evaluation has to accept that our use of technology might not always be the best solution). However, the overall focus on students not technology was refreshing. Hurrah!

Finally for today, I’d like to talk about where I believe the limits currently sit in the use of short-answer free-text questions.

I have written questions where the correct response requires three separate concepts. For example, I have written a question which asked how the rock in photograph shown was formed. (Incidentally this is granite, photographed near Lands End in Cornwall, but I’d never say that in a question, otherwise students just Google the answer). A correct answer would talk about the rock being formed from magma (first concept), which has cooled and crystallised (second concept) slowly (slow because it has cooled within the Earth rather than on the surface) (third concept). I haven’t managed to write a successful question with a correct answer that includes more than three separate ideas, but that doesn’t mean to say it can’t be done.

A second consideration in authoring short-answer questions is the number of discrete correct and incorrect responses. Here I think the limit came in another question based on one of my photographs, with thanks to the colleagues shown. We used this photograph in training people to write answer-matching and the question was simply ‘Who is taller?’ That might sound like a very straightforward question (until you get the bright-sparks who say ‘The prettier one’),  but writing a complete set of answer-matching for this question was a time-consuming and non-trivial task. Think about the correct answers: ‘The one on the right’, ‘The one with longer hair’, ‘The one carrying a brochure’, ‘The one wearing glasses’, [not 'The one holding a glass']..and so on…and so on.

The third limit is the serious one. Developing answer matching for the questions I’ve talked about was time-consuming but we got there in the end. However the correct answers are unambiguous – it is clear that the woman on the right-hand side of the photograph is taller than the one on the left. However in some subject areas, what is ‘correct is less well defined. I think that’s the real limit for questions of this type.

Hot on the heals of my previous post, I’d like to make it clear that human markers sometimes do better than computers in marking short-answer [less than 20 word] free-text questions.

 I have found this to be the case in two situations in particular:

1. where a response includes a correct answer but also an incorrect answer;
2. where a human marker can ‘read into’ the words used to see that a response shows good understanding, even if it doesn’t use the actual words that you were looking for. (more…)

Back to short-answer free-text questions. One of the startling findings of my work in this area was that computerised marking (whether provided by Intelligent Assessment Technologies’ FreeText Author or OpenMark PMatch) was consistently more accurate and reliable than human markers. At the time, I was genuinely surprised by this finding and so were the human markers concerned (one of whom had commented to me that he didn’t see how a computer could possibly pick up on some of the subtleties of marking). However others, especially those who know more about issues of reliability in the human marking of exams etc., were not at all surprised. And, reflecting on my own marking of batches of responses (where I can start by marking something in one way and find myself marking it differently at the end) and on the fact that I make mistakes in my work (not just when marking – think about the typos in this blog) however hard I try, I can see that human-markers have weaknesses! (more…)