e-assessment (f)or learning

I’m just catching up on my reading of New Scientist and in last week’s (13th July) issue I have found this from a ‘one minute’ interview with Anant Agarwal, president of the edX MOOC provider.

Question: What was the first course you offered – and what insights did it give you?

Answer: It was on circuits and electronics. In the first two weeks, about 70% of the students accessed the course video first, then did their homework. By midway through, 60% of students did their homework first, then watched the video. We learned that it’s more motivational to be given something to solve and then go get the knowledge, as needed.

I have recently submitted a review with the title ‘E-assessment: past, present and future’. I think it is quite good, but I don’t yet know whether others agree with me!  Imagine what I felt when I realised that Thomas Hench from Delaware County Community College was giving a paper at CAA2013 entitled ‘Electronic assessment: past, present and future’. Tom was similarly amused by our similar titles, but actually the papers are quite different. You’ll have to wait to see mine, but I’d like to say a bit more about Tom’s - it was one of the best papers of the conference.

Tom had used ERIC to investigate ‘e-markers’ in around 5000 abstracts relating to e-learning and e-assessment, and the paper gave the frequency with which e-marker appeared in each year (using a rolling three-year approach).  This enabled Tom to plot the ‘evolution’ of e-assessment, as you might do in considering the evolution of various genetic markers.

Each ‘e-marker’ comprised a ‘prefix’ (e.g. electronic, computer, mobile, assisted/aided/based, online, web) and an ‘element’ (e.g. instructions, teaching, learning, assessment, testing). The way the markers have evolved is fascinating. Back in 1985, what we had was basically ‘computer-aided instruction’. Since then there has been a marked increase in ’learning’ (Tom identified a shift from behaviourism to constructivism) and of the prefixes ’online’ and ‘electronic’, associated with the growth of the web.  See Figures 1 and 2 below:

Now look closely at Figure 2 and note the recent growth of the prefix ‘mobile’. The question is, how much of an impact is this latest shift going to have?

More reflections from CAA2013 (held in Southampton, just down the road from the Isle of Wight ferry terminal – shown)…

In the opening keynote, Don Mackenzie talked about the ‘rise and rise of multiple-choice questions’. This was interesting, because he was talking in the context of more innovative question types having been used back in the 1997s than are used now. I wasn’t working in this area in the 1997s so I don’t know what things were like then, but somehow what Don said didn’t surprise me.

Don went on to itentify three questions that each of us should ask ourselves, implying that these were the stumbling blocks to better practice. The questions were:

• Have you got the delivery system that you need?
• Have you got the institutional support that you need?
• Have you got the peer support that you need?

I wouldn’t argue with those, but I think I can say ‘yes’ to all three in the context of my own work – so why aren’t we doing better?

I think I’d identify two further issues:

1. It takes time to write good questions and this needs to be recognised by all parties;

2. There is a crying need for better staff development.

I’d like to pursue the staff development theme I little more. I think there is a need firstly for academics to appreciate that they can and should ’do better’ (otherwise people do what is easy and we end up with lots of multiple-choice questions, and not necessarily even good multiple-choice questions), but then we need to find a way of teaching people how to do better. In my opinion this is about engaging academics not software developers – and in the best possible world the two would work together to design good assessments. That means that staff development is best delivered by people who actually use e-assessment in their teaching i.e. people like me. The problem is that people like me are busy doing their own job so don’t have any time to advise others. Big sigh. Please someone, find a solution – it is beyond me.

I ended up talking a bit about the need for staff development in my own presentation ‘Using e-assessment to learn about learning’ and in her closing address Erica Morris pulled out the following themes from the conference:

• Ensuring student engagement
• Devising richer assessments
• Unpacking feedback
• Revisiting frameworks and principles
• and… Extending staff learning and development

I agree with Erica entirely, I just wonder how we can make it happen.

I suspect that this reflection from the 14th International Computer Aided Conference (CAA2013) may not go down well with all of my readers. I refer to the mention in several papers of the use of technology in teaching and learning as a ‘cargo cult’.

Perhaps I’d better start by saying what the term ‘cargo cult’ is being used to mean. Lester Gilbert (et al.) (2013) explained that ‘cargo cults refer to post-World-War II Melanesian movements whose members believe that various ritualistic acts will lead to a bestowing of material wealth’ and , by analogy, ‘cargo cult science is a science with no effective understanding of how a domain works’. Lester then quoted Feynman ( 1985):

‘I found things that even more people believe, such as that we have some knowledge of how to educate. There are big schools of reading methods and mathematics methods, and so forth, but if you notice, you’ll see the reading scores keep going down–or hardly going up–in spite of the fact that we continually use these same people to improve the methods. There’s a witch doctor remedy that doesn’t work. [This is an] example of what I would like to call cargo cult science.’

I’m not sure that my understanding is the same as Lester Gilbert’s or Richard Feynman’s, but the point that struck me forcably was the reminder of the ritualistic, ‘witch-doctor’ approach of much of what we do. Actually it doesn’t just apply to our use of technology. We have a mantra that doing such-and-such or using such-and-such a technical solution will improve the quality of our teaching and the quality of our students’ learning, and we are very often low on understanding of the underlying pedagogy. We are also pretty low on evidence of impact, but we keep on doing things differently just because we feel that it ought to work – or perhaps that we hope that it will.

Tom Hench ended his presentation (which I’ll talk about in another post)  by saying that we need ‘research, research and research’ into what we do in teaching. I agree.

Feynman, R (1985). Cargo cult science. In, Surely You’re Joking, Mr. Feynman! W W Norton.

Gilbert, L., Wills, G., Sitthisak,O. (2013) Perpetuating the cargo cult: Never mind the pedagogy, feel the technology. In Proceedings of CAA2013 International Conference, 9th-10th July, Southampton.

This will probably be my final post relating to the Assessment in Higher Education Conference and it relates to an excellent presentation I went to entitled ‘Feedback without Tears: students’ emotional responses’. The presentation was given by Mike McCormack from Liverpool John Moores University and two of his students and, perhaps as you would expect from a teacher of drama and two drama students, it was excellent.

The work described was an HEA funded project in which students interviewed their fellows about positive and negative emotional responses to feedback. Most students reported having had positive and a negative emotional responses to feedback received, and some of the negative reactions were long-lasting and unresolved.

Some of the tentative conclusions and recommentations are that:

• wherever possible, feedback should be given in a verbal discussion;
• lecturers should ‘have the time’ to discuss feedback with students (and make sure this is clear to their students);
• lecturers should give suggestions for improvement rather than focusing on what is wrong;
• we should avoid confusing disparity between marks and feedback;
• we should ensure that students understand that we can’t reward effort alone;
• we should be aware of students’ perceptions of power imbalance and of differing understandings of specialist discourse.

One of the suggestions made was that drama is an emotionally charged subject, so perhaps students in a different subject area – like physics – would be less emotional. That may be true, generally, but on the highly scientific basis of a sample of one physicist (me!) I don’t think it is. Physics students may react less emotionally to drama students, but that doesn’t mean that they are not sometimes hurt by feedback received. Now, whether that is necessarily a bad thing is another matter all together – sadly, dealing with negative feedback is something we all (as actors or physicists or teachers) need to be able to do in ‘real life’. It was an extremely interesting and thought-provoking presentation.

I said I’d post on two topics from the Assessment in Higher Education Conference. This is actually another one (my ‘second’ topic will follow), but noticing (a) Tim Hunt’s excellent summary of some of the things I wanted to say and (b) that this is my 200th post, I wanted to talk about something that I care about passionately. I’m picking up on ideas from Margaret Price’s keynote at the conference and from Tim’s summary, but this is essentially my own take on…Assessment literacy.

We wonder why some of our students plagiarise. We wonder why, when they are allowed to repeat iCMA questions as often as they want to, some students click through so as to get the ‘right answer’ for entry text time, so as to get the credit – without looking at our lovingly crafted feedback. The simple answer is that many of our students don’t share our understanding of what assessment is for. We may think we are firmly in the  ’assessment for learning’ mode, but if our students don’t understand that, what’s the point?

This is related to the problem that arises when students don’t understand an assessment task or the feedback provided, but the problem I’m describing is at a higher level. I’m talking about students simply not ‘getting’ what we think assessment is for – we want assessment to drive learning, but they remain driven by marks. To be fair to our students, I think that in many cases I’m talking about students simply not ‘getting’ what assessment is for because we don’t tell them – so perhaps there is an easy solution. I think the same is true of other aspects of teaching and learning, and it would help if we remembered that our students are not necessarily ‘like us’, so sometimes we need to explain our motivations more explicitly.

I drew another point from Margaret’s keynote that I’d like to mention. We are too much driven by NSS scores. In that, perhaps we are very like our students, driven by marks…I suppose it is too much to hope that a day might come when we actually cared about our students and their learning rather than University X’s ranking in some artificial league table. Ah well, we can hope.

As discussed in my previous post, the Assessment in Higher Education Conference was excellent. I helped Tim Hunt to run a ‘MasterClass’ (workshop!) on ‘Producing high quality computer-marked assessment’ and, with Janet Haresnape, ran a practice exchange on the evaluation of our faculty-wide move to formative thresholded assessment. As a member of the organising committee I also ran around chairing sessions, judging posters etc. and I have to say I loved every minute of it. I see from the conference website that another delegate has said it was the best conference they have ever attended, and I would agree with this.

I could go talk more about a number of the presentations I heard but for now I will just reflect on two themes. Here’s the first.

I have read a fair amount about the use of audio files and/or screencasts to give feedback and enjoyed the presentation from Janis MacCallum (and Charlotte Chalmers) from Edinburgh Napier University on ‘An evaluation of the effectiveness of audio feedback, and of the language used, in comparison with written feedback’. One of Janis and Charlotte’s findings is that many more words of feedback are given when the feedback is given as an audio file. Another point, widely made, is that students like audio feedback because they can hear the tone of the marker’s voice. In the unlikely event of finding spare time, the use of audio feedback is something I’d like to investigate in the context of the OU’s Science Faculty.

There is a sense in which oral assessment (i.e. assessing by viva) is just the next step. There are issues, especially to do with student anxiety and possibility of examiner bias. However, if you are there with a student, you can tease out how much they know and understand. I find it an exciting possibility. Gordon Jouglin from the University of Queensland, who is an expert on oral assessment, gave an excellent keynote on the subject (though being a dim-twit I didn’t understand his title: ‘Plato versus AHELO: The nature and role of the spoken word in assessment and promoting learning’). His slides are here. Lots to think about.

The 5th Assessment in Higher Education Conference will run in 2015 – be there!

I am in a (very brief) lull between the Assessment in Higher Education Conference, CAA 2013, masses of work in my ‘day job’ and a determination to both carry on writing papers and to get some time off for walking and my walking website. The Assessment in Higher Education Conference was great and, hopefully before CAA 2013 starts on Tuesday, I will post about some of the things I learned. However first, I’d like to reflect on something completely different.

During the week I was at an Award Board for a new OU module. All did not run smoothly. The module requires students to demonstrate ‘satisfactory participation’, but we’d used a horrible manual process to record this. Not surprisingly, errors crept in. Now the OU Exams procedures are pretty robust and the problem was ‘caught’. We stopped the Award Board, all the data were re-entered and checked, checked and checked again and we reconvened later in the week – and brought the board to a satisfactory conclusion. My point is that people make mistakes.

Next I would like to reflect on the degree results of one of my young relations at a UK Russell Group University. She got a 2:1, which was what she was aiming for, but was devastated that she ‘only’ got 64%, not the 67% she was aiming for. Now this is in a humanities subject – how on Earth can you actually distinguish numerically to that level of precision?

My general point is that, given that humans make mistakes – and even if they don’t their marking is pretty subjective – why do we persist in putting such faith in precise MARKS. It just doesn’t add up. I am pretty confident that, at our Award Board, we made the right decisions at the distinction/pass/fail boundaries and I am similarly confident that my young relative’s degree classification is what was demonstrably achieved. I’d reassure those of you who have never sat on a Award Board that considerable care is taken to make sure that this is the case. However, at a finer level, can we be sure about the exact percentage? I don’t think so.

Listening to the radio this morning, my attention was drawn to a new medical test with an accuracy of ‘more than 99%’. I was left thinking ‘well, is that good enough?’ and then ‘so is a marking accuracy of 99% good enough for e-assessment questions?’ (Actually the situation with the medical test is not as simple as I’d thought – the idea is that this test would just be used to give a first indication. That’s better.)

Returning to e-assessment. Well no, on one level 99% accuracy is not good enough. What about the 1%? But the e-assessment is likely to be considerably more accurate than human markers. And students are more likely to be confused by something in the question (which may be as simple as reading an addition sign as a division) than they are to be marked inaccurately by the system. Clearly this is something that we must continue to monitor, and to do our best to improve accuracy of marking (whether by humans or computers). And it matters even in purely-formative use, because if our answer matching is not correct we will not be giving the correct feedback message to the students.

I gave a webinar yesterday in the ‘Transforming Assessment’ series, on “Short-answer e-assessment questions: six years on”. The participants were lovely and I was especially pleased that there were lots of people I didn’t know. There is a recording at http://bit.ly/TA5J2013 if you’ve interested.

The response yesterday was very encouraging, but I remain concerned that more people are not using question types like this. However I rest my case that you need to use real student responses in developing answer matching for questions of the type we have written. That’s fine for us at the OU, with large student numbers, but not necessarily for others.

Then, in the feedback from participants, someone suggested that they would value training in writing questions and developing answer matching. I would so much like to run training like this, but simply don’t have the time.

But, thanks to Tim Hunt, we have another suggestion. I have recently used the Moodle Pattern Match question type to write very much simpler questions, which require a tightly constrained single word answer – like the one shown below.

If you are interested in using Pattern Match, writing questions like this would give a simple way in - and you’d probably get away with developing the questions without the need for student responses beforehand (though I would still monitor the responses you do get).