The figures below show, for each question, the number of students who got it right at first attempt (yellow), second attempt (green), third attempt (blue), or not at all (maroon). So the total height of each bar represents the total number of students who completed each question.

You can spot the differences for yourself and I’m sure you will be able to work out which module is which! However I thought you’d like to know that questions 24-27 are on basic differential calculus. Obviously still some work to do there…

[pg10] [we should] ‘approach the teaching of science like a science. That means applying to science teaching the practices that are essential components of scientific research and that explain why science has progressed at such a remarkable pace in the modern world.
The most important of these components are:
• Practices and conclusions based on objective data rather than—as is frequently the case in education—anecdote or tradition.This includes using the results of prior research, such as
work on how people learn.
• Disseminating results in a scholarly manner and copying and building upon what works. Too often in education, particularly at the postsecondary level, everything is reinvented, often in a highly flawed form, every time a different instructor teaches a course. (I call this problem “reinventing the square wheel.”)
• Fully utilizing modern technology. Just as we are always looking for ways to use technology to advance scientific research, we need to do the same in education.’
[I'm not sure I necessarily agree with the final point - I'd use technology when, and only when, that is beneficial to the student experience.]

Relative to this, the point I want to emphasise sounds timid:

[pg13] ‘Even the most thoughtful, dedicated teachers spend enormously more time worrying about their lectures than they do about their homework assignments, which I think is a mistake.’

But it is oh so true – certainly in my own institution, relative to the time and effort that goes into developing our (excellent) teaching resources, we put so little time and effort into getting assessment right. I think that’s a mistake! Your institution may be different of course, but I doubt that many are.

Wieman, C. (2010). Why not try a scientific approach to science education? Change: The Magazine of Higher Learning, 39(5), 9-15

The top figure (above) is depressingly familiar. The students who submit early all do very well – they probably didn’t need to study the module at all! The rest are rushing to get the assignment done, just before the due date – and lots of them don’t do very well.

I am very pleased with the lower figure. Here students are doing the assignment steadily all the while it is available – and with the exception of a small number who were probably prompted to have a go on the due date by a reminder email we sent, they do pretty similarly, irrespective of when they submitted. This is how assignments should perform!

I’m aware that my interpretation may seem simplistic, but we have other evidence that the first batch of students are overcommitted – they are also younger and have lower previous qualifications – so it all fits.

Finally, following yesterday’s JISC webinar on learning analytics I’m beginning to think that this is how I should be describing the work that I’ve previously categorised as ‘Question analysis’ and  ’Student engagement’. However we describe this sort of analysis, we must do more of it – it’s powerful stuff.

Recently, I had cause to take this slightly further. We have two interactive computer-marked assignments (iCMAs) that test the same material, and that are known to be of very similar difficulty. Some of the questions in the two assignments are exactly the same, most are slightly different.  But when we see very different patterns of use, this can be attributed to the fact that the two iCMAs are used on different modules, with very different student populations.

Compare the two figures shown below. These simply show the number of questions started (by all students) on each day that the assignment is open. The first figure shows a situation where most questions are not started until close to the cut-off date. The students are behind, struggling and driven by the due-date (I know some of these things from other evidence).

The second figure shows a situation in which most questions are started as soon as the iCMA opens – the students are ready and waiting! These students do better by various measures – more on this to follow.

‘The testing effect represents a conundrum, a small version of the Heisenberg uncertainty principle in psychology: Just as measuring the position of an electron changes that position, so the act of retrieving information from memory changes the mnemonic represention underlying retrieval – and enhances later retention of the tested information.’
Roediger, H. L., III, & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1, 181-210. pg 182.

Knight, P. (ed) (1995) Assessment for learning in Higher Education. Kogan Page in association with SEDA. pg13 (introduction)

Oh how much I missed in my last post!

Others, far better informed than me, have also reflected on the general principles of the use of the automated marking of essays. See, for example, Michael Feldstein, Audrey Watters and Justin Reich.

I think the interest was probably sparked by an ‘Automated Student Assessment Prize’ competition last year, funded by the William and Flora Hewlett Foundation. Phase 1 of this was to do with the automatic marking of essays and the results are here.

Things began to get interesting when the New York Times announced that edX were going to use automatic grading of essays. The report may or may not have been accurate, but concerns are being expressed about so-called ‘robo-marking’. See for example Les Pereman and the Professionals against Machine Grading of Essays group.

On Michael Feldstein’s blog ‘e-Literate’, Elijah Mayfield of LIghtSIDE Labs has now said more about why the NYT (and possibly the edX claim) is wrong. The LIghtSIDE labs approach seems very sensible – human marked essays are used as the basis for machine learning about good features of essays; this is used to provide feedback about the good and weak features of essays submitted by students and hence to peer review. Grading is still done by humans.

Some questions:
- I don’t know what edX are actually planning to do, in other words, how much can we believe the NYT report?
- I’d love to know more about the technology being used (by edX, LIGHTside, anyone) and whether they are marking essay content, style or both?

Vojak, C., Kline, S., Cope, B., McCarthey, S. and Kalantzis, M. (2011) New Spaces and Old Places: An Analysis of Writing Assessment Software. Computers and Composition, 28, 97-111.

This is about computer-based marking of essays. I should start by pointing out that this is not an area that I know a huge amount about – the issues are surprisingly different from those in the computer-based marking of short-answer questions. ‘E-rater’ is perhaps the best-known essay-marking software, but there are lots of others, some of which are discussed in the following review articles:

Dikli, S. (2006) An overview of automated scoring of essays. Journal of Technology, Learning and Assessment, 5(1).

Valenti, S. & Neri, F. & Cucchiarelli, A. (2003). An overview of current research on automated essay grading. Journal of Information Technology Education, 2, 319-330.

Some of the systems mark for content, some mark for essay-writing style, and some mark for both. As I understand it, when marking for content, this is in some senses easier to do than when you are marking an essay than when you are marking a shorter answer. In a short answer things like word order and negation can be extremely important (see my previous post on this here). However, in longer answers, if all the (probably large number) of keywords or their synonyms are there, then the essay-writer is likely to have right idea.

I don’t pretend to begin to understand the technologies used to mark writing style. I’m sure that they are very clever and they appear to have good success rates i.e. the things that they measure are good proxies for a the things that human markers of essays look for.

However Vojak et al (2003) raise an extremely important question: should we be asking our students to write for a computer rather than for a human marker? This question has philosophical overtones, and again I rapidly get out of my depth. It is an interesting point, given that writing is essentially a social activity.

More pragmatically, there are cases where people have succeeded in fooling an essay-marking system by submitting a gibberish essay and getting a good mark. Now, one of the things I learnt early in my e-assessment career (from my ‘mentor’ Phil Butcher) was that students don’t try to fool the system – this is something that academics do! Over the years I have found Phil’s advice on this point to be absolutely right. However, if word were to get out that a class’s essays were being marked on the strength of, say, the number of commas they contained, I, guess, I, wouldn’t, be, very, happy.

Returning to the more philosophical, I’d also feel uneasy if I felt that my carefully crafted words were being ‘read’ by a computer not a human. Despite the fact that I blog primarily to keep my own thoughts in order, as I write I do feel that I am trying to convince some flesh-and-bones reader of what I am trying to say, and to engage in debate. Is it odd that I feel like that, given that I am very happy to mark our students’ sentence-long answers automatically, and to give them computer-generated feedback. To get out of the logical hole I’m digging myself into, I’d argue that the marking and feedback on our short-answer questions is actually done by me, the question author. I set up the rules and wrote the feedback interventions iteratively, in the light of careful inspection of hundreds and sometimes thousands of real student responses.

Carol Bailey from the University of Wolverhampton gave an inspirational keynote, about the issues confronting international students when producing essays etc. to our (UK) standards. It is salutory to remember just how little shared experience we have with these students and how challenging it can be for them to produce written work to our standards  – though Carol ended with a lovely story from a student who had improved beyond his or her wildest dreams.

That brings me to what, for me, was the over-riding message from the conference: we have so little shared understanding with many of our students (not just international ones) of the purpose and nature of assessment. I don’t mean just in subtle, minor ways – many of our students don’t know why they are doing our assignments (and do we?), so is it surprising that they do things we don’t want them to do, like plagiarising? Let’s start making our expectations clear to our students and attempting to engage them in dialogue, before as well as after our assignments. Let’s make that the focus, rather than seeking to ‘get them’ when they have copied something from the web without proper referencing. Yes, we need to explain to students that this is wrong, but we also need to make sure that they know how to do better.

Carol also introduced me to Xpert, so this blog will now contain lots of images with Creative Commons attributions!

The first thing to note is that there was no obvious relationship between questions that were reported as unsound and questions that actually had a problem. Students reported as unsound questions that we thought were fine (notwithstanding the fact that, in general, if students really think there’s a problem with a question, then I think there is a problem with the question, albeit that the answer-matching is usually fine and the difficulty is likely to be that the student has not understood the feedback). More significantly, at this stage we definitely had questions that were unsound, but students didn’t tick the unsound button.

When we started using the short-answer free-text questions for real, the Module Team were not keen on having an unsound function for this type of questions but not for others (fair enough) and others were not keen to introduce an unsound function on all questions, because we would then be obliged to investigate and respond every time the button was ticked (also fair enough). So, in the early presentations, where I was not sure about the answer matching for the short-answer free-text questions, we checked the marking of ALL responses. Clearly this is not sustainable.

In our old computer-marked assignments, if sufficient students ticked the unsound button, the question was zero weighted and we could do that for iCMAs too. However I like the idea of an ‘unsound’ button simply because of its potential to defuse the angst. It would give students a way of saying ‘this isn’t right’ without having to contact the module team separately. Some students do this of course (and a lot of curriculum manager time is spent in dealing with the queries) but these are the angry students who have an axe to grind – it all feels a bit unbalanced.  But an unsound button might open the floodgates – students might tick it just because it is there. Tricky. An answer might be to provide an unsound button that notified the student’s own tutor rather than the module team (given that in 99% of cases, the student would be revealing a misunderstanding that their tutor could help with).

The underlying issue here is student confidence, or lack of it, in the computer’s marking. What we haven’t really felt able to do is to remind our students just how accurate computer marking is and, relatively, how inaccurate human marking is. Hmmm.