e-assessment (f)or learning

Whether or not you’re coming to our seminar ‘Self-assessment: stratgies and software to stimuate learning’ on 11th June, there are lots of things worthy of discussion.

Many of the speakers have provided links that you might like to look at:

Gwyneth Hughes on Ipsative Assessment

Tony Gardner-Medwin on Certainty-based Marking and CBM in Moodle

Peerwise (Paul Denny is in New Zealand so we will be showing a video) and a JISC-funded project that is evaluating the use of Peerwise at Edinburgh University.

Jon Rosewell on ‘Assessing with confidence’

And, from Phil Butcher et al at the Open University, ‘eAssessment at the Open University with open source software.’

Tony Gardner-Medwin and I have identified lots of questions related to self-assessment – click here for the full list. I particularly like the following:

• Must assessments always encourage ‘learning (and teaching) to the test’? Can self-assessment capitalise on the truism ‘assessment drives learning’ in a more constructive way?
• Should the learner or teacher be in charge of what self-assessments to do, and when?
• Is instant feedback important? (or is a delay better)
• Are explanations of answers important as feedback?
• Should students doing self-assessments be encouraged to work together?
• Should individual students’ marks on self-assessments  be ignored by teachers?
• Are self-assessments most useful after study (‘revision’) or as a stimulus to continuing study?

 There are lots of other questions about self-assessment (including many we haven’t identified) and I suppose the mother and father of all of them are ‘What do we mean by self-assessment?’ and ‘What does it matter?’ - hopefully we’ll have a good discussion on 11th June. However if you would like to discuss any of these questions (or anything else relating to self-assessment) now, whether or not you are able to join us on 11th June, please do so by using ‘comments’ on this post.

I’m pleased to hear that OCR have apologised for errors in adding up marks for GCSE and A-level papers last year. It doesn’t seem right that the whistleblower remains suspended, but I don’t know the details so perhaps I shouldn’t comment.

When I first heard about this latest case of human error in exam marking, I was amazed that we are still reliant on human arithmetic (though I know that addition of scores is meant to be checked by someone else – and that payment for this is included in the payment of GCSE and A-level markers). However I suppose that if markers were required to enter their scores into a computerised system of some sort (to enable the computer to check the arithmetic, or to do the arithmetic in the first instance) there would still be transcription errors – and it would take time and so slow down the marking process.

The important point is that, however much their work is checked and however much they are encouraged not to make mistakes and/or punished for making them (according to The Guardian, some examiners have had their contracts terminated), human markers are fallible. They are fallible when marking, they are fallible when adding scores. I’m a human, I make mistakes. The sooner we are honest with ourselves and admit that, the better.

I have a huge amount of work to do, yet this afternoon I have found myself enthralled in a number of coincidental conversations about the role of ‘incentives’ in getting students to engage with assessment, or dare I say, with learning.

With permission, I am copying  parts of Phil Langton’s latest message:

I’m coming rapidly to the conclusion that the mistake we make is to fail to put ourselves in the students’ shoes.  We ask lots of questions and expect that by weight of numbers of sheer dumb luck the  students will begin to see the world as we do.

Wrong!

What drove my own learning in my early years was my upbringing – to be obedient and respectful of my elders and to anything to the best of myability.  This was what I knew my parents would expect of me.  Right there is a problem that has become larger and larger. [true of me too Phil]

 Later, when I had more interests and I was old enough to indulge them without fear of arrest, I worked hard only on those things that were of interest or that would bite me if I ignore them (ie examinations).  The examinations at the time were not of a type that one could do well in simply by some cramming of facts – there was some understanding required.

This is the difference between using a foreign language dictionary and being able to speak the language – at least a bit. [I've never been much good at languages, but I think this is true too.]

We have an increasingly difficult job to do to hook the students’ interest - there are just so many attractive alternatives….  Being theatrical only gets you so far. Personally I think we need to stop asking lots of little bread-crumb questions and instead throw the problem at them in full.  We have to get science students everyday to observe and think like scientists and not have them simply learn and remember what other scientists have done; to simply vomit up the history science to date – ignoring all the blind alleys and wrong turns that were made along the way.  That does not a scientist make!

 I think it was Aristole who is supposed to have said, “education is lighting a fire, not filling of a bucket”. [this quote seems to have been attributed to Yeats or Plutarch but whatever, it's great!]

thanks Phil.

Building on work that others have done, in particular my colleagues Joy Manners and John Bolton in the OU Department of Physical Sciences, we are encouraging a move across the Science Faculty to what we are calling ‘formative thresholded assessment’.

To understand this, you need a bit of history. The OU has a long history of ‘double duty’ continuous assessment i.e. assessment where we think that the giving snd receiving of feedback is important, but where the mark also counts. However we are now trying something a bit different. What we are essentially saying is that students need to get above a certain threshold overall on their continuous assessment, but their continuous assessment score will not contribute to their overall module (or qualification) result.

The initial driver for this was to reduce the huge amount of resource we put into creating new tutor-marked assignments for each presentation of a module – and, like everyone else, just occasionally(!) we make mistakes and end up with assessment tasks that are not very good. With formative thresholded assessment, we can re-use assignments and make sure that these are as good as they possibly can be. But hopefully, we can also put the emphasis back on the formative nature of the assignments.We will still be using the usual software to discourage collusion or plagiarism, but again the emphasis will be on teaching good academic practice. 

We acknowledge that we still need to do all that we can to encourage students to engage in all assignments. Suggested methods for doing this include:

• Making it clear that doing all the assignments will help students to prepare for the exam or end-of-module assignment
• Increasing tutor-marked assignment (TMA) and interactive computer-marked assignment (iCMA) weightings as the module progresses (so students are not tempted to think they have ‘done enough’ and so don’t need to submit work)
• Module Teams are advised to retain firm cut-off dates for TMAs and iCMAs, again to encourage student engagement.
• Most importantly, just encouraging a culture that appreciates that is is useful to do the assignments.

Will it work? We’ll see.

I’m just back from my second attendance at a JISC Learning and Teaching Practice Experts Group meeting. It was excellent – it is inspiring to be surrounded by people who know such a lot about learning and teaching and, more importantly, actually care about the student experience.

Half the day was spent discussing some of the early outcomes from the new JISC Assessment and Feedback Programme. I was involved in an unsuccessful bid to this programme, but can truly say that I am happy not to have been successful (not sure what my colleagues think!).  I’m just SO busy and it is nice to learn from others rather than being in the thick of it.

At today’s meeting we heard from a selection of ‘Stand B’ (Evidence and Evaluation) projects but for me the highlight was the summary from Lisa Gray ‘What are we learning about technology-enhanced assessment and feedback practices?’. What follows are some of my notes, with some thoughts from subsequent discussion. Apologies for anything I’ve misrepresented:

Educational principles are rarely enshrined in strategy/policy. [In practice I think this means that it is the summative function of assessment that finds its way into policy and paperwork.]

Devolved responsibility makes to difficult to achieve parity. [This is an interesting one - we might want individual academics to have ownership of and be happy with their assessment practice, but if they are then the student experience will vary from course to course.]

Assessment and feedback practice does not reflect the reality of working life.

Administrative staff are often left out of the dialogue.

Traditional tasks such as essays/exams still predominate.

Timeliness of feedback is an issue. [our group talked about this quite a lot. This is an issue, but perhaps we need to make bigger changes so that feedback on some very specific thing in one module is less significant than feedback that is more generic and so more relevant at a later date]

Curriculum design issues inhibit longitudinal development. [again, we need a more joined up approach]

We have funding from the HEA for a seminar on self-assessment that we (Tony Gardner-Medwin and me) are running at the OU in Milton Keynes  on 11th June 2012.

It’s a really interesting topic and we have some excellent speakers lined up. How does testing affect learning – for good or ill? What part does ipsative assessment have to play? How about certainty-based marking? And there will be an update on the Moodle quiz engine. Click on OU_prog  for the  full programme or on http://www.ucl.ac.uk/~ucgbarg/OU_workshop.htm for more information (including booking information).

Do come!

I’m indebted to Phil Langton to pointing me towards an article by Steven Schwartz in Times Higher Education. The article, Get back in the saddle, explores the way in which failure and life’s knocks can lead to success in the future.

The theme is similar to that discussed in comments on a previous post of mine. I think it is certainly the case that we sometimes molycoddle our students. It doesn’t seem right to be deliberately hard or harsh in our grading or feedback, but I think we should be honest. If a student’s work is not up to the mark, we should tell them. It may be painful for all, but if the article is to believed (and it rings true) this approach may be just what is needed in order to encourage our students to succeed in the future.

I have posted before about the difficulties that students have with fractions and the problems that this causes. Many others recognise the same difficulty.

On a related point, students also have problems with reciprocation, and sometimes it is simply that they don’t recall that x^-2=1/x²

So the analysis of the question shown on the right should not be too surprising. The question is very well answered, but the ‘correct’ option that students are least likely to select is the one shown highlighted (top right). This is present in about 86% of all answers, but all the other correct options are present in more than 90% of answers.

The question shown was originally planned to assess students’ ability to estimate, but since we can’t prevent them from using a calculator, I adapted it to test calculator use as well.

43% of the analysed 14943 responses were correct, and the errors made were depressingly familiar:

8.1% were numerically correct but expressed to the nearest order of magnitude not to 1 significant figure.

3.1% were numerically correct but expressed to 2 significant figures.

In 2.9% of responses, the square root on the numerator had only been applied to the first number (4 in the example shown)

2.9% had used 10 to the plus 6 instead of 10 to the minus 6 in the denominator (which may have been caused by dividing rather than multiplying by 10 to the minus 6).

So it remains the simple things that cause the problems. Sig figs, calculator use, precedence…

I’ve said before that students are not good at giving answers to an appropriate number of significant figures. But what do they do wrong? The question shown on the left provides some insight.

The correct three options are B, C and E and they are the three most commonly selected. However options C and E are selected more frequently than option B. The two incorrect responses are selected an approproximately equal number of times – perhaps due to guessing; this question is in formative-only use.

Things get more interesting when you just look at responses from students who use all three responses before getting the question right – or who fail to get it right at all. Whilst 69.2% of these responses include Option C and 65.1% include Option E, only 56.4% include Option B – and an almost identical proportion of responses (56.0%) include the incorrect Option A. So leading and trailing zeroes cause problems.