On being chased by a bull

Posted on November 2, 2013 by Sally Jordan

I just came across this quote:

If you are in a field and are being chased by a bull you will probably perform above your usual level at jumping the gate, but below your usual level at undoing the combination lock. Anxiety may enhance motor skills, but it depresses complex intellectual performances.

The authors of the book where I found the quote (Bell et al. 1983) attribute this to the Yerkes-Dodson Law. After a bit of Googling, I’m not sure that’s right, but it is a lovely quote. I came across it whilst checking through old notes on papers for something I’m writing on students’ mathematical misunderstandings. However, it is deeply relevant in the context of students’ exam anxiety.

Bell, A.W., Costello, J. and Kuckermann, D. (1983) A review of research in mathematical education. Part 1: Research on learning and teaching. NFER-Nelson.

Posted in exam anxiety, quotes | Tagged , , | Leave a comment

Better engagement with continuous assessment, better exam score

Posted on October 26, 2013 by Sally Jordan

This is unlikely to be much of a surprise. The middle horizontal line in each of the box plots shown below indicates the median exam score and the vertical height of each box indicates the inter-quartile range. The solid line shows most of the range and the points are outliers.

Reading from the right, there are separate plots for students who submitted all the tutor-marked assignments (TMAs), then those who missed one, two etc.

The message is extremely clear. Students who do all the TMAs do better in the exam. This is unlikely to be a simple case of one thing causing the other, and the outliers indicate that some students can do well in the exam even after missing several TMAs, whilst others submit all the TMAs but make a complete mess of fail the exam.

However the overall picture is clear.

The picture is also entirely consistent from module to module, whatever the module’s assessment strategy.

Posted in assessment analytics | Tagged | Leave a comment

Significant figures and decimal places

Posted on October 2, 2013 by Sally Jordan

Following the finding that students struggle with significant figures and decimal places, we added a question that tested fairly basic rule-based understanding (of stuff that I thought we were teaching quite well in the Maths for Science book).

The question is shown on the left hand side.

I have to say that I’ve been quite surprised by what it is that students got wrong. You may like to think about what you’d expect before reading on.

Using the variant of the question shown as the example, about 95% of the 300 or so students on the presentation we looked at knew that 0.074 is to 3 decimal places, and 87% knew that 0.074 is to 2 sig figs.

86% knew that 7.20 is to 2 decimal places but only 78% knew that 7.20 is to 3 sig figs. The most common error (in 8.4% of responses) was to give this as 2 sig figs – and an additional 3.5% thought 7.20 was given to 2 sig figs and 1 decimal place.  In other words, students do not appreciate that the final zero is there for very good reason!

Posted in mathematical misunderstandings, significant figures | Tagged , | Leave a comment

E-assessment : Past, present and future 2

Posted on October 2, 2013 by Sally Jordan

Apologies for my recent lack of activity. We have a module whose student numbers have risen from 440 to 1075, so I have been interviewing, appointing and staff developing in every available moment. Well, that and beginning to get a firmer steer on what OU Science Faculty assessment strategy might look like in the future. More on that asap.

For now, I can report that my review E-assessment: past, present and future (not to be confused with Tom Hench’s very fine paper of the same name) is now available on the New Directions website – here – though the paper version is not out yet.  Unfortunately there is a typo in the introduction (introduced during copy editing – grrrr – but I should have spotted it) but if you can get past that I hope you’ll find the review useful. I’ve had some really nice comments about it, and to my amazement and delight it is showing as the ‘most read’ New Directions paper.

I won’t give the game away by saying too much about the paper (I want you to read it so that I don’t get knocked off that ‘most read’ position!) but I would say  that my conclusions/predictions/wishes for the future are:

1.That a beneficial side effect of MOOCs is that they are forcing the assessment community to consider appropriate methodologies for assessing huge numbers of informal learners – but we mustn’t let our standards slip in the rush to deliver assessment at scale, speed and at low cost.

2. That we should make full use of learning analytics and ‘assessment analytics’ in finding out more about the misunderstandings of individual students and cohorts of students.

3. That the boundaries between teaching, assessment and learning are becoming blurred.

4. That we should make full use of computers in assessment when that is appropriate, but not when it isn’t. To quote myself: ‘We should use computers to do what they do best, relieving human markers of some of the drudgery of marking and freeing up time for them to assess what they and only they can assess with authenticity’.

Posted in e-assessment | Tagged | Leave a comment

Quote of the day

Posted on August 20, 2013 by Sally Jordan

“While using computers to score essays may have seemed like science fiction when first conceived (Page 1966) such application may seem remarkably mundane when compared to contemporary capabilities of computers to win game shows , respond amicably to our vocal commands and drive cars. That dismissive view fades, of course, when we recall that writing is among the most complex of human behaviors, both vehicle and subject of thought and culture.”

Elliot, N. & Williamson, D.M. (2013) Assessing Writing special issue: Assessing writing with automated scoring systems. Assessing Writing 18, 1-6.

Page, E. B. (1966). The imminence of grading essays by computer. Phi Delta Kappan, 48, 238–243.

Posted in essays, quotes | Tagged , | 1 Comment

Quote of the day

Posted on August 10, 2013 by Sally Jordan

I thought I’d quoted this one before but it seems not. I can remember reading this for the first time, perhaps 5 years ago, and being amazed that it was written as long ago as 1995. The whole chapter feels very modern – or perhaps we are just slow to learn.

Students can, with difficulty, escape from the effects of poor teaching, they cannot (by definition if they want to graduate) escape the effects of poor assessment.

Boud, D (1995) Assessment and learning: contradictory or complementary? In P. Knight (ed) Assessment for learning in higher education. Kogan Page in association with SEDA. pg 35.

Posted in quotes | Tagged , | 2 Comments

Learn before testing or test before learning?

Posted on July 20, 2013 by Sally Jordan

I’m just catching up on my reading of New Scientist and in last week’s (13th July) issue I have found this from a ‘one minute’ interview with Anant Agarwal, president of the edX MOOC provider.

Question: What was the first course you offered – and what insights did it give you?

Answer: It was on circuits and electronics. In the first two weeks, about 70% of the students accessed the course video first, then did their homework. By midway through, 60% of students did their homework first, then watched the video. We learned that it’s more motivational to be given something to solve and then go get the knowledge, as needed.

Posted in MOOC, testing effect | Tagged , , | 3 Comments

E-assessment: past, present and future

Posted on July 12, 2013 by Sally Jordan

I have recently submitted a review with the title ‘E-assessment: past, present and future’. I think it is quite good, but I don’t yet know whether others agree with me!  Imagine what I felt when I realised that Thomas Hench from Delaware County Community College was giving a paper at CAA2013 entitled ‘Electronic assessment: past, present and future’. Tom was similarly amused by our similar titles, but actually the papers are quite different. You’ll have to wait to see mine, but I’d like to say a bit more about Tom’s – it was one of the best papers of the conference.

Tom had used ERIC to investigate ‘e-markers’ in around 5000 abstracts relating to e-learning and e-assessment, and the paper gave the frequency with which e-marker appeared in each year (using a rolling three-year approach).  This enabled Tom to plot the ‘evolution’ of e-assessment, as you might do in considering the evolution of various genetic markers.

Each ‘e-marker’ comprised a ‘prefix’ (e.g. electronic, computer, mobile, assisted/aided/based, online, web) and an ‘element’ (e.g. instructions, teaching, learning, assessment, testing). The way the markers have evolved is fascinating. Back in 1985, what we had was basically ‘computer-aided instruction’. Since then there has been a marked increase in ‘learning’ (Tom identified a shift from behaviourism to constructivism) and of the prefixes ‘online’ and ‘electronic’, associated with the growth of the web.  See Figures 1 and 2 below:

 

Now look closely at Figure 2 and note the recent growth of the prefix ‘mobile’. The question is, how much of an impact is this latest shift going to have?

Posted in e-assessment, mobile devices | Tagged , , , | 2 Comments

Staff engagement with e-assessment

Posted on July 11, 2013 by Sally Jordan

More reflections from CAA2013 (held in Southampton, just down the road from the Isle of Wight ferry terminal – shown)…

In the opening keynote, Don Mackenzie talked about the ‘rise and rise of multiple-choice questions’. This was interesting, because he was talking in the context of more innovative question types having been used back in the 1997s than are used now. I wasn’t working in this area in the 1997s so I don’t know what things were like then, but somehow what Don said didn’t surprise me.

Don went on to itentify three questions that each of us should ask ourselves, implying that these were the stumbling blocks to better practice. The questions were:

  • Have you got the delivery system that you need?
  • Have you got the institutional support that you need?
  • Have you got the peer support that you need?

I wouldn’t argue with those, but I think I can say ‘yes’ to all three in the context of my own work – so why aren’t we doing better?

I think I’d identify two further issues:

1. It takes time to write good questions and this needs to be recognised by all parties;

2. There is a crying need for better staff development.

I’d like to pursue the staff development theme I little more. I think there is a need firstly for academics to appreciate that they can and should ‘do better’ (otherwise people do what is easy and we end up with lots of multiple-choice questions, and not necessarily even good multiple-choice questions), but then we need to find a way of teaching people how to do better. In my opinion this is about engaging academics not software developers – and in the best possible world the two would work together to design good assessments. That means that staff development is best delivered by people who actually use e-assessment in their teaching i.e. people like me. The problem is that people like me are busy doing their own job so don’t have any time to advise others. Big sigh. Please someone, find a solution – it is beyond me.

I ended up talking a bit about the need for staff development in my own presentation ‘Using e-assessment to learn about learning’ and in her closing address Erica Morris pulled out the following themes from the conference:

  • Ensuring student engagement
  • Devising richer assessments
  • Unpacking feedback
  • Revisiting frameworks and principles
  • and… Extending staff learning and development

I agree with Erica entirely, I just wonder how we can make it happen.

Posted in conferences, staff development | Tagged , | Leave a comment

The Cargo Cult

Posted on July 11, 2013 by Sally Jordan

I suspect that this reflection from the 14th International Computer Aided Conference (CAA2013) may not go down well with all of my readers. I refer to the mention in several papers of the use of technology in teaching and learning as a ‘cargo cult’.

Perhaps I’d better start by saying what the term ‘cargo cult’ is being used to mean. Lester Gilbert (et al.) (2013) explained that ‘cargo cults refer to post-World-War II Melanesian movements whose members believe that various ritualistic acts will lead to a bestowing of material wealth’ and , by analogy, ‘cargo cult science is a science with no effective understanding of how a domain works’. Lester then quoted Feynman ( 1985):

‘I found things that even more people believe, such as that we have some knowledge of how to educate. There are big schools of reading methods and mathematics methods, and so forth, but if you notice, you’ll see the reading scores keep going down–or hardly going up–in spite of the fact that we continually use these same people to improve the methods. There’s a witch doctor remedy that doesn’t work. [This is an] example of what I would like to call cargo cult science.’

I’m not sure that my understanding is the same as Lester Gilbert’s or Richard Feynman’s, but the point that struck me forcably was the reminder of the ritualistic, ‘witch-doctor’ approach of much of what we do. Actually it doesn’t just apply to our use of technology. We have a mantra that doing such-and-such or using such-and-such a technical solution will improve the quality of our teaching and the quality of our students’ learning, and we are very often low on understanding of the underlying pedagogy. We are also pretty low on evidence of impact, but we keep on doing things differently just because we feel that it ought to work – or perhaps that we hope that it will.

Tom Hench ended his presentation (which I’ll talk about in another post)  by saying that we need ‘research, research and research’ into what we do in teaching. I agree.

Feynman, R (1985). Cargo cult science. In, Surely You’re Joking, Mr. Feynman! W W Norton.

Gilbert, L., Wills, G., Sitthisak,O. (2013) Perpetuating the cargo cult: Never mind the pedagogy, feel the technology. In Proceedings of CAA2013 International Conference, 9th-10th July, Southampton.

Posted in Cargo Cult, conferences | Tagged , , , | Leave a comment