This was the topic of my roundtable at the Earli/Northumbria Assessment Conference and I am very grateful to the 10 people who attended one of the two wonderful discussions we had on the topic.
The obvious answer is that, no, we don’t know what we mean by ‘quality’. We don’t even know what we mean by ‘e-assessment’, Having discussed this a bit, we moved on to discuss different aspects of ‘quality’. I note that both groups, whilst mentioning that validity and reliability are important, also emphasised the role of e-assessment in transforming learning, in particular through its ability to provide instantaneous feedback.
Words that experts use in the context of ‘quality’ in e-assessment include (in no particular order):
Reliable, Accurate, Valid, Effective, Practical (i.e. cost-effective), Consistent, Equitable, Secure, Technically reliable, User-friendly, Accessible, Appropriately aligned, Authentic, Authenticated (Plagiarism-proof), Transparent (students know what they are expected to do), Engaging, Satisfying, Appropriate to medium, At appropriate level,Feedback-rich, Relevant, Timely
Interviews with students lead me to think that they regard the following as particularly important:
- Fairness (though some are happy to trust us)
- Accessibility
- Technical reliability (though some see that as our problem)
- Clear instructions and questions
- Variety of test items
- Usefulness for learning
We had interesting conversations about the tension between the need for reliability and the need for authenticity. We also discussed the importance of writing clear and unambiguous questions.
Thank you to the person who reminded me that we should not be adding together marks from questions that assess different things (different ‘domains’) . This is absolutely true of course (and I was missing a point) but the challenge of carrying this through into practice is, I suspect, one that few of us entrely meet.
More generally, I am led to conclude that not only do we not know what we mean by ‘quality’ in e-assessment, we don’t know what we mean by ‘quality’ in assessment more generally. Lots more to think about here.