Use of STACK to generate formative assessment for level 3 Pure mathematics

Project leader(s): Hayley Ryder
Theme: Innovative assessment
Faculty: STEM
Status: Archived
Dates: October 2016 to November 2019

Highly Commended at the 3rd eSTEeM Scholarship Projects of the Year Awards 2020 under the category - Innovative/Original Approach to Teaching.

In a study of exam revision impact, Cross et al found that 83.9% of students rated access to a sample examination paper useful or very useful [Cross, Whitelock, and Mittelmeier, 2016]. Many mathematics modules have been running for a number of years and have a large number of available past examination papers. As a result, students studying mathematics at the OU have often developed revision strategies that assume access to a large number of past examination papers (together with solutions). On newer modules this resource does not exist, and students have cited lack of past papers as a reason for deferring or for not taking a new module. Mathematics examination-like questions and solutions are time-consuming and expensive to write and check; therefore, it is not easy to produce a large number of sample papers.

Many mathematics modules use a computer algebra system called STACK to generate online short practice questions (SPQs) with feedback and solutions. The STACK system can be programmed so that the feedback produced for incorrectly answered questions depends on the mistakes made by the individual. These questions are used to build quizzes (SPQ-quizzes). For this project we used STACK to produce sets of long examination-like practice questions (ELPQs) with worked solutions and tailored feedback for a relatively new module (M303). These long examination-like questions were then used to build a randomly generated quiz that resembled a past exam paper (ELPQ-quiz) so that students could generate multiple instances of an example examination-like quiz, with feedback and example solutions, for practice.

We used a mixed methods approach to our evaluation of this project, performing a quantitative analysis of the engagement with these questions by 660 students, sending a questionnaire to just over 200 students and carrying out a qualitative analysis of semi-structured interviews undertaken with 12 students.

A thematic analysis of the structured interviews showed lack of past papers as a strong theme, along with the existence of established revision strategies that relied on access to many past papers.

The results of the quantitative analysis showed that a higher percentage of students engaged with the ELPQ quiz than engaged with the traditional SPQ quizzes. The students that did engage with the ELPQs used them at a higher intensity. In addition, a significantly higher percentage of students who engaged well with the ELPQs maintained or improved their exam result for M303: Further pure mathematics, as compared to their score in the level two precursor module, M208: Pure mathematics.

Over 80% of students responding to the survey, taking the exam, and using the ELPQs, either agreed or strongly agreed that the ELPQ-quiz helped them structure their revision and felt that the ELPQs were helpful when they saw the exam. Over 90% of the respondents who used the ELPQs agreed or strongly agreed that quizzes similar to the ELPQ-quiz should be implemented on other mathematics modules.