Usage of Early Alerts Indicators on two level 1 modules

Project leader(s): Carol CalvertAlison BromleyChris Hughes
Theme: Supporting students
Faculty: STEM
Status: Archived
Dates: November 2018 to December 2019

The Open University (UK) has developed two systems of predictive modelling to identify students at risk. The main function of one system was to predict whether a student would pass their module whilst the second system was primarily designed to generate weekly predictions on whether a student was likely to submit their next Tutor Marked Assignment. The Early Alerts Indicators combined the two systems within one graphical interface and this interface is now made available to Associate Lecturers and module teams across all undergraduate modules. This eSTEeM project was designed to establish Associate Lecturers views on how to maximise their use of the current Early Alerts data and to identify improvements based on their recommendations.

The views of the Associate Lecturers’, who were recruited as volunteers to the project, were generally positive about the Early Alerts Indicator information. The ALs identified several areas for improvement and amongst these were two key areas: around timing and around the transparency/simplicity of how the probabilities of passing the module are generated.

Whilst the indicators can be made available earlier this comes at a cost in accuracy. Pre module start probabilities have an accuracy of 73%-75% and accuracy increases to around 85% as soon as information on the first assignment is available. The accuracy of predictions on whether a student will pass are in the high 90%’s shortly before the module ends. Simplifying the variables used in the prediction models involves only around 3%-5% percentage points loss in accuracy. Thus, pre-start probabilities of success can be generated, with an accuracy of around 70%, by considering for a student

if they are, they are new or continuing
which module are they on
how much have they engaged pre module start with the VLE

And

if they are a new student then how high a workload have they signed up to
or if they are a continuing student how positive is their prior success within the OU

Associate lecturers were asked to consider contacting students on the basis of the Early Alerts Indicators and they reported students were generally very happy to be contacted. Students were also asked for their views and they had few reservations of predictions being generated and used as the basis for their tutor to contact them.

The pass rates of the students in tutor groups of the ALs involved in the project showed no consistent differences to the pass rates of those not involved in the project.