Piloting OU Analyse and the Student Probabilities Model on 12 STEM Modules

Project leader(s): Carlton WoodSteve Walker
Theme: Supporting students
Faculty: STEM
Status: Archived
Dates: May 2017 to August 2019

The eSTEeM project Piloting OU Analyse and the Student Probabilities Model on 12 STEM Modules was established in November 2017 and aimed to explore whether, and how, these two learning analytics tools, could contribute to one of the four priorities - the use of data and analytics - outlined in the STEM Retention and Progression Plan 2017/18 set to assist STEM in reaching its institutional targets.

We sought to manage this project from inside STEM where we have a greater level of buy-in from module teams and were able to position ourselves as a more neutral observer, interested in the teaching and learning aspect of learning analytics. We did, however, work closely with the Early Alerts Indicator (EAI) project team who did the technical set-up, delivered the tutor training and played an ongoing advisory role for some module teams.

In this summary we use the term learning analytics, and its acronym LA, in accordance with the widely accepted definition provided at the First International Conference on Learning Analytics (LAK 2011):

Learning analytics is the measurement, collection, analysis, and reporting of data about learners and their contexts, for the purposes of understanding and optimizing learning and the environments in which it occurs.

A subset of this definition can be seen in the more recent development of predictive learning analytics (PLA) that uses machine learning and artificial intelligence approaches to predict student behaviour and/or outcomes.

We have attempted to be scrupulous in our use of the terms here, in particular to avoid confusion that engagement with LA in the EAI dashboard necessarily equates to engagement with the PLA. One does not necessarily follow the other.

11 STEM module teams and their tutors were given access to the EAI dashboard and the 3 types of LA in it:

TMA submission scores (PI level for tutor group) and rates of submission (aggregated for module level)
VLE engagement data (PI level for tutor group and aggregated for module level)
Predictive learning analytics (PLA) generated on a weekly basis by OU Analyse (OUA) machine learning algorithms that predict whether or not a student will submit their next tutor-marked assignment (TMA).

One other STEM module sent their tutors spreadsheets containing PLA generated by the student probabilities model (SPM) which produces predictions of whether an individual student will reach specific milestones (different points in a course presentation or between courses) such as completing and passing a module or returning in the next academic year.

All module teams and tutors were given some training in how to use their respective learning analytics approach during the 17J presentation.

Outputs from tutors were:

Prior to presentation - 7 tutors interviewed to establish how they already use data to support students.
After presentation - 38 tutors interviewed to establish how they had used the different types of LA available to them.
Usage data downloaded directly from the EAI dashboard.

And from module teams:

Prior to presentation – 12 module teams responded to implementation intention survey and interviewed to develop logic models that outlined expected short, medium, and long-term outcomes.
After presentation – 12 module teams interviewed to comment on the extent to which their expected outcomes had been realised.