Each of the projects in the Activity Data programme strand were asked to establish a hypothesis that we would test throughout the project. For RISE the hypothesis we chose is:
“That recommender systems can enhance the student experience in new generation e-resource discovery services”
This hypothesis was chosen quite carefully for a number of reasons. We’ve only recently implemented our Ebsco Discovery Solution aggregated search system so we are still in an evaluation stage and are really still assessing how students at the OU will get the best out of the new system. We have a particular perspective at the Open University in that the use that students make of our search systems varies widely from course to course. So we will particularly want to look at whether there is variation between the levels of students in their reaction to recommendations.
How do we plan to evaluate the hypothesis?
We are planning to approach the evaluation in three ways.
- By establishing some website metrics to allow us to assess how user behaviour is affected by the recommendations. We expect to build two versions of the search interface, one with recommendations and one without. This will allow us to A/B test the interfaces, so we can track the impact that different options make on users behaviour. Using Google Analytics we will track where users click and where they go.
- We will track the use that users make of the rating feature to see whether there is evidence that it is actively being used.
- We will actively encourage user feedback on the tools, carry out surveys with students and run a short series of focus groups to test the hypothesis.
As part of our evaluation work we will be looking to assess whether there are variations that can be ascribed to course or course level in how useful students find the recommendations to be, and whether there are circumstances where they are not useful. We will also be testing a variety of different types of recommendations and will aim to assess which are found to be most useful.
The evaluation report will detail the activities and results of the work to test the hypothesis and we will look at using Quora to record evidence.