Category Archives: Analytics

Learning analytics and ethics

2 Replies

Way back when, I did some thinking about the differences between approaches to ethics in the Arts and the Social Sciences. To generalise, the Social Sciences treat the Internet as space, whereas the Arts treat the Internet as text. As I noted at the time: if you view the Internet as a virtual space populated by human actors, then you need a human subject approach to ethics, with informed consent a big issue. If, on the other hand, you see the Internet as an accumulation of texts, then your concern is with data protection, copyright and intellectual property rights. One practical example of this is that giving the real name of a data source is typically unethical behaviour in the Social Sciences, while failing to give the real name of a data source is typically unethical behaviour in the Arts.

So ethical behaviour is not a given – it is context dependent.

Extending this to learning analytics, a one-size-fits-all approach to ethics won’t necessarily work. Ethical research behaviour depends on what we are doing, on what we are trying to do and on what those involved in the research believe we are trying to do. The ethics discussion at #lasi13 suggested many of us are trying to different things – so maybe our approach to ethics will need to vary according to context.

Much of the discussion about the ethics of learning analytics this morning was framed in terms of learning as an economic transaction. The student contributes time, effort and money to an educational institution and, if this transaction is successfully completed, the student should emerge with a proxy for learning in the form of a certificate.

This view of learning is associated with  a view of data as a commodity to be owned and exchanged. In order for this transaction to be successfully completed, some exchange of data (attendance figures, grades, etc) is essential, and each party to the contract has rights and responsibilities in relation to the data.

So that implies a contractual perspective on ethics. My own work is in a different context – in informal or barely formal learning settings. Learning through social media, open educational resources, MOOCs, virtual worlds… The transaction is not overtly economic, the outcomes are more varied, the data have a different role. There is less sense of an obligation on either side. I suspect this means that the ethical concerns and priorities will be different, and that negotiating them will take us in different directions.

So one ethical code for learning analytics may prove impossible, we may need to shift from one to another according to context.

Educational Data Mining for Technology-Aided Formative Assessment

1 Reply

Notes on seminar by Dr Ilya Goldin

Dissertation ‘Peering into peer review with Bayesian models’

Interested in how we can help students who are learning to analyse open-ended problems. How do we help them to do peer review? Peer review removes the instructor from the interaction beteen students. How do we keep the instructor within the loop?

Students need feedback that explains to them their current level describes target performance and suggests ways of getting there.

Rubrics are used in peer review to inform assessors of criteria, to support reviewers in their evaluation, and to give a structure to the feedback received by the author. They state the criteria of interest and define each criterion.

When dealing with open-ended problems you need to focus on argumentation. Generics rubrics can be replaced by domain-relevant rubrics or by problem-specific assignments. However, the rubric is then more limited in its scope.

Experiment was run with 58 law students. Each essay received four peer reviews, these were passed on to the authors (who had adopted pseudonyms), and then the authors gave feedback on the feedback. Assessed pre- and post-measures on a quiz. Students received one of two rubrics – one that was domain specific and concept oriented and one that was domain relevant and argument oriented.

Domain relevant was focused on issue identification, argument development, justified oveall conclusion and writing quality. For each dimension you were given one anchored rating and 1 comment. eg 3 – develops few strong, non-conclusory arguments, and neglects counter-arguments. (Prior research suggests that if people just give a rating, these tend not to be as well justified.)

Problem-specific rubric was focused on breach of non-disclosure, trade secret misappropriation, violation of right of publicity, and two examples of idea misappropriation. Here, an example of a review might be

3 – identifies claim, but neglects arguments pro/con and supporting facts; some irrelevant facts and arguments.

This rating scale could be used with many problems, if you were aware what the key issues were.

If students were taken as individuals, and you looked at an average of what peer review scores were, they were not helpful for predicting instructor scores, However, if you worked on the basis that scores in the class were likely to be related to other scores in the class, then it was possible to predict instructor scores.

Assessing learner analytics

Leave a reply

Learner analytics use the experiences of a community [network?] of learners to help individual learners in that community to identify more effectively learning content from a potentially overwhelming set of choices.

Analytics and recommendations have generative power. The object recommended many not yet exist – it may be something that the learner must construct or that is constructed from the recommendation.

Analytics can be assessed from numerous perspectives, including: accuracy, adaptivity,  behaviour,  confidence, coverage,  coverage,  diversity,  drop-out rate,  effectiveness of learning,  efficiency of learning,  learner improvement,  novelty,  precision (comparing resources selected by user with those selected by algorithm),  prediction accuracy,  privacy,  reaction of learners,  recall (ratio of relevant results to those presented by algorithm) results,  risk,  robustness,  satisfaction, scalability,  serendipity,  trust, user preference,  utility.

(MUPPLE seminar – Hendrik Drachsler)