Notes on seminar by Dr Ilya Goldin

Dissertation ‘Peering into peer review with Bayesian models’

Interested in how we can help students who are learning to analyse open-ended problems. How do we help them to do peer review? Peer review removes the instructor from the interaction beteen students. How do we keep the instructor within the loop?

Students need feedback that explains to them their current level describes target performance and suggests ways of getting there.

Rubrics are used in peer review to inform assessors of criteria, to support reviewers in their evaluation, and to give a structure to the feedback received by the author. They state the criteria of interest and define each criterion.

When dealing with open-ended problems you need to focus on argumentation. Generics rubrics can be replaced by domain-relevant rubrics or by problem-specific assignments. However, the rubric is then more limited in its scope.

Experiment was run with 58 law students. Each essay received four peer reviews, these were passed on to the authors (who had adopted pseudonyms), and then the authors gave feedback on the feedback. Assessed pre- and post-measures on a quiz. Students received one of two rubrics – one that was domain specific and concept oriented and one that was domain relevant and argument oriented.

Domain relevant was focused on issue identification, argument development, justified oveall conclusion and writing quality. For each dimension you were given one anchored rating and 1 comment. eg 3 – develops few strong, non-conclusory arguments, and neglects counter-arguments. (Prior research suggests that if people just give a rating, these tend not to be as well justified.)

Problem-specific rubric was focused on breach of non-disclosure, trade secret misappropriation, violation of right of publicity, and two examples of idea misappropriation. Here, an example of a review might be

3 – identifies claim, but neglects arguments pro/con and supporting facts; some irrelevant facts and arguments.

This rating scale could be used with many problems, if you were aware what the key issues were.

If students were taken as individuals, and you looked at an average of what peer review scores were, they were not helpful for predicting instructor scores, However, if you worked on the basis that scores in the class were likely to be related to other scores in the class, then it was possible to predict instructor scores.