Does the use of Grammarly impact detection rates from Generative-AI detection tools?

Project leader(s): Mike Richards
Theme: Technologies for STEM learning
Faculty: STEM
Status: Current
Dates: January 2024 to June 2025

Generative AI tools can produce compelling text, even to University assessment questions. Our research has demonstrated that this poses a risk to the OU, as current versions of ChatGPT can produce material sufficient to pass undergraduate modules. We also tested two types of detection software including the standard OU TurnItIN AI detection service. This software produced mixed results https://oro.open.ac.uk/89325/.

This provisional work took a simplistic view, submitting material completely generated through ChatGPT. We recommend students use Grammarly – or other similar tools – to improve their writing skills. Grammarly now uses similar technology to ChatGPT to improve writing. It is currently unknown whether AI detection tools can distinguish between writing produced by generative AI and material which has been improved by the use of generative AI.

Our proposal is to run a study with the following steps:

Obtain ethical and data protection sign-off.
Download 10 random student scripts apiece from multiple C&C modules covering a range of study levels and assessment types from the 21J cohort (pre-release of ChatGPT, as used in our previous studies).
Use Grammarly suggestions to progressively change the student submissions resulting in three levels of scripts – unchanged, minimal changes, significant changes.
We will generate a separate set of scripts which manipulate the student submissions using the AI-based Microsoft Word editor.
Upload the scripts to the MCT quality assurance area on TurnItIn ACQ have setup for our team as a testbed. We will record the resulting detection levels across the scripts.
Analyse the data.
Disseminate externally through a SIGCSE publication, and internally through seminars and the OU community of interest for generative-AI group on Teams.
Based on the results, we will engage with the study skills team that provides tool advice for students, and the group working on AI policy for the OU.