Natural Language Processing to identify student misunderstandings from free-response assessment questions in physics

The Open University has a rich history in using Natural Language Processing (NLP) to support short answer assessment. Early work by Butcher & Jordan (2010) and Willis (2015) demonstrated how NLP could be applied to automatic assessment of students' free text answers, with Haley (2009) showing how Latent Semantic Analysis could be used to identify conceptual gaps in students' learning. Subsequently, Parker's (2020) work within the Open University's School of Physical Sciences (SPS) has developed and tested a version of the well-known Force Concept Inventory (Hestenes et al, 1992) based on automatically marked free-response items rather than the conventional multiple-choice questions. In computing, Pandey's (2022) work developed Deep Learning architectures to improve the state of the art in marking performance. In tandem, research across the University has developed a world-leading understanding of issues in the use of technology to support education (Howson et al. 2020).

This project will extend the work in automatic assessment in Computing, and its application to physics learning in SPS, by building on Willis' (2015) work using Inductive Logic Programming (ILP) as the basis for the Machine Learning aspects of the work. By using the symbolic approach of ILP rather than the statistical approach of deep learning, we can build models of students' understanding which can be analysed to identify conceptual gaps (the aim of explainable AI systems). By combining these symbolic models with the ideas in the Force Concept Inventory, we aim to improve students' learning by being able to more accurately diagnose students' misunderstandings and provide more explanatory feedback.

The project will have three themes:

1. The Force Concept Inventory as a tool for automatic teaching.

This work package will address questions such as:

• How do students typically approach the questions set on the Force Concept Inventory?
• What language do they use?
• Are there cues which indicate (mis)understandings?
• How should the computer-marking techniques be extended to provide insight into student’s misconceptions?

This work package requires the sound understanding of physics, and experience of teaching physics concepts and diagnosing students' grasp of ideas.

1. Inductive Logic Programming as the framework for automatic assessment.

This work package will address questions such as:

• What are the limits on short, free-response questions that can be accurately marked by a computer?
• How can conceptual models be developed with machine learning, and how can symbolic and non-symbolic systems work together to maximise accuracy and explainability?
• Can an automatic marking system provide an equivalent level of explanation of its awarded marks to a human marker?

This work package will build on existing work in automatic assessment, with a particular focus on how to handle increasingly complex questions.

1. Embedding into a teaching environment.

This work package will address questions such as:

• How can automatic assessment systems be effectively (ie. from a pedagogical perspective) embedded into different teaching and learning environments?
• What are the requirements for the various stakeholders (teachers, students and computational implementors)?
• How can computer-marking be democratised, so that domain experts can set meaningful questions and evaluate student responses without requiring expertise in computer-marking?

This work package will build on Howson's work in delivering innovative pedagogy to stakeholders.

References:

• Butcher, Philip, G.; Jordan, Sally, E. (2010), A comparison of human and computer marking of short free-text student responses, Computers & Education, 55, 489 – 499.
• Haley, Debra Trusso (2009), Applying latent semantic analysis to computer assisted assessment in the Computer Science domain: a framework, a tool, and an evaluation. PhD thesis, The Open University. Available at Open Research Online.
• Hestenes, David; Wells, Malcolm; Swackhamer, Gregg (1992), Force concept inventory, The Physics Teacher, 30(3), 141 – 158.
• Howson, Oliver; Adeliyi, Adeola; Willis, Alistair; Hirst, Tony; Charlton, Patricia; Gooch, Daniel; Rosewell, Jonathan; Richards, Mike; Wermelinger, Michel; Piwek, Paul; Savage, Simon; Lowndes, Charly; Thomas, Elaine and Smith, Andrew (2020), Best Practices in using Technological Infrastructures. The Institute of Coding.
• Pandey, Suraj Jung (2022), Modelling Alignment and Key Information for Automatic Grading. PhD thesis, The Open University. Available at Open Research Online.
• Parker, Mark (2020), Establishing Physics Concept Inventories Using Free-Response Questions, PhD thesis, The Open University. Available at Open Research Online.
• Willis, Alistair (2015), Using NLP to Support Scalable Assessment of Short Free Text Responses, Proc. 10^th Workshop on Innovative Use of NLP for Building Educational Applications, 243 – 253.

Qualifications required:

Either: 

• 2:1 or above, MPhys or other integrated science masters in physics, astronomy, computer science or related disciplines 
• First class honours BSc in physics, astronomy, computer science, or related disciplines 
• 2:1 BSc in physics, astronomy, computer science, or related disciplines, plus a Masters-level qualification in a relevant area 

Contacts:

For more information about the project, please contact the follwoing academisc:

Dr Alistair Willis, Dr Jonathan Nylk and Mr Oli Howson