Skip to content

Toggle service links
  1. eSTEeM
  2. Projects
  3. Themes
  4. Technologies Stem Learning
  5. Development and evaluation of a software tool for automated Java specification marking

Development and evaluation of a software tool for automated Java specification marking

Project leader(s): 
Anton Dil
Faculty: 
STEM
Status: 
Archived

Students studying M250, our second year object-oriented programming module using Java, are required to complete Java programs according to detailed syntactical, structural, functional and stylistic specifications.

Although software tools exist for code syntax, functionality and style checking, tools for structural specification checking are not widely available. The long-term goal of this project is to raise awareness of these various aspects of correctness in our assessment of students’ code and to support automated assessment of these aspects of code quality for tutors and students.

The project focused particularly on the development and evaluation of a structural specification tool (known as CheckM250), deployed in the 2017J presentation of M250, to allow tutors to check to what extent students’ code met a specification. The tool was provided for use in the module IDE, BlueJ, alongside traditional tutor marking notes. The project also explored the use of automated marking in the module Virtual Learning Environment (VLE), for quick feedback to students, and overcame technical obstacles in this context. 

Tutor surveys and interviews were used to gather feedback on CheckM250 and on other kinds of marking tool support and traditional resources.

Automated structural checks on code were found to have multiple use cases:

  • supporting human markers in assessing students’ code;
  • for markers to assess their own marking (as a kind of e-monitoring tool);
  • in online assessment for automated marking of students’ code;
  • as a step in determining if software functional tests can proceed;
  • for students to perform checks on their code before submitting it for marking;
  • for question setters to check completeness of questions set for students. 

There was evidence of tutors favouring the use of marking tools, or of their distrusting them, or finding them an obstacle. This appeared to depend less on the tool itself than on a predisposition for or against the use of tools. Similarly, tutors’ comparative rating of tools as aids to themselves versus as aids to students appeared to depend on the tutors’ disposition towards tools.

Most tutors using CheckM250 found it to be useful, and some reported that it increased their accuracy in marking. Tutors not using the tool cited lack of time and the simplicity of the assignment it was trialled on. Some reservations were expressed about reliance on automated marking tools, both for markers and for students.  The marking software was also shown to be useful in the VLE for automated student feedback.

The results provided indicators of topics that should be discussed with tutors and students in this context:

  • how automated code marking tools may best be used in tutor and student workflow;
  • how the outputs of the tools should be interpreted;
  • the potential benefits and pitfalls of automated marking;
  • the relationships between the outputs of various automated marking tools.

The project has also suggested ways forward in developing automated marking tools for Java code. 

Related resources

Dil, A., Truby, S. and Osunde, J. (2018) Development and evaluation of a tool for Java structural specification testing. eSTEeM Final Report. (PDF)

Dil, A., Truby, S. and Osunde, J. (2018) Development and evaluation of a tool for Java structural specification testing. Appendix A, Java specification checking: software notes. (PDF)

Dil, A., Truby, S. and Osunde, J. (2018) Development and evaluation of a tool for Java structural specification testing. Appendix B, Java specification checking: survey and interview results. (PDF)

Dil, A. and Truby, S. (2018) Evaluation of a tool for use on M250 “Object-oriented Java Programming”. Presentation from the 7th eSTEeM Annual Conference, 25-26 April 2018, Milton Keynes. (PowerPoint)

Dil, A., Osunde, J. (2018) Evaluation of a tool for Java Structural Specification Testing. Paper presentenced at the 10th International Conference on Education Technology and Computers, October 2018, Tokyo. (PDF)

 

Projects